Enhancing Video-based Learning Using Knowledge Tracing: Personalizing Students’ Learning Experience with ORBITS
Year: 2023

Abstract:
As the world regains its footing following the COVID-19 pandemic, academia is striving to consolidate the gains made in students’ education experience. New technologies such as video-based learning have shown some early improvement in student learning and engagement. In this paper, we present ORBITS predictive engine at YOURIKA company, a videobased student support platform powered by knowledge tracing. In an exploratory case study of one master’s level Speech Processing course at the Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI) in Abu Dhabi, half the students used the system while the other half did not. Student qualitative feedback was universally positive and compared the system favorably against current available methods. These findings support the use of artificial intelligence techniques to improve the student learning experience.

heading: 1 Introduction
text: Looking ahead to the post-pandemic tertiary education landscape, higher education institutions ought to innovate by incorporating effective technology to maximize a more personalized video-based learning experience. One of "the most recurring challenges towards online learning" (Munoz et al., 2021, p.2) is disengagement and absence of participation. A poll of 350 university students taking synchronous (fully online) Zoom classes indicated that approximately 94% had moderate to considerable difficulty with online learning (Peper et al., 2021). Recently,
investigators have examined the effects of Videobased Learning (VBL) on student learning and engagement (Ou et al., 2019; Poquet et al., 2018). Ou et al. (2019) and Sablić et al. (2021) have for instance established that it can have a positive impact on learning, and “students’ perceptions of video effectiveness significantly predicted how they perceived the overall effectiveness of the course” (Ou et al., 2019, p. 99). Studies by Tripodi (2018) on first year osteopathic students in Australia and Lacey and Wall (2021) on three B.Sc. undergraduate student groups (microbiology) in Ireland have also indicated that VBL stimulated interest and improved performance, motivation, and engagement, as it increased exam confidence and decreased exam anxiety. An additional benefit of VBL was “improved communication with their mentees and greater ability to demonstrate the experiments to their student groups” (Lacey and Wall, 2021, p.8). So far, however, there has been little discussion in the published literature about the use of AI-powered video-based learning support platforms at the postgraduate level to personalize learning paths and improve the achievement of intended learning outcomes.
Knowledge tracing involves creating models that track students' understanding over time, enabling accurate predictions of their future performance. Advancements in this area would allow personalized resources to be recommended based on individual needs, while identifying content that may be too easy or challenging, allowing for skipping or postponing. Machine learning solutions could potentially extend the benefits of high-quality personalized instruction to anyone worldwide, at no cost.
100
The knowledge tracing problem is inherently challenging due to the complexity of human learning, encompassing both the human brain and accumulated knowledge. Therefore, employing sophisticated models appears to be suitable.
There are two primary aims of this study: 1. To investigate student use of and engagement with ORBITS, a video-based learning (VBL) software powered by a predictive engine that uses knowledge tracing at a graduate-level research intensive university in Abu Dhabi, United Arab Emirates, and 2. To assess the extent to which ORBITS improved students’ perception of their learning experience.
This investigation takes the form of a case study. The findings should make an important contribution to the field of knowledge tracing - powered video-based learning.
This paper is organized as follows: First, we review relevant literature pertaining to Videobased Learning (VBL); the next two sections present the methodology and the results of the research, respectively. Section four discusses the results, while section five concludes.

heading: 2 Background

heading: 2.1 Video-Based Learning (VBL)
text: Research on video-based learning (VBL) has seen substantial growth in the last few years as a result of the launch of a) educational platforms using video (e.g., Khan Academy, LinkedIn Learning, MOOC platforms such as Edx, Coursera, FutureLearn), b) short-video hosting services (e.g., Tik Tok, Snapchat) on multiple devices, c) new features to existing apps (e.g., Instagram Reels, YouTube Shorts), and d) the adoption of lecturecapture platforms (Panopto, Echo360, Kaltura) by tertiary institutions around the world. As 1.5 billion students across 165 countries (UNESCO, 2020) were asked to return home, academic staff was requested to move all their courses fully online and use videoconferencing platforms such as Zoom, Skype, WebEx, Blackboard Collaborate Ultra or Microsoft Teams, video consumption in the past two and a half years has increased exponentially. In fact, as the COVID threat receded ZOOM or Teams meetings are still more common than physical meetings and students still prefer to attend their classes online. Torre et al. (2022) argued that “multimedia content and video-based learning are
expected to take a central role in the post-pandemic world” (p.1). Research by Calonge et al. (2019) indicated how crucial videos (and analytics) were for student retention in a fully online course. Considering recent events, it is indeed becoming extremely difficult to ignore the importance of VBL and its impact on learning and teaching (Navarette et al., 2023). A study by Cheristiyanto (2021) on 119 high school teachers of economics in the Indonesian context indicated for instance that VBL had had a positive impact on students’ learning outcomes. Another study by Schmitz et al. (2021) on the use of a flipped classroom model and video learning in a surgical course by 58 adult students (29 in the control group) showed that students in the test group preparing through the video-based online platform reached significantly higher scores in their written exams. Additionally, results of a survey by Davey et al. (2020), sent to all higher specialist orthopedic trainees in Ireland, also indicated high levels of satisfaction and positive outcomes when it concluded that “over 90% of trainees agreed that the video-based distance learning is of the same quality or an improvement of previous utilized teaching styles” (p.2089). It is now well established from a variety of studies, that the personalization of the learning experience, with immediate and customized instruction or feedback, based on students’ needs and interests, a) increases cognitive, emotional, and behavioral engagement, and b) improves motivation, performance, and learning outcomes. User preference modelling, knowledge tracing, and item-based collaborative filtering have been extensively used by e-commerce websites such as Alibaba, Amazon, or social media platforms and companies such as Netflix, Hulu, Instagram, or YouTube to predict and recommend videos. Pandey and Karypis (2019) defined knowledge tracing as “the task of modeling each student's mastery of knowledge concepts (KCs) as (s)he engages with a sequence of learning activities” (p.1). Collaborative filtering analyses the similarities between users and items selected (behavior, preference) to personalize suggestions. Recent articles by Zhang (2022), El Aouifi et al. (2021), Wang (2021), Li and Ye (2020), and Madani et al. (2019) showcased for instance a userbased collaborative filtering algorithm applied to a personalized learning platform. Finally, a review of the recent research literature on video-based learning by Navarrete et al. (2021, p.8) argued that
recent approaches to forecasting learning success mainly build upon deep learning techniques (e.g., multilayer perceptrons, gated recurrent units, RNNs, and LSTMs).

heading: 2.2 Intelligent Tutoring System
text: Intelligent tutoring systems (ITSs) can successfully teach skills (like algebra, computer programming, or medical diagnosis) using learning-by-doing principles, track progress, and provide learners with personalized feedback and materials adapted to their level of understanding.
Given a learner’s history of past interactions with an ITS, a performance model can be developed to estimate the current level of a learner’s knowledge to predict future performance. Performance models have three major purposes: (1) enabling adaptive behavior of the instructional policy, (2) displaying the learner’s estimated knowledge as a means of learning support and (3) generating interpretable and actionable intelligence. Most adaptive instructional policies used in practice today rely on an estimate of a learner’s performance. They either require a learner to become proficient in one topic before allowing him/her to proceed to the next one, or sequence items based on some notion of optimal difficulty. The performance model also provides interpretable and actionable insights to learning designers, educators, and educational researchers to develop the ITS further. However, building an ITS is often very time-consuming (Weitekamp et al., 2020). Researchers started exploring and implementing different methods for quantizing performance prediction for learning including Bayesian Knowledge Tracing (BKT) or Deep Knowledge Tracing (DKT) (Piech et al. 2015).
BKT is considered the baseline approach for knowledge tracing. Other research indicated that DKT also showed a 25% gain in prediction accuracy, whereas classical statistical models could match the accuracy under a constrained environment. As such, there was no standard method of predicting the learning performance of a learner. Knowledge tracing using self-attention (Wang et al., 2022) identifies the key concepts from the student's past activities that are relevant to the given Knowledge Concept (KC) and predicts his/her mastery based on the relatively few KCs that it picked (Pandey et al., 2019). Since predictions are made based on relatively few past activities, it handles the data sparsity problem
better than the methods based on RNN. For identifying the relevance between the KCs.
Given the ability of recurrent neural networks (RNNs) to use information from an input in a prediction at a much later point in time, we hypothesized that RNN models, particularly the Long Short-Term Memory (LSTM) network, could provide significant improvements in prediction accuracy regardless of the conditions and topic considered for knowledge tracing.
Further, the accuracy of prediction and stability of parameters may impact the usability of a learning performance model. For ITS to provide actionable insights, the stability of parameters has more importance than the accuracy of prediction, whereas the accuracy of prediction impacts the performance of the adaptive behavior of ITS.
We found that the existing methods fail to overcome the lack of performance, due to a larger data size: The unsolved vanishing gradients problem hampers learning of long data sequences. The gradients carry information used in the RNN parameter update. When the gradient increasingly becomes smaller, the parameter updates become insignificant, which indicates that no significant learning is happening. The existing DKT models fail to reconstruct the observed input. As a result, even when a student performs well on an assessment, the prediction of that assessment mastery level decreases instead, and vice versa. The predicted performance for assessments across time steps is not consistent. This is undesirable and unreasonable because a student’s performance is expected to transition gradually over time.
Figure 1: Screenshot of ORBITS that displays student view of the platform.

heading: 3 Methodology
text: This work proposes a personalized learning environment named ORBITS as shown in figure 1 that gathers and traces the knowledge state of the learner. We built a new model that is beyond the standard deep learning-based model based on Long Short-Term Memory (LSTM) with a new network architecture and a combination of methodologies to solve the challenges in question. Beyond the standard approach mentioned above, we built the following four methods:
1. Question to topic mapping: the standard approach is to feed the data at the question level which will result in millions of combinations of feature space dimensionality. Since the number of topics is less in order of magnitude than the number of questions, in our model, we built an encoding layer of topics rather than questions to decrease the dimensionality of unique questions. The mapping between questions and topics is achieved before the training, so, at inferencing time when the question is predicted, the model will measure the knowledge state of its topics and all the dependent topics accordingly.
2. Representative subset of the feature space: This improves our model performance. However, the feature space still needed to be reduced. For such a large feature space, the standard approach of the one-hot encoding became impractically large. Therefore, the existing DKT models only work on specific subjects for hundreds of topics and as mentioned earlier, are unable to scale to thousands of topics. Thus, we reconstructed the input from a series of new topic-based sampling measurements to decrease the high dimensionality of the feature space. We achieved it by sampling lowdimensional representations of a one-hot highdimensional vector. Sampling was done by picking topics that are dependent on other topics, as explained in the next step #3, our topic-based encoding layer. This means that based on fewer answers to questions/topics, the answer will be predicted, which will decrease the sparsity significantly.
3. Topic-based encoding: Takes two topics/questions answers, turns them into a matrix where the answers of one topic/question form the columns, and the answers of another topic/question form the rows to understand how this topic relates to others. This improves the model predictions as it can identify all the dependencies among the
knowledge states of the topics and can measure these dependencies inherently in the model.
4. Student knowledge context: We hypothesized that the standard approach lacks accuracy in high dimensionality since it does not take the learning context into account. And since the objective of the model is to predict what the student needs to learn on the next topic knowledge state, context is key. We, therefore, go beyond the standard approach to capture the context of the student knowledge state. Since students tend to forget topics, in what is often referred to as cognitive load (Hultberg et al., 2008), we want to preserve the knowledge state context of the topics that are answered to emphasize the recent knowledge states that have been answered. We went beyond the standard approach and ordered the input to relate a student's future interaction with topic/question to their past interaction. In this case, the model creates a representation that learns about the learning context across the topics from historic responses. The model architecture is shown in figure 2.

heading: 3.1 Scaling
text: We were looking to scale multiple thousands of topics and still achieve an AUC that is not less than ~0.7. Personalization needs inspired researchers to propose AI models to understand the knowledge state which is the “Knowledge Tracing” area. Recently, the spotlight has shone on comparisons between traditional, interpretable models such as Bayesian Knowledge Tracing (BKT) and its variants, and complex, opaque neural network models such as Deep Knowledge Tracing (DKT).

heading: 3.2 Research Design
text: The research used in this article is an exploratory qualitative case study (Yin, 2018). This case study was conducted at a graduate-level research intensive university in the United Arab Emirates.
The purpose of this case study was to explore students’ adoption of ORBITS. The research questions being examined in this study were:
• How do students engage with the ORBITS platform?
• How do students perceive their learning experience, as affected by ORBITS predictive engine? This study focused on contemporary events as seen through the eyes of the participants (students using ORBITS), supplemented by engagement data from ORBITS and a student feedback survey. The case study presented here is of an exploratory nature and uses an embedded, single-case design. This design allows for the exploration of several units of analysis within the case. Specifically, there are two embedded units of analysis: 1. Student engagement with ORBITS, and 2. Student perception of the learning experience.

heading: 3.3 Context of the Study and Participants
text: This study involved postgraduate students in a course on speech processing within the MSc in Natural Language Processing (NLP).
The course is compulsory for postgraduate students in a variety of disciplines at Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI) in Abu Dhabi. Participants were informed of the study’s purpose, the procedures involved, and were asked to sign a consent form. Pseudonyms (Participant - P 1, Participant - P 2, etc.) were used throughout the study to protect the participants’ anonymity.

heading: 4 Results
text: This section illustrates the different methods that are used to improve the results.
1. Question to topic mapping: After analyzing the results, we found that the feature space still needs to be reduced. For such a large feature space, the standard approach of the one-hot encoding became impractically large, and the predictions are not consistent. This led us to find a better approach to decrease the feature space.
2. Representative subset of the feature space: We decreased the high dimensionality of the feature space by reconstructing the input from a series of new topic-based sampling measurements. After analyzing the results, we found that the sampling approach misses important topics that are
prerequisites to the next predicted topic in the training phase. This led us to find a better approach to selecting the prerequisite topics.
3. Topic-based encoding: After analyzing the results, we found that prediction performance starts with high accuracy but decreases over time. We made a further analysis to capture this behavior across students or per 1 student. We were able to segregate one student's results and found the prediction performance improves. This led us to find a better approach to address the student context issue.
4. Student knowledge context: After analyzing the results, we did several experiments on another four datasets as shown in Table 1 below to fine-tune the model hyper-parameters and make sure the model performance is solid. This led us to find a better approach for training the model on a large scale.
5. Machine learning operations (ML Ops) pipeline: Testing model against real-world datasets
We then implemented an end-to-end machine learning operations pipeline to facilitate large-scale training of hundreds of our models and modification of the model hyperparameters. The end-to-end pipeline was configured to handle the different steps to build the Knowledge Tracing system from pre-processing raw data, training a model, and deploying the system on a cloud platform. We built a benchmarking tool to compare the different techniques with the baseline approach, as shown in Table 1 below.
# of Topics /Skills BKT (baseline) AUC (%)
DKT AUC (%)
SelfAttention AUC (%)
ASSIST2009
124 0.630 81.81 ± 0.10 84.20 ± 0.10
https://sites.google.com /site/assistmentsdata/ho me/2009-2010assistmentdata?authuser=0
ASSIST2015
100 0.630 72.94 ± 0.05 82.09 ± 0.03
https://sites.google.com /site/assistmentsdata/da tasets/2015assistments-skillbuilder-data
ASSISTmentsChall
102 0.640 72.29 ± 0.06 75.70 ± 0.32
https://sites.google.com /view/assistmentsdatam ining
We also surveyed students to gather feedback about their experience using ORBITS. Table 2 shows their comments.
Student feedback, albeit from a small sample, was overwhelmingly positive. Students spoke of the advantages the ORBITS platform provided them in their acquisition of course content (e.g., transcriptions), especially the speed (accuracy and responsiveness) with which they were able to locate specific content for learning and revision purposes. One student noted that “the customized report on understanding and knowledge of a specific topic is amazing”. Students also noted that the user interface aided efficient and strategic use of their time.

heading: 5 Discussion
text: Defining the input sequence and the way it is architected in the LSTM is the key to defining how the LSTM is used. Hence, we used LSTM in a different manner by defining the sequence in two ways: horizontally across topics and vertically across learning time per topic.
The topic-based encoding provides a sequence of topics based on their dependencies as prerequisites. Student knowledge context provides the chronological order of the student learning sequence in each topic over time and hence, across topics. This unique sequence definition enabled us to go beyond the standard approach by using LSTM in a unique manner in knowledge tracing. These two sequences would not have been possible without (1) the Question to topic mapping that enabled us to work at the level of the topic rather than questions, and (2) the Representative subset of the feature space that enabled the capability of selecting prerequisite topics.

heading: 6 Conclusion
text: The aim of the present research was to examine student use of and engagement with ORBITS and the extent to which ORBITS predictive engine improved students’ perception of their learning experience. Positive initial feedback from participants indicated that ORBITS’s ease of use, responsiveness, and usefulness were the three main factors of learning satisfaction. In fact, previous research has shown that learning satisfaction often correlates with performance and achievement of learning outcomes. Based on our initial exploratory findings, we can conclude that students using ORBITS are more engaged, more satisfied with their learning experience and may achieve higher assessed learning outcomes than students not using ORBITS.
Limitations
This study reports early results and was based on a limited cohort of students from one master’s course. The generalizability of these results is therefore subject to certain limitations. Follow on studies will test the system with larger samples and different disciplines to add weight to any significance of the results. Notwithstanding the relatively limited sample, this work offers valuable insights into how a video-based student support
platform that uses knowledge tracing improved graduate students’ perception of their learning experience.
As the students finish their course, we will collect additional quantitative data in the form of final student grades. These will be compared between students who used the system and those who did not. We will also compare the final grades of students who used the system between the course in which they used it and other courses in which they did not. It also remains unclear what influence the course content has on the students’ experience. To examine this element further, the system should be tested on multiple courses imparted by different lecturers, and in varying subject fields. There may be an inherent positive opinion of AIpowered technologies by students at an AI university. To test that hypothesis, the system should be provided to students at other universities in subject areas not related to AI. Several questions, however, remain to be answered. Further research should be undertaken to test several hypotheses, for instance, whether Perceived Ease of use (PEoU) and Perceived usefulness (PU) would predict the Attitude towards Usage (AtU) of ORBITS.

heading: Future Work
text: Given a learner’s history of past interactions with an ITS, a performance model can be developed to estimate the current level of a learner’s knowledge to predict future performance. Performance models have three major purposes: (1) enabling adaptive behavior of the instructional policy, (2) displaying the learner’s estimated knowledge as a means of learning support and (3) generating interpretable and actionable intelligence. Most adaptive instructional policies used in practice today rely on an estimate of a learner’s performance. They either require a learner to become proficient in one topic before allowing him/her to proceed to the next one, or sequence items based on some notion of optimal difficulty. The performance model also provides interpretable and actionable insights to learning designers, educators, and educational researchers to develop the ITS further. However, building an ITS is often very time-consuming (Weitekamp et al. , 2020). Researchers started exploring and implementing different methods for quantizing performance prediction for learning including Bayesian Knowledge Tracing (BKT) or Deep Knowledge Tracing (DKT) (Piech et al. 2015). BKT is considered the baseline approach for knowledge tracing. Other research indicated that DKT also showed a 25% gain in prediction accuracy, whereas classical statistical models could match the accuracy under a constrained environment. As such, there was no standard method of predicting the learning performance of a learner. Knowledge tracing using self-attention (Wang et al. , 2022) identifies the key concepts from the student's past activities that are relevant to the given Knowledge Concept (KC) and predicts his/her mastery based on the relatively few KCs that it picked (Pandey et al. , 2019). Since predictions are made based on relatively few past activities, it handles the data sparsity problem
better than the methods based on RNN. For identifying the relevance between the KCs. Given the ability of recurrent neural networks (RNNs) to use information from an input in a prediction at a much later point in time, we hypothesized that RNN models, particularly the Long Short-Term Memory (LSTM) network, could provide significant improvements in prediction accuracy regardless of the conditions and topic considered for knowledge tracing. Further, the accuracy of prediction and stability of parameters may impact the usability of a learning performance model. For ITS to provide actionable insights, the stability of parameters has more importance than the accuracy of prediction, whereas the accuracy of prediction impacts the performance of the adaptive behavior of ITS. We found that the existing methods fail to overcome the lack of performance, due to a larger data size: The unsolved vanishing gradients problem hampers learning of long data sequences. The gradients carry information used in the RNN parameter update. When the gradient increasingly becomes smaller, the parameter updates become insignificant, which indicates that no significant learning is happening. The existing DKT models fail to reconstruct the observed input. As a result, even when a student performs well on an assessment, the prediction of that assessment mastery level decreases instead, and vice versa. The predicted performance for assessments across time steps is not consistent. This is undesirable and unreasonable because a student’s performance is expected to transition gradually over time. Figure 1: Screenshot of ORBITS that displays student view of the platform