# Emotion-Based Adaptive Learning Systems

Sai Prithvisingh Taurah, Jeshta Bhoyedhur, Roopesh Kevin Sungkur

► To cite this version:

Sai Prithvisingh Taurah, Jeshta Bhoyedhur, Roopesh Kevin Sungkur. Emotion-Based Adaptive Learning Systems. 2nd International Conference on Machine Learning for Networking (MLN), Dec 2019, Paris, France. pp.273-286, 10.1007/978-3-030-45778-5_18. hal-03266463

HAL Id: hal-03266463

https://inria.hal.science/hal-03266463

Submitted on 21 Jun 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Emotion-based Adaptive Learning Systems

Sai Prithvisingh Taurah ¹, Jeshta Bhoyedhur ² and Roopesh Kevin Sungkur ³

¹ Tylers Ltd, Mauritius, kta@tylers.mu

²LSL Digital, Mauritius

³Department of Software and Information Systems, University of Mauritius
Reduit, Mauritius, r.sungkur@uom.ac.mu

**Abstract.** Right from our primary school to professional academic level, the classical education system modus operandi, forces us to follow a series of predefined steps to climb the stairs of academic levels. Traditionally those predefined steps forces students to go through the beginner level to advanced level and then specialized in a specific level. The main problem was that the teaching styles and content delivery was not tailored to every learning styles and student personalities. The traditional education system is moving towards adaptive learning system where students are not bound only to one predefined set of contents. Therefore the traditional “one size fits all” approach is no longer valid as it were before. Each student has their curriculum based on their unique needs and personality. Adaptive learning may be referred as the process of creating unique learning experience for each and every learner based upon the learner's personality, interests and performance. This research presents a novel approach of adaptive learning by presenting an emotion-based adaptive learning system where the emotion and psychological traits of the learner is considered to provide learning materials that would be most appropriate at that particular instance of time. It shall demonstrate an intelligent agent based expert system using artificial intelligence and emotion detections capabilities to measure the user learning rate and find an optimum learning scheme for the latter.

**Keywords:** Adaptive learning, personalisation, emotion, neural networks, machine learning.

# 1 Introduction

Given the uproar of distance learning through Massive Open Online Courses (MOOC), top universities in the world such as Harvard and MIT joined the craze to propagate knowledge. Despite the millions of subscription for the Harvard MOC, only 10 % of students were completing the courses. Feedback from student's show that the content of the courses did not suit their current knowledge level and that the way the course content was presented decelerated their learning rate. Hence the concept of “one size fits all” was questioned by researchers who brought forward the concept of learning styles and prior knowledge relationship to learning process. The idea of adaptive learn-
ing system was the answer to mitigate the dropout rate from MOOC. The idea of adap-
tation is described as the concept of making changes in the educational environment to
match variety in the learner needs and abilities to sustain suitable context for interac-
tion. Adaptive hypermedia systems build a model of the goals, preferences and
knowledge of each individual user; this model is used throughout the interaction with
the user in order to adapt to the needs of that particular user (Brusilovsky et al., 2000).
One aspect of adaptive system is adaptive learning, which is the use of technology to
derive correct learning pattern for the different learner's intrinsic and extrinsic factors
and delivering the contents in a personalized way. Components of an adaptive learning
system include a content model, a learner model and an instructional model.

1.1 Content Model

This refers to the way the specific topic, or content domain, is structured, with thor-
oughly detailed learning outcomes. It is responsible for adaptation of the content as per
the user requirements for better interaction. The structure of the domain knowledge
relies on symbolic methods. This is often represented as a semantic network of domain
concepts, or generally elementary pieces of knowledge for the given domain related
with different kinds of links (Oxman and Wong, 2014).

1.2 Learner Model

This is also known as the Student Model. The aim is to guide the tutor in taking the
pedagogical decisions better adapted to a learner. It models the statistical implications
of the knowledge, complications and misapprehensions of the person. It reflects the
learner's understanding in a particular field and is prone to changes. The learner infor-
mation can also be stored such as name, personality style, learning style, age etc.

1.3 Instructional Model

It is the interface provided by the system depending on the individual differences such that the learning process is facilitated. Information from both learner and content model is used to deliver responsive response.

2 Literature Review

2.1 Types of Adaptation

The different types of adaptation are briefly discussed in the section below.

**Adaptive Interaction:** Adaptation occurs at the graphical user interface and are planned to simplify the user's interaction with the system, without, however changing in any way the learning content itself. Examples: alternative color scheme, font sizes to accommodate user preferences (Paramythis and Loidi-Reisinger, 2003).
**Adaptive Course Delivery:** Adaptations are envisioned to customize a course to the individual learner. The intention is to adjust the gap between course contents and the user requirement so that the ideal learning result is attained (Paramythis and Loidi-Reisinger, 2003). Examples of adaptation in this category are dynamic course (re-structuring; adaptive navigation; and, adaptive selection of alternatives course material (Brusilovsky, 2001). Adaptive navigation tends to show the content of an on-line course in enhanced order, where the enhancement criteria considers the learner's background and performance.

**Dynamic Courseware Generation:** It generates a customized course by considering explicit learning goals, as well as, the basic level of the student's knowledge. The system with dynamic generation studies and adapts to the students' advancement during his interaction with the generated course in real-time.

**Content Discovery and Assembly:** Application of adaptive methods in the discovery and assembly of learning material from the content model (Paramythis and Loidi-Reisinger, 2003). Information collected on the user learning style and prior knowledge on the corpus are the parameters that allow rules defined to be triggered.

**Adaptive collaboration Support:** It involves apprehending adaptive support in learning processes that includes communication between multiple persons (Paramythis and Loidi-Reisinger, 2003).

## 2.2 Emotions

"Emotions are basic psychological systems regulating an individual's adaptation to personal and environmental demands. Emotions are closely related to cognitive, behavioral, motivational and physiological processes; therefore they are generally important for learning and achievement" (Seel, 2012; Khalfallah and Slama, 2018). An influential research on human behaviour put forward that the learning process is more effective when it is associated with positive relations than it is with building negative one. Several researches have indicated that one way for effective learning to take place is by having positive emotions while learning takes place (Corradino and Fogarty, 2017; Fatahi, 2019; Lane and D'Mello, 2019). Furthermore brain imaging has showed that these positive emotions are very important to efficient learning; instructional styles that backs up positive emotions have been correlated with more efficient cognitive processing (Hinton et al., 2008). Researchers claimed that positive mood assists difficult cognitive functions that require elasticity, integrations and use of cognitive material such as memory, classification, creative problem solving, decision-making and learning (Febrilia et al., 2011). However from the results of a recent research, it was stated that good mood does not really guarantee that the student is able to focus. On the other hand, it does show that a bad mood do affect learning process subsequently.
## 2.3 Studies in the field

Several adaptive learning tools have been developed till now. These tools focus on different aspects that contribute to learning. Thus they use one of these approaches: adaptive content, adaptive assessments, and adaptive sequences or they use a combination of two of these approaches stated (EdSurge, 2016). Knewton is a web learning platform which focuses on adaptive sequences. It records feedback and responds to changes on a real time basis. According to Knewton (2016), the learning materials are built on thousands of observations consisting of theories, structure, and difficulty level. Knewton analyses these learning materials and uses sophisticated algorithms to render the most appropriate content to the user. Knewton also added that data are collected from a network of students and these data are recorded, analysed and applied to optimize the next output to each student.

## 2.4 Machine Learning Techniques

A number of different learning styles classification algorithms have been used since the last decades in adaptive learning system. As per (Truong, 2016) which has reviewed 51 studies, the following were the most used in the last ten years: Bayesian network, Rules (Association rules), Neural Network, and Naïve Bayes Network.

### Bayesian Network

"A Bayesian network is a graphical model that encodes probabilistic relationships among variables of interest" (Heckerman et al., 1995). Bayesian networks can be used to establish learners profile according to activities that they are selected and realized.

### Rule based algorithm

Rule based system also known as expert system uses rules knowledge representation for knowledge coded into the system. A rule based system is a way of encoding a human expert's knowledge in a fairly narrow area into an automated system.

### Neural Network

Artificial neural network models have specific properties such as capability to adapt, learn or to cluster data. ANN has been modeled from the human cortex but in a less complex way. It contains several nodes arranged in layers (input, hidden and output layer). Activation of a layer is done by the activation function.

### Naïve Bayes

"Naive Bayes classifiers, a family of classifiers that are based on the popular Bayes' probability theorem, are known for creating simple yet well performing models" (Raschka, 2014). A Naïve-Bayes classifier is built by using the training data to approximate the probability of each group according the examples.
# 3 Proposed Solution

## 3.1 Overview of System

The system developed is an online adaptive learning platform which takes into consideration the human psychological factor and human emotional behavior. To increase the efficiency of the system and provide contribution to the field, a multimodal approach for adaptivity is chosen. Upon registration, the user will be provided with a prior knowledge test which is performed to situate the user knowledge level in the domain model space and after a learning style questionnaire is used to identify the user learning style preference, and then, At this point the personalized learning path is generated for optimum learning rate which is achieved by a neural network. During the learning phase, the user emotion is tracked to identify the user current mood that is bored, neutral, surprise among others. This data is used to predict a time table showing exactly when it is optimum for him to study. This is done by a second neural network. After each section completed the user will take a test whereby if his performance is low a reinforcement rule will apply where he will be given additional personalized content to master this section. Those methods will not require resource intensive hardware except a camera and access to a good internet connection. These allow for a wider audience to be acquired and educational academies can use the system at minimal cost.

## 3.2 Architecture of Proposed System

Three tier architecture is privileged where the system is broken down into Presentation, Application, and Data tiers. This helps in the maintainability of the system and the agility to cope with changing requirements. On the Presentation layer a web interface is provided based on the bootstrap library for adaptivity on different screen sizes. Data tier consist of a content and learner model. The content model contains the learning objects in different version and the learner model contains the student static data such as personal information and dynamic data like student performance and learning paths. Springboot has been used as the web development framework it acts as a middleware between the presentation tier and the data tier. At the application layer, two neural network form the core logic of the system. The content prediction neural network take input such a student performance, learning style and prior knowledge to predict best learning object to learn. The time table neural network uses recorded emotion of user during hourly interval and performance to predict favorable hours for student to learn. Hence a personalized time table is created for the user which suggest hours of days best to learn.
<figure><img src="image_2.png" /><figcaption>Fig. 1. System Architecture</figcaption></figure>

3.3 Choice of Final Tools

IntelliJ Ultimate was the IDE of choice in conjunction with tomcat server, XAMPP and MySQL database. SpringBoot Framework was used as the backbone for the web application. This bundle provide ease of development for web application using Spring technologies such as Spring security, Spring MVC, Hibernate and view resolvers such as Thymeleaf. Libraries used were bootstrap and JQuery for frontend and Neuroph for the neural network on the server side. Microsoft cognitive emotion API was used for emotion detection.
<figure><img src="image_3.png" /><figcaption>Fig. 2. System Description</figcaption></figure>

## 3.4 Modules in the proposed system

The modules proposed for the system are outlined below.

Module 1: Predict user's learning content (Training of Data Set and testing the neural with dataset to get the predicting of learning content)

Module 2: Psychometric model

Module 3: Predict user's time table

Module 4: User prior knowledge

Module 5: Monitoring emotions of user while learning takes place

Module 6: Displaying time table (html)
# 4 Results and Interpretation

## 4.1 Training of Data Set

The personalised content page for the learner is shown below

<figure><img src="image_4.png" /><figcaption>Fig. 3. Personalised Content Page</figcaption></figure>

In this training of dataset, nine inputs, ten neurons and four outputs are used to build the neural network. The neural is run for each interval learned to get the classification of the interval. The average emotion and performance for the specific time interval is used as input.
Each emotion is represented as:

<table><caption>**Table 1. Emotions**</caption><thead><tr><td>Emotion</td><td>Binary representation</td></tr></thead><tbody><tr><td>Anger</td><td>00000001</td></tr><tr><td>Contempt</td><td>00000010</td></tr><tr><td>Disgust</td><td>00001000</td></tr><tr><td>Fear</td><td>00010000</td></tr><tr><td>Happiness</td><td>00010000</td></tr><tr><td>Neutral</td><td>00100000</td></tr><tr><td>Sadness</td><td>01000000</td></tr><tr><td>Surprise</td><td>10000000</td></tr></tbody></table>

The performance is represented as double integer. An example of input to the neural network would be 0,0,0,1,0,0,0,0,0,0.80

<table><caption>**Table 2. Output**</caption><thead><tr><td>Output</td><td>Meaning</td></tr></thead><tbody><tr><td>001</td><td>Most Favorable</td></tr><tr><td>010</td><td>Favorable</td></tr><tr><td>100</td><td>Not Favorable</td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>**Fig. 4. Input and Output using Neural Network**</figcaption></figure>
## 4.2 Results

A number of experiments have been performed to see if the emotion of the learner is correctly detected. Since the monitoring works in background, this module has been tested where the face of the person is visible on the page. The image in left is the live streaming and the one in the right is the image shot. An alert is displayed when the emotion is recognized.

<figure><img src="image_6.png" /><figcaption>Fig.5. Testing the emotion monitoring system</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig.6. Testing the emotion monitoring system - 2</figcaption></figure>

Figure 6 below shows a personalized time table where learning would be most conducive for a specific learner.
<figure><img src="image_8.png" /><figcaption>Fig. 7. Personalised Time Table</figcaption></figure>

# 5 Discussions

## 5.1 Testing the Accuracy

For the accuracy of training data set, the total network error for each iteration of the neural network is observed. As the number of iteration performed increased, the total network error decreases. The figure below shows the relations between iteration performed a total network error.
<figure><img src="image_9.png" /><figcaption>Fig. 8. Accuracy</figcaption></figure>

The table below shows the total network error for the iteration performed. It shows the same concept as the graph above. At the first iteration, the total network error was 0.5009843578823818, which is not really good for a start however at the 493th iteration the total network error is 0.009950455956534508 which is really good.

<table><caption>Table 3. Total network error at each iteration</caption><thead><tr><td>Iteration number</td><td>Total network error</td></tr></thead><tbody><tr><td>1</td><td>0.5009843578823818</td></tr><tr><td>2</td><td>0.47491693836473853</td></tr><tr><td>3</td><td>0.4696483547660897</td></tr><tr><td>4</td><td>0.46478039443432384</td></tr><tr><td>5</td><td>0.45985672557279855</td></tr><tr><td>..</td><td>..</td></tr><tr><td>20</td><td>0.3932929797443801</td></tr><tr><td>..</td><td>..</td></tr><tr><td>50</td><td>0.2555228021830213</td></tr><tr><td>..</td><td>..</td></tr><tr><td>100</td><td>0.11904842671535672</td></tr><tr><td>..</td><td>..</td></tr><tr><td>150</td><td>0.08475942876046237</td></tr><tr><td>..</td><td>..</td></tr><tr><td>200</td><td>0.07397413704542873</td></tr><tr><td>..</td><td>..</td></tr></tbody></table>
<table><tr><td>250</td><td>0.06740893346060287</td></tr><tr><td>..</td><td>..</td></tr><tr><td>300</td><td>0.054403972658013805</td></tr><tr><td>..</td><td></td></tr><tr><td>350</td><td>0.03493140578900644</td></tr><tr><td>..</td><td></td></tr><tr><td>400</td><td>0.021648853428465915</td></tr><tr><td>..</td><td></td></tr><tr><td>450</td><td>0.013942979512677475</td></tr><tr><td>..</td><td></td></tr><tr><td>493</td><td>0.009950455956534508</td></tr></table>

## 5.2 Critical Analysis of Proposed System

The proposed system uses a form of supervised learning because it is based on storing data from the user in a database and then generating data. A questionnaire has been designed to get the learning profile of the user. Based on the answers provided, the learning profile was determined and saved in the database for later for generating learning content. Whenever the user is learning, his/her emotions are determined and an algorithm is used to determine the emotion expressed for the longest time period in the interval. The average emotion for the time interval, together with the performance is used to predict if the time interval learned is favourable, most favourable or not favourable using Neural Network. For each test and for each question, the probability that a particular learner gets the right answer is calculated using the Item Response Theory (IRT) model, with the knowledge level and difficulty of question as parameters. The expected performance is calculated using those probabilities (all questions have the same marks). Actual performance is then compared with expected one. Any increase or decrease in performance is reported and the user profile is updated. Additionally the learning profile of each user is updated each time the system is used. For example, when a test is taken, the learning profile is updated by using reinforcement rule to lower the content level if performance is low.

## Comparison against Knewton adaptive learning by Knewton

This section compares the proposed system with Knewton which had been introduced in the previous sections of this report. Knewton uses the unsupervised learning method to get its data points whereas the proposed system uses supervised learning method. While Knewton delivers courses based on the most recent student profile, the proposed system uses learning style to determine which learning content is most appropriate for the user. However, one major advantage is that Knewton depends on a large volume of data to do precise clustering and prediction whereas the proposed system accommodates for smaller user population and does not require a large sample size to determine learning profiles. Most importantly the proposed system does something that Knewton does not. The proposed system predicts the customised timetable of the learner by
adding a new dimension to adaptivity which is that of emotion. The system recognizes the learner's emotion while learning takes place and this emotion is evaluated to get a favourable time interval for effective learning to take place.

## 5.3 User Testing

### Testing of non-adaptive system scenario

Student was provided a PowerPoint presentation on the topic to learn and upon completion was given a multiple choice test to ascertain his newly acquired knowledge. Student were allowed fifteen minute to learn the content and five minute to do the test. Google form was used to create PowerPoint and questionnaire. Since the topic to be learned was C# programming, users with no prior background in programming was chosen. A sample of 17 students was used.

<figure><img src="image_10.png" /><figcaption>Fig. 9. Results for non-adaptive scenario</figcaption></figure>

<table><caption>Table 4. Statistical Analysis of non-adaptive scenario</caption><thead><tr><td>Statistical Operation</td><td>Explanation</td><td>result</td></tr></thead><tbody><tr><td>Mean</td><td>Average score of student</td><td>5.78</td></tr><tr><td>Mode</td><td>Maximum occurrence of a score</td><td>2,7,9</td></tr><tr><td>Median</td><td>Middle value separating the distribution</td><td>6</td></tr></tbody></table>
<table><tr><td>Variance</td><td>How far a dataset is spread out</td><td>6.47</td></tr><tr><td>Standard Deviation</td><td>How spread out numbers are</td><td>2.54</td></tr></table>

### Testing of -adaptive system scenario:

Student performance was tested using the adaptive system. A sample of 17 students with no prior knowledge in programming was used. Student was given fifteen minute to learn the content and five minute to complete the test. The topic to be learned was C# programming. Test Question was the same as in the scenario of the non-adaptive system scenario below.

<table><caption>**Table 5. Statistical Analysis of adaptive scenario**</caption><thead><tr><td>Statistical Operation</td><td>Explanation</td><td>result</td></tr></thead><tbody><tr><td>Mean</td><td>Average score of student</td><td>6.18</td></tr><tr><td>Mode</td><td>Maximum occurrence of a score</td><td>7</td></tr><tr><td>Median</td><td>Middle value separating the distribution</td><td>5</td></tr><tr><td>Variance</td><td>How far a dataset is spread out</td><td>2.10</td></tr><tr><td>Standard Deviation</td><td>How spread out numbers are</td><td>1.45</td></tr></tbody></table>

### Discussion of result:

Student given personalized learning content had an average score above passing mark and are nearly clustered. The result was satisfactory given users were put under a time limit to learn a whole new concept. User feedback was used to improve details in each section and more examples were suggested.

## 6 Conclusion

Nowadays e-learning application is being very responsive but there is a problem as each individual's needs is different. The one-size-fits-all is not the solution to build learning platform as each learner is different. These were what has been confirmed at the end of testing phase. It is found to be true that adaptivity in software is the key for future application building success. This research brings forward a novel aspect of
adaptivity by considering another intrinsic factor that is that of the learner's emotion. Current research have so far been concentrating on intrinsic factors such as learner's prior knowledge, learning pace and learning style. Machine learning techniques such as neural network to make prediction of personalized content and classification of time interval proved to be very useful. The accuracy of the proposed emotion-based adaptive learning system is also a conclusive factor. Future works may include the use of body language recognition and human gesture recognition to get the mood of the user. Additionally identifying hand gesture to navigate through content and speech-based assessment of user would be interesting features to implement in the future to make the content rendering mechanism more adaptive and responsive.

## References

1. Brusilovsky, P., Stock, O. and Strapparava, C. (2000). Adaptive hypermedia and adaptive Web-based systems. Berlin: Springer.

2. Corradino, C. and Fogarty, K. (2017). Positive Emotions and Academic Achievement - Applied Psychology OPUS - NYU Steinhardt. [online] Steinhardt.nyu.edu. Available at: http://steinhardt.nyu.edu/appsych/opus/issues/2016/spring/corradino_fogarty [Accessed 9 August. 2019].

3. EdSurge. (2016). Software | Adaptive Learning | EdSurge. [online] Available at: https://www.edsurge.com/research/special-reports/adaptive-learning/software [Accessed 29 July. 2019].

4. Fatahi, S. (2019). Educ Inf Technol 24: 2225. https://doi.org/10.1007/s10639-019-09868-5

5. Febrilia, I. & Warokka, A. & Abdullah, H.H. (2011). University Students' Emotion State and Academic Performance: New Insights of Managing Complex Cognitive. Journal of e-Learning & Higher Education. 2011. 10.5171/2011.879553.

6. Heckerman, D., Geiger, D. and Chickering, D. (1995). Learning Bayesian networks: The combination of knowledge and statistical data. Machine Learning, [online] 20(3), pp.197-243. Available at: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-95-06.pdf [Accessed 7 August 2019].

7. Hinton, C., Miyamoto, K., & Della-Chiesa, B. (2008). Brain research, learning and emotions: Implications for education research, policy and practice. European Journal of Education, 43(1), 87-103.

8. Khalfallah, J. and Slama, J.B.H. (2018). The effect of emotional analysis on the improvement of experimental e-learning systems. Computer Applications in Engineering Education. Wiley Online Library. https://doi.org/10.1002/cae.22075

9. Knewton adaptive learning. (n.d.). 1st ed. [ebook] Knewton. Available at: https://www.knewton.com [Accessed 30 July. 2019].

10. Lane H.C., D'Mello S.K. (2019) Uses of Physiological Monitoring in Intelligent Learning Environments: A Review of Research, Evidence, and Technologies. In: Parsons T., Lin L., Cockerham D. (eds) Mind, Brain and Technology. Educational Communications and Technology: Issues and Innovations. Springer, Cham.

11. Oxman, S. and Wong, W. (2014). White Paper: Adaptive Learning Systems. DV X Innovations DeVry Education Group.

12. Paramythis, A. and Loidi-Reisinger, S. (2003). Adaptive Learning Environment and e-Learning Standards. In R. Williams (Ed.), Proceedings of the 2m1 European Conference on
e-Learning (ECEL2003), Glasgow, Scotland, 6-7 November, pp.369-379. Academic Conferences International Reading 2003; ISBN: 0-9544577-4-9.
13. Raschka, S. (2014). Naive Bayes and Text Classification. [online] Sebastian Raschka's Website. Available at: http://sebastianraschka.com/Articles/2014_naive_bayes_1.html [Accessed 7 August 2019].
14. Seel, N. (2012). Encyclopedia of the sciences of learning. 1st ed. [S.l.]: Springer, pp.166-167.
15. Truong, H.M. (2016). Integrating Learning Styles and adaptive e-learning system: current developments, problems and opportunities. Computers in Human Behaviour. Volume 55, Part B, February 2016, Pages 1185-1193.