SUGISAKAMASANORI

Journal of Robotics, Networking and Artificial
Life Vol. 10(4); March (2024), pp. 357–361
ISSN (Online): 2352-6386; ISSN (Print): 2405-9021
https://alife-robotics.org/jrnal.html

Research Article

YOLOv5 Based Student Engagement and Emotional States
Detection in E-ClassesShuai Wang1,2, Abdul Samad Shibghatullah3, Kay Hooi Keoy4, Javid Iqbal5,
Institute of Computer Science and Digital Innovation, UCSI University, Cheras, Kuala Lumpur, Malaysia
Department of Physics and Electronic Engineering, Yuncheng University, Yuncheng, Shanxi, China
3
College of Computing & Informatics, Universiti Tenaga Nasional, Kajang, Selangor, Malaysia
4Graduate Business School, UCSI University, Cheras, Kuala Lumpur, Malaysia
5Department of Computing and Information Systems, Sunway University, Petaling Jaya, Selangor, Malaysia
12ARTICLE INFO
Article History
Received 28 November 2023
Accepted 18 July 2024
Keywords
Cyberbullying detection
Emotion recognition
Deep learning
E-learning
YOLO

ABSTRACTThe rapid expansion of E-learning environments has highlighted the critical issue of cyberbullying
within digital classrooms. This study introduces a novel approach for early detection ofcyberbullying by analyzing student engagement and emotional states in real time. Our SERYOLO model fuses an advanced You Only Look Once version 5 (YOLOv5) with a StudentEmotion Recognition system, enriched by sophisticated methodological improvements. It features
Soft NMS to refine the Non-Maximum Suppression (NMS) process, embeds the Channel
Attention (CA) module to augment the network's backbone, and employs Enhanced Intersection
over Union (EIOU) for bounding box regression. This method proactively detects changes instudent engagement and emotional states, providing an effective mechanism for the early
detection and management of cyberbullying in E-learning environments.

© 2022 The Author. Published by Sugisaka Masanori at ALife Robotics Corporation Ltd.
This is an open access article distributed under the CC BY-NC 4.0 license(http://creativecommons.org/licenses/by-nc/4.0/).

mockery, threats, and other malicious behaviors. The
virtual classroom (or online classroom), a new arena forboth teachers and students, poses potential risks ofcyberbullying that can severely impact the emotional andpsychological well-being of its participants, as well as
hinder the educational experience. Moreover,cyberbullying has been linked to a decline in student
engagement. The emotional state is also vulnerable tofluctuations due to the stress and anxiety induced by thepresence of cyberbullying, leading to a less conducive
atmosphere for educational growth and exchange.
In online learning classrooms, communication between
students and teachers is facilitated solely through screens,
limiting the educator's assessment of comprehension to
the students' facial expressions and emotional states.
Analyzing these expressions can assist teachers in better
understanding the students' levels of engagement and in
making timely adjustments to their teaching methods [1].

1. Introduction
In the era of the big data revolution, propelled bymachine learning (ML) and artificial intelligence (AI),and with the rapid growth of virtual learning, onlinesocial media platforms have become an integral part ofmodern educational frameworks. The unexpected adventof the COVID-19 in 2020 has significantly alteredtraditional approaches to learning, work, and daily life.
The previously unchallenged model of face-to-face
instruction is now confronted with significant obstacles.
Many academic institutions have turned to online
instruction as a practical alternative. However, this shift
to digital learning environments has brought forth a rangeof challenges, with cyberbullying being particularly
prominent. Cyberbullying, alternatively termed as online
harassment or digital aggression, involves the intentionaluse of electronic means to cause harm through aggression,
Corresponding author E-mail: 1002268166@ucsiuniversity.edu.my357

algorithms, and detecting emotional cues. These
approaches aim to anticipate and mitigate bullying
incidents in digital learning environments.
Additionally, numerous studies highlight theimportance of educational initiatives and improving
cyberbullying awareness among faculty and students.
These efforts are essential for creating a safe andsupportive
onlinelearning
environment[6].
Implementing these strategies not only protects against
the harmful effects of cyberbullying but also reinforces
the commitment to maintaining the dignity and wellbeing of all participants in virtual education.

In 1971, Paul Ekman et al. [2] carried out comprehensive
studies of human facial expressions, distinguishing six
fundamental emotional expressions: happiness, surprise,fear, sadness, disgust, and anger. These basic emotions
are pivotal for interpreting students' reactions in theclassroom. Positive emotions like happiness and surpriseindicate active engagement and willingness to learn,
while negative emotions such as sadness, anger, fear, anddisgust suggest a lack of interest or inattention. Neutral
expressions may suggest moderate engagement. Thus,
monitoring students' facial expressions in an online
setting is crucial for effective teaching and learninginteractions.
Amidst these challenges, educational entities and digitalplatforms are increasingly focusing on the effective
identification and mitigation of cyberbullying within thevirtual classroom milieu. The advent of facial expression
recognition (FER) technology, with its prospects foremotion analytics and affective state monitoring, hasgarnered considerable interest. Despite the burgeoning
interest, the extant research leveraging the YOLO familyfor facial expression recognition remains sparse. This
paper introduces an innovative approach to facialexpression recognition (FER) predicated on an enhanced
YOLOv5 architecture, intended for the prompt
identification and amelioration of cyberbullying in theonline classroom setting.
Our paper's key contributions are: (1) Enhanced SoftNMS: Replaces the standard NMS, enabling the captureof subtle expressions crucial for detecting student
emotional states. (2) CA Mechanism: Improves therecognition of complex behaviors in scenarios involving
multiple students. (3) Loss Function Enhancement:
Utilizes EIOU and Focal Loss to effectively mitigate thechallenges posed by sample imbalance.

2.2. YOLOv5
In May 2020, Ultralytics LLC introduced YOLOv5 [7],an efficient convolutional neural network designed forswift image processing, capable of remarkable inferencespeeds, reaching up to 140 frames per second (FPS), thus
satisfying the stringent real-time demands for video
analysis. This model, crafted with Python, is a testament
to the progress in lightweight deep learning architectures.
YOLOv5 is available in four scalable variants: YOLOv5s,YOLOv5m, YOLOv5l, and YOLOv5x, all sharing afundamental design philosophy. These variants aredistinguished by parameters such as width_multiple anddepth_multiple parameters, which regulate the network's
capacity and complexity, respectively. Consequently,
each variant offers a unique balance between model size
and accuracy, allowing for tailored selection based on thecomputational and accuracy requirements of specificapplications. Among these, the Yolov5s variant, with its
reduced depth and parameter count, stands out as an
optimal candidate for real-time applications such as facial
expression analysis, where rapid inference is paramount.
By optimizing network structures and training strategies,
YOLOv5 enables rapid and accurate facial expression
recognition in videos, crucial for assessing student
engagement and emotional states in classrooms.
Research shows that integrating YOLOv5 with emotion
classification models can enhance the analysis of
students' emotional states, aiding teachers inunderstanding learning experiences and adjusting
instructional methods to improve quality.
For cyberbullying detection, YOLOv5 can be combinedwith technologies like natural language processing (NLP)
to identify and prevent bullying behaviors in online
environments. Analyzing students' online behaviors andlanguage expressions allows YOLOv5 to promptly detect
potential bullying incidents and facilitate timely
interventions.
In this research, we employ YOLOv5s version 6.0 as afoundation for further advancements. We present the2. Related work
2.1. Cyberbullying in E-classes
The rise of online learning, accelerated by the COVID19 pandemic, has brought both opportunities andchallenges to the academic community. A significant
challenge is cyberbullying, which includes different
types of online harassment within virtual classrooms,
such as verbal abuse, intimidation, misinformation, andexclusion from classroom activities [3], [4]. Thesebehaviors have profound impacts on students' academic
performance, mental health, and social relationships [5].
Addressing cyberbullying in virtual classrooms iscritical. Research has explored various methods toidentify and prevent such behavior, including analyzing
behavioral patterns, parsing linguistic content, examining
social network structures, applying machine learning358

efficacy of the traditional NMS approach. The ultimate
calculation is expressed in Eq. (1).

SER-YOLO model, a groundbreaking methodology thatharnesses the sophisticated YOLOv5 framework toaccurately detect cyberbullying incidents via facial
expression analysis in digital classrooms and online
learning spaces. This model represents a significant step
towards creating a safer and more emotionally intelligent
virtual educational experience.

𝑖𝑖𝑖𝑖𝑖𝑖(𝐵𝐵,𝑏𝑏𝑖𝑖 )2 )𝑠𝑠𝑖𝑖 = 𝑠𝑠𝑖𝑖 𝑒𝑒 −𝜎𝜎, ∀𝑏𝑏𝑖𝑖 ∉ 𝐷𝐷

(1)This method ensures that some high-scoring boundingboxes are preserved as accurate detections in subsequent
computations.
3.2.2. Coordinate attention module

3. Methodology

In our research, we introduce the Coordinate Attention(CA) mechanism [9], a novel approach that integrates
spatial coordinates with channel attention. Our
experiments substantiate that incorporating the CA
module into the core network framework enhances
detection accuracy. This improvement is particularly
noticeable in classroom environments with multiple
students, especially those seated in distant rows from thecamera's view. These advancements are credited to theCA module (see Fig. 1), which strengthens channel
characteristics in the feature map, enabling the capture ofmore precise and influential data. The networkarchitecture with the CA module integrated into thebackbone is illustrated in Fig. 2.

The current literature on facial expression recognition
leveraging the YOLOv5 framework, while not extensive,
presents a compelling opportunity for monitoring student
engagement and emotional states in online educationalspaces. Our contribution to this field builds upon theexisting YOLOv5s-6.0 model by introducing a series ofenhancements that significantly elevate its performanceon experimental datasets.
3.1. Data collection and processing
To precisely discern the facial expressions associatedwith varying degrees of students' comprehension—
namely, mastery, confusion, and non-mastery—weconducted a questionnaire-based survey. This study
aimed to explore the correlation between these levels ofunderstanding and the corresponding facial expressions
exhibited. The survey spanned higher vocational colleges
and universities, yielding a total of 520 responses (287male, 233 female). We focused on seven fundamental
emotional expressions: anger, surprise, disgust,happiness, fear, sadness, and neutral.
In this research, we utilized both the Fer2013 dataset
and a proprietary collection of student facial expressions
captured within an educational setting. The Fer2013
dataset [8], known to include non-facial and incorrectlylabeled images, underwent a meticulous process ofselection, cleansing, and filtration to enhance theprecision of our experiments, culminating in a refined setof 11,000 images. Concurrently, our in-house dataset wascurated by surveilling classroom videos, compiling 668
images that were subsequently labeled after extracting
relevant frames from the recorded sessions.

Fig. 1 The mechanism of CA.

3.2. YOLOv5 Enhancements
3.2.1. Optimized NMS Mechanism
The methodologies of NMS and its variant, Soft NMS,
rely on the confidence scores derived from classification
predictions, with higher scores signifying greater
localization precision. For the advancement of detection
accuracy in this research, we have implemented anenhanced version of Soft NMS, which surpasses theFig. 2 The enhanced network of YOLOv5s-6.0.

359

3.2.3. Refined EIOU Loss

3.9% increase, while mAP@0.5 rose from 85.9% to88.9% after 150 epochs, marking a 3.0% gain. These
results confirm SFER-YOLO's enhanced effectiveness indetecting and tracking student engagement.

Our approach enhances the original loss function byintegrating the Efficient-IOU (EIOU) loss function tocalculate height and width-related loss values, thereby
improving upon conventional aspect ratio estimation
based on CIOU. Furthermore, we have incorporated
Focal Loss to mitigate class imbalance, thereby achieving
more balanced optimization across diverse samples. The
formulas for EIOU and Focal Loss [10] are provided inEq. (2) and (3), respectively.
𝐿𝐿𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸 = 𝐿𝐿𝐼𝐼𝐼𝐼𝐼𝐼 + 𝐿𝐿𝑑𝑑𝑑𝑑𝑑𝑑 + 𝐿𝐿𝑎𝑎𝑎𝑎𝑎𝑎= 1 − 𝐼𝐼𝐼𝐼𝐼𝐼 +𝜌𝜌2 (𝑏𝑏,𝑏𝑏𝑔𝑔𝑔𝑔 )(𝑤𝑤 𝑐𝑐 )2 +(ℎ𝑐𝑐 )2+𝜌𝜌2 (𝑤𝑤,𝑤𝑤 𝑔𝑔𝑔𝑔 )(𝑤𝑤 𝑐𝑐 )2

𝐿𝐿𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹 (𝑝𝑝𝑡𝑡 ) = −𝛼𝛼(1 − 𝑝𝑝𝑡𝑡 )𝛽𝛽 𝑙𝑙𝑙𝑙( 𝑝𝑝𝑡𝑡 )+𝜌𝜌2 (ℎ,ℎ𝑔𝑔𝑔𝑔 )(ℎ𝑐𝑐 )2

Table 1 The hyperparameter settings of model training.
Hyper-parametersresolution
optimizer
lr0
lrf
weight_decay
epochbatch_size
IOU threshold

(2)
(3)

where, α is a factor serves to address sample imbalance,and 𝛽𝛽 adjusts the weights between hard and easy samples.

Value
640x640
Adam
0.01
0.1
0.0005300
160.5

4. Experimental results

Our study's experimental setup featured a 64-bit
Windows 10 OS, powered by an 11th Gen Intel Core i511400H processor and an NVIDIA RTX 3050 graphics
card. We utilized PyTorch 1.8 for deep learning, within aPython 3.8 version environment. Table 1 details thehyperparameter settings used during the model training
process. The training process spanned 300 epochs.
Fig. 3 illustrates the overall SER-YOLO method. The
model training utilized pre-trained weights (yolov5s.pt)
and a configuration file (SER.YAML). Logs and weight
files were saved throughout the training process.
For comprehensive assessment, we conducted extensiveexperiments on the self-constructed dataset and Fer2013.
Table 2 compares the performance of the YOLOv5 andSFER-YOLO models on experimental datasets [11].
SFER-YOLO significantly outperforms the baseline
YOLOv5. Precision improved from 83.4% to 87.3%, aFig. 3 The overall process of SER-YOLO model

Table 2 Experimental results on our self-constructed and FER-2013 dataset
Self-constructed Dataset
EmotionYOLOv5

FER-2013

SER-YOLO

YOLOv5

SER-YOLOPmAP@0.5PmAP@0.5PmAP@0.5PmAP@0.5

happy

0.816

0.853

0.851

0.887

0.863

0.918

0.876

0.911

sad

0.845

0.867

0.889

0.909

0.636

0.594

0.669

0.626

neutral

0.842

0.857

0.879

0.871

0.694

0.799

0.722

0.515

all

0.834

0.859

0.873

0.889

0.731

0.770

0.756

0.784

360

5. Conclusion
This research addresses the pivotal issues of inadequate
real-time detection capabilities and the lack of timelinessin identifying students' facial expressions within the
intricate backdrop of e-learning settings, concurrently
focusing on the mitigation of cyberbullying incidents.
Our SER-YOLO model introduces an enhanced soft
NMS to replace the conventional method, enrichesfeature extraction through an optimized CoordinateAttention (CA) mechanism, and refines bounding box
representation by adopting the EIoU_loss function
instead of CIoU_loss. The empirical findings indicate anotable 3.9% enhancement in precision (P) and a 3.0%
increase in mAP@0.5 on our self-constructed dataset.
Subsequent studies will broaden the scope to encompass
not only facial expressions but also body postures andgestures, thereby providing a more comprehensive
understanding of the dynamics within online classrooms
and strengthening cyberbullying prevention throughdeeper insights into student participation and emotionalstates.

10. C. Liu, et al., "Powerful-IoU: More straightforward and fasterbounding box regression loss with a nonmonotonic focusing
mechanism", Neural Networks, 170, 276-284, 2024.
11. S. Wang, A. S. Shibghatullah, J. Iqbal, K. H. Keoy, "OnlineClassroom Student Engagement Analysis using Enhanced
YOLOv5", ICAROB, 974-978, 2024.

Authors Introduction
Mr. Shuai Wang
He is currently pursuing Ph.D. in Computer
Science at the Institute of Computer
Science and Digital Innovation (ICSDI),UCSI University, Kuala Lumpur, Malaysia.
He also works in the Department of Physics
and Electronic Engineering at Yuncheng
University, Yuncheng, Shanxi, China. His researchinterests include computer vision, machine learning, andmulti-modal learning.
Dr. Abdul Samad Bin Shibghatullah
He earned his Ph.D. in Computer Sciencefrom Brunel University, Uxbridge, UK.
He is now an Associate Professor
Universiti Tenaga Nasional, Malaysia. His
research interests focus on optimization,modeling, and scheduling.

Acknowledgment
This study was supported by the 2023 Applied Research
Project of Yuncheng University (Project no. YY-202312).
References
1.

2.
3.

4.
5.

6.
7.

8.

9.

Dr. Keoy Kay Hooi
He earned his Ph.D. from Sheffield Hallam
University, UK, in 2006. He is currently an
Associate Professor at the Centre for
Business Informatics and Industrial
Management, Graduate Business School(GBS), UCSI University, Malaysia. His
research interests include machine learningand block chain.

J. Bao, X. Tao, Y. Zhou, "An Emotion Recognition Method
Based on Eye Movement and Audiovisual Features in MOOC
Learning
Environment", IEEETransactions
onComputational Social Systems, 2022.
P. Ekman, W.V. Friesen, "Constants across cultures in theface and emotion", Journal of personality and socialpsychology, 17.2 (1971), 124.
M. Khairy, T.M. Mahmoud, A. Omar, T. Abd El-Hafeez,"Comparative performance of ensemble machine learning for
Arabic cyberbullying and offensive language detection",
Language Resources and Evaluation, 58(2), 695-712, 2024.
A. Bozyiğit, S. Utku, E. Nasibov, "Cyberbullying detection:
Utilizing social media features", Expert Systems withApplications, 179: 115001, 2021.
R. Kumar, A. Bhat, "A study of machine learning-basedmodels for detection, control, and mitigation of cyberbullying
in online social media", International Journal of Information
Security, 21(6), 1409-1431, 2022.
G.W. Giumetti, R.M. Kowalski, "Cyberbullying via social
media and well-being", Current Opinion in Psychology, 45,101314, 2022.
K. Zhou, et al., "Evaluation of BFRP strengthening andrepairing effects on concrete beams using DIC and YOLO-v5
object detection algorithm", Construction and Building
Materials, 411, 134594,2024.
P. Giannopoulos, I. Perikos, I. Hatzilygeroudis, "Deeplearning approaches for facial emotion recognition: A casestudy on FER-2013", Advances in hybridization of intelligent
methods: Models, systems and applications, 1-16, 2018.
Y. Li, M. Zhang, C. Zhang, H. Liang, P. Li, "YOLO-CCS:
Vehicle Detection Algorithm Based on Coordinate Attention
Mechanism", Digital Signal Processing, 104632, 2024.

Dr. Javid Iqbal Thirupattur
He is a Senior Lecturer at Sunway
University, Malaysia, holding a Ph.D. inInformationandCommunication
Technology from the National Energy
University, Malaysia. His research interests
are in multimedia, augmented reality, andvirtual reality.

361