A Multimodal Human-Computer Interaction for Smart Learning System

AbstractAbstractThe rise of digitalization and computing devices has transformed the educational landscape, making traditional teaching methods less productive. In this context, early and continuous user interaction is crucial for designing and developing effective learning applications. The field of Human-Computer Interaction (HCI) has seen significant technological growth, enabling educators to provide quality educational services through smart input and output channels. However, to prevent students from discontinuing their studies and help them grow their careers, a multimodal HCI approach is needed. This paper proposes a multimodal deep learning multi-layer Convolutional Neural Network (CNN) to improve the educational experience. Our designed system aims to create a promising solution for improving the educational experience and enabling educators to provide high-quality educational services to students. Our implementation results show promising real-time performances, including a high success rate in a constriction learning concept, a quality interaction experience, and enhanced educational services. We evaluated the accuracy of five multimodal inputs, including Finger Touch (FT), Hands Up (HU), Hands Down (HD), Voice Command (VC), and Click/Typing (CT). The results indicate an average accuracy of 90.8%, 87%, 88.6%, 91.8%, and 87%, respectively, demonstrating the effectiveness of our proposed approach.Keywords: Human-Computer Interactionmultimodal HCImultilayer CNNdeep learning modelsmart Learning SystemView correction statement:Correction Disclosure statementNo potential conflict of interest was reported by the author(s).Ethical approvalWe did not use animals and Human participants in the study reported in this work.Consent formFor this type of study consent for publication is not required.Informed consentFor this type of study informed consent is not required.Additional informationFundingThe authors did not receive support from any organization for the submitted work.Notes on contributorsTareq Mahmod AlzubiTareq M. Alzubi is an Assistant Professor at Al-Balqa Applied University, Jordan. Received PhD degree in Computer Graphics from Santiago De Compostela University, Spain. Tareq works and researches in multi-disciplinary fields involving Human-Computer Interaction, Computer Vision, and Machine Learning.Jafar A. AlzubiJafar A. Alzubi is an Associate Professor at Al-Balqa Applied University, Jordan. Received PhD degree in Advanced Telecommunications from Swansea University - UK. Jafar works and researches in a multi and interdisciplinary environment involving Machine Learning, Cyber Security, and Networks Security. which resulted in publishing more than sixty articles.Ashish SinghAshish Singh is an Assistant Professor, the School of Computer Engineering, Kalinga Institute of Industrial Technology, Bhubaneswar, Odisha. He has completed his M.Tech and PhD degree from NIT Patna, India. He is working in the field of Healthcare Security, Edge Computing, Cloud Computing, and the Internet of Things.Omar A. AlzubiOmar A. Alzubi received his PhD degree in Computer and Network Security from Swansea University, United Kingdom. Currently, he is a professor at Al-Balqa Applied University, Jordan. Professor Alzubi research interests include cyber security, machine learning, and networks. His cumulative research experience resulted in the publishing more than fifty articles.Murali SubramanianMurali Subramanian received his PhD from VIT University in 2017. He is presently serving her alma mater as an Associate Professor. He has published around 25 papers in various International Journals. His research interests include Data Aggregation in Wireless Sensor Network, Data analysis, Artificial intelligence & Biologically Inspired Optimization Algorithms.Correction StatementThis article was originally published with errors, which have now been corrected in the online version. Please see Correction (http://dx.doi.org/10.1080/10447318.2023.2220184)