JAIST Repository
https://dspace.jaist.ac.jp/Title

遠隔学習プロセスにおける学習者のエンゲージメントに関する自動認識と分析

Author(s)SHOFIYATI NUR KARIMAH

CitationIssue Date

2024-03

Type

Thesis or Dissertation

Text version

ETDURLhttp://hdl.handle.net/10119/19068

RightsDescription

Supervisor: 長谷川 忍, 先端科学技術研究科, 博士

Japan Advanced Institute of Science and Technology

氏

名

SHOFIYATI NUR KARIMAH

学

位の種類

博士（情報科学）学

位記番号

博情第 527 号

学 位 授 与 年 月 日
論文題目論 文 審 査 委 員

令和 6 年 3 月 22 日
Automatic Recognition and Analysis of Learners’Engagement inDistance Learning Process長谷川 忍

北陸先端科学技術大学院大学教授

小谷 一孔

同教授

池田 心同教授

岡田 将吾

同准教授

村上 正行

大阪大学教授論文の内容の要旨

Engagement is an essential component of the learning processes associated with positive
learning outcomes. Measuring learner engagement in learning processes is important for providing
insights for enhancing learning activities. Because the learning paradigm has shifted to enable moredistance learning practices, machine learning-based automatic engagement estimation methods havebeen proposed as a new way to measure learner engagement. Nevertheless, most existing methods arebuilt standalone and have yet to be integrated into actual distance learning practice. Furthermore,
implementing automatic engagement estimation should ensure technological and ethical impact
responsibilities.
This study aims to provide an intermediary knowledge and solution to analyse learners’
engagement in the distance learning process by addressing the main research question: ”How do
educators or education institutions safely apply automatic engagement estimation in their distancelearning process?” A systematic review is conducted to gain basic knowledge of the current trend of
automatic engagement estimation in the literature to achieve this goal. The engagement types,
datasets, and methods are defined and theoretically investigated. Secondly, the technical investigation
to understand the basic requirement for automatic engagement estimation is done by building anengagement estimation module using deep learning methods. We introduce a design principle forend-to-end integration of real-time automatic engagement estimation in distance learning practice.

Thirdly, we introduce a design principle for the ethical implementation of automatic engagement
estimation so that the technology can benefit actual distance learning in practice.
From the literature review, we found that clearer engagement definitions and cues are crucial fordeveloping an applicable automatic engagement estimation. However, there is no clear taxonomy to define
engagement, especially for distance learning implementations. Therefore, we introduced a taxonomy of engagement
definitions and cues, categorized the engagement datasets, and conducted method categorization, which mainlyutilised machine learning-based methods. The combination of a clear definition of engagement and suitablemachine learning methods allows learners’ engagement during learning activities to be measured automatically,
including in human-human interactions, human-computer interactions, and human-robot interactions.
Two deep learning models were experimented with, i.e., long short-term memory (LSTM) andconvolutional neural network (CNN), and a publicly available engagement dataset. However, we found that classic
machine learning would be the best practice, especially for real-time engagement estimation, while LSTM is lessfeasible for practical implementation compared to CNN from a runtime perspective. Furthermore, a framework forreal-time automatic engagement estimation is proposed for implementation in distance learning practices.
Furthermore, we introduce system designs and prototypes for both an asynchronous and a synchronous setting.
We propose the design of RAMALAN, a real-time engagement assessment for asynchronous distancelearning, and MeetmEE (pronounced as ’meet me’), a real-time video conference integrated with automatic
engagement estimation for synchronous distance learning. The MeetmEE prototype was deployed in a pilot
experiment to evaluate the MeetmEE system design. A total of 20 participants joined the experiment in a one-hour
meeting session with the author via MeetmEE online either as educators (n = 13; 65%) or learners (n = 7; 35%) with
60%. The participants completed two survey forms (Forms A and B) based on their roles in their affiliations. The
experiment results of Form A demonstrate that most of the responses were very positive to the automaticengagement estimation concept, represented in MeetmEE. MeetmEE is favourable for 70% of the participants,
where, for educators, this technology will motivate them to improve their teaching strategies and give support totheir students, while students can measure their own engagement as well. Furthermore, the results of Form B
showed a positive evaluation, demonstrating that MeetmEE is sufficient, particularly in scales of stimulation,

attractiveness, perspicuity, and novelty. However, MeetmEE is perceived as relatively low in terms of dependability
and efficiency.
Finally, the user evaluation results are considered to construct the design principle of ethicalimplementation. The automatic engagement estimation implementation’s design principle incorporates technicaland operational measures. While the current automatic engagement estimation studies focused on only the ICT pointof view instead of the feasibility of the actual education process, the development of an engagement estimation
design principle incorporated with its real-time application in the distance learning process is a part of the originalityof this research. We believed that this contribution would be beneficial in designing a broader distance learning
framework where the learners’ internal state and affective factors are considered.

Keywords: Distance learning, automatic engagement estimation, emotional engagement, WebRTC, designprinciple.

論文審査の結果の要旨
本論文は，コロナ禍を経て日常化した遠隔学習プロセスにおける学習者のエンゲージメント分析を実現するための要件に関する文献調査を行い，カメラ映像によるリアルタイム推定が可能な分析手法を実装し，開発したシステムを評価したものである．文献調査では PRISMA モデルに基づき，様々な先行研究におけるエンゲージメントの定義，データセット，測定法，推定アプローチを広範に調査し，それらの研究と実運用上の（特にリアルタイム性，教員へのフィードバックの観点から）ギャップを明らかにした．分析手法の実装については，データの前処理手法と機械学習や深層学習による手法を組み合わせ，精度と処理時間のバランスを考慮した手法に基づく同期型・非同期型遠隔学習のためのリアルタイム推定システムを開発した．深層学習ベースの手法は既存の SOTA にせまる精度を達成し，機械学習の手法は極めて短時間の処理で，新たなユーザのデータに対しては SOTA手法と遜色のない精度を達成できることを示した．さらに，開発したシステムを実際に利用して教育者・学習者による評価を行い，提案手法の有用性と改善点について議論した．開発したシステム
は 1 対 1 を対象としたものであり，1 対多の遠隔学習に向けたさらなる拡張と処理の高速化，結果の信頼性は求められるが，提案手法の必要性やわかりやすさについては十分な評価を得た．さらに，顔情報を含むデータ分析を行う立場から，プライバシー保護に関する倫理的なリスクについて検討を行い，エンゲージメント推定技術導入のための，技術的・運用的な設計原理をまとめた．
本研究は学習者のエンゲージメント推定技術を遠隔学習の実践的な観点から活用するための諸課題に取り組んでいる点に新規性があり，特にリアルタイム性が要求される同期型遠隔学習の文脈で効果が期待される提案を行っている．一方で，このようなエンゲージメント推定を利用することで教育効果がどのように改善するかについては文献調査による裏付けはあるものの，提案手法に基づく評価実験は今後の課題となっており，提案手法の遠隔学習におけるインパクトについての議論は限定的である．
以上，本論文は，特に同期型の遠隔学習環境における学習者の前向きな心的状態であるエンゲージメントのリアルタイム推定手法を提案・評価したものであり，実践的な観点からの学術的貢献が大きい．よって博士（情報科学）の学位論文として十分価値あるものと認めた．