Rodrigo, M. M. T. et al. (Eds.) (2021). Proceedings of the 29th International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Preparations for Multimodal Analytics of an
Enactive Critical Thinking Episode
Rwitajit MAJUMDARa*, Duygu ŞAHINb, Yuanyuan YANGc & Huiyong Liaa
Academic Centre for Computing and Media StudiesbGraduate School of Informatics, Kyoto University, Japanc
Graduate School of Social Informatics, Kyoto University, Japan*majumdar.rwitajit.4a@kyoto-u.ac.jp
Abstract: Multimodal analytics provides a different perspective to understand and characterise
learning tasks and potentially model the process. However, the logistics and the analysis process
are often cumbersome to easily adopt the approach. We had earlier developed a platform thatenables us to synchronise learning environment interaction data and signal streams from various
physiological sensors. Also a framework was proposed that looked at critical thinking as aconstituent of its interactions and developed an embodied cognition narrative. In this study weused ENaCT, the online critical thinking task environment and studied an episode of criticalthinking that a participant attempted in it. We collected physiological data, specifically signals
from EEG and GSR sensors along with the interaction logs within the ENaCT environment. We
present an initial analysis of the multimodal data.
Keywords: Multimodal analytics, thinking task, GOAL system, ENaCT framework.

1. Background
Multimodal data is often collected in the context of analysis of learning tasks. However, the set of datacollected, the purpose of its analysis and the method of processing the data vary for different contexts.
For instance, Andrade, A. (2017) collected multimodal data in the context of mathematics learning
trajectories. To explain the embodied actions, fine-grained logs of hands and gaze data, depicting
motion, were mapped against the levels of student performance. In another recent study, also in thedomain of mathematics in motion-based learning environment (Lee-Cultura, Sharma & Giannakos
2021), explorations were made with video-based data encoding and involve eye-tracking andphysiological sensors to analyse play and problem-solving behaviours of kids. Overall such approaches
focused on the learning episodes and aimed to investigate the patterns and the predictive capabilities ofthe signals gathered during the learning tasks. Other research focused on development of tools foranalysis of multimodal data in educational context (Di Mitri et al.,2019) or application of specificanalysis methods in the context of learning. For instance, earlier using EEG sensors and applying
wavelet decomposition methods researchers analysed challenge level, need of skill level, engagement,
stress and focus (Miyauchi & Kawasaki, 2018; Conrad & Bliemel, 2016). More recently event-related
potentials (ERP) of a sensory stimulus was used to analyse attention, sense of task complexity andlanguage-related cues (Huang et al., 2020).

1.1 Research Objectives
To our knowledge limited work focused on specifically characterizing episodes of critical thinking withmultimodal data. In this study we utilize ENaCT as a theoretical framework to investigate theinteractions and signals captured during the critical thinking task. In this pilot, we aimed to establish adata collection and analysis pipeline of physiological signals to be integrated into the ENaCT
framework. In the current article, we present the visualisations of that data to answer the researchquestion: What are the interactions and corresponding changes in the physiological attributes andbehaviours during an interaction enabled thinking episode?

292

1.2 Embodied Narrative of Critical Thinking (ENaCT) Framework
The ENaCT framework (Figure.1) is an integrated framework for aligning learning design and learning
analytics of critical thinking (Mishra et al. 2020). It adopts an embodied and enactive cognition
perspective, explicitly considering that the affordances of (technology-enhanced) environments and theactions that a person performs on these affordances, are integral to a person’s thinking. Specifically, it
does not consider actions in the task environment and generated artefacts as products of critical thinking,rather that the generated artefacts also generate new affordances (and hence thinking possibilities) forthe problem solver. The ENaCT framework uses Paul & Elder’s (2008) “Elements of Thought” to build
the task model for design and analytics. As seen in Figure 1, the framework has two parts; the conceptual
part models how one’s actions with the artefacts in the environment relate to the execution and learningof critical thinking, and an analytics part which models how these actions and artefacts relate to theassessments and models of performance and learning. Connecting these two parts is the task model that
describes how different actions constitute the execution or application of the elements of thought. In
this work we explore how the physiological signals can be used to characterise the levels of behaviors
observed during the task.

Figure 1. ENaCT framework and the system architecture

2. Materials and Methods
2.1 ENaCT Interface and Critical Thinking Task
In the ENaCT environment the participants are given a specific critical thinking task which involves
responding to a focus issue while integrating information from multiple representations. The responses
were scaffolded according to the Paul and Elders (2008) elements of thoughts. Figure 2 provides thedifferent elements that were present and its corresponding use during the task.

293

Figure 2. Critical thinking task environment (Majumdar et al. 2021)

2.2 Data Collection and Technology Architecture
2.2.1 Interactions form the online task environment
The technology is based on the Learning Evidence and Analytics Framework (Ogata et al. 2018) where
the learning tools are integrated into a central learning management system with LTI protocol, andlearner interactions and created artefacts are logged in a learning record store. The interaction logs
enable tracing problem-solving narratives in the ENaCT environment. Figure 3 highlights the system
connections.

2.2.2 Sensor data
It is possible to collect high resolution physiological data related to the variation of the electricalproperties of the skin by electrodermal activity (EDA) sensors and the electrical activity of the brain byelectroencephalography (EEG) sensors. The data pipeline captures heart rate, EDA and temperature
data from Empatica E4 devices and EEG data from Emotiv insight devices.

2.2.3 Architecture to synchronize data streams
We use the GOAL architecture to synchronize the data from the online learning environment and thephysiological sensors (Majumdar et al. 2020). A data upload option caters to any sensors that provide
CSV files with time information. All the data (interactions and sensor generated) is linked to only amachine readable id of the user and synchronized on a common timeline to be uploaded on the activity
record store. Figure 3 provides the overview of the technology architecture and the set of data collected.

Figure 3. System architecture for data collection during the study294

2.3 Research Methods and Participants
A phenomenographic research approach was chosen for the study (Marton 1981, Jan Larsson & Inger
Holmström, 2007). Only one episode of thinking was observed in detail with the above mentioned
multiple data sources. There were 5 participants (2F) who participated independently in the session.
Four of them were graduate students and one was a postdoc researcher. For the current analysis weselected the data of only the postdoc researcher (Par-2). Par-2 was selected based on the level ofengagement in the activity, the notion of being an expert amongst the selected participants for a thinking
task and lastly the level of detailed reflections drawn at the end of the session. Further the physiologicalsignals for that participant were also of high quality (EEG signals average quality above 85%).

3. Visualised Data of the Critical Thinking Episode3.1 Interactions During the Activity
Figure 4 presents the interactions of the participant within the ENaCT environment during the CT
episode. The interaction logs extracted from the database were grouped into the following task-relevant
action categories: OpenHome (landing on ENaCT platform), OpenTask (opening the task page),
information-interaction (Info_Int), and artefact creation (Art_Cre). Info_Int are the actions users
performed with the information affordances, while Art_Cre are the actions that users performed within
the expression affordances. Art_Cre logs were further separated into two types: Art_Cre_acc, when theuser simply accessed the panel to create or modify their text-response, and Art_Cre_sub when the textresponse was submitted. Summary is when the user opens the summary panel and OpenSurvey (opening
post survey after task). The data is plotted using Tableau (v.2021.2.1) on a timeline which is represented
as the horizontal axis. The value is the local time.
The participant spent 15 minutes 19 seconds in the ENaCT environment. In total there were 39
interactions. From figure 4 it is seen that the participant had the 15 information interaction in two
grouped periods of time. 5 interactions were related to accessing the answering panels and 14
submissions were made. However, regarding the excessive submission clicks for the 5 fields, weverified with the screen capture video. It was found that due to a server error to send the log theparticipant repeatedly clicked the submit button in a quick succession generating extra 9 interaction logs
for one of the submission fields. In the Figure 4 plot those successive clicks were removed.

Figure 4. Interactions in the ENaCT environment295

3.2 EEG Data Processing
The EEG signals were processed to extract the emotional aspects. Currently we utilized theEmotivPro (v2.7.2) software’s algorithm for extracting the emotion levels and select the scaled
normalised values. The data from the EMOTIV Insights sensor was collected and the software
extracted bandpowers and computed engagement, excitement, interest, relaxation and stress levels. The
data is then masked with the task cues from the ENaCT environment. While the sensor sampled at ahigher frequency, the emotion levels were computed each second and visualised in Figure 5 as thenormalized and scaled value. The average value of each of the emotions across the activity are denotedby the reference line in each pane.

Figure 5. The analysis of the EEG data

3.3 EDA data processing
The EDA data provides information regarding the intensity of the emotional state or response asemotional arousal cues. In order to preprocess the signal and get components with their peak
information, we used a Python toolbox called Neurokit2 by Makowski et al. (2021).
The EDA signal basically consists of two additive processes namely tonic and phasic signals.
Tonic signals are the ones fluctuating very slowly and depicting the general trend whereas the phasic
signals are varying very fast and representing the skin conductance responses. In figure 6A, the firstwindow shows the cleaned EDA signal whereas the second window depicts the phasic component withall the related peak information and the third window shows the tonic component. In figure 6B, thetonic and phasic signals are marked by the intervals gathered from the ENaCT environment.
In a minute the average number of peaks for the intervals in order are 5, 4, 14, 16 and 4.

296

A.

B.

Figure 6. The analysis of the EDA

3.4 Eye Tracking of The Participant
Eye tracking gaze data was directly visualized as heatmap using the eye tracker software and overlaid
on the screen capture. As the task involved multiple interfaces and scrolling the analysis of the eye
tracking data would require further pre-processing in the future. Currently, it provides an indication of
the focus in the different areas of interactions and is used for visual exploratory data analysis. The
snapshot of the heatmap in different sections of the interface is presented in Figure 7.

297

Figure 7. Screenshot of heatmap of fixation of eye gaze

4. Discussion and Future Work
4.1 Thinking Episode of Participant 2 (Par-2)
The interactions of the participant (Par-2) as seen in Figure 4 highlighted that there was an interface
exploration phase in the beginning. The main sets of interactions were in the information panels area tocheck what kind of information was presented. The questions given as prompts were already seen aslabels so there were no click interactions required there. This was also confirmed from the eye tracking
video and in the post task reflections during the interview with the participant.
Further from the interaction data, Par-2 did not go back and forth with the information panels
and the artefact creation panels. To respond and note What are the interpretations of the data?, therevisit to the information panel was seen. Par-2 mentioned the intention was to only get the trend of therelative values presented in the different information panels to answer that and hence back and forthdetails were not considered. However, we have seen different behaviors in our previous novice studies(Mishra et al. 2021, Majumdar et al. 2021) where participants who had more frequent back and forthactions also tended to respond with relatively better insights about the interpretations of the data theyobserved.
Par-2 also had a distinct viewing summary panel action at the end. While one of our design
conjectures was participants would access the summary panel to take the affordance of their own input
to further think (Majumdar et al. 2021), however as observed in this case too, the summary was accessedat the end of the task attempt to kind of verify the artefacts created.
For the analysis of the EEG and EDA data, an essential step is to create temporal masks for
tagging specific activity periods. In figure 5 we have labelled the 5 masks as follows: task orientation where the participant reads the requirements of the task; information interactions - interactions with the298

given information panel; Artefact creation and submit - accessing the artefact panel, typing andsubmitting response; View summary - viewing summary panel; Mixed interaction - having multiple
panels open on screen and the interval where we can not indicate distinctly whether it is a informationinteraction or artefact creation action. The following mixed interaction phase was identified while thedata was being tagged after synchronising. Currently it required further manual identification from thescreen recording to distinct the phases. However, that would limit the scaling of the analysis process
and hence we created the distinct state to label such a situation.

4.2 Analysis Approaches to Thinking Episodes with Physiological Data
While preparing to analyse the collected data, we also reviewed some of the multiple ways to analyse
the physiological signals in previous studies (See Table 1). Conventionally, EEG data is analysedthrough either event related potentials (ERP) or frequency bands. ERP is mainly reflecting the average
response to a task with a certain duration of 1,5 - 2 seconds. Through this component it is possible toassess the attention and learning levels, the complexity of tasks as well as language related processes
such as semantics (Huang et al., 2020). The frequency band approach involves time-frequency
decomposition. The resultant bands on theta, alpha, beta and gamma ranges can be further processed
for their power, ratio, entropy, topography and pattern change to get cues about challenge levels oftasks, need of skill, the levels of attention, engagement and focus as well as stress (Miyauchi &Kawasaki, 2018; Conrad & Bliemel, 2016).
As for the EDA data, a common approach involves the raw signals to be segregated in phasic
and tonic components. From the phasic components, it is possible to relate the emotional arousal withthe stimulus and gather information about stress, joy, boredom, excitement, interaction levels,
anticipation, decision making. The higher the peaks, the higher the arousal and the more there is achange in the phasic component patterns the more interaction takes place (Rico-Olart et al. 2020).
Table 1. Physiological Sensor Data and It’s Analysis
Sensor (Signal)

Analysis approach

What it helps to determine

Reference paper

EEG(voltage from 7
points)event-related
potentials (ERP)

attention, learning, sense ofcomplexity, language-related
cues

Huang et al., 2020

Waveletdecomposition

challenge level, need of skill
level, engagement, stress, focus

Miyauchi & Kawasaki,
2018; Conrad &Bliemel, 2016

Phasic and Tonic
state separation

Emotional cues related withexcitement and stress levels

Rico-Olart et al., 2020

GSR (Skin
Conductance)

In our current approach, we utilized the frequency band processes as well as EDA data
variability to gather info about engagement, excitement, focus, interest, stress levels, and cognitive load
during the given task. When this info is further analyzed through statistical comparison methods, itwould characterize the individual differences in the thinking task in its different phases.

299

Figure 8. Data processing flow
Further analysis approach includes the utilization of other features of the collected sensor and possibly
fusing the information together to then interpret. At this stage additional analysis and data tagging withemotional cues are possible through the preset machine learning algorithms. The data processing flow
including all steps are depicted in Figure 8.

4.3 Current Challenges and Limitations
This pilot made us aware of the current challenges and limitations of the data and experimental setting
as follows. Since the task was designed as a problem solving task related to executing critical thinking,it is more complicated than a perceptual task that is often the usual settings of the experiments used in
the literature using physiological sensors. Thus the processing and interpretation of the data is notstraightforward and requires specific mapping of the analysis masks to the given context. Further, given
the complexity of the task and the time taken to attempt it, there is a lot of room for individual
differences in responding. That makes it challenging to compare the data within subjects. Due to thoseindividual differences, each subtasks denoted by the masked phases in figure 6 are also not uniform in
the time space. For the pilot study there were a limited number of subjects, and often the data gatheredfrom the sensors had high noise or unexpected data loss, thus any statistical procedures can not beapplied for comparison.

4.4 Contribution and Future Works
This is an initial exploration to collect multimodal data during a critical thinking task. Both the task
environment design and the data analysis was done from the perspective of (inter)actions involved inthinking. We have created a platform and data pipeline for synthesising physiological, behavioral andinteraction data during such an activity. Here we presented the visualised data collected during the pilot
session as a demonstration of the variation of the trends of the signals along with the progress of thetask.
Next we aim to analyse the details of the physiological and behavior data of the participants.
The processing of the EDA data and the EEG data would be done to find whether there are any
differences in the level of physiological activities in the information interaction and the artefact creation
phases to characterise the critical thinking activity from the perspective of the ENaCT framework.

300

Acknowledgements
This project was partially funded by the SPIRITS2020 grant of Kyoto University and JSPS
KAKENHI Grant-in-Aid for Early-Career Scientists 20K20131

References
Andrade, A. (2017, March). Understanding student learning trajectories using multimodal learning analytics
within an embodied-interaction learning environment. In Proceedings of the Seventh International
Learning Analytics & Knowledge Conference (pp. 70-79).
Conrad, C., & Bliemel, M. (2016). Psychophysiological Measures of Cognitive Absorption and Cognitive Loadin
E-Learning
Applications.
Proceedings
of
the
ICIS
2016.
Retrievedfromhttps://aisel.aisnet.org/icis2016/Human-ComputerInteraction/Presentations/9
Di Mitri, D., Schneider, J., Klemke, R., Specht, M., & Drachsler, H. (2019, March). Read between the lines: An
annotation tool for multimodal data for learning. In Proceedings of the 9th international conference onlearning analytics & knowledge (pp. 51-60).
Huang, H.-W., King, J.-T., & Lee, C.-L. (2020). The New Science of Learning. JSTOR. Retrieved fromhttps://www.jstor.org/stable/26981740
Jan Larsson & Inger Holmström (2007) Phenomenographic or phenomenological analysis: does it matter?
Examples from a study on anaesthesiologists’ work, International Journal of Qualitative Studies on
Health and Well-being, 2:1, 55-64, DOI:10.1080/17482620601068105
Lee-Cultura, S., Sharma, K., & Giannakos, M. (2021). Children’s play and problem-solving in motion-basedlearning technologies using multi-modal mixed methods approach. International Journal of ChildComputer Interaction, 100355.
Majumdar R., Şahin D., Kondo T., Li H., Yang Y.Y., Flanagan B. and Ogata H. (2021) Enabling Multimodal
Reading Analytics through GOAL Platform. in the Companion Proceedings of LAK 2021
Majumdar, R., Kothiyal, A., Mishra, S., Pande, P., Li H, Yang Y.Y., Ogata H. and Warriem J.M. (2021) Designof a Critical Thinking Task Environment based on ENaCT framework. In 2021 International Conference
on Advanced Learning Technologies (ICALT) (pp. 294-296).
Makowski, D., Pham, T., Lau, Z. J., Brammer, J. C., Lespinasse, F., Pham, H., … Chen, S. H. A. (2021).
NeuroKit2: A Python toolbox for neurophysiological signal processing. Behavior Research Methods
2021, 1–8. https://doi.org/10.3758/S13428-020-01516-Y
Marton, F. (1981). Phenomenography—describing conceptions of the world around us. Instructional science,10(2), 177-200.
Mishra, S., Majumdar, R., Kothiyal, A., Pande, P., Warriem, J.M. (2021) Tracing Embodied Narratives of CriticalThinking. In: Roll I., McNamara D., Sosnovsky S., Luckin R., Dimitrova V. (eds) Artificial Intelligencein Education. AIED 2021. Lecture Notes in Computer Science, vol 12749. Springer, Cham.
https://doi.org/10.1007/978-3-030-78270-2_48
Miyauchi, E., & Kawasaki, M. (2018). Electroencephalogram oscillations support the involvement of taskunrelated thoughts in the mechanism of boredom: A pilot study. Neuroscience Letters, 677, 72–77.
https://doi.org/10.1016/J.NEULET.2018.04.034
Mu, S., Cui, M., & Huang, X. (2020). Multimodal Data Fusion in Learning Analytics: A Systematic Review.
Sensors (Basel, Switzerland), 20(23), 6856. https://doi.org/10.3390/s20236856
Rico-Olarte, C., López, D. M., Becker, L., & Eskofier, B. (2020). Towards classifying cognitive performance bysensing electrodermal activity in children with specific learning disorders. IEEE Access, 8, 196187–
196196. https://doi.org/10.1109/ACCESS.2020.3033769

301