<figure><img src="image_1.png" /></figure>

HAL
open science

# Multimodal learning analytics for studying creative problem-solving with modular robotics

Axel Palaude, Margarida Romero, Thierry Viéville

► To cite this version:

Axel Palaude, Margarida Romero, Thierry Viéville. Multimodal learning analytics for studying creative problem-solving with modular robotics. [Research Report] RR-9467, Inria & Labri, Université Bordeaux. 2022, pp.27. hal-03621919v4

HAL Id: hal-03621919

https://inria.hal.science/hal-03621919v4

Submitted on 31 Aug 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution - ShareAlike 4.0 International License
informatics mathematics
Inria

# Multimodal learning analytics for studying creative problem-solving with modular robotics

Axel Palaude, Margarida Romero, Thierry Viéville

## RESEARCH REPORT

Nº 9467

March 2022

MNEMOSYNE

ISSN 0249-6399 ISBN INRIA/RR-9467-FR+EN
<figure><img src="image_3.png" /></figure>
<figure><img src="image_4.png" /></figure>

# Multimodal learning analytics for studying creative problem-solving with modular robotics

Axel Palaude<fnref n="1" />, Margarida Romero<fnref n="2" />, Thierry Viéville<fnref n="3" />
MNEMOSYNE

Research Report N° 9467 — March 2022 — 27 pages.

**Abstract:** Creative problem solving is a complex process that is being studied through a diversity of tasks. CreaCube is an open ill-defined task whereby the player is required to engage in a creative problem-solving activity (Romero et al., 2017) that we try to analyze with computational models (Alexandre, 2020a). To represent the behavior of the subject through the activity evolution, we analyze the activity through a coding schema considering the observables for this tangible problem solving activity. Observables correspond to different states of the artifact of behaviors of the participant at a given time. In this study we introduce the learning analytic strategy corresponding to a temporal sequence of observables. Through this sequence we aim to infer the participants' internal state based on a different sequence of observables.

Through this study we aim to advance in the learning analytics strategy of a tangible problem-solving task with educational robotics. Our main goal is to be able to both collect data more easily, avoiding as much as possible the manual analysis of the video recording of the activity, and propose enriched observables. To this end, we refined the observables' model as detailed in (Mercier et al., 2021) and added new observables and decomposed existing ones into more specific ones based on the learner and task model. We distinguished observables with automatable data collection and those which require manual identification. In the end we discuss the relevance of this new version of CreaCube by discussing to what extent it offers additions to actual data analysis and ongoing research on this subject.

**Key-words:** creative problem-solving, learning analytics, educational robotics, data collection

<footnote n="1">¹ Axel Palaude, Inria Mnemosyne Team, AldE - axel.palaude@inria.fr</footnote>

<footnote n="2">² Margarida Romero, LINE, Université Côte d'Azur - margarida.romero@unice.fr</footnote>

<footnote n="3">³ Thierry Viéville, Inria Mnemosyne Team, UCA LINE Laboratory, thierry.vieville@inria.fr</footnote>
# Analyse des données multimodales d'apprentissage pour l'étude de la résolution créative de problèmes via une activité de robotique modulaire

**Résumé** : La résolution créative de problèmes est un processus complexe mis à l'étude via une grande variété de tâches. CreaCube est une tâche ouverte et mal-définie par laquelle une personne est engagée dans une activité de résolution créative de problème (Romero et al., 2017) que nous essayons de représenter et d'analyser avec des modèles informatiques (Alexandre, 2020a). Afin de représenter le comportement de la personne engagée dans l'activité, nous analysons le déroulement de celle-ci à travers un codage comportant différentes données nommées observables. Les observables correspondent aux différents états des artefacts observables lors de l'activité (comportements, émotions, matériel) à un moment donné. Dans cette étude, nos données d'apprentissage correspondent à des séquences d'ensembles d'observables. Via ces séquences, nous espérons pouvoir inférer l'état interne d'une personne engagée dans l'activité à partir de la séquence d'observables lui étant associée.

Via cette étude, nous visons un approfondissement d'une stratégie d'analyse d'apprentissage pour CreaCube, une tâche de résolution de problème tangible avec de la robotique éducative. Notre objectif principal est d'être capable à la fois de collecter plus simplement et rapidement des données sur le déroulement de l'activité, en évitant autant que possible le recours à l'analyse manuelle des enregistrements vidéo de l'activité, et de proposer un nouveau cadre d'observables plus riche. Pour cela, nous proposons d'affiner le modèle des observables détaillé dans (Mercier et al., 2021) et d'ajouter de nouvelles observables relatives à l'évolution de la tâche extérieure à la personne engagée. Nous proposons également une distinction entre les observables dont la collecte lors des expériences peut être automatisée et celles dont la collecte requiert une identification manuelle. Enfin, nous discutons de la pertinence et de la faisabilité de ces changements en réfléchissant à une nouvelle version de CreaCube et ce qu'elle peut offrir de plus aux analyses sur les données actuellement menées sur le sujet.

**Mots clés** : résolution créative de problème, analyse de l'apprentissage, robotique éducative, collecte de données
<table><tr><td>1.Introduction</td><td>6</td></tr><tr><td>1.1. The CreaCube task</td><td>6</td></tr><tr><td>1.2. Creative problem solving modeling</td><td>10</td></tr><tr><td>1.3. An observer model to collect data</td><td>11</td></tr><tr><td>2.Observables refinement</td><td>13</td></tr><tr><td>2.1. Constraints to let CreaCube remain the same</td><td>13</td></tr><tr><td>2.2. A new framework</td><td>15</td></tr><tr><td>2.3. Classification of observables</td><td>20</td></tr><tr><td>3.Discussion and future work</td><td>22</td></tr><tr><td>Bibliography</td><td>24</td></tr></table>
# 1.Introduction

CreaCube is a creative ill-defined problem-solving task (Romero et al., 2019). Through this task, the learner-player is engaged in a problem-solving activity with almost no information but the final goal. The learner-player engages in a process of creative problem-solving skills required both to engage in posing the problem, advancing through the problem-solving process through different iterations of enactive interactions with the tangible material in order to disambiguate the problem situation and advance through the interactive process of creating and evaluating an artifact with four robotic cubes until they are assembled in a way that corresponds to the constraints of the task goal. In this context, perception of technological affordances in the robotic cubes (Kalmpourtzis & Romero, 2020) and their relation to prior knowledge and the strategies to explore and acquire new insights from interactions with the unknown material (Romero et al., 2018) are required to advance in the knowledge constructions which will allow the learner-player to solve the CreaCube ill-defined task.

## 1.1. The CreaCube task

**Description of the task** The CreaCube task setup consists of four different modular cubelets (red, white, blue and black) presented on a table (with a black point and a red point on it) in front of the player. The instructions, repeatable at will through a push button, consist in one sentence : « You need to build a vehicle composed of 4 pieces that moves autonomously from the red point to the black point. You can listen to the instructions again by clicking again on the button. ». A configuration of this activity is shown in Figure 1. The activity is not limited in time (but the player can decide to stop the activity at any moment) and no help is given to the player by any means. The activity is developed twice. The first occurrence of the activity has an average duration of 173 seconds (sd=224 seconds) and the second occurrence of the activity has an average duration of 62 seconds (sd=79 seconds) based on the analysis of 425 participants.

<figure><img src="image_5.png" /><figcaption>Figure 1 : An example of configuration of CreaCube. The four cubes must be assembled in order to make them move autonomously from the red point to the black point</figcaption></figure>

**Hardware description** Each cube is a modular robotic component named *Cubelet*<fnref n="4" /> and has different technological affordances and are enacted into different functional behavior during the activity depending on the way the cubes are assembled. We describe below each of the cubes :

- The white cube has 5 magnetic faces that are used to connect it with other cubes, and one face with two wheels. As we want to determine when the player sees the wheels for the first time (see next section for

<footnote n="4">⁴ https://modrobotics.com/</footnote>
more details), the wheels are hidden from the player's vision in the beginning. The wheels do not move by default, they must be "activated".

- The blue cube has 5 magnetic faces, and one face with two elements: a switch "O/I" and a plug. The switch must be turned "on" to supply electricity to other cubes. The plug can be used to refill the cube, and is not relevant during the activity.

- The black cube has 5 magnetic faces and one face with two "eyes" that hide a directional infrared distance captor that can detect obstacles in the direction of the face and send a signal to other cubes depending on the measured distance

- The red cube has 6 magnetic faces with no specific visible technological affordances. Depending on its position, the inverter cube changes the behavior of the structure depending on where it is situated, acting more or less as an approximate "inverter" of the signal sent by the black cube.

The general configuration of a goal structure (i.e. a structure that can move autonomously from the red point to the black point) consists of a balanced structure of the four cubes connected together with the magnetic faces, with the wheels on the table to allow movement and the switch on the "on" position. Depending on where and in which direction the black and red cubes are placed, the structure can have a lot of different types of movement: linear, accelerating, no movement at all etc.

**Context of the study** The protocol developed within the ANR CreaMaker with the CreaCube task aims to study computational thinking through a creative problem solving task (Romero, 2017; Romero et al., 2019) The CreaCube activity can solicit most of them. For instance, the activity can be solved by one or more participants (Romero et al., 2021). CreaCube is a creative problem solving task engaging computational thinking competencies.

**Computational Thinking components** Romero et al. (2017) describe the computational thinking (CT) competency as "a coherent set of cognitive and metacognitive strategies engaged in (complex) systems identification, representation, programming, and evaluation" and consider six different components. Barnabé et al. (2020) applied those components to the CreaCube activity as shown in figure 2 to represent solicited CT competency and its components during this problem-solving activity.

<figure><img src="image_6.png" /><figcaption>Figure 2. CT components in relation to the computational framework as developed in (Romero et al., 2017) by (Barnabé et al., 2020).</figcaption></figure>

**Problem-solving** There are different approaches when tackling problem-solving study.

From an information processing theory of human reasoning, (Newell, 1981) defines a problem space as the "fundamental organizational unit of all human goal oriented symbolic activity" (p.696) and a problem as "a set of initial states, a set of goal states, and a set of path constraints. The problem is to find a path through the space that
starts at any initial state, passes only along paths that satisfy the path constraints, and ends at any goal state" (p. 695). In this model, we can consider a problem-space as an automata with a problem-solving process as a word:

-Initial state(s) and final state(s) correspond to the beginning and the goal states of the problem;

-States correspond to each possible state of the problem

-Transitions from one state to another correspond to a transformative action modifying the problem state from one configuration to another.

The CreaCube task can be decomposed in different states representing all possible configurations of the four cubes, the initial state being the four disconnected cubes and the final states being every structure that allows the resolution of the task by being able to move autonomously and reach the target point. Operators are represented by actions taken by the subject, such as "connecting the white cube to the blue cube".

From a socio-cultural perspective, within a PISA framework for instance (OECD 2013), problem-solving is modeled as a process including four different stages, that are nor linear nor sequential, especially for complex problem-solving requiring subdivisions of the problem space:

-exploring and understanding the problem to identify the components of a situation and their structure;

-representing and formulating the problem to organize and model the situation efficiently;

-planning and executing the problem to engage in the solution creation;

-monitoring and reflecting the problem to engage in the solution evaluation and improvement.

Prior knowledge can allow a better identification of the problem and the use of relevant knowledge to solve it. Wiley (1998) determined that having a certain domain knowledge can "act as a mental set, promoting fixation in creative problem-solving attempts". Non-relevant prior-knowledge should be inhibited to avoid fixations to early ideas or the use of inappropriate knowledge for the task.

The development of problem-solving competencies can change the way the problems are tackled, allowing for instance the inhibition of non-relevant knowledge and the generation of novel ideas to reach the goal. As such, the development of these competencies modify the way the different phases are tackled. For instance, Brand-Gruwel et al. (2005) observe that expert problem solvers engage more time in the analysis of the problem and are better at regulating the problem-solving process than novice problem-solvers.

CreaCube is an ill-defined problem. Greenwald (2000) defines ill-defined problems as being "unclear and raises questions about what is known, what needs to be known, and how the answer can be found. Because the problem is unclear, there are many ways to solve it, and the solutions are influenced by one's vantage point and experience".

As such, the task cannot be solved with an initial phase of exploration, and the subject is not aware of any prior routine or algorithm to solve it.

**Creativity** While some primary teachers consider creativity to be specially related to the arts (Capron Puozzo, 2016), research on creativity is considered as a transversal competency and is present even in STEM disciplines. Creativity has already been studied in education science (Leroy et al., 2021), and in many other disciplines of interest such as cognitive neuroscience (Alexandre, 2020b), computational science (Oltețeanu, 2020), but also philosophy (Gaut, 2010). Here we consider creativity for the CreaCube activity as an individual process that aims to design a solution that is both novel and appropriate in regard to the subject and the task.

**Techno-creative computational thinking** As illustrated by Figure 3, creativity, problem-solving and computational thinking skills are intersecting with each other. Creativity can be considered as a process engaged in decision-making and in problem-solving competency (Romero et al., 2022). On one hand, computational thinking can be solicited for problem-solving activity in which computer science knowledge is engaged. On the other hand, Jacob and Warschauer (2018) consider problem-solving as a process within the computational thinking competency. Finally, creativity and computational thinking can intersect when the computational situation allows a margin of creativity to decide the final chosen method, system or solution.
<figure><img src="image_7.png" /><figcaption>Figure 3: Techno-creative computational thinking as the intersection of creativity, problem solving and computational thinking (Romero et al., 2022).</figcaption></figure>

The CreaCube activity is soliciting computational thinking competencies as mentioned in Figure 2, and is an ill-defined problem-solving activity. As such, the activity is more likely to solicit creativity skills, as there is no known method to solve the problem. As such, the CreaCube activity fits into the techno-creative computational thinking category.

To sum up, the CreaCube activity, by design, is an ill-defined problem-solving activity soliciting computational thinking skills. The learner-subject, by manipulating the different cubes without prior knowledge of their role or the resolution method, adopts different behaviors engaging the aforementioned skills. In order to study these different competencies and the way each learner solicits them, we need now to model the subject and its behavior in order to be able to connect our models.

## 1.2. Creative problem solving modeling

In order to better understand the creative problem-solving process we consider the limits of behavioral observations in limited samples.

First, indirect measures like surveys and questionnaires are typically easy-to-apply, and low-cost tools, while allowing an alternative to assess issues difficult to identify by observation. However, they are potentially misleading (Gris & Bengtson, 2021), as they are coming from subjects who potentially add more biases (lying, agreeing, responding in a socially desirable fashion, etc.).

Direct measures are more complex and time-consuming, as it implies, for instance, to identify what is to be measured, or determine what are appropriate data-recording procedures. They are more precise, but they are limited in application. For instance, identifying a behavior may require a lot of different direct measures.

In any case, the observation of the subject's behavior is an important issue to take into account.

For the CreaCube activity, an observation model is used to represent its evolution in relation to the player.

## 1.3. An observer model to collect data

Our main goal is to represent the evolution of the mental state of the player during this creative problem-solving task: what are the player goals and beliefs, what the player learns, how the player behaves etc.

However, we do not have direct access to the evolution of the internal state of the subject during the activity, but rather to external cues.
An observer's role during an experiment is to describe the external state of the experiment: how the experiment changes over time, what are the subject's actions etc. The observer can collect any accessible (e.g. visible) data about the experiment.

We do model the observer by what variables the observer is collecting during the experiment. We call these variables *observables*.

A set of observables at a given time corresponds to a representation of the experimental state at this given time. As observables may change over time, the evolution of a set of observables corresponds to the representation of the evolution of the experimental state over time.

In order to represent the evolution of the CreaCube experiment, we will use an observer model composed of different in-task observables as detailed by Mercier et al. (2022).

**In-task observables** (Mercier et al., 2022) detailed in-tasks observables about the activity. As shown in Figure 3, there are different types of observables :

F** and AS** correspond respectively to complete and partial structures of the cubes;

U** correspond to what the user is doing physically while B** and E** correspond to the mental state of the user representing respectively behaviors and emotions of the subject;

A** correspond to visual affordance and if they are discovered by the subject

T** correspond to discrete categorisation of the experimental states;

S01 and S02 correspond to the state of the switch

P** correspond to the different problems related to a structure stopping it from a "goal" configuration.

FL** correspond to changes made by the subject on a cube when the subject tries the same configuration (either the cube is rotated or not).

<figure><img src="image_8.png" /><figcaption>Figure 4 : Observables of the CreaCube activity by (Mercier et al., 2022).</figcaption></figure>

They are recorded with the precise instant of occurrence, with an offline measure with a video recording of the experiment. In this paradigm, either a human observer or a machine learning algorithm may identify the in real-time experimental state.
Observables are completed with surveys before and after the activity filled by participants. The pre-activity survey contains questions about the subject (age, familiarity with technology etc.) and the after-activity survey contains questions about what the subject did during the activity (e.g. "Did you tried to solve the task the fastest possible way?") and what he did learn about the cubes (e.g. "Can you describe the role and features of the white cube?). Also, participants are aware of the experiment, so they know what is recorded.

**Data and data collection limits** We are convinced that some observables have some flaws :

-Some observables lack precision : AS** observables take into account the general configuration, but the color of connected cubes may be relevant. As we saw earlier, the black-blue-white structure is able to move autonomously, which is not the case for other 3-cubes configurations. For the subject, the reaction to the 3-cube structure may not be the same if the structure can move or not. U** observables are not able to distinguish a "Hands up with two connected cubes" from a "Hands up with two disjoint cubes", which seems to be relevant to determine the subject's actual action.

-Behaviors' observables seem to be overlapping (e.g. B02 and B03) while not being enough to represent all behaviors relevant to the task.

We are also convinced that the method to collect observables is not optimal. First, the identification of observables by a human-observer does not seem to be very efficient in terms of time resources. Even trying to train a machine-learning algorithm requires a prior data set that is time-consuming to create.

Our main goal in this report is to propose some corrections to the data collection by proposing a new framework of observables without changing the means of the CreaCube task.

## 2.Observables refinement

In order to make the model more precise, we will try both to complete the observables currently used and to refine the coding of the observables already detailed in (Mercier et al., 2022), i.e. the learning traces. At the experimental level, the main challenge is to reliably collect a large amount of data. For this, we will give a classification of observables based on the separation between observables whose identification is manual (identified by a human observer) and those whose identification can or could be automated. We will also add new automatable observables in order to refine what can be taken into account with those already existing. However, the activity should remain the same, and adding elements to observe on the activity may change the way the subject tackles it.

To dodge this kind of problem, we will first address problems to take into account before transforming the observables framework. Then we will present a classification of existing observables, in order to present a new observables framework in a third part.

### 2.1.Constraints to let CreaCube remain the same

Our main constraint is that the CreaCube activity must remain the same. To be more precise:

**(C) Changes made to the activity should not change the behavior of the subject towards the activity.**

We want to keep the compatibility with existing work on the CreaCube activity. For instance, if we add a visible physical element to the cubes in order to collect new observables, it will become a neutral or negative affordance. Either way, a new affordance may imply new ways of thinking from the subject of the activity, thus changing current learning representation models of the subject.

This constraint can be decomposed in more precise ones that we need to discuss.

**Visible external sensors** When people are aware of being observed, their behavior may change, and they can distort both what they do and what they say (Kazdin, 1982). Even when the observer is actually a video camera, people distort their behaviors. (Constantinou et al., 2005) showed that the presence of a video camera impacts task performance. The original activity records only the subject's hands, and even if it already affects the subject's behavior, we will try to avoid adding another bias by adding other external sensors like video cameras.

**(C1) Visible sensors added for data collection should not change the behavior of the subject towards the activity.**
Filming the subject is one example of the (C1) constraint not respected, but it is not the only one.

For instance, we think that the posture of the subject towards the activity may be a good indicator for behavioral and emotional observables. A subject slumped in his chair may be less likely to be in an active trial/error behavior, and more likely to feel boredom. However, as it implies to be able to record movements of the body during the activity, either with a camera or sensors directly on the subject, we consider that it does not fulfill (C1) and thus will not be added in the framework detailed in the next section.

An objection to this constraint is that the subject may get accustomed to sensors even on the subject's body, in a way that the subject forgets the existence of the sensors. This way, we could suppress the (C1) constraint. We did not explore this possibility here, but we will discuss this possibility later in the article.

**Invisible sensors** We could argue that if (C1) does not allow adding visible sensors because it may change behaviors, a way to get around the problem is to add non visible sensors. For instance, we could add a camera to film the subject without the subject knowing. However, it is stated that the subject must be aware of the measurements before the beginning of the activity for consent reasons. If the subject is aware of a camera's presence somewhere around, the subject's behavior may be changed the same way as if there were visible cameras. We change the constraint (C1) to take the subject's awareness about measurements into account:

**(C1) Sensors either visible or not, added for data collection, should not change the behavior of the subject towards the activity.**

**Visible internal modifications** By visible internal modifications, we are talking about sensors that are situated on the activity, either on cubes or on the table, or complete changes of the activity. An extreme example would be the addition of a fifth cube, completely changing the dynamics of the activity, but little changes may be impactful too: adding a visible sensor on the table or cubes may add a new affordance to take into account. That leads to a second constraint:

**(C2) No visual modification should be done on the activity, because it may change the way the subject learns or acts.**

This constraint implies that we should not change the appearance of the activity because the actual appearance of it already has a specific meaning, and specific set of possible physical manipulations.

Meaning-wise, the general appearance of an activity is important because it has a specific meaning (that may be different for each subject as it depends on culture and previous experiences): different colors and shapes have different meanings. So it appears that changing the color of one cube for instance does not respect (C2), because the semantic of each color is distinct, so a new color may change the hypothesis formulated by the subject. For instance, if all cubes are black, a possible assumption to make is that they are identical.

Manipulation-wise, changing the possible manipulations of the cubes may change the way the subject tackles the activity. Comparing two cubes is more difficult if the subject cannot take them both in hands for example, so the subject may be encouraged by this constraint to adopt other behaviors.

For instance, changing the size of the cubes may be beneficial for us : bigger cubes may contain more internal sensors or have more space onto the faces of the cubes to have bigger affordances that could be easier to detect. However, even if the meaning remained the same, bigger cubes may imply that the subject cannot have the four cubes in hand at the same time, thus preventing the subject from doing the same manipulations as the original activity. This constraint applies also for modifications not related to cubes. If we decide to put the cubes in a box instead of on a table (in order to film manipulations from all angles for example), the subject's arms movements will be limited, thus changing once again the manipulations done in comparison to the original activity.

Now that the constraints are determined, we then create a new framework of observables that is more precise while respecting the constraints.
## 2.2. A new framework

The first step to create a more precise set of observables is to refine existing ones, preferably without changing anything to the activity for the moment.

In order to do this refinement, we need to identify which observables are lacking precision and decide which ones are refinable given our constraints. For this, we will take a look at each category of observables, one by one.

**Final structures.** For each final structure shape, there are a lot of different configurations, depending on the position of each cube and where specific cube faces are situated (mostly wheels and eyes).

For instance, we will consider the structure F000, where all cubes are aligned. There are 24 different configurations, some of them are detailed in Figure 4. For each configuration, the wheels may touch the ground or not, and sensors can see obstacles (if directed to the bottom where the table is) or not (if placed in front of the line for instance). For each configuration, goal structures can exist but are not systematic (depending on the orientation of the wheels for instance). This classification is already done by Mercier et al. (2022)

<figure><img src="image_9.png" /><figcaption>Figure 4: 12 configurations for the F000 structure detailed by (Mercier et al., 2022). The configuration is coded by four letters S (sensors), W (wheels), B (battery) and I (inverter), and the letter T (or F) indicates if the structure can be a goal-structure, depending on the orientation of the wheels and the sensor, and if the switch is turned on.</figcaption></figure>

**Partial structures.** Partial structures observables indicate how many cubes are connected with each other when not being on a final structure: none (AS00), two (AS01), three (AS02) and four but with two unconnected pairs (AS03). We propose two improvements to this classification.

First, we consider the shapes of partial structures. As soon as three cubes are connected, the shape can either be a line or a bend. This separates AS03 into AS03-L (lined shape) and AS03-B (bended shape).

Second, we consider which cubes are connected to each other:

- AS00 does not change;

- There are 6 different possible pairs for AS01. To code each cube, we use an initial letter: I (inverter, red cube), W (wheels, white cube), B (battery, blue cube) and S (sensors, black cube). When coding pairs (as there is no order), we will code a pair by two letters. AS01 is decomposed in AS01-IW, AS01-IB, AS01-IS, AS01-WB, AS01-WS and AS01-BS;

- There are 3 different possible pairs of pairs for AS03, as there is no order between pairs and there is no order in each pair because there is no spatial information about the partial structure. AS03 is decomposed in AS03-IW-BS, AS03-IB-WS and AS03-IS-WB;

- There are 12 possible 3-cube structures for AS02, as we distinguish the middle cube from other ones. Given the separation of AS02 depending on shape, that gives 24 possible 3-cube structures in total. We encode those by putting the middle cube in the middle of the coding. For example, bended shapes are encoded AS02-B-IWB, AS02-B-IWS, AS02-B-BWS, AS02-B-WIB, AS02-B-WIS, AS02-B-BIS, AS02-B-IBW, AS02-B-IBS, AS02-B-WBS, AS02-B-ISW, AS02-B-ISB and AS02-B-WSB.

In the end, instead of 4 observables, we have a total of 34 different observables for partial structures.

We could differentiate structures (either partial or complete) to be more precise depending on the orientation of each cube. In particular, faces with specific elements such as wheels or sensors are important to differentiate because their orientation changes the movement of the structure (e.g. wheels not touching the ground cannot
make the structure move). However, the observation of the orientation of each cube during the activity given existing data may be difficult depending on the hand movement (e.g. moving a cube in hand to observe it with the hand blocking the camera's gaze).

**Switch** As the switch can only have two configurations, we do not need to add new observables to this category. However, we replaced B01 and B02 by S01 and S02 (for switch) as it was redundant with behavioral observables.

**Affordances.** Affordances are determined by an element from the cubes that suggest an action. Affordances observables correspond to the moment the affordance suggests the action. From the original activity, 5 observables where determined :

- Wheels from the white cube that suggest movement

- Magnets from all cubes that suggest connection

- The switch from the battery cube that suggests its pushing

- The “eyes” from the sensor cube that suggest the vision of the cube (i.e. sensors)

- The sensors from the sensor cube suggest the placement of the cube in the structure.

As we are not able to determine all affordances exactly, we will not change this category.

**Emotions** Emotional observables are based on a specific model of emotion representation using the wheel developed by (Plutchik, 1980). The model, available Figure W, consists of 8 main types of emotions that are divided into subcategories. Actual observables differentiate only main categories for a total of 8 observables. A possible refinement of this model would be to consider separately the different emotions that are represented together in the actual model, like joy and serenity for instance.

<figure><img src="image_10.png" /><figcaption>Figure 5 : Emotions' wheel described in (Plutchik, 1980).</figcaption></figure>
**Behaviors** There are different models representing problem-solving strategies. For instance, (Missier & Fum, 2002) consider "*two main classes of problem solving strategies (memory-based vs rule-based)*", but we can make other distinctions between strategies, depending on formal education, type of problem etc. The general model considered at the moment for the CreaCube activity consists of five strategies considered as behaviors of the subject regarding the activity :

- Trial/error: the subject tries different configurations, most of them do not work, so the subject adapt the structure to achieve a goal (e.g. making a sub-structure consisting of the black, blue and white cubes move)

- Analytical/systemic: the subject analyzes the system and tries to understand how the system (or a subsystem) of the activity works (e.g. watching the white cube, and concluding that because there are wheels, the structure may move thanks to it).

- Hypothesizing: the subject makes hypotheses about the system or the subsystem or changes pre-existing ones to achieve a good level of understanding the system (e.g. making the hypothesis that the white cube moves forward when moved backwards by the subject, or contradicting this hypothesis when tested)

- Ego preservation and complaining: even if those behaviors may not be active towards the resolution of the problem, we could consider them as (unsuccessful) strategies regarding the activity. For instance, a subject complaining to an experimenter may hope that the experimenter will give the solution or a hint.

As this category depends on a chosen model of behaviors and strategies, we cannot refine it without changing the entire behavior model, so we will not change it in this report.

However, there are different

However, we will discuss later how our refinement of observables could lead to a new model of behavior with different techniques such as clustering.

**Problems** Actual problem observables were determined by looking at the existing data and manually classifying non-winning structures. As such, we are not capable of producing something as exhaustive as every combination of goal structures for instance, so we decided not to change anything to these categories. It could be however the subject of a future work.

**Experimental states** T** observables indicate states of the experiment. It goes from T01 to T02 or T03 depending on the end of the experiment, which can be an abandon (T02) or a success (T03). As there are only those two possibilities, we leave this category unchanged.

**Cube repositioning** FL** observables indicate a repositioning of a cube in a structure, without changing its place in the structure: either the cube's orientation is changed (the cube is rotated but remains at the same place in the configuration) or the cube is repositioned exactly the same way (the manipulation did not change the structure at all).

Changes in cubes orientation are of one of those two types, but the modification to the structure may change the coding of final structures (going from a "goal"-configuration to a configuration with one or more problems and vice versa)

**User-related manipulations** U** observables are related to manipulations done by the subject. These manipulations are related either to the instructions (U01 and U02 corresponding respectively to playing and stop playing instructions) or the cubes, giving indications about how many cubes are manipulated. By assembling partial structure observables and user-related manipulations, we think that we could have all necessary information to reconstruct the scene of manipulation at a given moment. However, this is contradicted by the actual model for two reasons :

- When it comes to 2 cubes in hand and a partial structure with two connected cubes and two disconnected cubes, we are not able to identify if cubes in hand are connected or not connected. Making the distinction is interesting for behavior identification reasons: the subject may not have two connected cubes and two disconnected cubes for the same reason. For instance, with two disconnected cubes, the subject may be looking at them one by one, which is not possible if they are connected.

To resolve this problem, we separate the U05 into two observables, because it is the only one which has this problem (1 cube is necessary disconnected, 3 and four cubes are entirely identified by the partial structures), namely U05-C (connected) and U05-D (Disconnected)
When it comes to partial structures with same-size elements, we are not able to distinguish which element is in hand and which element is not. For instance, if we take the partial structure AS00 with no connection between cubes, if the user has two cubes in hand, we are not able to determine which ones are in hand.

To resolve this problem, we could refine again partial structure to have a specific order and consider, for instance, that U** observables consider the first one, two or three cubes in hand (for instance, AS01-IW could be separated into AS01-IW and AS01-BS and if 2 cubes are in hand, those cubes are necessarily IW for the first one and BS for the second one). Another solution could be the color specification of cubes for each cube in hand. As it will be a very tedious solution, we instead propose a new class of observables named *handled*.

**Handled cubes** In order to know which cubes are in hand at a given time, we add the class of observables *handled*. There are 4 H** observables, one for each cube, indicating if the cube is handled or not by the subject:

- HW: "The white cube (wheels) is in hand";

- HI: "The red cube (inverter) is in hand";

- HB: "The blue cube (Battery) is in hand";

- HS: "The black cube (Sensors) is in hand".

We note that these observables may be used to determine some U** observables, as the number of verified H** observables correspond to the number of cubes in hand. For instance, if only H01 is verified, it implies that we are in the configuration U04 "Hands up with 1 cube".

With the refined partial structure observables detailed previously, we are also capable of identifying if the cubes in hand are connected or not without adding new observables. For instance, if we are in the configuration AS01-BS, H01 and H02, not only we are in the configuration U05: we are more precisely in the configuration U05-D because the connected cubes of the partial configuration are black and blue and the handled cubes are white and red.

**Conclusion** To sum up added observables, Table 1 sums up all additions and modifications made to the observables framework.

It appears that changes made here do not concern the analysis of the subject but rather the analysis of cubes. With the actual camera filming only cubes and hands, new observables may be identified (except maybe the green light affordance, see the Discussion section), even with already filmed experiments. This way, new observables do not break previously established constraints, as no sensor is added (so (C1') is valid) and no changes are done on the activity (so (C2) is valid).

The new framework appears to be more precise. However, a precise framework may not be useful. To justify the relevance of this refinement, we need to explain how the new framework is useful to either collect more precise data and analyze it.

The collection of more precise data comes from the non-suppression of any existing observables: the entire set of observables collected from previous experiments with the previous framework is included onto the new one. As such, we can guarantee that the new framework is at least as precise as the previous one.

In order to discuss contributions of the new framework to data analysis, we will then classify observables from the framework into two categories ("device-related" and "subject-related"), and suggest how the easy collection of the first category will help the analysis of the second one.
<table><caption>Table 1: Summary of modifications of the observables framework. c-letters stand for the cube ID (depending on its role, W, B, I or S) and ** stand for numbers.</caption><thead><tr><td>Category</td><td>Name</td><td>Statues</td><td>Coding</td><td>Number</td><td>Refinement</td></tr></thead><tbody><tr><td>F</td><td>Figures</td><td>Untouched</td><td>(Mercier et al., 2022)</td><td>(Mercier et al., 2022)</td><td></td></tr><tr><td>AS</td><td>Partial Structures (0, 2 or 3 cubes connected)</td><td>Extended</td><td>**-cccc(-S/B)</td><td>34</td><td>Which cubes are connected, do they form a bend or a line?</td></tr><tr><td>S</td><td>Switch</td><td>Untouched</td><td>**</td><td>2</td><td></td></tr><tr><td>AF</td><td>Affordances</td><td>Untouched</td><td>**</td><td>5</td><td></td></tr><tr><td>E</td><td>Emotions</td><td>Untouched</td><td>**</td><td>8</td><td></td></tr><tr><td>B</td><td>Behaviors</td><td>Untouched</td><td>**</td><td>5</td><td></td></tr><tr><td>U</td><td>User manipulations</td><td>Extended</td><td>**(-C/D)</td><td>8</td><td>U05 : connected pair or not?</td></tr><tr><td>P</td><td>Problems</td><td>Untouched</td><td>**</td><td>6</td><td></td></tr><tr><td>H</td><td>Handled cubes</td><td>Added</td><td>**</td><td>4</td><td>Is a cube handled?</td></tr><tr><td>FL</td><td>Repositioning of a cube</td><td>Untouched</td><td>**</td><td>2</td><td></td></tr><tr><td>T</td><td>Experimental state</td><td>Untouched</td><td>**</td><td>3</td><td></td></tr></tbody></table>

## 2.3. Classification of observables

We decided to classify observables from the new framework in two distinct categories: subject-related and Device-related.

**Device-related observables** Device-related observables are observables that do not take into account the internal state of the subject to be determined. They can be identified by sensors or a manual identification pointed only on cubes. It corresponds to the evolution of the cubes and the general structure. As such, this category contains Partial structure observables AS, Final structures observables F, Switch-state observables S, but also Handled observables H and User-related observables U and experimental states T, because they do not require assumption about the player's internal state but rather the physical actions taken, such as pressing the button to play instructions, having one cube in hand, or succeeding to the activity.

As these observables cannot be subject to interpretation (they are the most precise direct measures we have), we think that it is possible to automate the collection of device-related observables. We will discuss in the next section what are the solutions we think of to begin with.

Also, even if we think that it is rather inefficient, the current method of data collection is enough to properly collect these observables, and past experiments videos are compatible with the collection of device-related observables from our new framework.

**Subject-related observables** Subject-related observables are observables that require assumptions by the observer, either the internal state of the subject or the state of the activity to be determined. It corresponds to the player's emotion observables E**, the behavior observables B**, but we also consider problem observables P** (because the identification of the problem comes from the observer) and affordance observables AF** (because even if affordances are situated on the device, the observables correspond to their identification by the subject, thus requiring an assumption of changes at an internal state level).
**Manipulation reconstitution** A set of observables is collected at a given time (at a given time t, we collect the activity state for device-related observables and the player's internal state for subject-related observables). With a given sampling, we are able to describe an experiment by the succession of sets of observables (t0, t1, t2 etc.). As such, we may be able to reconstitute the entire physical manipulation done by a subject if the sampling step is low enough, such as the evolution of the subject's internal state during the activity.

However, it is more difficult to determine the player's internal state than the physical state of the experiment during the activity. We did the distinction between device-related and subject-related observables because we think that the first may help the collection of the others in two possible ways :

-First, with manual collection of subject-related observables, we may be able to train a machine learning algorithm capable of inducing the subject-related observables from device-related observables. As an example, consider that the successive grabbing of each cube is classified by human observers as an "Analytical/System" observable (B02). The succession of "No cube in hand" (U03) and "One cube in hand" (U04), and the succession of the four different Handled observables H may imply that the subject is having an "Analytical/System" behavior, without having access to any subject-related observable.

-Second, from the identification of patterns in the manipulation, a new classification of behaviors may emerge if different frequent patterns are related to the same behavior in our actual framework. This is also the reason why we did not want to refine most of the subject-related observables, as the analysis of the device-related ones may change them over time.

**Conclusion** We think that the new framework of observables is relevant because it allows us to make the physical reconstitution of an entire manipulation. Moreover, we think that this reconstitution may be useful to also better identify the player's internal state, either by training an algorithm capable of inducing the player's internal state from the physical reconstitution, or by allowing us to refine the model with currently unknown manipulation patterns. However, this framework is far from complete, and we will now discuss the limits of this framework.

## 3. Discussion and future work

In the last section, we detailed a new framework, more precise than the actual one. However, it has some limits that must be pointed out before deciding to consider it or not in future works.

**A discussion about premises** Our main constraint was that the CreaCube activity remained the same. The main justification to this constraint is that we do not want to throw away existing data. For instance, the activity is not redoable (as for problem-solving activities in general, a familiar activity is not resolved the same way as a completely new one), so it requires finding new subjects. However, the activity already has other external factors that can relativize this constraint. For instance, even with a perfect environment with no external factors whatsoever, depending on the subject's condition (illness, fatigue, etc.) the experiment may be different from day to day with the same subject. Another example is that there are already external factors such as a camera for the experiment recording. As such, the addition of the constraint is only a constraint of caution, but we could consider that, as there are already cameras and sensors, we could add other sensors.

Moreover, it appears that the problem of external sensors is not necessarily a problem. It seems that external sensors, even when they are noticed by the subject (for instance when they are situated directly onto the subject with something like eye-tracking goggles, are ultimately forgotten. As such, we could consider the (C1') constraint as irrelevant, and then be able to collect more data about the subject. We gave the example of the subject's position during this activity, it is something that could be observed with new sensors if we get rid of this constraint. However, these new sensors should be able to prove their non-influence on the subject's behavior.

Another way of thinking about the problem is trying to minimize every external factor possible. Even then, our constraint is not necessarily the best, because the activity already has sensors that are (even if they are not likely to be) disturbing for the player. This constraint could have led us to change the activity protocol to try to identify and minimize external factors.
In any case, the premises are not set in stone. We think that ours may be too cautious but were good constraints as a first look at the observables framework.

**A discussion about choices** When it comes to refining device-related observables, the first question is: where should we stop the refinement? We could consider that the best data about the external activity is the exact evolution of the cubes, whether they are pivoted or moved a little for instance, implying that the refined model we proposed is not enough. First, it is exact that this refinement may not be enough, and may be the object of a future work. We could refine more, but then we will be confronted with new questions, about the time sampling of observables (some micro-movements of the cube may not be collected) or the precision of our sensors. Our choice was motivated by the reconstruction of the manipulation, not in the exact spatialized reconstruction of the manipulation, because we are trying to study general behaviors when a subject is confronted to a problem-solving activity, so we are trying to identify general patterns.

We also decided to not insist on subject-related observables. As already detailed before, the player's internal state representation models are neither complete nor definitive. For instance, even if we take the categorization of emotions from Plutchik (1980), we could create a new framework with a new emotions representation model by changing E observables and without changing anything else. This implies that our framework is flexible, because it may be adapted to different behavioral or emotional models and help to determine if a model is more relevant than another. To sum up, we chose not to change emotions or behavior observables because we want the framework to be agnostic in terms of the player's internal state model.

Moreover, a future work could be to use our model without subject-related observables to confirm or infirm models of behaviors or emotions during the CreaCube activity. For instance, we could verify if the identification of specific behaviors correspond to specific manipulations of the cubes : a "trial and error" behavior could correspond to a manipulation in which the cubes are assembled in structures of four, then disassembled and assembled again differently. We could compare manipulation patterns with identified behaviors in order to determine an adequate model of behaviors.

**A discussion about feasibility** We already justified that we can use this framework with the actual protocol, because every refinement may be identified with the manual labeling of recordings. However, in order to justify the constraint (C) and maybe be able to remove the camera, we started to design a new version of the material for the CreaCube task, identical from the outside but different from the inside, allowing us to collect data we thought we needed to automatically collect observables. It appears that it may be difficult to create a new version that is not too expensive without changing the physical form of the activity. We are working on two different perspectives for the CreaCube task :

- A new version of the activity inside a box with sensors to detect the spatial position of each cube during the activity. We are trying to implement algorithms for automatic deduction of device-related observables thanks to the evolution of the spatial position over time. For instance, we created a rule-based algorithm to identify H observables and U observables.

- Another version of the experiment adding eye-tracking goggles and data to concentrate on the player's internal state. For instance, a player may be handling a cube but be looking at another one. The eye-tracking technology may help us to identify behaviors and affordance more precisely. As we already discussed, adding eye-tracking goggles may not influence the subject too much while bringing a lot of new information about the player.

Through this reconsideration of the observables but also the device we aim not only to improve the indicators but also improve the way of automatisation the data collection.

**Acknowledgement:** This work is a collaboration between the ANR CreaMaker<fnref n="5" /> (ANR-18-CE38-0001) and the AEx AIDE<fnref n="6" />.

<footnote n="5">⁵ https://creamaker.wordpress.com</footnote>

<footnote n="6">⁶ https://team.inria.fr/mnemosyne/en/aide/</footnote>
Bibliography

Alexandre, F. (2020a, September 7). Creativity explained by Computational Cognitive Neuroscience. ICCC'20 - International Conference on Computational Creativity. https://hal.inria.fr/hal-02891491

Alexandre, F. (2020b). Creativity explained by Computational Cognitive Neuroscience. ICCC'20 - International Conference on Computational Creativity, 4. https://hal.inria.fr/hal-02891491

Barnabé, S., Denet, L., Manrique, M., Menon, D., Pascual, E., Romero, M., & Viéville, T. (2020). *A low-cost tabletop game to collect learning analytics during computational thinking using unplugged or tangible activities* [Report]. Inria. https://hal.inria.fr/hal-03040909

Brand-Gruwel, S., Wopereis, I., & Vermetten, Y. (2005). Information problem solving by experts and novices: Analysis of a complex cognitive skill. *Computers in Human Behavior*, 21(3), 487-508.
https://doi.org/10.1016/j.chb.2004.10.005

Capron Puozzo, I. (2016). *La créativité en éducation et formation*. De Boeck Supérieur.
https://www.deboecksuperieur.com/ouvrage/9782807301894-la-creativite-en-education-et-formation

Constantinou, M., Ashendorf, L., & McCaffrey, R. J. (2005). Effects of a Third Party Observer During Neuropsychological Assessment: When the Observer Is a Video Camera. *Journal of Forensic Neuropsychology*, 4(2), 39-47. https://doi.org/10.1300/J151v04n02_04

Gaut, B. (2010). The Philosophy of Creativity. *Philosophy Compass*, 5(12), 1034–1046.
https://doi.org/10.1111/j.1747-9991.2010.00351.x

Greenwald, N. L. (2000). Learning from Problems. *Science Teacher*, 67(4), 28–32.

Gris, G., & Bengtson, C. (2021). Assessment Measures in Game-based Learning Research: A Systematic Review.
*International Journal of Serious Games*, 8, 3–26. https://doi.org/10.17083/ijsg.v8i1.383

Jacob, S., & Warschauer, M. (2018). Computational Thinking and Literacy. *Journal of Computer Science Integration*, 1. https://doi.org/10.26716/jcsi.2018.01.1.1

Kalmpourtzis, G., & Romero, M. (2020). Artifactual Affordances in Playful Robotics. In I. Marfisi-Schottman, F. Bellotti, L. Hamon, & R. Klemke (Eds.), *International Conference on Games and Learning Alliance* (pp. 316–325). Springer International Publishing. https://doi.org/10.1007/978-3-030-63464-3_30

Kazdin, A. E. (1982). *Single-Case Research Designs—Methods for clinical and applied settings*. Oxford University
Press. //global.oup.com/ushe/product/single-case-research-designs-9780190079970

Leroy, A., Romero, M., & Cassone, L. (2021). Interactivity and materiality matter in creativity: Educational robotics for the assessment of divergent thinking. *Interactive Learning Environments, 0*(0), 1–12.
https://doi.org/10.1080/10494820.2021.1875005

Mercier, C., Romero, M., Alexandre, F., & Vieville, T. (2022). An ontology to formalize a creative problem solving task. *IEEE Transactions on Cognitive and Developmental Systems, Submiteed*.

Mercier, C., Roux, L., Romero, M., Alexandre, F., & Viéville, T. (2021). Formalizing Problem Solving in Computational Thinking: An Ontology approach. *2021 IEEE International Conference on Development and Learning (ICDL)*, 1–8. https://doi.org/10.1109/ICDL49984.2021.9515660

Missier, F. D., & Fum, D. (2002). Declarative and Procedural Strategies in Problem Solving: Evidence from the Toads and Frogs Puzzle. In *Proceedings of the Twenty-Fourth Annual Conference of the Cognitive Science Society*. Routledge.

Newell, A. (1981). The Knowledge Level: Presidential Address. *AI Magazine*, 2(2), 1–1.
https://doi.org/10.1609/aimag.v2i2.99

Oltețeanu, A.-M. (2020). *Cognition and the Creative Machine: Cognitive AI for Creative Problem Solving*. Springer International Publishing. https://doi.org/10.1007/978-3-030-30322-8

Organisation for Economic Co-operation and Development. (2013). *Draft PISA 2015 Collaborative Problem Solving Framework*.

Plutchik, R. (1980). Chapter 1—A GENERAL PSYCHOEVOLUTIONARY THEORY OF EMOTION. In R. Plutchik & H. Kellerman (Eds.), *Theories of Emotion* (pp. 3–33). Academic Press.
https://doi.org/10.1016/B978-0-12-558701-3.50007-7

Romero, M. (2017). CreaCube, analyse de la résolution créative de problèmes par le biais d'une tâche de robotique modulaire. *Journées Nationales de la Recherche en Robotique, JNRR 2017*, 1.

Romero, M., David, D., & Lille, B. (2019). CreaCube, a Playful Activity with Modular Robotics. In M. Gentile, M. Allegra, & H. Söbke (Eds.), *Games and Learning Alliance* (Vol. 11385, pp. 397–405). Springer International Publishing. https://doi.org/10.1007/978-3-030-11548-7_37

Romero, M., DeBlois, L., & Abajyan, P. (2018). *CréaCube, comparaison de la résolution créative de problèmes, chez des enfants et des adultes, par le biais d'une tâche de robotique modulaire*. *Mathematice 61*. En ligne: http://revue.sesamath.net/spip.php?article1104. 61.

Romero, M., Freiman, V., & Rafalska, M. (2022). *Techno-creative problem solving (TCPS) framework for transversal epistemological and didactical positions: The cases of CreaCube and the Tower of Hanoi task*.
Romero, M., Lepage, A., & Lille, B. (2017). Computational thinking development through creative programming in higher education. *International Journal of Educational Technology in Higher Education, 14*(1), 1–15.
https://doi.org/10.1186/s41239-017-0080-z

Romero, M., Lille, B., & Patiño, A. (Eds.). (2017). *Usages créatifs du numérique pour l'apprentissage au XXIe siècle* (1st ed.). Presses de l'Université du Québec. https://doi.org/10.2307/j.ctt1vw0rkx

Romero, M., Vieville, T., & Heiser, L. (2021). *Analyse d'activités d'apprentissage médiatisées en robotique pédagogique*. https://doi.org/10.13140/RG.2.2.10953.16489

Wiley, J. (1998). Expertise as mental set: The effects of domain knowledge in creative problem solving. *Memory & Cognition, 26*(4), 716–730. https://doi.org/10.3758/BF03211392
informatics mathematics
Inria

RESEARCH CENTRE
BORDEAUX - SUD-OUEST

351 Cours de la Libération
Bâtiment A29
33405 Talence Cedex France

Publisher
Inria
Domaine de Voluceau - Rocquencourt
BP 105 - 78153 Le Chesnay Cedex
inria.fr
ISSN 0249-6399