# From CreaCube to EscapeCube: trying to add internal sensors to collect data automatically

Axel Palaude, Lucie Bannay, Baptiste Bernard, Thierry Viéville

## ► To cite this version:

Axel Palaude, Lucie Bannay, Baptiste Bernard, Thierry Viéville. From CreaCube to EscapeCube: trying to add internal sensors to collect data automatically. [Technical Report] RT-0517, Inria & Labri, Université de Bordeaux. 2022, pp.18. [hal-03687237v2](https://hal.inria.fr/hal-03687237v2)

HAL Id: hal-03687237

https://inria.hal.science/hal-03687237v2

Submitted on 31 Aug 2022

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_1.png" /></figure>
informatics mathematics
Inria

# From CreaCube to EscapeCube : Trying to add internal sensors to collect data automatically

Axel Palaude, Lucie Bannay, Baptiste Bernard, Thierry Viéville

## TECHNICAL REPORT

N° 0517

May 2022

MNEMOSYNE

ISSN 0249-6399 ISRN INRIA/RT-0517-FR+EN
<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

# From CreaCube to EscapeCube: trying to add internal sensors to collect data automatically

Axel Palaude<fnref n="1" />, Lucie Bannay, Baptiste Bernard<fnref n="2" />, Thierry Viéville<fnref n="3" />
MNEMOSYNE

Technical Report N°0517 — May 2022 — 18 pages.

**Abstract:** CreaCube is a problem solving activity requiring both computational thinking and creative problem-solving, in which the subject tries to assemble four cubes into one structure with a specific performance. We made the assumption that device-related observables such as cubes position evolution in space may be used to automatically determine subject-related observables which can then be interpreted in relation to certain behaviors or emotions. This article details the design of new material for the CreaCube activity in order to create a new set of modular robotic cubes. This paper describes the analysis developed, step-by-step, and what pitfalls and failures we faced during the analysis of the requirements for developing new modular cubes. The goal of developing a new set of robotic cubes aims to facilitate automatic data collection but also to ensure a large diffusion of the task to collect more data automatically. First, the analysis of existing Human Robot Interactions (HRI) within the CreaCube task is developed through UML charts based on the instructions of the task and the anticipation of the HRI. Second, we discussed the physical elements of the new set of cubes, aiming at an easy open-source manufacturing for anyone interested in activities for the development of computational thinking. Third, we implemented a conditional statement algorithm trying to automatically collect device-related observables despite an unknown set of unpredicted but possible behaviors.

**Key-words:** creative problem-solving, learning analytics, educational robotics, data collection

<footnote n="1">¹ Axel Palaude, Inria Mnemosyne Team, AldE - axel.palaude@inria.fr</footnote>

<footnote n="2">²Lucie Bannay, ENSC Bordeaux; Baptiste Bernard, ENSEIRB-MATMECA Bordeaux</footnote>

<footnote n="3">³ Thierry Viéville, Inria Mnemosyne Team, UCA LINE Laboratory, thierry.vieville@inria.fr</footnote>
## De CreaCube à EscapeCube : une tentative d'ajout de capteurs internes pour permettre une collecte de données automatique

**Résumé** : CreaCube est une activité de résolution de problème qui requiert à la fois un recours à la pensée informatique et à la résolution de problème créatif, lors de laquelle le sujet doit assembler quatre cubes en une seule structure avec une propriété de mouvement. Nous avons fait l'hypothèse que les observables relatifs à l'état du support physique comme l'évolution de la position des cubes dans l'espace pourraient être utilisés pour déterminer automatiquement les observables relatifs au sujet comme les comportements ou les émotions. Ce rapport détaille le développement d'une nouvelle version de CreaCube. Il décrit pas à pas les étapes mises en place pour ce développement, en détaillant les différents obstacles qui se sont posés durant celui-ci. L'objectif est de développer un nouvel ensemble de cubes robotiques permettant de faciliter la collecte automatique des données. L'analyse de l'interaction homme-machine (IHM) de la tâche CreaCube est développée à travers des schémas UML basés sur les instructions de la tâche et l'anticipation de l'IHM. Ensuite, nous discutons des éléments physiques de nouveaux cubes, en visant une production facile et open-source pour un accès facilité pour une large diffusion de la tâche pour quiconque est intéressé dans le développement de la pensée informatique. Enfin, nous implémentons un algorithme conditionnel afin de déduire les observables relatifs au support à partir des données collectées par les nouveaux cubes, malgré un ensemble de conditions non exhaustif.

**Mots clés** : résolution créative de problème, analyse de l'apprentissage, robotique éducative, collecte de données
**Description of the project** 6

1.1. Context and framework 6

1.2. Basic requirements 7

**Conception of EscapeCube** 8

2.1. Modeling of the new cubes set 8

2.2. Practical cases and discussion about physical components 10

2.3. Data processing using conditional statements: from raw data to task observables 13

**Conclusion** 15

**Bibliography** 16
# 1. Description of the project

## 1.1. Context and framework

The CreaCube task engages a participant in an ill-defined problem solving activity with four different cubelets (red, white, blue and black) presented on a table (with a black point and a red point on it) to a player. The instructions, repeatable at will, consist in one sentence : « build a vehicle made up of four pieces that moves by itself from the red point to the black point ». The setup of the activity is shown in Figure 1. The black cube has a face with sensors, the blue cube has a switch on a face to activate a battery and the white cube has wheels on a face that need to be alimented and receive instructions from sensors to move at a certain speed.

<figure><img src="image_4.png" /><figcaption>Figure 1 : A configuration of the CreaCube activity</figcaption></figure>

We can use the CreaCube activity in order to study some transversal competencies, also known as 21st-century skills, including problem solving, collaboration, creativity and computational thinking (Romero et al., 2017). The data collection of CreaCube experiments consists of sets of observables, and a set of observables represents the state of the activity at a given time. In this context, an observable represents a specific element of the scene : the actual configuration of the cubes or the player's behavior. Observables require a decision based on direct analysis of the situation or indirect one (through video analysis) or other data such as learning analytics provided by the modular cubes can inform these observables.

In this article, we will use the observable framework that was developed in (Palaude et al., 2022). This framework consists of two categories :

- Device-related observables : Observables that are related to the physical manipulation of the activity. It contains mainly observables about the configuration of the cubes (what cubes are connected, what cubes are in hand, etc.). These observables can be considered as modular robotic learning analytics at specific moments or data streams if they are generated within a certain time frame with a certain regularity to inform about the longitudinal CPS process during the task;

- Subject-related observables : Observables that are not related directly to the physical manipulation of the activity. It contains mainly observables about the player's internal state (behavior, emotions, what did the player see etc.).

The main assumption behind this classification is that subject-related observables may be deduced by the analysis of the evolution of device-related observables. As device-related observables are not subject to interpretation (a
cube is in hand at a given time or not), this assumption implies that we may be able to collect subject-related
observables without the bias of an external human observer.

This research report summarizes the design and conception of a new physical version of CreaCube taking into
account the new set of observables detailed in (Palaude et al., 2022).

## 1.2. Basic requirements

Our main goal is the data collection of observables from the CreaCube activity with non-intrusive sensors, with a
setup that is transportable, autonomous, cheap and easily reproducible. This goal may be decomposed into three
main issues: (1) learning analytics from each modular robotic cube; (2) transportability and autonomy; (3)
Reproducibility and price.

**1. Learning analytics from each modular robotic cube.** Through these observables we aim to analyze the
CreaCube activity through non-intrusive sensors. We make the following assumption : If we want to collect
relevant data, we need to create an activity that is the same from a subject's point of view.

One possibility is to integrate the sensors inside the new version of the cubelet, which induces a modification of their size, which has been considered as acceptable by the learning science colleagues.

Beyond, for instance, adding a fifth cube devoted to the collection of data is not conceivable, because its presence
in the activity would add an element that the subject can take into account during the experiment.

As such, we call intrusive sensors the physical elements devoted to the data collection that may influence the
subject's behavior during the manipulation.

We point out that the original CreaCube activity already has what we could call an intrusive sensor. A camera
pointed at the hands of the subjects (and the cubes) records the activity to allow a manual (or automatic) detection
of observables. It is not an intrusive sensor because it is included within the original CreaCube activity
organization, but as it is not considered as an existing element during data evaluation, we will try to pass on it.

Given these elements, it seems that it will be necessary to change internal components of the cubes, the table with
the black and red points or the instructions device in order to achieve the data collection.

**2. Transportability and autonomy.** Since we want to collect enough data to perform relevant statistics without
hundreds of hours of manual analysis of video, we want to be able to easily set up an automatic experiment,
leading to two different constraints : transportability and autonomy.

The transportability indicates that the setup may be used in different environments where the experiment may take
place : during scientific events, in libraries or schools etc. As such, the setup should not be too fragile, sensible to
external conditions like humidity and not too cumbersome. We will take the original cubes from CreaCube as a
reference for resistance as they are already resistant to most of the factors we will describe later. They are also not
cumbersome, easily stored in a handbook. Our final product will comply with such requirements, in addition to the
fact that a drastic change in physical form of the activity may contradict our previous statement about intrusivity to
the activity.

The autonomy indicates that the setup should be able to collect data without the presence of a specialized
observer. This overlaps with the next point, as autonomy in data collection is key to allow people not specialized in
the CreaCube activity to reproduce it without an extended knowledge of its data collection. In other words, third
party partners should easily perform the experiment in an unbiased way.

**3. Reproducibility and price.** Another consequence of our general objective is the fact that we want the activity to
be easily reproducible. As the activity tackles computational thinking and creativity we think the activity could be
adapted by teachers, educators, etc. The idea is to achieve a win-win situation, where the activity itself is used for
those who want to do it, and data can be collected to be analyzed afterwards if people want to, thus creating a
case of participatory science.

However, this idea requires making a project that is open (with potential models, 3d-printable tools etc.) and as
cheap as possible.
The CreaCube activity itself, on a minimal setup (instructions printed on paper and no red or black points), only requires the four cubes from the *Cubelet* robotic cube kit, which are sold for about 40 euros each. Thus, our budget should not exceed 150 to 160 euros. Our cubes will not include some functionalities of cubelets that are not necessary for the activity, but additional sensors may increase the price.

To sum up, we want the new activity to remain the same from the subject's perception. As such, we needed to consider non intrusive sensors. However, as an assumed goal is to easily spread the activity outside laboratory experimental situations, we also needed to consider constraints of transportability, autonomy, reproducibility and price, constraints that do not apply for set laboratory experiments controlled by experimenters.

With all these points in mind, we will now detail the steps of the conception of the project, that we called EscapeCube (as an escape game shop was a potential target for a spreading of the activity).

## 2. Conception of EscapeCube

The conception of the project consists of three steps : modeling the new activity with the constraints previously detailed, choosing the physical components for the setup and its assembly and programming the autonomous collection of data.

### 2.1. Modeling of the new cubes set

While in use, the EscapeCube activity has two main actors. First and foremost, the subject of the activity, the player, manipulates the cubes to achieve the goal given by the instructions. Second, the experimenter is interested in data about the activity. Figure 2 is the use-case diagram of the project following the Unified Modeling Language (UML) standard, that was used for the duration of this part of the conception.

<figure><img src="image_5.png" /><figcaption>Figure 2 : UML Use-case diagram of EscapeCube</figcaption></figure>

These two actors do not necessarily interact with each other, and thus actions are not necessarily linked together :
- The player **does the CreaCube activity**. As the cubes are freely manipulated, they require robustness for cubes manipulation (transportability) and autonomy from the battery.

- The experimenter is interested in the **Data collection** of the activity. This collection requires a **Data recording** setup. We made the distinction between **individual data** and **group data** because we are interested in data from only one subject at a time, but the activity may be played by groups, leading to different data that we must be aware of (in order not to include them in the analysis of one-person manipulations). The experimenter is the actor interested in the data collection, and it will require an external computer to retrieve it to guarantee the autonomy of the cubes, either by a wireless connection (bluetooth, wi-fi) or by a plug connected to the cubes when they are not manipulated. We also consider that the experimenter needs to take care of the **Maintenance** of the cubes (mainly moving out data and charging the battery), but it could be done by an external actor (e.g. a teacher presenting the activity to the class).

It appears that a third actor, the facilitator, could have been added, taking up the role of the maintenance of the activity material for instance, but also the preparation of the activity (disposition of the cubes, general explanations before the start of the activity), but its role may overlap with the experimenter's. It could be a teacher, an animator etc. The facilitator is not necessarily interested in data collection, but rather in the playful side of the activity for instance. We will consider different scenarios in the next section where we will consider a facilitator.

We will consider now a situation of what we think will be a typical sequence of the EscapeCube activity, represented in Figure 3.

<figure><img src="image_6.png" /><figcaption>Figure 3 : UML Sequence diagram of the activity representing the timeline of interactions.</figcaption></figure>
The subject needs to listen to the instructions, and the first listening is situated at the beginning of the activity. It implies that the data collection must begin when these instructions are played for the first time. As the player may quit without succeeding in the activity, we cannot consider an automatic ending to the activity without considering either a time limit (or inactivity time limit) or a stop button. As the stop button is more flexible and representative than a time limit because it accounts for the desire of the subject to stop the activity, we considered a stop button. The announcement of the end of the experiment is indicative to allow the activity to be done again, thus preventing a player change in the middle of the activity.

While creating the sequence diagram, we noticed that the activity instructions are given by a device external to the cubes, and thus put a start and a stop button there. In the original CreaCube activity, it is not a clear element, as it may be a piece of paper with instructions written on it, or a recorder with a listenable record of the instructions. As it is not a defined element, we did not consider the device as a part of the unchangeable CreaCube activity (in the late version of the activity, the most frequent device is an external listenable record of the instructions). The same goes for the surface where the cubes are placed. As such, it appears that those elements may be pivotal to the transformation of CreaCube into EscapeCube, as well as the internal structural organization of the cube (which can be modified freely).

However, it also implies that the CreaCube activity already has blurry elements in it that could change the behavior of the cube depending on which one is chosen. Maybe a recorder may be analyzed as a potential part of a solution from the subject's perspective, whether a simple piece of paper may not. As such, we could question the relevance of our first and main point justifying the creation of a new version of the activity.

With these two diagrams, we saw that a general comment about the activity is that it may be used in a wide variety of situations, and that these situations may have an impact on how the cubes should work. In order to go a step further and discuss the physical implementation, we will now present different practical cases for the project and what they imply for the cubes.

## 2.2. Practical cases and discussion about physical components

We consider three different “typical” experimental situations in which the EscapeCube project and cubes could take place. These situations were created to discuss design constraints by covering a large panel of possible situations with typical predictable constraints. Our 3 situations are the following :

- An experimental situation similar to already existing data situations.

- An educational and playful situation in a place like a classroom or a media library.

- A fully playful situation in a place like an escape game.

### a. Research lab experimental situation

In this situation, the subject is confronted with the activity in a controlled environment which is situated in a research laboratory.

Before the activity, the experimenter prepares the cubes original disposition beforehand, and gives only necessary information required by the experimental protocol.

During the activity, the subject is alone, and so the data collection corresponds fully to actions taken by the one subject of the activity, as no external noise (like advice from other people) will change the subject's behavior.

Given the situation and the actual experimental protocol, the subject is fully aware of the measures. Furthermore, as the subject is doing the activity in a laboratory, there is an awareness of the scientific and educational purpose, in addition to the playful feature of the activity.

After the activity, the experimenter can gather data about the experiment and prepare the activity for another subject.

This situation is the control situation. It corresponds to the activity with minimum noise and constraints, as the experimenter and the environment can be adapted to fit a new version of the cubes.
This experimental condition is the one requiring the least constraints for the experimental material. A prototype that is not usable in this control situation has no chance to be developed fully.

### b. "Media center" situation

In this situation, the subject is confronted with the activity in an uncontrolled environment like a library.

Before the activity, the facilitator-experimenter prepares the cubes original disposition beforehand, and gives only necessary information required by the experimental protocol.

During the activity, the subject is alone. However, there can be a potential noise in this public environment, and elements from the environment (bookshelves, pen, paper etc.) may influence the subject's way of thinking. Moreover, the subject may already have seen or heard cues about the activity before being confronted with it (during someone else's take on the activity for instance). As such, the subject's potential behavior may be influenced.

Given the situation and with the presence of the facilitator-experimenter, the subject is aware of the measurement beforehand. As the subject may not be present specifically for the activity (school trip for instance), the playful nature of the activity may be more important than scientific and educational purposes, unlike in the control situation.

After the activity, the experimenter-facilitator may gather data about the experiment and prepare the activity for another subject.

This situation is less controlled than the previous situation. The main purpose behind the organization of the activity is the educational and scientific purpose behind the activity. The potential variations in the environment (depending on where the activity takes place, which time of the day it is) may influence the behaviors and slightly modify data. This situation may correspond to the activity done in class, with potential noise, external elements and potential cues obtained beforehand by other subjects of the activity.

### c. Uncontrolled experimental situation

In this activity, the subject is confronted with the activity in an uncontrolled environment, which is a game center like an escape game or an outreach activity.

As there is no experimenter nor facilitator, the subjects are entirely dependent on EscapeCube specific elements like listenable instructions and on previous subjects to have the correct original disposition.

During the experiment, the subject is not alone. As the place is a place for groups to come, the subject may be influenced by other group members during the manipulation, either because they may participate during the subject's activity or because other group members are previous subjects that the subject may have seen doing the experiment.

This comes in addition to the potential noise to the experiment (by the form of sound noise or visual noise like elements from other activities, games etc.). As such, the subject's behavior is likely to be influenced by a lot of external factors.

Given the situation, the subject may not be aware of the measurements beforehand, and this is one role of the EscapeCube set to make the subject aware of it. Furthermore, this activity may be a distraction for subjects waiting for other things to do in the place, so the playful nature is the most important one of the EscapeCube set for subjects of this situation.

The data is gathered at given moments by a researcher, for instance after the center is closed. As such, the researcher cannot be considered as a potential facilitator.

This situation assumes real autonomy of the EscapeCube set because it cannot suppose the existence of a facilitator to make sure that the data is properly collected. The design of EscapeCube must take into account a lot of elements : possible biases given by the presence of a group, a way to make the subject aware of the measurement (the scientific outreach of the activity) and a way to help players to put the elements back in order for the next subjects. The set should also be able to detect when the manipulation of a player is not eligible data (when, for instance, cubes are already assembled before the beginning of the activity).

### d. Implications and propositions for the activity
We identify three key elements of the situations :

- **Installation**: in the first two situations, a facilitator may be present to install the activity and in particular the cubes in the correct initial way. However, the absence of the facilitator in the "game center" situation implies that the setup should include a way to have the activity prepared for a subject. Our proposition is that the setup should include a way to help players to put the elements back in order for the next subjects, with for instance detailed instructions to put the cubes in place. The instructions could be vocal or written. Moreover, subjects should not be able to see the cubes before the start of the activity. To this end, we propose a box containing the cubes, opened at the start of the activity.

Finally, as we will see in the next subsection, the cubes need to have autonomous batteries. We can imagine that the initial positions in the box may be used as plugs to recharge the cubes while the device is not used.

- **Distraction**: Depending on the situation, subjects may be influenced by external distractions such as other people. However, setting apart a vocal or written encouragement to focus, we cannot assure that the subject is fully involved in the activity. We may be able to detect the physical intervention of other people sometimes, but as we do not record the voice of the subject, we cannot detect tips or remarks given by other people<fnref n="4" />.

- **Scientific engagement**: The scientific purpose behind the activity may not necessarily be clear. In the first two situations, the facilitator can help the scientific purpose to be perceived by the subject. Without the facilitator, once again, a written or oral informative speech can be added to the setup. The implication of the subject of the experimentation with respect to both the science outreach objective (i.e., computational thinking initiation) and the learning science objective (i.e., better understand how we solve a creative ill-posed problem, in order to contribute to meta learning issues, that is better learn how to learn), are discussed in, e.g., (Barnabé et al., 2020).

<figure><img src="image_7.png" /><figcaption>Figure 4 : Schema of the activity box that contains the cubes for the activity.</figcaption></figure>

<footnote n="4">⁴ This project focuses on single-subject activities, so any cue of another person's presence (except the facilitator) is prohibited.</footnote>
To sum up, the general idea for the activity; represented in Figure 4, is a box containing the cubes and a voice transmitter. The box is used as a battery charger and to hide the cubes from a new subject.

The voice transmitter has three records. The first one contains instructions for installation, information about the scientific purpose of the experiment and general encouragement to do the activity alone. The second one gives the instructions relative to the activity, and the third one gives instructions to prepare the cubes for the next subject. The voice transmitter may be replaced by written instructions.

We could have considered another element like a connected table with devices to measure the distance from the cube during the activity. However, we did not consider it here because of the transportability constraint.

We will now discuss the physical components of the cubes in order to discuss the estimated budget required for EscapeCube.

### e. Physical components of cubes

In order to determine the price of a cube, we need to determine its specifications. We will consider each cube as identical to the red cube (the only cube without supplementary components like wheels or a switch) for the sake of the estimation.

First, we cannot add new elements to the activity because it would change the perception of the subjects. Hence, elements like the bluetooth hat of the original cubelets are not usable.

Every sensor should be situated inside the cube, or be invisible for the subject if placed outside.

Among observables we need to determine with this data collection, we need to determine if a cube is in hand and/or connected to other cubes. Other device-related observables depend on each cube's specifications. We think that we will be able to determine these observables thanks to the spatial position of each cube and how these positions change over time. We will present a conditional statement algorithm for this sake in the next section.

These sensors imply different physical elements to add to the cubes :

- An internal battery for each cube (as sensors must be able to work even when disconnected to the others);

- Sensors allowing to determine the position over time (accelerometer for instance);

- Sensors allowing to determine for each face if it is connected to another cube or not (contact sensors for instance);

- A microcontroller connected to sensors to gather data (allowing to connect other sensors relative to cubes' specificities).

In addition to the price of an individual cube, we need to take into account the other elements like the general device box allowing the portability of the activity discussed in the previous section, or specific sensors like the one allowing to check the evolution of the switch of the blue button over time. That said, it appears that the price of this new device will be too expensive for the budget we fixed. For the sake of the activity only, it will be more interesting to buy the original cubelets for the "game center" situation for instance.

However, we will go on the development of the activity as a more expensive budget is not necessarily an abandon point for research. For instance, this activity may still be used in the "research lab" situation. We will now detail the data exploitation process to determine device-related observables.

## 2.3. Data processing using conditional statements: from raw data to task observables

Our goal is to determine device-related task observables. In the previous section, we discussed potential physical constraints. In this section, we will present device-related task observables, and conditional statements allowing us to convert collected data into these observables.

### a. Device-related observables

In this section, we detail device-related observables. We consider five different types of observables: final structures, partial structures, state of the switch, handling of a cube and user manipulations.
The state of the switch can be directly obtained by a sensor inside the blue cube with the switch. We can get the evolution of the switch over time this way.

Partial and final structures (cubes configurations) correspond mainly to the evolution of the connections between cubes. As we are able to determine the evolution of the connected face over time, we are able to easily reconstruct these structures over time.

Handled cubes correspond to the evolution of the cubes' status. One cube can either be in hand or not in hand. As we didn't consider pressure sensors on the cubes because the technology is too expensive, we need to deduce it from the evolution of the position over time.

User manipulations correspond to general manipulations done by the subject during the activity. It includes the number of cubes the subject has in hand, and when the subject is listening to the instructions for the first time or once again. The first ones are immediately given by the number of cubes with the "handled" status, and the second ones may be detected within the voice transmitter.

Within these observables, handled observables seem to be the ones that require data exploitation. In the next section, we present one possible algorithm of handling detection.

## b. Conditional statements

The algorithm presented here is able to determine which cubes are in hand or not over time, given the evolution over time of the connections between the cubes and the spatial position of the cubes over time.

We will consider three coordinates: two for the position over the table's plan and one for the altitude.

The first naïve idea is to consider that every cube that is not on the spatial positions corresponding to the table is in hand. However, it fails when considering that cubes can be stacked. This algorithm introduces a set of conditional statements to distinguish situations in which the cubes may be in hand from those in which they may not:

**Over limit height.** If the cube is higher than four cubes, it is considered in hand.

This case makes sure that a cube is necessarily in hand if its altitude is more than the theoretical maximum height for non-handled cubes. This theoretical maximum corresponds to the 4 cubes stacked on top of each other, thus equal to the height of the four cubes.

**Increasing altitude.** If the cube's altitude is increasing, the cube is considered in hand.

A cube cannot go up without a helping hand. Thus, if the cube's altitude is increasing in comparison to its previous position, then it is in hand.

**Immobile cubes.** If the cube's altitude is at the minimum level and does not register any movement, it is considered not in hand.

This case corresponds to the naive one. If the cube is on the table and is immobile, it is not in hand. However, we cannot only specify a constant altitude, as the subject may be moving the cubes on the experimental hand, without raising them.

**Constant movement.** If the cube has a constant altitude and moves constantly for few samples, the cube is considered not in hand.

This case takes into account autonomous structures such as the final one that are moving autonomously. As they need to be stable in order to not fall off, if the altitude is constant despite the movement of other coordinates, the cube may be moving autonomously. If this movement is constant within the previous samples (depending on the sample rate), then it is not moved by the subject.
**Propagation of immobility** If two cubes are connected and one of these cubes is considered "not in hand" after the propagation phase, both are considered "not in hand".

This case allows us to take into account immobile cubes stacked on top of other ones. The propagation of immobility allows to let immobile objects to remain not in hand and cubes added to a structure by the subject to be not in hand anymore.

This algorithm was tested on different test and sample data<fnref n="5" />. We decided to use this kind of algorithm because we did not have access to training data for machine learning algorithms for instance. However, we cannot affirm that this algorithm is precise enough, as some specific situations may be incorrect with this algorithm. We will now discuss the limits of this algorithm by, among other, looking at these specific situations.

### c. Discussion

The previous algorithm is able to determine whether a cube is in hand or not depending on the position of each cube. For this, it considers a finite number of events like when the cubes are moving autonomously or when they are handled. However, this technique is limited because of the nature of the CreaCube experiment. As an ill-defined problem, the subject is not limited in their actions. As such, the subject is capable of doing actions that we are not capable of anticipating. To illustrate this, we will consider two situations in which the previous algorithm's cases lead to a false result.

**The fall of the cubes.** When the cubes fall, for instance because there is imbalance in the final structure created by the subject, the cubes may fall on the table and bounce on the table, thus indicating that they are going up. As such, the cubes may be considered in hand when they are not.

**Inconstant movement.** Given the nature of the sensor's cube, a final structure may not have a constant movement while moving, for instance because the subject's hand's position relative to the sensor varies over time. As such, the autonomous movement of a structure may be similar to the moving of the structure by the subject and may not be identified properly.

These two cases are examples of limits of a rule-based algorithm for an ill-defined problem. In the current state, we could just create new cases to take into account those two examples, but the point is that we are not able to list the cases exhaustively. As such, it seems that this kind of algorithm is not a good method to tackle this identification.

Moreover, it appears that, for some cases, the sample rate or the precision of sensors may make the case obsolete or invalid. For instance, the "immobile cube" case, where a cube that is not moving since the last sample is considered not in hand, may lead to invalid results when considering a sensor that is not precise enough to determine that a cube in an immobile hand (the hand is necessarily moving a little) is in hand. As such, this algorithm should be reconsidered each time different sample rates or different components are used, and we will prefer more general algorithms for this.

## Conclusion

The conception of the EscapeCube project faced a lot of different challenges. As such, it appears that the project is not viable right now. In this section, we will sum up the different pitfalls and failures of the project and propose different ideas and future work to move on the issue of data collection.

First, we faced a physical component problem. As we didn't want to transform the activity, the choice of the components was limited and the budget we fixed was exceeded. This excess implied a loss in terms of potential data, as the reduced price in comparison to the original cubes was an argument for potential facilitators (teachers, educators etc.) to choose this version instead of the original cubes.

<footnote n="5">⁵ Available at : https://github.com/BaptisteBenard/creacube_projet_s9
RT N° 0517</footnote>
However, it is possible that the constraint of creating a new activity that is identical from the subject's point of view to the original CreaCube activity is not as important as it seems, for two reasons :

- Extrinsic elements, such as how the subject feels, what is the prior knowledge of the subject etc. may be more important factors of behavioral differences between subjects.

- The point behind this constraint is to ensure that behavioral differences between subjects of CreaCube and subjects of EscapeCube are not caused by the physical components themselves. However, the point of CreaCube is not to study subjects' behaviors specifically about the CreaCube activity but to better understand skills about creativity and problem-solving. As such, some points of the original activity may already be profitable or detrimental to the subject's behaviors, like, for instance, any visual cue such as the visible presence of screws on the original cubes.

As such, the presence of differences between activities may not be that detrimental to data<fnref n="6" />. In particular, cubes including the elements we described in a previous section would probably have been bigger than the original cubes<fnref n="7" /> without causing much trouble for the subjects. The same argument applies when it comes to the presence (or absence) of external elements like one or more cameras.

We also tried to implement a rule-based algorithm to identify whether a cube is in hand or not. Because of the open nature of the activity, it is impossible to predict an exhaustive list of cases (what if the cubes fall on the table, below the table etc.). Moreover, it appears that some cases of this algorithm depend on sensors' precision and sample rate, potentially leading to invalid results if they change in the future. Other types of algorithms should be considered.

If we are able to change the perspective, we could imagine a setup with gloves able to detect the gripping of the cubes, with an immediate detection of whether a cube is in hand or not.

In the end, as we determined that external devices may be added to the CreaCube activity, we are working on the idea of adding an eye-tracking device recording the eye movements of a subject during the experiment to the already existing camera recording the hands during the experiment. This way, we will be able to have an additional data stream from the subjective lived experience during the activity. In this context, the subject may be concentrated on a different cube than the handled one for instance. We still consider that external devices may influence the behavior of a subject, but for the specific eye-tracking device (which are goggles that the subject puts on), it appears that, after a short amount of time (that will be spent before the start of the activity), the subjects do not pay attention to it anymore, as if they forgot its existence.

**Acknowledgement:** This work is a collaboration between the ANR CreaMaker<fnref n="8" /> and the AEx AIDE<fnref n="9" />, and students from the ENSEIRB-MATMECA school<fnref n="10" />.

Margarida Romero, Chloe Mercier and Frédéric Alexandre are gratefully acknowledged for their precious advice during the execution of this work and this report reviewing.

<footnote n="6">⁶ On the contrary, it is possible that some minor changes to the physical components may help the data collection.</footnote>

<footnote n="7">⁷ Bigger cubes are an example of potential beneficial changes to the activity : as they are more difficult to manipulate, maybe we could observe the evolution of the body posture of the subject because of larger movements, and use these large movements as new cues for the data analysis of the subject's behaviors.</footnote>

<footnote n="8">⁸ https://creamaker.wordpress.com ANR CreaMaker (ANR-18-CE38-0001) project</footnote>

<footnote n="9">⁹ https://team.inria.fr/mnemosyne/en/aide/</footnote>

<footnote n="10">¹⁰ https://enseirb-matmeca.bordeaux-inp.fr/fr</footnote>
Bibliography

Barnabé, S., Denet, L., Manrique, M., Menon, D., Pascual, É., Romero, M., & Viéville, T. (2020). *A low-cost tabletop game to collect learning analytics during computational thinking using unplugged or tangible activities* [Report, Inria]. https://hal.inria.fr/hal-03040909

Palaude, A., Romero, M., & Viéville, T. (2022). *Multimodal learning analytics for studying creative problem-solving with modular robotics* (Research Report RR-9467; p. 27). Inria & Labri, Université Bordeaux.
https://hal.inria.fr/hal-03621919

Romero, M., Lepage, A., & Lille, B. (2017). Computational thinking development through creative programming in higher education. *International Journal of Educational Technology in Higher Education, 14*(1), 1-15. https://doi.org/10.1186/s41239-017-0080-z
informatics mathematics
Inria

RESEARCH CENTRE
BORDEAUX - SUD-OUEST

351 Cours de la Libération
Bâtiment A29
33405 Talence Cedex France

Publisher
Inria
Domaine de Voluceau - Rocquencourt
BP 105 - 78153 Le Chesnay Cedex
inria.fr
ISSN 0249-6399