<figure><img src="image_1.png" /></figure>

HAL
open science

# Should We Use Educational Robots to Introduce Students to Computational Thinking? Insights From Two Experimental Studies

Kevin Sigayret, Nathalie Blanc, André Tricot

<figure><img src="image_2.png" /></figure>

## To cite this version:

Kevin Sigayret, Nathalie Blanc, André Tricot. Should We Use Educational Robots to Introduce Students to Computational Thinking? Insights From Two Experimental Studies. Journal of Computer Assisted Learning, 2025, 41 (4), 10.1111/jcal.70074. hal-05101389

HAL Id: hal-05101389

https://hal.science/hal-05101389v1

Submitted on 6 Jun 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_4.png" /></figure>

ORIGINAL ARTICLE

OPEN ACCESS

# Should We Use Educational Robots to Introduce Students to Computational Thinking? Insights From Two Experimental Studies

Kevin Sigayret ID | Nathalie Blanc ID | André Tricot ID

Laboratoire Epsylon UR 4556, Université Paul-Valéry Montpellier 3, Montpellier, France

**Correspondence:** Kevin Sigayret (kevin sigayret@univ-montp3.fr)

**Received:** 20 June 2024 | **Revised:** 12 December 2024 | **Accepted:** 23 May 2025

**Funding:** This work was supported by Académie de Montpellier.

**Keywords:** cognitive load | computational thinking | educational robotics | programming | scratch | unplugged programming

## ABSTRACT

**Background:** Teaching programming and computational thinking is becoming a major issue in many education systems. Numerous approaches are possible, but very few studies compare these different ways of implementing programming and computational thinking learning.

**Objectives:** We compared three ways of teaching programming and computational thinking to grade 5 students: unplugged activities, block-based programming software and educational robotics.

**Method:** This paper re-exploits previously published data on the comparison between unplugged and plugged-in (using the block-based programming software Scratch) learning, by adding a new experimental condition centred on the use of Scratch associated with an educational robot, and incorporates new mixed model statistical analyses. Based on these results, we conducted a second experiment to explore the cognitive load associated with a learning situation on Scratch, with or without an educational robot.

**Results:** Our first experiment revealed that novice grade 5 students using Scratch without a robot learned better programming concepts and skills, compared to students involved in unplugged activities or using Scratch with a robot. Experiment 2 showed that the detrimental effect of using a robot is probably due to an increased extraneous cognitive load in the Scratch with a robot condition. However, robots had a large positive effect on students' motivation. Gender differences were also observed, as girls showed slightly better learning performance but were less motivated than boys when using Scratch without a robot.

**Conclusions:** Overall, these results provide key elements for understanding the advantages and disadvantages of using robots to sustain computational thinking learning in children.

## 1 | Introduction

Learning to program generally covers different activities with an identical goal: developing student's ability to produce algorithms, that is, successions of actions that constitute steps towards the resolution of an algorithmic problem. Computer programming would be intrinsically complex and cognitively

costly, as it would require mastery of several skills and its learning would involve high interactivity between elements (Berssanette and de Francisco 2021). Despite its complexity, there is solid empirical evidence that programming skills can be learned in school (Scherer et al. 2020), and these are part of school curricula in most developed countries (Bourgeois et al. 2019; Conrads et al. 2017). The emergence of new visual

This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

© 2025 The Author(s). Journal of Computer Assisted Learning published by John Wiley & Sons Ltd.
## Summary

* What is already known about this topic

* Computational thinking and programming learning is now part of school curricula in most developed countries.

* Very few studies have been conducted to compare different ways of teaching programming and computational thinking, such as unplugged activities, block-based programming software or educational robotics.

* What this paper adds

* We conducted two studies to compare three ways of teaching programming and computational thinking with grade 5 students.

* The first study revealed that using Scratch software is more effective to teach concepts and skills, compared to unplugged activities or the use of an education robot associated with Scratch.

* The second study measured students' cognitive load, whether they use Scratch only or Scratch with a robot, and revealed an increase in extraneous cognitive load when the robot is used.

* Implications for practice and/or policy

* Programming and computational thinking learning seem most effective with novice grade 5 students when Scratch software is used without a robot.

* The combination of Scratch and a robot could have a detrimental effect due to an increase in cognitive load when students have to handle both tools simultaneously.

* However, robots have a positive effect on students' motivation.

* Girls performed slightly better than boys but, in the Scratch without robot condition, they were also less motivated.

* These results can guide teachers' practices.

programming languages in the 2000s has also made learning easier (Maloney et al. 2010). Programming could improve cognitive abilities (Psycharis and Kallia 2017) and develop computational thinking (Buitrago Flórez et al. 2017; Lye and Koh 2014; Sun et al. 2022).

Computational thinking (CT) is a way of solving problems or designing systems based on the fundamental concepts of computer science (Wing 2006). Currently, there is no real consensus among researchers on a precise CT definition (Tang et al. 2020). However, Román-González et al. (2017) identified three types of CT definitions: general definitions rather focused on the set of skills involved in problem solving; operational definitions that provide a framework by categorising CT into cognitive processes (such as abstraction, decomposition, generalisation, etc.); pedagogical definitions that list a set of concepts and skills to teach in an educational context (e.g., the computational concepts, practices and perspectives of Brennan and Resnick 2012). There are a large number of activities for teaching programming and CT, but three approaches are the most widely used: unplugged activities (with no digital tool), block-based visual programming languages and educational robotics (Piedade and Dorotea 2022).

In this article, we re-examine some of the data we published previously (Sigayret, Blanc, et al. 2022; Sigayret, Tricot, et al. 2022) and compare them to new data. In this previous publication, two groups of students had to learn programming and CT in two different conditions: unplugged versus with Scratch block-based programming software. A year after implementing our study in schools, we ran the experiment again, using the same protocol, with a group of students using a Thymio educational robot, connected to Scratch. This article therefore takes the previously published data and compares them with this new experimental condition. We also took our statistical analyses a step further by running mixed models which include the class each student was in as a random effect of the model, which seems more appropriate for our study than the classic analyses of variance previously conducted. Finally, we present here the results of a second experiment, which aims to explain some of the results observed in the first experiment.

The two experiments we conducted aim to provide some answers regarding the following research questions:

- Which way of teaching programming (plugged-in or unplugged, with or without robot) enhances learning performance?

- Which way of teaching programming (plugged-in or unplugged, with or without robot) enhances student's subjective experience (motivation, self-belief)?

- Does the impact of the way of teaching programming (plugged-in or unplugged, with or without robot) depend on students' gender (boys or girls)?

- Do digital tools (especially when robots are used in addition to block-based programming languages) imply a high cognitive load, potentially detrimental to learning?

## 2 | Literature Review

### 2.1 | Unplugged Activities

Unplugged learning to program does not involve any digital tool and uses student's body movements to represent computational concepts. It includes various activities using cards, board games, pictures or classic pen and paper. These activities usually consist of problems to be solved by manually following sequential steps (Brackmann et al. 2017). Bell et al. (2009) suggested that using activities away from computers to teach computer science is effective, as students would be able to think about questions they would not have asked themselves through simple programming activities, without the need to acquire technical skills beforehand. Unplugged programming would also have some major benefits: reducing the cognitive load associated with using a machine, facilitating sensory-motor interactions and building concrete analogies in relation to the abstract notions in programming (Romero et al. 2018). Bell and Vahrenhold (2018) also identified several benefits to the use of unplugged activities: students engaged with broader issues concerning computers in general, no need for available computer equipment, simpler activities to set up for a short time slot...
Several studies reported highly significant results in the devel-
opment of CT across a wide range of age groups, indicating an
absence of age-related restrictions on the use of unplugged activ-
ities (de Souza Oliveira et al. 2022). For example, pretest–posttest
comparisons show an undeniable improvement in CT skills for
students aged 6–8 (Zhang et al. 2021) and students aged 10–12
(Brackmann et al. 2017; Delal and Oner 2020). Brackmann
et al. (2017) considered that CT is primarily about mastering
problem-solving skills and processes, which can be disconnected
from computer programming. Similarly, Threekunprapa and
Yasri (2020) showed better understanding of concepts related
to programming and CT with 160 secondary school students,
but also a beneficial effect on students’ self-belief. In a literature
review, Battal et al. (2021) affirmed that significant gains have
been observed in students’ understanding, motivation, achieve-
ment and engagement in CT. Based on these studies, we could
assume that unplugged learning to program is an approach to
be recommended because it does not need any specific equip-
ment, can produce high learning performance and may even be
less conducive to stereotypes in relation to students’ gender (Bell
et al. 2009; Delal and Oner 2020).

However, Suárez et al. (2022) conducted a study which revealed
only a slight positive impact of unplugged activities on CT for
high school students. Likewise, Caeli and Yadav (2020) con-
sidered that it is not possible to reach the full potential of CT
through unplugged approaches alone. In addition, Huang and
Looi (2021) conducted a literature review and considered that
results are limited concerning CT development through un-
plugged activities. Some studies suggested that unplugged ap-
proaches can at most be as effective as plugged-in approaches.
For Huang and Looi (2021), most of the studies involving un-
plugged activities are not based on appropriate assessments and
are unable to separate certain confounding variables, as pro-
gramming is part of the intervention. Akiba (2022) argued that
hybrid unplugged/plugged-in activities are an ideal method for
promoting digital literacy. Therefore, the efficiency of unplugged
programming alone should be questioned, particularly in com-
parison with concurrent alternatives, like block-based program-
ming languages which are very often used in a school setting.

## 2.2 | Block-Based Programming Languages

Block-based programming has become increasingly used to
introduce students to programming and computing in general
(Weintrop 2019). Block-based languages differ from traditional
text-based languages by representing instructions as puzzle
pieces that provide the user with visual cues on how and where
commands can be dragged and dropped (Weintrop 2019). This
also prevents syntax errors and makes it possible to use natu-
ral language. Weintrop and Wilensky (2015) showed that high
school students perform significantly better with block-based
programming compared to text-based programming, but Xu
et al. (2019) conducted a meta-analysis which revealed only a
small effect in favour of block-based languages for cognitive and
affective variables. However, Scherer et al. (2020) found a mod-
erate effect in favour of visualisation tools, in particular visual
block-based programming. Similarly, Hu et al. (2021) reported
in their meta-analysis that the use of block-based programming
languages has a moderate positive effect on students' academic

achievement, compared with text-based languages, probably
due to their accessible and intuitive interface. These tools are
also likely to reduce the cognitive load associated with reading,
understanding and building programs, making it more acces-
sible for novice students (Sengupta et al. 2015 cited in Scherer
et al. 2020).

Numerous block-based educational programming software are available but Scratch software is by far the most widely used in the world (Lye and Koh 2014; Tang et al. 2020). Scratch is considered low floor, high ceiling (Olabe et al. 2011; Weintrop 2019), offering quickly mastered functionalities and the possibility of designing complex programs. It is also compatible with some educational robots to create a program on Scratch to control the robot (Olabe et al. 2011). Scratch's interface is particularly easy to learn (Maloney et al. 2010), could foster learning to program in grade 5 or 6 (Sáez-López et al. 2016) and could motivate students to continue practicing programming (Armoni et al. 2015; Ouahbi et al. 2015). Even for university students, Tsai (2019) showed an improvement in learning to program when Scratch was used, compared to more conventional computer science introduction. Scratch is also likely to develop CT skills and knowledge. Piedade and Dorotea (2022) observed a strong effect with 186 grade 4 students, in favour of the group using Scratch, compared to a passive control group. Moreover, Gökçe and Yenmez (2022) showed an increase in CT skills with 524 grade 5–6 students after 17 weeks of Scratch. In their literature review, Stewart and Baek (2023) affirmed that Scratch's ability to foster CT skills is no longer in doubt. So, it seems highly probable that block-based programming languages can foster programming and CT skills, particularly with novice students, thanks to its accessible interface which reduces the cognitive load of the learning task.

However, Jiang and Li et al. (2021) conducted a study with 336 grade 5 students programming on Scratch for 5 weeks and reported mixed results, as no significant difference was observed regarding algorithmic thinking and problem-solving skills, yet widely considered to be constitutive of CT. On the other hand, Aivaloglou and Hermans (2016) analysed almost 250,000 Scratch projects and reported a high proportion of bad programming practices and a lack of complexity in most programs, which confirms that Scratch is mainly used for initial exposure to programming. Despite results highlighting benefits of using Scratch, some studies seem to show the limits of this software to develop core CT skills and reveal that its potential could be misused. However, block-based programming languages such as Scratch offer the possibility to be associated with an educational robot, which might increase Scratch's efficiency in transmitting fundamental knowledge and skills in programming and CT.

## 2.3 | Educational Robots

Educational robotics can be defined as the interaction between
a robotic mechanical device, a programming software and a stu-
dent (Dos Reis et al. 2015), and appear to have a positive impact
on a range of cognitive abilities, such as visuospatial ability and
spatial knowledge (Julià and Antolí 2016; Mari 2022), metacogni-
tive abilities (Socratous and Ioannou 2019), executive functions
(Di Lieto et al. 2017), or creativity (Park et al. 2015; Zawieska
and Duffy 2015; Noh and Lee 2020). Educational robotics would be beneficial to increase students' academic performance, collaborative work, engagement or problem-solving (Anwar et al. 2019; Athanasiou et al. 2019; Talan 2021). According to Talan (2021), educational robotics can be more effective than other traditional methods, as it facilitates learning by concretising abstract concepts and giving students an active role stimulating their interest, attention and motivation. Indeed, robots' affective impact has been corroborated by Kim and Lee (2016), who showed greater interest when students aged 11–12 used robots, even 3 months after their introduction. Robots also appear to facilitate transfer of acquired knowledge to a new context (Chiazzese et al. 2019). In their meta-analyse on learning to program, Scherer et al. (2020) noticed an increase in effect size when robots are used, although it is still too scarcely studied.

Ioannou and Makridou (2018) conducted a literature review on robotics and its impact on CT acquisition and showed a positive effect, with significant gains in problem-solving, collaboration and communication skills, and interest in programming and STEM disciplines in general. Again, the authors concluded that research into CT development through robotic activities is needed. Angeli and Valanides (2020) found an increase in CT skills when using with young children, aged 5–6. Other studies highlighted CT development through robotics activities with students aged 3–6 (García-Valcárcel-Muñoz-Repiso and Caballero-González 2019), 8–10 (Chiazzese et al. 2019; Noh and Lee 2020) or in secondary school (Valls Pou et al. 2022). Ching et al.'s review (2018) considered that there is strong evidence for positive outcomes of developing CT through programming in young learners using educational robots. Finally, Zhang et al.'s meta-analysis (2021) on the impact of educational robotics on CT among K-12 students concluded that robots have a moderate positive effect on CT skills. Generally speaking, the body of literature in regards with the use of robotic devices to increase programming and CT learning seem really enthusiastic but several authors highlighted a lack of empirical data, which does not allow for definitive conclusion. Furthermore, the benefits of using Scratch alone, especially the accessible interface which could reduce the cognitive load, may disappear when associated with a robot, as managing both a programming software and a material robot might be overwhelming for young and novice students. All the studies mentioned above shed light on the potential benefits of unplugged activities, block-based programming languages and educational robots, but which of this approach or tool is to be recommended for novice students who do not yet master basic programming and CT skills?

## 2.4 | Comparative Studies

To date, very few studies have compared experimentally and directly these three main approaches to teaching programming and CT. Wohl et al. (2015) used three different modalities, including one based on unplugged activities and another based on the use of Scratch, with 28 Students aged 5–7. The results showed a better understanding of algorithm, logical prediction and debugging in the unplugged group. Hermans and Aivaloglou's study (2017) involved 35 students aged 8–12. The first group of students followed four plugged-in programming sessions, while the second followed four unplugged sessions.

Then, all students followed four plugged-in sessions, using Scratch software. No significant difference was found in the understanding of programming concepts. However, self-belief was higher when students started with unplugged activities. In del Olmo-Muñoz et al.'s study (2020), 84 grade 2 students were initially separated into two groups (plugged-in and unplugged) and then reunited later with plugged-in activities. The experimental group (which started with unplugged activities) showed the best progression in CT development and, while students' motivation decreased over time, this decline was lower in this group. Recently, Fanchamps et al. (2024) showed that unplugged games could be as effective as plugged-in activities to develop CT skills, with 35 grade 5–6 students. Namli and Aybek (2022) also reported no difference between visual block-based programming software and unplugged activities, in terms of self-belief. Their study involved 82 grade 5 students, and only the group using the software improved significantly. Therefore, studies focused on students engaged in unplugged activities first, then plugged-in activities, tend to show benefits of this practice, but studies comparing each approach alone do not reveal contradictory results. However, as we discussed earlier, unplugged and plugged-in programming have very different characteristics, and the results from a few studies should not lead us to stop exploring these differences.

Regarding the comparison between presence and absence of educational robots, Merkouris and Chorianopoulos (2015) compared one group performing only computer-based activities and another that includes robotics activities, with 36 students aged 12–13. The use of robots led to more positive emotions and behaviour (excitement, satisfaction, engagement, interest, etc.), but no difference on programming performance was observed. Similarly, Kert et al. (2020) compared an experimental group that followed programming activities via educational robots with a control group that used block-based visual programming software, for 10 weeks, with 78 grade 6 students (aged 11–12). They noted that robots seem responsible for a greater increase in students' learning performance and self-belief. And students were better able to make connections between the concepts covered in this group. Bellegarde et al. (2019) conducted a comparative study to introduce programming to more than 200 kindergarten students, aged 5–6, using different teaching tools. Three learning modalities were used during these experiments: a tablet, a robot and unplugged activities. Using a pretest-posttest comparison, the researchers highlighted that the use of two of the three modalities seems to be the most interesting, in particular the tablet+ robot combination, for which the progression is the highest. Interestingly, when these two media were used individually, students' performance were significantly lower. Then, current research seems to encourage the association between robotic devices and computers or tablets.

As comparative experiments are still too rare, because they have involved very different students in different contexts, new comparative studies are absolutely necessary. Webb et al. (2017) recommended further research to identify the strengths of these approaches, and del Olmo-Muñoz et al. (2020) suggested that future studies feature larger sample sizes and longer intervention durations. Zhang and Nouri (2019) noted that experimental results are too rarely replicated, which weakens research validity. They call for interventions to fill a gap, namely to compare
different ways of learning the same skill. Up to now, most stud-
ies measure increases in general CT with transfer tasks, often
not directly related to programming and to what has been
learned specifically during the teaching sequence. Brackmann
et al. (2017) suggested that unplugged programming learning
may have certain limitations and that computers may be needed
to further develop certain skills. Indeed, Ching et al. (2018) con-
sidered that the feedback enabled by programming software
could facilitate students' ability to test different hypotheses and
refine their ideas. Moreover, they argued that robotic tools in
particular are more apt to enable complex programming activ-
ities and the mastery of computational concepts, although ro-
bots would require a greater initial investment for learners and
teachers. For Ching et al. (2018), unplugged activities involving
the manipulation of concrete objects are certainly easy to imple-
ment, but also much more limited.

When introducing novice students to programming, it is cru-
cial to find a good balance between ease of implementation,
a manageable cognitive load and a high potential for learning
complex knowledge and skills, while ensuring that students are
motivated and engaged in the activities. The current gaps in the
literature do not allow conclusions to be drawn on which ap-
proach should be preferred. Given that unplugged programming
is supposed to increase learning by reducing cognitive load, and
that the same argument is used by several authors to support
block-based programming, it seems important to consider cogni-
tive load and how it has been theorised in education.

## 2.5 | Cognitive Load Theory

Cognitive load theory 'aims to explain how the information
processing load induced by learning tasks can affect students'
ability to process new information and construct knowledge
in long-term memory' (Sweller et al. 2019, 1-2). It is based on
the premise that cognitive processing is strongly constrained
by the limits of our working memory, which can only process
a small number of items simultaneously. When the cognitive
load is too high, it impedes learning and transfer to long-term
memory. Teaching methods and materials therefore need to be
adapted to avoid distractions that unnecessarily overload cogni-
tive processing. In practice, cognitive load theory enables us to
eliminate irrelevant information and focus on what is relevant,
to optimise learning situations. Sweller et al. (1998) proposed
to distinguish three categories of cognitive load for effective in-
structional design.

Intrinsic cognitive load refers to the complexity of the informa-
tion processed and to the interactions between these different
elements. This complexity depends on the learner's prior knowl-
edge. The more a learning situation involves a high degree of
interaction between several elements, the greater the risk of
cognitive overload. There are therefore only two ways to reduce
intrinsic cognitive load: by lowering complexity or by improving
learner's expertise. Extraneous cognitive load is determined by
the way in which information is presented and corresponds to all
the information present in the learning situation but unneces-
sary to learning. Extraneous load can be reduced with effective
pedagogical design by removing unnecessary information and
minimising interactions between the elements to be learned.

Finally, germane cognitive load refers to the working memory
resources devoted to managing intrinsic load rather than ex-
traneous load. The exact characterisation of germane load is
still a matter of debate (Sweller et al. 2019), but it is generally
considered to be the one allowing the redistribution of working
memory resources from activities extrinsic to the learning task
to activities intrinsic to that task.

There are some theoretical arguments concerning cognitive
load's impact on programming and CT learning. For example,
Romero et al. (2018), assume that unplugged activities, because
they do not involve any digital tool, would logically eliminate the
extraneous cognitive load linked to the manipulation of these
tools. However, block-based visual programming languages
could also reduce this extraneous load for children, compared
to classic text-based languages. Instructions are in a natural
language and keyboarding is not necessary to drag-and-drop
instructions. Educational robots are supposed to be motivating,
but the simultaneous management of a programming software
and a robot could imply a high extraneous load. Nevertheless,
robots' ability to concretise the abstract elements of program-
ming could potentially reduce learning complexity. Therefore,
the cognitive load associated with the different tools and activi-
ties in programming remains an open question.

Cognitive load is an abstract concept that has very concrete im-
plications for the way students learn. Assessing it in a school
context may be challenging. The first attempt to directly mea-
sure cognitive load was with a single-item scale (Paas 1992),
particularly quick and easy to use. This scale is still one of the
most used measures of cognitive load today, both for its reliabil-
ity and its practical usefulness. Participants are asked to indicate
their own estimate of the mental effort invested in a task, on
a symmetrical scale ranging from 1 to 9. Leppink et al. (2013)
also designed a widely used scale based on subjective measures
that distinguishes between the three types of cognitive load: the
Cognitive Load Scale (CLS). Subsequent studies have confirmed
the validity of this tool in different contexts (Andersen and
Makransky 2021; Hadie and Yusoff 2016). Klepsch et al. (2017)
have also conducted two studies to validate their scale, which
is distinct from CLS, but focuses on the same goal. Other em-
pirical measures can be used to assess cognitive load: physio-
logical measures, task performance, task completion time or
completion of a primary and secondary task (Paas et al. 2003).
These measures are considered more objective, but also more
time-consuming and complicated to set up. Chen et al. (2011)
compared four distinct methods for assessing cognitive load in a
mental arithmetic task and concluded that subjective measures
appear to be the most effective. Previous studies already showed
that there was considerable evidence that subjective measures
are reliable, non-intrusive and more sensitive than physiolog-
ical measures (Ayres 2006; Paas et al. 1994). Mutlu-Bayraktar
et al. (2019) also noted that subjective measures of cognitive load
in multimedia learning are the most widely used.

## 2.6 | Our Studies

To overcome the research limitations regarding the relative dif-
ferences between unplugged and plugged-in activities, with or
without a robot, for introducing young students to programming
and CT, we conducted two experiments in which the concepts and skills to be developed were strictly similar between our experimental groups.

Experiment 1 compares three groups of grade 5 students learning programming and CT using either unplugged activities, only Scratch software or a Thymio robot connected to Scratch. It measures differences related to students' learning performance (mastery of computational concepts and ability to solve algorithmic problems) and subjective experience (motivation and self-belief). In view of the particularly incomplete and contradictory literature on the relative effectiveness of these three approaches, we were not in a position to make any experimental predictions.

Experiment 2 compares two groups of grade 5 students learning programming and CT using either only Scratch software or a Thymio robot connected to Scratch. It measures differences related to students' cognitive load in order to explain results observed in Experiment 1. We assumed that the cognitive load would be higher in the Scratch + Thymio group.

# 3 | Experiment 1

## 3.1 | Method

### 3.1.1 | Participants

We recruited 306 grade 5 students (155 girls) in southern France. Classes were randomly divided into three groups: six classes in the unplugged group (N=133 students), four classes in the Scratch group (N=84) and four classes in the Scratch + Thymio group (N=89). The three groups were gender balanced. We ensured that all students were novices in programming by giving them a short questionnaire before the beginning of the study, assessing their knowledge of key terms in programming (algorithm, instruction, loop, etc.). No student was able to (even roughly) define or explain these key terms. We also checked that no programming lessons were given in the lower grades in each participant school. Each teacher was randomly assigned to one group. Therefore, this could be considered a cluster randomised study, as the randomisation was made at the class/teacher level and not at the student level. Indeed, randomisation at the student level is very complicated to set up in an authentic context and does not reflect the usual setting in schools. Fourteen teachers with several years of professional experience took part. They were given 10 programming sessions to implement in their classroom. We held meetings with teachers prior to the experiment to ensure that they understood and approved the importance of following as closely as possible the flow of the session to enable rigorous comparison between the different experimental groups. The teachers voluntarily chose to participate in response to a recruitment request we made.

### 3.1.2 | Material

In the unplugged group, paper instructions were used. These instructions reproduced Scratch blocks: the text and colours of the blocks were the same. During the learning sessions, students were invited to assemble these 'paper blocks' in the same way as

blocks are assembled on Scratch, to build their algorithms. In the Scratch group, students used the educational programming software Scratch 3.0 and learned to manipulate block-based instructions on computers. In the Scratch + Thymio group, students used a Thymio II wireless robot in addition to the Scratch software. Thymio is a small educational robot that can be programmed using various programming languages, including Scratch, supposed to be attractive to young children and teenagers, regardless of gender (Riedo et al. 2013).

We ensured every class had at least one computer for every three or four students, and provided the robots in the Scratch + Thymio group (six robots per class). The 10 programming sessions conducted in each experimental group were designed with the help of an educational advisor, specialised in digital technologies and programming courses in elementary school. The concepts, skills and activities developed during this session were identical in all experimental groups. The only major difference between the two groups lay in the use (or absence of use) of Scratch and Thymio robot.

### 3.1.3 | Instruments

To measure students' knowledge and skills, we designed two tests. Based on a literature review we made (Sigayret, Blanc, et al. 2022; Sigayret, Tricot, et al. 2022), we identified several limitations to readily available tools regarding our particular context. Indeed, there are no instruments validated in the French language and in French classrooms. Moreover, these instruments do not distinguish between knowledge and understanding of computational concepts and the ability to use these concepts to solve an algorithmic problem, a distinction which is fundamental, in particular with regard to French curricula related to programming. Finally, most of the reliable and robustly validated instruments to assess CT development, such as Bebras tasks, use items involving the transfer of knowledge by asking students to solve problems unrelated to programming. Because our objective is to assess programming concepts and skills in a programming context, assessing transfer skills slightly differs from our objective.

The Mastery of computational concepts Test assesses student's ability to know and understand five key computational concepts that we identified based on the works of Brennan and Resnick (2012) and Tchounikine (2017): algorithm, instruction, loop, conditional instruction and variable. We relied on the Computational Thinking Test (CTt, Román-González 2015) to evaluate loops and conditional instructions. While the CTt is a dependable tool, its items solely focus on practical exercises where students must mentally execute programs and select correct solutions, limiting its scope. We applied Bloom's revised taxonomy (Krathwohl 2002) to diversify our assessment items, compared to the CTt. Our Mastery of Computational Concepts Test comprises 17 multiple-choice items, structured as follows: four CTt items translated into French; five items prompting students to identify a specific computational concept from a set of propositions; four items in which students have to break down a program into its constituent elements (algorithm, loop, condition, variable); four items in which they have to complete sentences with the names of the five concepts, which sometimes
<figure><img src="image_5.png" /><figcaption>**FIGURE 1** | For this item, students had to pick which programs contain a loop instruction.</figcaption></figure>

requires them to make links between these concepts. Figure 1 shows an example of the five items prompting students to identify a specific computational concept.

We also developed the Algorithmic Test to assess students' ability to solve algorithmic problems, targeting higher levels of programming skills, according to Bloom's revised taxonomy (Krathwohl 2002), especially the 'Create' level. The test comprises four exercises in which students must write an algorithm as a solution to a given problem. The first two exercises are the most concrete, requiring students to program a character's movements on a grid, while the last two are the most abstract, focusing on programming a character to count indefinitely or to count time like a stopwatch. In each exercise, we assessed the ability to produce an algorithm with relevant instructions and to use loops, conditional instructions or variables when needed. While students are explicitly instructed to use loops, conditional instructions or variables in the first two exercises, they have to identify these solutions themselves in the last two more challenging exercises. Figure 2 shows the path to follow in the first exercise.

We validated both the Mastery of computational concepts Test and the Algorithmic Test using an independent sample of 621 grade 4 to 9 students (aged 9–15). We found high reliability indicators ($\omega$ = 0.85 for both tests) and a positive correlation between students' level of programming practice and their performance on our tests ($r$ = 0.61 for the Mastery of Computational Concepts Test; $r$ = 0.67 for the Algorithmic Test), demonstrating concurrent validity (Sigayret 2023). It should be noted that the Mastery of computational concepts Test originally comprised 22 items,

<figure><img src="image_6.png" /><figcaption>**FIGURE 2** | For this exercise, students had to program a character named 'BOT' to go from one point to another, which involved using loops and incrementing a variable when 'BOT' reached the finish.</figcaption></figure>

five of which were eventually deleted as statistical analysis revealed them to be unnecessary. Both tests are available, translated into English, in Appendices S1 and S2.

To measure students' motivation and self-belief, we used the SAL scale (Students' Approaches to Learning, Marsh et al. 2006), which we adapted to our context. We only retained items related to motivation and self-belief. The SAL scale is generally used to assess self-belief in mathematics and reading. We adapted these items to self-belief in programming. We used a pretest-posttest comparison. In the pretest, some questions were reworded to address students who had never taken part in programming activities, to assess motivation and self-belief a priori, before the learning sessions. The pretest contained 18 items, while the posttest contained 20. Item 19 and 20 were excluded from the pretest because novice students were unable to answer them,
without having participated in programming activities. The questions were grouped into three distinct categories: general self-belief (nine items), self-belief related to programming activities (six items) and motivation (three items in the pretest, five items in the posttest). Questions were answered on a four-point scale. For questions 1–6, students used a frequency scale ('almost never', 'sometimes', 'often', 'almost always'), for the other items, students used an agreement scale ('agree', 'somewhat agree', 'somewhat disagree', 'disagree'). Our motivation and self-belief test is available, translated into English, in Appendix S3.

### 3.1.4 | Procedure

The pretests and posttests were administered during the first and last sessions. Indeed, it was not possible to measure mastery of computational concepts and ability to solve algorithmic problems before the intervention because our participants were all novices in programming and clearly unable to simply understand the questions asked. Between these two, eight sessions were devoted to programming. Students were introduced to the fundamental computational concepts needed to practise programming and learned to build algorithms to program movements and interactions with objects during the first four learning sessions. Students were then asked to conduct a creative project in order to use the concepts previously learned during the last four teaching sessions. A description of all sessions is available in Appendix S4. Figure 3 summarises the experimental design.

In each experimental group (Unplugged, Scratch, Scratch + Thymio), students worked in small groups (three or four students per group). At the end of sessions 2, 3 and 5; 5 min were devoted to writing up a summary of the main elements seen in class. This written trace was exactly the same in all three groups, attesting to a common conceptual learning process between the three groups.

## 3.2 | Results

Mixed models are today considered a standard in data analysis, particularly when these involve repeated measures or clustered data, even though this type of analysis has only recently

begun to be widely used in psychology (Muradoglu et al. 2023). The term 'hierarchical model' is also frequently used to refer to mixed models. The notion of hierarchy refers to nested datasets, particularly conducive to mixed models (e.g., in education, the 'student' variable is nested within a 'class' variable). Since we have continuous data, analyses based on linear mixed models with two fixed effects (experimental group and gender) and a random effect (class) were conducted. For repeated measures (Motivation and self-belief), we included the pretest in our model as a covariate, whose influence impacts our dependent variable: the posttest. For each of our dependent variables (mastery of computational concepts, ability to solve algorithmic problems, motivation and self-belief), we measured the impact of experimental group, gender and the interaction between experimental group and gender. For the sake of brevity, we will only detail significant differences observed in this experiment.

We removed outliers using interquartile range method and standardised our data. Therefore, the calculation of the standardised coefficient $\beta$ is a reliable estimate of the effect size (Sommet and Morselli 2021). According to Cohen (1992), an effect size is large when greater than 0.50, medium between 0.30 and 0.50 and small between 0.10 and 0.30. We also generated Q-Q plots and followed Judd et al. (2018) recommendations to check the normality of our distributions. For each dependent variable, we checked the relevance of our model by using the Likelihood Ratio Test for random effects. This compares our model with a null model, for which there is no fixed effect, only the random effect of the class. We calculated the Akaike Information Criterion to ensure our model is better suited than the null model. Finally, we calculated the Intraclass Correlation Coefficient (ICC) associated with the random effect of our 'class' variable. The analyses were performed on Jamovi software, version 2.3.26.0, using the 'General Analyses for Linear Models in jamovi' module (Gamlj).

### 3.2.1 | Mastery of Computational Concepts

Table 1 shows the Mastery of Computational Concepts Test scores (means and standard deviations), in the experimental groups, by gender: the total score (/26 points) and the sub-scores concerning the mastery of the concept of algorithm (/6), instruction (/4), loop (/7), conditional instruction (/6) and variable (/3).

<figure><img src="image_7.png" /><figcaption>**FIGURE 3** | Design of the sessions.</figcaption></figure>
The ICC was 0.19, meaning that 19% of the variance observed in the total score is explained by inter-subject variability (here, variability between our different classes) and 81% by intra-subject variability (here, variability within the same class, between students). Results showed a main effect of experimental group (F(2, 275)=3.72, p=0.05) and gender (F(1, 276)=6.29, p=0.01). Girls performed significantly better than boys but this effect is small (β=-0.27, p=0.01, 95% CI [-0.47; -0.06]). Students in the Scratch group seemed to master computational concepts better than the other two groups (β=-0.55, p<0.05, 95% CI [-1.00; -0.11]). For the concept of loop, there was no significant effect of the group (F(2, 275)=2.50, p=0.12). However, the simple quadratic effect was significant and of medium size (β=-0.46, p=0.05, 95% CI [-0.88; -0.04]), indicating that the Scratch group was better than the other two concerning this concept. Here again, girls seemed to outperform boys, but this effect was small (F(1, 276)=6.40, p=0.01, β=-0.27, 95% CI [-0.49; -0.06]). For the concept of conditional instruction, the group effect was significant (F(2, 275)=8.16, p=0.01) and of medium size (β=-0.41, p=0.01, 95% CI [-0.65; -0.16]). Once again, the Scratch group performed better than the other two. Finally, for the concept of variable, the group effect was significant (F(2, 275)=7.82, p<0.01), still in favour of the Scratch group and of large size (β=-0.66, p<0.01, 95% CI [-1.01; -0.30]).

### 3.2.2 | Algorithmic Problem Solving

Table 2 shows the Algorithmic Test scores (means and standard deviations), in the experimental groups, by gender: the total score (/26 points) and the sub-scores concerning the ability to produce an algorithm made up of relevant instructions (/10), to use loops (/6), to use conditional instructions (/4) and to use

variables (/6). Table 2 also shows the scores for the first two exercises (/11) and the last two exercises (/15).

25% of the variance was explained by variability between classes, and 75% by variability within the same class, between students (ICC=0.25). Students in the Scratch group showed a significantly higher ability to solve algorithmic problems than the other two groups, with a large size for the total score (β=-0.75, p<0.01, 95% CI [-1.21; -0.29]), the algorithm score (β=-0.58, p<0.05, 95% CI [-1.07; -0.10]) and the loop score (β=-0.60, p<0.05, 95% CI [-1.05; -0.15]), the conditional instructions score (β=-0.54, p<0.01, 95% CI [-1.87; -0.21]) and the variable score (β=-0.81, p<0.01, 95% CI [-1.23; -0.39]). It is also interesting to observe that there was no superiority of the Scratch group concerning the first two exercises, which were the simplest (β=-0.27, p=0.30, 95% CI [-0.77; 0.22]), whereas the superiority of this group over the other two was particularly marked concerning the last two exercises, which were the most complex (β=-0.83, p<0.01, 95% CI [-1.29; -0.37]).

### 3.2.3 | Motivation and Self-Belief

Table 3 shows the pretest scores for students' motivation and self-belief in the experimental groups, by gender: total score (/54) and sub-scores for general self-belief (/27), self-belief related to programming activities (/18) and motivation (/9).

Table 4 presents the same statistics as Table 3, this time relating to the posttest scores for student's motivation and self-belief. Statistics for items 19 and 20 (/6), only present in the posttest, are not included in the total score and in the Motivation sub-total. In this way, the total score and the motivation sub-score were

<table><caption>TABLE 1 | Means (and standard deviations) of Mastery of Computational Concepts Test scores in the three experimental groups, by gender.</caption><thead><tr><th>Group</th><th>Genre</th><th>Total</th><th>Algorithm</th><th>Instruction</th><th>Loop</th><th>Condition</th><th>Variable</th></tr></thead><tbody><tr><td rowspan="2">Unplugged</td><td>Female</td><td>14.2 (4.29)</td><td>4.08 (1.31)</td><td>2.41 (0.90)</td><td>3.20 (1.90)</td><td>3.22 (1.39)</td><td>1.31 (0.91)</td></tr><tr><td>Male</td><td>14.0 (5.66)</td><td>3.86 (1.55)</td><td>2.64 (1.07)</td><td>3.13 (2.17)</td><td>3.04 (1.58)</td><td>1.30 (1.00)</td></tr><tr><td rowspan="2">Scratch</td><td>Female</td><td>18.5 (4.11)</td><td>4.20 (1.27)</td><td>3.24 (0.73)</td><td>4.73 (1.82)</td><td>4.15 (1.26)</td><td>2.15 (0.76)</td></tr><tr><td>Male</td><td>17.2 (5.16)</td><td>4.03 (1.40)</td><td>2.95 (1.12)</td><td>4.04 (1.89)</td><td>3.99 (1.38)</td><td>2.18 (0.97)</td></tr><tr><td rowspan="2">Scratch + Thymio</td><td>Female</td><td>16.3 (5.02)</td><td>4.23 (1.56)</td><td>2.87 (0.92)</td><td>3.79 (1.89)</td><td>3.72 (1.54)</td><td>1.69 (0.89)</td></tr><tr><td>Male</td><td>14.4 (4.66)</td><td>3.79 (1.59)</td><td>2.67 (0.66)</td><td>3.13 (1.69)</td><td>3.51 (1.41)</td><td>1.33 (1.01)</td></tr></tbody></table>

<table><caption>TABLE 2 | Means (and standard deviations) of Algorithmic Test scores in the three experimental groups, by gender.</caption><thead><tr><th>Group</th><th>Gender</th><th>Total</th><th>Algorithm</th><th>Loop</th><th>Condition</th><th>Variable</th><th>Ex 1+2</th><th>Ex 3+4</th></tr></thead><tbody><tr><td rowspan="2">Unplugged</td><td>Female</td><td>9.80 (5.02)</td><td>6.26 (2.26)</td><td>2.03 (1.79)</td><td>0.92 (1.09)</td><td>0.58 (1.03)</td><td>6.97 (2.26)</td><td>2.83 (2.26)</td></tr><tr><td>Male</td><td>9.16 (4.40)</td><td>5.93 (2.19)</td><td>1.69 (1.49)</td><td>1.09 (1.08)</td><td>0.44 (0.76)</td><td>6.63 (2.39)</td><td>2.53 (2.94)</td></tr><tr><td rowspan="2">Scratch</td><td>Female</td><td>14.0 (5.03)</td><td>7.65 (1.63)</td><td>2.77 (1.61)</td><td>1.52 (1.22)</td><td>2.06 (1.63)</td><td>7.20 (1.78)</td><td>6.81 (4.58)</td></tr><tr><td>Male</td><td>14.4 (5.70)</td><td>7.78 (1.93)</td><td>3.08 (1.67)</td><td>1.47 (1.31)</td><td>2.11 (1.56)</td><td>7.16 (2.25)</td><td>7.28 (4.73)</td></tr><tr><td rowspan="2">Scratch + Thymio</td><td>Female</td><td>8.81 (5.39)</td><td>5.93 (2.21)</td><td>1.49 (1.87)</td><td>0.54 (0.93)</td><td>0.85 (1.51)</td><td>5.32 (2.73)</td><td>3.49 (3.81)</td></tr><tr><td>Male</td><td>8.34 (4.91)</td><td>5.80 (2.38)</td><td>1.69 (1.50)</td><td>0.45 (0.92)</td><td>0.62 (1.27)</td><td>5.43 (2.56)</td><td>2.91 (3.22)</td></tr></tbody></table>
<table><caption>**TABLE 3** | Means (and standard deviations) of motivation and self-belief pretest scores in the three groups, by gender.</caption><thead><tr><th>Group</th><th>Gender</th><th>Total</th><th>General self-belief</th><th>Programming self-belief</th><th>Motivation</th></tr></thead><tbody><tr><td rowspan="2">Unplugged</td><td>Female</td><td>34.7 (8.95)</td><td>18.3 (4.36)</td><td>11.3 (3.54)</td><td>5.49 (2.35)</td></tr><tr><td>Male</td><td>35.4 (8.71)</td><td>17.9 (4.68)</td><td>12.1 (3.28)</td><td>5.54 (2.39)</td></tr><tr><td rowspan="2">Scratch</td><td>Female</td><td>36.3 (7.36)</td><td>18.9 (5.32)</td><td>12.0 (2.67)</td><td>5.34 (2.25)</td></tr><tr><td>Male</td><td>35.7 (8.49)</td><td>17.3 (4.69)</td><td>12.3 (3.43)</td><td>6.57 (2.13)</td></tr><tr><td rowspan="2">Scratch + Thymio</td><td>Female</td><td>33.8 (6.77)</td><td>16.5 (3.91)</td><td>11.6 (3.83)</td><td>5.73 (2.52)</td></tr><tr><td>Male</td><td>33.6 (7.84)</td><td>15.2 (5.10)</td><td>11.7 (3.44)</td><td>6.81 (2.04)</td></tr></tbody></table>

<table><caption>**TABLE 4** | Means (and standard deviations) of motivation and self-belief posttest scores in the three groups, by gender.</caption><thead><tr><th>Group</th><th>Genre</th><th>Total</th><th>General self-belief</th><th>Programming self-belief</th><th>Motivation</th><th>Motivation 19 + 20</th></tr></thead><tbody><tr><td rowspan="2">Unplugged</td><td>Female</td><td>35.9 (9.32)</td><td>19.4 (4.95)</td><td>12.1 (4.21)</td><td>4.69 (2.14)</td><td>2.66 (1.60)</td></tr><tr><td>Male</td><td>36.1 (7.58)</td><td>18.7 (4.27)</td><td>12.2 (3.31)</td><td>4.86 (4.72)</td><td>2.93 (1.71)</td></tr><tr><td rowspan="2">Scratch</td><td>Female</td><td>36.3 (7.91)</td><td>19.0 (6.23)</td><td>12.6 (2.93)</td><td>4.72 (2.31)</td><td>3.20 (1.52)</td></tr><tr><td>Male</td><td>37.7 (9.28)</td><td>17.9 (5.09)</td><td>13.5 (4.13)</td><td>6.81 (1.75)</td><td>4.37 (1.55)</td></tr><tr><td rowspan="2">Scratch + Thymio</td><td>Female</td><td>40.4 (10.4)</td><td>18.4 (4.25)</td><td>12.2 (3.92)</td><td>6.07 (3.13)</td><td>3.90 (2.13)</td></tr><tr><td>Male</td><td>43.7 (8.66)</td><td>19.5 (5.01)</td><td>13.5 (3.54)</td><td>6.96 (2.51)</td><td>4.27 (1.91)</td></tr></tbody></table>

comparable between the pretest and the posttest because they were based on the same number of items.

ICC = 0.2, which means that only 2% of the observed variance can be explained by differences between classes. In other words, our random 'class' effect on motivation and self-belief was almost null. The experimental group had a significant effect on motivation and self-belief, in favour of the Scratch + Thymio group (F(2, 234) = 16.53, p < 0.001). The greatest difference concerned the unplugged and Scratch + Thymio groups, with a large effect size (β = 0.75, p < 0.001, 95% CI [0.48; 1.03]). Regarding general self-belief, there was a significant effect of the experimental group (F(2, 222) = 6.56, p < 0.01), especially with the Scratch + Thymio group, which was the only one to show a moderate increase (β = 0.35, p < 0.01, 95% CI [0.11; 0.59]) compared with the pretest. Regarding motivation, there was a significant effect of experimental group (F(2, 233) = 4.10, p = 0.05) in favour of the Scratch + Thymio group (β = 0.54, p < 0.05, 95% CI [0.17; 0.91]). A small gender effect can also be observed in favour of boys, who appeared more motivated than girls (F(1, 234) = 9.18, p < 0.05, β = 0.28, 95% CI [0.06; 0.50]). Moreover, results showed a marginal interaction effect between our two variables (group and gender, F(1, 230) = 2.71, p = 0.07). A more precise analysis, focusing on pairwise comparisons, showed that gender differences were only observable in the Scratch group (β = 0.57, p < 0.05, 95% CI [0.08; 1.06]). Finally, the analysis concerning Items 19 and 20 revealed very similar results to what we observed concerning the other motivational items present in both the pretest and posttest: a small gender effect in favour of boys (F(1, 244) = 6.42, p = 0.01, β = 0.30, 95% CI [0.07; 0.53]) and a marginal interaction effect (F(2, 240) = 2.36, p = 0.10). As before, a more precise comparison showed that the gender effect

is only visible in the Scratch group (β = 0.55, p < 0.05, 95% CI [0.04; 1.06]).

## 4 | Experiment 2

The aim of this second experiment was to evaluate students' cognitive load in Scratch versus Scratch + Robot sessions, in order to explain Experiment 1 results. We assumed that the lower results in Scratch + Robot condition could be linked to an increase in cognitive load.

### 4.1 | Method

#### 4.1.1 | Participants

We recruited 116 grade 5 students (63 girls) in France. Each participating class was split into two groups. In each class, around half the students followed the Scratch session without robot (59 students), while the other half worked on Scratch associated with an educational robot (58 students). The two versions of the session (with or without robot) were not conducted at the same time. When one half of the class was doing the session, the other half was in another room, not engaged in programming activities. Students were randomly assigned to the two experimental groups. We therefore controlled for the 'class effect', as students from each class attended both versions of the session. The two groups were gender balanced. We ensured all students were novices in programming.

We also controlled for the 'teacher effect', since each teacher had to carry out both versions of the session in his or her
classroom and thus worked in the two experimental groups. Four teachers with several years of professional experience took part. They received the two programming sessions they had to implement in their classroom. A meeting was scheduled with them before the experiment to ensure that they understood and approved of the importance of following as closely as possible the flow of the session to enable rigorous comparison between the different experimental groups. The teachers voluntarily chose to participate in response to a recruitment request we made.

### 4.1.2 | Material

In both groups, students used the educational programming software Scratch 3.0 and learned to manipulate blocks (instructions) on computers. In the group using a robot, the Thymio II robot was used, as in our Experiment 1. We made sure every class had one computer for every two students and provided the robots in the Scratch + Robot group (one robot for every two students). The two programming sessions carried out in each experimental group were designed with the help of an educational advisor, specialised in digital technologies and programming courses in elementary school, and are largely based on some of the activities already implemented in our first experiment. The concepts, skills and activities developed during this session were identical in both experimental groups. The only major difference between the two groups lay in the use (or absence of use) of the Thymio II educational robot.

### 4.1.3 | Instruments

Cognitive load was measured twice, in the middle of the second programming session attended by the students, and at the end of the same session. For this purpose, we used two scales, based on subjective self-reported measures. First, the Klepsch et al. (2017) scale, which distinguishes between the three types of cognitive load (intrinsic, extraneous and germane), thus guaranteeing a precise and targeted assessment. Based on a small number of items, the Klepsch et al. (2017) scale seems suited to short interventions, like the one we implemented (two sessions, approximately 1 h and 45 min).

However, we had to adapt the Klepsch et al. (2017) scale to make its use relevant in our context. Indeed, the scale was initially validated with adults (whereas our participants are grade 5 students) in the German language before being translated into English. We therefore faced a double challenge of adapting this scale in French for a younger audience. To this goal, we conducted a translation/retranslation process by six PhD students with good English skills, to produce a translation that was as accurate as possible. We then benefited from the help of a Full Professor in Germanic Studies and Language Sciences, who was also a cognitive load specialist. With her advice and suggestions, we translated the Klepsch et al. (2017) scale from the source language (German) to refine our translation and adapted it to grade 5 students. We then tested the scale with 12 students at the end of elementary school to check that the vocabulary used was understandable for 9-10-years old children.

This instrument consists of seven Likert scales. Initially, each item offered seven possible answers. We reduced this total to just five, as Bouranta et al. (2009) noted that a 5-point Likert scale is less confusing and elicits more responses. Furthermore, Royeen (1985) suggested using Likert scales based on a reduced number of points for children, and 5-point and 7-point scales generally produce the same average score when one of the two is rescaled (Dawes 2008).

We also used the Paas (1992) scale exclusively in the second assessment, at the end of the second session. This scale measures the mental effort invested in the entire session. We felt it appropriate to use it because of its ease of use and reliability, which has been demonstrated in numerous studies measuring cognitive load. It also provides a measure of total cognitive load that is not an addition of the three types of cognitive load. For the Paas (1992) scale, we retained the 9-point Likert scale, as it is based on a single item. A reduction in the number of possible answers could have resulted in a loss of information that would have been far more detrimental on a single-item scale.

### 4.1.4 | Procedure

Assessments were administered twice during the second programming session: in the middle and at the end. Each assessment concludes one of the two main activities that made up this session. In session 1, students were introduced to some of the fundamental computational concepts needed to practise programming and learned how to build algorithms to program the movements of a virtual object on Scratch. So, students in both groups had a first introductory session to Scratch without a robot. In the second session, one of the two groups continued to work exclusively on Scratch, while the second group was introduced to the Thymio II robot. In each experimental group, students worked in pairs. A description of all sessions is available in Appendix S5.

## 4.2 | Results

We performed a mixed-model analysis to observe the influence of two fixed effects (experimental group and gender) and one random effect (class) on our dependent variables: total cognitive load composed of the three types of cognitive load (measured using the Klepsch et al. 2017 scale) and subjective mental effort (measured using the Paas 1992 scale). Cognitive load was assessed twice. As there was no significant difference between the two assessments, we calculated the mean for each student. The preliminary analyses are identical to those for the previous experiment: outliers' removal (interquartile range method), data standardisation, graphical analysis of data's normality. Our analyses were conducted on Jamovi software, version 2.3.26.0, using the Gamlj module. For the sake of brevity, we will only detail significant differences observed in this experiment.

Table 5 shows our measures of cognitive load for the two experimental groups by gender: total cognitive load (/28), intrinsic load (/8), germane load (/8), extraneous load (/12) and subjective mental effort (/9).
<table><caption>TABLE 5 | Means (and standard deviations) for cognitive load (Klepsch et al. 2017) and subjective mental effort (Paas 1992), by gender and group.</caption><thead><tr><th>Group</th><th>Gender</th><th>Total load</th><th>Intrinsic load</th><th>Germane load</th><th>Extraneous load</th><th>Mental effort</th></tr></thead><tbody><tr><td rowspan="2">Scratch + Thymio</td><td>Female</td><td>14.1 (4.05)</td><td>3.48 (1.79)</td><td>5.38 (1.38)</td><td>5.22 (2.38)</td><td>6.11 (1.59)</td></tr><tr><td>Male</td><td>12.7 (4.12)</td><td>3.43 (1.96)</td><td>5.34 (1.86)</td><td>3.96 (1.97)</td><td>5.17 (2.36)</td></tr><tr><td rowspan="2">Scratch</td><td>Female</td><td>12.9 (4.28)</td><td>3.39 (1.69)</td><td>5.56 (1.38)</td><td>3.94 (2.44)</td><td>4.43 (1.94)</td></tr><tr><td>Male</td><td>10.8 (5.28)</td><td>2.82 (1.93)</td><td>5.07 (1.96)</td><td>2.89 (2.40)</td><td>4.00 (2.43)</td></tr></tbody></table>

A significant effect was observed on total cognitive load ($F(1, 112) = 5.83, p < 0.05, \beta = 0.42, 95\% CI [0.08; 0.76]$), which was higher for students in the Scratch + Thymio group. Similarly, cognitive load was higher for girls than for boys ($F(1, 112) = 4.82, p < 0.05, \beta = 0.38, 95\% CI [0.04; 0.71]$). For extraneous cognitive load, our results highlighted an effect of experimental group ($F(1, 112) = 7.90, p < 0.01, \beta = 0.49, 95\% CI [0.15; 0.84]$) and gender ($F(1, 112) = 7.37, p < 0.01, \beta = 0.47, 95\% CI [0.13; 0.82]$). Extraneous cognitive load was higher in the Scratch + Thymio group and for girls. Finally, we observed a significant and large impact of the experimental group on mental effort ($F(1, 114) = 13.72, p < 0.001, \beta = 0.65, 95\% CI [0.30; 0.99]$), which was significantly higher in the Scratch + Thymio group.

## 5 | Discussion

In our two studies, we aimed to compare different ways to teach programming and CT to grade 5 students, using unplugged activities, Scratch software or a Thymio robot associated with Scratch. Results of the first experiment revealed that students using Scratch without a robot performed better regarding mastery of computational concepts and ability to solve algorithmic problems. Thus, we designed a second experiment, comparing students using Scratch with or without a robot and measuring cognitive load, to attempt explaining these results. Experiment 2 showed that extraneous cognitive load was higher for students using Scratch with a robot.

### 5.1 | Experiment 1

Our first experiment's main results revealed higher performance in the Scratch group, compared to the unplugged and Scratch + Thymio groups. Regarding mastery of computational concepts, this effect is quite large but is not observed on the two most basic concepts of algorithm and instruction. This group effect is medium for the concepts of loop and condition, and large for the concept of variable. Therefore, this sub-score analysis shows that students using only Scratch are better able to master the three concepts that are generally considered to be quite complex for novices to grasp, even with visual block programming languages (Zhang and Nouri 2019), especially the concept of variable (Hermans and Aivaloglou 2017; Tchounikine 2017). Likewise, students in the Scratch group are better able to solve algorithmic problems, compared with the two other groups, with a large effect on all the sub-scores. The largest effect size also concerns variables. The differences between the groups were only observed on the last two exercises, more complex and requiring greater mastery of variables. Thus, it seems that the

more complex the concepts and skills addressed, the more effective an approach using block-based programming software without robot is effective.

These results contradict previous studies, highlighting an absence of difference between unplugged and plugged-in activities (Hermans and Aivaloglou 2017; Romero et al. 2018), a superiority of unplugged programming (del Olmo-Muñoz et al. 2020; Wohl et al. 2015), no difference with or without robot (Merkouris and Chorianopoulos 2015), or a superiority of approaches using robots (Bellegarde et al. 2019; Kert et al. 2020). However, we must point out that most of these few studies use approaches combining unplugged and plugged-in activities and involved smaller sample sizes and younger students. However, our results are in line with Namli and Aybek (2022), who showed a superiority of plugged-in approaches with regard to learning performance, and with Huang and Looi (2021) who have been critical of unplugged activities.

Our results also show an increase in students' motivation and general self-belief when robots are used. Self-belief related to programming increased marginally in all conditions, but never significantly. A positive effect of digital tools is observed on students' motivation and engagement in programming, which had already been observed with Scratch software (Armoni et al. 2015; Ouahbi et al. 2015; Sáez-López et al. 2016) and with educational robots (Ching et al. 2018; Kaloti-Hallak et al. 2015; Kim and Lee 2016; Riedo et al. 2013), particularly a preference for approaches combining software and robots, compared with a software-only approach (Merkouris and Chorianopoulos 2015; Kert et al. 2020). Other authors considered unplugged activities to be conducive to a more positive attitude, higher motivation and greater self-efficacy but did not rely on empirical data to support their statement (Battal et al. 2021; Delal and Oner 2020; Threekunprapa and Yasri 2020). Still, we can question the existence of a potential 'novelty effect' (Cheung and Slavin 2013), following the introduction of new robots in the classroom, which did not necessarily disappear completely at the end of the 10 programming sessions, and which can explain the increase in motivation.

The superiority of the Scratch without robot group could be explained by a too high cognitive load when managing Scratch and a robot simultaneously (see the Section 5.2 for more discussion), and by the benefits of block-based programming languages over unplugged activities, particularly the possibility to visualise in real time the execution of a program students created. Our findings have practical implications and should be used by teachers as a warning to not introduce educational robots too early in the teaching sequence, as a plugged-in approach without robot to programming learning
may be more suited for young beginners, who do not yet master basic programming concepts and skills. However, robots could be useful later, to prevent a decrease in motivation, to expand students' discovery of the different contexts in which programming is necessary and, maybe, to enhance programming skills further.

Women are generally less attracted to programming (Gökçe and Yenmez 2022; Weintrop and Wilensky 2019), although this effect is smaller in elementary school (Román-González et al. 2017). In our experiment, gender effects on performance are small or uncertain. However, we highlighted a slightly better mastery of computational concepts for girls. In light of the contradictory literature on differences between girls and boys in programming (del Olmo-Muñoz et al. 2020; Li et al. 2021; Noh and Lee 2020; Relkin et al. 2020; Wang et al. 2022), showing low to medium effect sizes of this variable, it would be more cautious to state that no conclusions can be drawn from our results and that gender influence on performance is at most limited. We showed no significant interaction effect between experimental group and gender on performance, contrary to what has been observed before: positive impact of Scratch on girls' performance (Scherer et al. 2020) or on boys' performance (Jiang and Li 2021) and of educational robotics on boys' performance (Zhang et al. 2021).

However, in the Scratch group, boys are more motivated than girls while gender differences are not significant in the other two groups. We can infer that an exclusively virtual approach is unfavourable to girls' motivation. A study by Wu and Su (2021) revealed that girls outperformed boys on one learning variable (pattern recognition) and boys on another (algorithm design), only when learning to program without using a virtual programming environment. This confirms that the work environment is a factor that modulates students' learning experience differently according to gender. Cultural and social factors might be involved to explain these differences and we encourage future research to replicate our experiments and investigate these factors. Our results also confirm the absence of differences between boys and girls in the unplugged group (Delal and Oner 2020), often considered less conducive to gender stereotypes (Bell et al. 2009). Teachers should be aware that some approaches are gender-sensitive. While unplugged activities seem to reduce this gap between girls and boys, our results do not show that it is the most efficient approach. Therefore, researchers and educators should investigate ways to boost girls' motivation without compromising learning outcomes. Robotic activities also seem to increase girls' motivation, in line with what had been argued by Gunes and Kucuk (2022). Girls appear to be just as motivated as boys in the Scratch + Thymio group, as already shown by Merkouris and Chorianopoulos (2015). The positive motivational impact of Scratch on girls, hypothesised by Tikva and Tambouris (2021), is not found here. On the contrary, it is the boys who benefit.

## 5.2 | Experiment 2

Our second experiment's results show a higher overall cognitive load for students using Scratch and a Thymio robot,

compared with those only using Scratch, which confirms our predictions. More precisely, this increase concerns only extraneous load. It is quite logical, according to Sweller et al. (1998), as intrinsic load represents the complexity of the information to be learned. Here, the concepts, skills and tasks are very similar between our two groups, with and without robot. Extraneous load on the contrary corresponds to all the information unnecessary for learning. This result is therefore probably due to the need to master certain technical aspects, specific to the handling of the robot associated with a software, and to the complexity of managing both the software and the physical robot. Likewise, subjective mental effort invested in the learning task is significantly higher in the Scratch + robot group. Our findings suggest that the detrimental effect of educational robots on students' learning performance observed in Experiment 1 may be at least partly explained by the fact that the cognitive load associated with the use of a robot and a visual programming software is too high for grade 5 students, who are novices in programming. Educational robots' attractiveness relies mainly on their many functions (lights, sounds, movement, obstacle detection, etc.), as stated by Riedo et al. (2013). These undoubtedly contribute to their appeal and to students' interest in robots, but they can also be considered distracting, as the variety of functions does not guarantee a better mastery of programming and CT skills. Indeed, Garner (2002) considers programming to be a discipline with a very high intrinsic cognitive load, which implies that the extraneous load should be as low as possible to increase learning performance.

Furthermore, we also observed a significant effect of gender on overall cognitive load and particularly on extraneous load. Girls reported higher scores than boys. The result is not so surprising: Chen et al. (2021) showed that, in a robotic learning context with primary school students, boys reported a lower level of cognitive load than girls. In their study, boys performed slightly better than girls, but this difference was not significant. However, girls did not perform less well than boys in the two plugged-in groups in Experiment 1. This could be explained by other factors, not investigated in our research, enabling girls to raise their performance level to boys', despite a subjectively judged higher cognitive load. We also believe that a gradual narrowing of the gender gap in cognitive load assessment over the course of the sessions is the most credible hypothesis. Indeed, our Experiment 1 consisted of 10 sessions, compared with just two sessions in Experiment 2 (and only one involving robots).

## 5.3 | Limitations and Future Studies

In Experiment 2, the intervention was quite short (two sessions, as opposed to the 10 sessions in Experiment 1) but we really controlled teacher and class effect (Hattie 2003). In Experiment 1, for practical reasons, it was difficult for teachers to conduct 10 sessions in half classes so that several experimental conditions could be present within the same class. We also chose to control the skills and concepts taught to ensure that our experimental groups were comparable. However, certain pedagogical practices may be more suited to unplugged or plugged-in learning, with or without a robot. Our results
should not be generalised to all ways of teaching programming. Although our intervention was quite long in Experiment 1, results of a full year's training and learning in programming and CT could be different from those we observed during a shorter period of time. Many studies indicate a decrease in effect size when the duration of an intervention aimed at developing CT increases (Chen et al. 2023; Sun et al. 2022; Zhang et al. 2021). Lu et al. (2023) found that the ideal intervention is quite short, between 4h and 1 week. However, long-term studies are rare and the effects of teaching programming over a long period are still largely unknown (Chen et al. 2023). In addition, to ensure that performance assessment was strictly the same across our experimental groups, all students were required to compose on paper. Nevertheless, we can regret not allowing the students to produce algorithms in the same condition as they did throughout the sessions. This could have created a bias though, as it would have been difficult to consider these three different assessments as strictly equivalent.

For the purpose of our two experiments, we designed and validated two scales, to assess mastery of computational concepts and ability to solve algorithmic problems. We also used already existing scales (SAL scale, Klepsch et al.'s scale) that we had to adapt to French language and to our grade 5 students population. We carefully adapted these scales, especially by using a translation-retranslation process, by collaborating with a full professor specialised in cognitive load, education and fluent in German (which was the original language of the Klepsch et al.'s scale) and by pretesting this adaptation with 12 children. However, because the scales' adaptation did not strictly follow a psychometric validation process, this has to be mentioned as a limitation for these studies. On the other hand, our study could have benefited from qualitative data, collected in the form of observations or semi-structured interviews with students and teachers, which could have shed light on potential reasons to explain the differences we observed between our experimental groups. Our second experiment may explain why students in the scratch + robot group showed lower performance in our first experiment, compared to the scratch-only group. Qualitative data could have either confirmed this hypothesis or led us to explore new possible explanations. In view of that, mixed method studies are required to take an in-depth look at the determinants of programming learning and CT development. Even if it seems tedious, the community has to be encouraged to include qualitative data when using an experimental setup in the future.

Furthermore, it is important to remind the reader that our results were observed in a particular context. Future research should investigate if these results hold across different populations (from different socio-economic backgrounds, culture or countries) and should measure the impact of prior exposure to technology and classroom resources on programming and CT learning. In our two experiments, we did not really investigate why robots have a positive influence on students' motivation. Talan (2021) already suggested that using educational robots give students an active role, stimulating their interest, attention and motivation. But it would be interesting to explore which aspects of the interaction with the robot (e.g., tactile, visual) made learning more engaging in future research. Moreover, our results showed that learning performance and motivation were decorrelated. It would be interesting to check if long-term

studies show different results, as higher motivation could translate as better programming skills over time. Our programming sessions used step-by-step instructions but, if we knew at this time that robots would increase extraneous cognitive load, we could have made more effort in order to reduce this cognitive load in this particular condition, for example by setting up a scaffolding process. So, future research should investigate how the extraneous cognitive load associated with educational robots could be reduced.

Furthermore, the state of the art provides too few answers to the question of whether the introduction of educational robots might be more beneficial for older students or students with greater prior exposure to programming concepts and skills. Anwar et al. (2019) conducted a systematic review of the literature on educational robotics considering 147 studies published between 2000 and 2018. All of these protocols included students whose academic level ranges from primary school to the last years of high school (K-12 students). Generally speaking, the incorporation of educational robotics would present an advantage in increasing students' academic success, as well as certain professional skills (collaborative work, engagement, problem-solving, etc.). Pellas (2024) also showed that the use of robots can be beneficial to programming and CT skills and motivation, even for university students. Merkouris and Chorianopoulos (2015) and Kert et al. (2020) showed benefits of robots for students aged 11-13, and Bellegarde et al. (2019) for younger students aged 5-6; but, as we wrote earlier, comparative studies are lacking. Nevertheless, it would be crucial to identify a potential age threshold above which educational robotics' efficiency increases, with regards to programming and CT learning.

It would also be interesting to apply our comparative experimental design to combined approaches (unplugged then plugged-in, plugged-in without robot then with robot). Sun et al. (2021) conducted a meta-analysis and found an increase in size effects when these approaches were used, first unplugged, then plugged-in. Based on our results and on Sun et al.'s study (2021), we recommend using first unplugged activities, then plugged-in activities without robot and finally, introducing robots. Moreover, Chen et al. (2023) recommend gamifying unplugged activities to generate greater motivation and engagement. Our unplugged activities were rather playful, but it would still be worth testing whether other types of activity might produce greater or lesser effects. Although Scratch is still the most popular visual block-based language, there are other educational software programs that can develop similar skills (Alice, Kodu, App Inventor, Blockly, etc.). It would be interesting to measure the impact of different software on learning performance, even if Scratch seems more effective than Alice and App Inventor (Hu et al. 2021) or Blockly (Seraj et al. 2019) in fostering programming skills. In our studies, we used Thymio robots for practical and methodological reasons (since these robots can be programmed using Scratch, enabling a relevant comparison with Scratch without robot), and also because no study to date has shown the superiority of one robot over another in teaching programming and CT. However, other alternatives do exist (Robot Turtle, Lego Mindstorms, mbot, Blue Bot, etc.) and the contributions of these different robots deserve to be better documented. For example, the mbot robot can be assembled in different configurations, unlike Thymio, and can also be controlled remotely via block-based visual software to learn
programming (Pisarov and Mester 2019), while producing a high
level of satisfaction among students and encouraging them to con-
tinue programming activities (Cheng and Hsiao 2021). Previous
researches seem to discourage the use of text-based languages
for young or novice learners (Hu et al. 2021; Scherer et al. 2020;
Xu et al. 2019), but virtual robotic toolkits, which allow users to
design, program, and simulate robots in a virtual environment,
could also be a valuable option.

6 | Conclusion

Learning programming and CT skills is becoming a major
issue. Despite the lack of a consensual CT definition, its devel-
opment is undoubtedly the primary objective of programming
activities in school. We questioned the relative effectiveness of
various tools used to teach programming and CT, and made
comparisons based on quantitative empirical data between un-
plugged activities, block-based programming software and edu-
cational robots. Our studies took place in an authentic context,
namely in a typical classroom environment, in accordance with
French school curricula. The same concepts and skills were
taught across our experimental groups. Our findings highlight
a superiority in students' learning performance when using the
Scratch programming software, without robot. In fact, adding
a robot to Scratch could be detrimental to learning, at least for
primary school students who are new to programming and CT.
This could be explained by an increase in extraneous cognitive
load, as measured in our Experiment 2. However, robots are also
the most likely to maintain or arouse students' motivation. The
main gender differences show that motivation was higher for
boys only when students used Scratch without a robot. Further
studies are needed to explore the influence of approaches com-
bining several tools or other tools not used in our experiments.

Author Contributions

**Kevin Sigayret:** conceptualization, investigation, writing – original draft, methodology, writing – review and editing, formal analysis.

**Nathalie Blanc:** conceptualization, investigation, methodology, writing – review and editing, formal analysis, supervision. **André Tricot:** conceptualization, investigation, methodology, formal analysis, supervision, writing – review and editing.

Acknowledgements

We want to thank the DRANE (Délégation Régionale Académique au numérique éducatif) of the Montpellier Academy for their support and all the teachers and students who participated in our studies.

Conflicts of Interest

The authors declare no conflicts of interest.

Data Availability Statement

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

References

Aivaloglou, E., and F. Hermans. 2016. “How Kids Code and How We Know: An Exploratory Study on the Scratch Repository.” In *Proceedings*

of the 2016 ACM Conference on International Computing Education Research, 53–61. ACM. https://doi.org/10.1145/2960310.2960325.

Akiba, D. 2022. “Computational Thinking and Coding for Young Children: A Hybrid Approach to Link Unplugged and Plugged Activities.” *Education in Science* 12, no. 11: 793. https://doi.org/10.3390/educsci12110793.

Andersen, M. S., and G. Makransky. 2021. “The Validation and Further Development of a Multidimensional Cognitive Load Scale for Virtual Environments.” *Journal of Computer Assisted Learning* 37, no. 1: 183–196. https://doi.org/10.1111/jcal.1247.

Angeli, C., and N. Valanides. 2020. “Developing Young Children’s Computational Thinking With Educational Robotics: An Interaction Effect Between Gender and Scaffolding Strategy.” *Computers in Human Behavior* 105: 105954. https://doi.org/10.1016/j.chb.2019.03.018.

Anwar, S., N. A. Bascou, M. Menekse, and A. Kardgar. 2019. “A Systematic Review of Studies on Educational Robotics.” *Journal of Pre-College Engineering Education Research (J-PEER)* 9, no. 2: 2. https://doi.org/10.7771/2157-9288.1223.

Armoni, M., O. Meerbaum-Salant, and M. Ben-Ari. 2015. “From Scratch to “Real” Programming.” *ACM Transactions on Computing Education* 14, no. 4: 1–15. https://doi.org/10.1145/2677087.

Athanasiou, L., T. A. Mikropoulos, and D. Mavridis. 2019. “Robotics Interventions for Improving Educational Outcomes—A Meta-Analysis.” In *Technology, and Innovation in Learning, Teaching and Education: First International Conference, TECH-EDU 2018, Thessaloniki, Greece, June 20–22, 2018, Revised Selected Papers 1*, 91–102. Springer International Publishing. https://doi.org/10.1007/978-3-030-20,954-4_7.

Ayres, P. 2006. “Using Subjective Measures to Detect Variations of Intrinsic Cognitive Load Within Problems.” *Learning and Instruction* 16, no. 5: 389–400. https://doi.org/10.1016/j.learninstruc.2006.09.001.

Battal, A., G. Afacan Adanır, and Y. Gülbahar. 2021. “Computer Science Unplugged: A Systematic Literature Review.” *Journal of Educational Technology Systems* 50, no. 1: 24–47. https://doi.org/10.1177/00472395211018801.

Bell, T., J. Alexander, I. Freeman, and M. Grimley. 2009. “Computer Science Unplugged: School Students Doing Real Computing Without Computers.” *New Zealand Journal of Applied Computing and Information Technology* 13, no. 1: 20–29.

Bell, T., and J. Vahrenhold. 2018. “CS Unplugged—How Is It Used, and Does It Work?” In *Adventures Between Lower Bounds and Higher Altitudes: Essays Dedicated to Juraj Hromkovič on the Occasion of His 60th Birthday*, 497–521. Springer. https://doi.org/10.1007/978-3-319-98,355-4_29.

Bellegarde, K., J. Boyaval, and J. Alvarez. 2019. “S’initier à La Robotique/Informatique en Classe de Grande Section de Maternelle. Une Expérimentation Autour de L’utilisation Du Robot Blue Bot Comme Jeux Sérieux.” *Review of Science, Mathematics and ICT Education* 13, no. 1: 51–72. https://doi.org/10.26220/rev.3105.

Berssanette, J. H., and A. C. de Francisco. 2021. “Cognitive Load Theory in the Context of Teaching and Learning Computer Programming: A Systematic Literature Review.” *IEEE Transactions on Education* 65, no. 3: 440–449. https://doi.org/10.1109/TE.2021.3127215.

Bouranta, N., L. Chitiris, and J. Paravantis. 2009. “The Relationship Between Internal and External Service Quality.” *International Journal of Contemporary Hospitality Management* 21, no. 3: 275–293. https://doi.org/10.1108/09596110910948297.

Bourgeois, A., P. Birch, and O. Davydovskaia. 2019. “Digital Education at School in Europe.” *Eurydice Report. Education, Audiovisual and Culture Executive Agency, European Commission. Available From EU Bookshop.*

Brackmann, C. P., M. Román-González, G. Robles, J. Moreno-León, A. Casali, and D. Barone. 2017. “Development of Computational Thinking
Skills Through Unplugged Activities in Primary School." In *Proceedings of the 12th Workshop on Primary and Secondary Computing Education*, 65-72. ACM. https://doi.org/10.1145/3137065.3137069.

Brennan, K., and M. Resnick. 2012. "New Frameworks for Studying and Assessing the Development of Computational Thinking." In *Proceedings of the 2012 Annual Meeting of the American Educational Research Association, Vancouver, Canada*, vol. 1, 25. ACM. http://scrat.ched.gse.harvard.edu/ct/files/AERA2012.pdf.

Buitrago Flórez, F., R. Casallas, M. Hernández, A. Reyes, S. Restrepo, and G. Danies. 2017. "Changing a Generation's Way of Thinking: Teaching Computational Thinking Through Programming." *Review of Educational Research* 87, no. 4: 834-860. https://doi.org/10.3102/0034654317710096.

Caeli, E. N., and A. Yadav. 2020. "Unplugged Approaches to Computational Thinking: A Historical Perspective." *TechTrends* 64, no. 1: 29-36. https://doi.org/10.1007/s11528-019-00410-5.

Chen, B., G. H. Hwang, and S. H. Wang. 2021. "Gender Differences in Cognitive Load When Applying Game-Based Learning With Intelligent Robots." *Educational Technology & Society* 24, no. 3: 102-115.

Chen, P., D. Yang, A. H. S. Metwally, J. Lavonen, and X. Wang. 2023. "Fostering Computational Thinking Through Unplugged Activities: A Systematic Literature Review and Meta-Analysis." *International Journal of STEM Education* 10, no. 1: 1-25. https://doi.org/10.1186/s40594-023-00434-7.

Chen, S., J. Epps, and F. Chen. 2011. "A Comparison of Four Methods for Cognitive Load Measurement." In *Proceedings of the 23rd Australian Computer-Human Interaction Conference*, 76-79. ACM. https://doi.org/10.1145/2071536.2071547.

Cheng, Y. H., and J. M. Hsiao. 2021. "Exploring the Intention to Continuance of Learning Programming at Elementary School of Rural Area by the Mbot Robot." In *Proceedings of the International Conference on Artificial Life and Robotics, Online, Japan*, 21-24. ACM. https://alife-robotics.co.jp/members2021/icarob/data/html/data/OS/OS9/OS9-7.pdf.

Cheung, A. C., and R. E. Slavin. 2013. "The Effectiveness of Educational Technology Applications for Enhancing Mathematics Achievement in K-12 Classrooms: A Meta-Analysis." *Educational Research Review* 9: 88-113. https://doi.org/10.1016/j.edurev.2013.01.001.

Chiazzese, G., M. Arrigo, A. Chifari, V. Lonati, and C. Tosto. 2019. "Educational Robotics in Primary School: Measuring the Development of Computational Thinking Skills With the Bebras Tasks." *Informatics* 6, no. 4: 43. https://doi.org/10.3390/informatics6040043.

Ching, Y. H., Y. C. Hsu, and S. Baldwin. 2018. "Developing Computational Thinking With Educational Technologies for Young Learners." *TechTrends* 62: 563-573. https://doi.org/10.1007/s11528-018-0292-7.

Cohen, J. 1992. "Statistical Power Analysis." *Current Directions in Psychological Science* 1, no. 3: 98-101. https://doi.org/10.1111/1467-8721.ep10768783.

Conrads, J., M. Rasmussen, N. Winters, A. Geniets, and L. Langer. 2017. *Digital Education Policies in Europe and Beyond: Key Design Principles for More Effective Policies*. Publications Office of the European Union. https://doi.org/10.2760/462941.

Dawes, J. 2008. "Do Data Characteristics Change According to the Number of Scale Points Used? An Experiment Using 5-Point, 7-Point and 10-Point Scales." *International Journal of Market Research* 50, no. 1: 61-104. https://doi.org/10.1177/147078530805000106.

de Souza Oliveira, P. H., R. D. Medina, F. B. Nunes, and F. Herpich. 2022. "Analysis of Unplugged Tools Used for the Development of Computational Thinking in the Teaching of Algorithms: A Review of the Literature." *Renote* 20, no. 1: 243-252. https://doi.org/10.22456/1679-1916.126670.

del Olmo-Muñoz, J., R. Cózar-Gutiérrez, and J. A. González-Calero. 2020. "Computational Thinking Through Unplugged Activities in Early Years of Primary Education." *Computers & Education* 150: 103-832. https://doi.org/10.1016/j.compedu.2020.103832.

Delal, H., and D. Oner. 2020. "Developing Middle School Students' Computational Thinking Skills Using Unplugged Computing Activities." *Informatics in Education* 19, no. 1: 1-13. https://doi.org/10.15388/infedu.2020.01.

Di Lieto, M. C., E. Inguaggiato, E. Castro, et al. 2017. "Educational Robotics Intervention on Executive Functions in Preschool Children: A Pilot Study." *Computers in Human Behavior* 71: 16-23. https://doi.org/10.1016/j.chb.2017.01.018.

Dos Reis, W., H. Sereno, M. Do Amaral, and P. Dos Reis. 2015. "Educational Robotics as an Instrument of Formation: A Public Elementary School Case Study." In *VI Workshop de Robótica Educacional*, vol. 6, 70-75. ACM. http://www.natalnet.br/wre2015/wre2015.pdf#page=69.

Fanchamps, N., E. Van Gool, L. Slangen, and P. Hennissen. 2024. "The Effect on Computational Thinking and Identified Learning Aspects: Comparing Unplugged smartGames With SRA-Programming With Tangible or On-Screen Output." *Education and Information Technologies* 29, no. 3: 2999-3024. https://doi.org/10.1007/s10639-023-11,956-6.

García-Valcárcel-Muñoz-Repiso, A., and Y. A. Caballero-González. 2019. "Robotics to Develop Computational Thinking in Early Childhood Education." *Comunicar: Media Education Research Journal* 27, no. 1: 63-72. https://doi.org/10.3916/C59-2019-06.

Garner, S. 2002. *Reducing the Cognitive Load on Novice Programmers*, 578-583. Association for the Advancement of Computing in Education (AACE). https://www.learntechlib.org/primary/p/10329/.

Gökçe, S., and A. A. Yenmez. 2022. "Ingenuity of Scratch Programming on Reflective Thinking Towards Problem Solving and Computational Thinking." *Education and Information Technologies* 28: 5493-5517. https://doi.org/10.1007/s10639-022-11,385-x.

Gunes, H., and S. Kucuk. 2022. "A Systematic Review of Educational Robotics Studies for the Period 2010-2021." *Review of Education* 10, no. 3: e3381. https://doi.org/10.1002/rev3.3381.

Hadie, S. N., and M. S. Yusoff. 2016. "Assessing the Validity of the Cognitive Load Scale in a Problem-Based Learning Setting." *Journal of Taibah University Medical Sciences* 11, no. 3: 194-202. https://doi.org/10.1016/j.jtumed.2016.04.001.

Hattie, J. A. C. 2003. "Teachers Make a Difference: What Is the Research Evidence?" In *Building Teacher Quality: What Does the Research Tell Us ACER Research Conference, Melbourne, Australia*. ACM. https://ediforrm.santillana.com.mx/abc/docs/hattie.pdf.

Hermans, F., and E. Aivaloglou. 2017. "To Scratch or Not to Scratch? A Controlled Experiment Comparing Plugged First and Unplugged First Programming Lessons." In *Proceedings of the 12th Workshop on Primary and Secondary Computing Education*, 49-56. ACM. https://doi.org/10.1145/3137065.3137072.

Hu, Y., C. H. Chen, and C. Y. Su. 2021. "Exploring the Effectiveness and Moderators of Block-Based Visual Programming on Student Learning: A Meta-Analysis." *Journal of Educational Computing Research* 58, no. 8: 1467-1493. https://doi.org/10.1177/0735633120945935.

Huang, W., and C. K. Looi. 2021. "A Critical Review of Literature on "Unplugged" Pedagogies in K-12 Computer Science and Computational Thinking Education." *Computer Science Education* 31, no. 1: 83-111. https://doi.org/10.1080/08993408.2020.1789411.

Ioannou, A., and E. Makridou. 2018. "Exploring the Potentials of Educational Robotics in the Development of Computational Thinking: A Summary of Current Research and Practical Proposal for Future Work." *Education and Information Technologies* 23: 2531-2544. https://doi.org/10.1007/s10639-018-9729-z.
Jiang, B., and Z. Li. 2021. "Effect of Scratch on Computational Thinking Skills of Chinese Primary School Students." *Journal of Computers in Education* 8, no. 4: 505-525. https://doi.org/10.1007/s40692-021-00190-z.

Judd, C. M., G. H. McClelland, C. S. Ryan, D. Muller, and V. Yzerbyt. 2018. *Analyse Des Données: Une Approche Par Comparaison de Modèles. De Boeck Superieur*.

Julià, C., and J. Ò. Antoli. 2016. "Spatial Ability Learning Through Educational Robotics." *International Journal of Technology and Design Education* 26: 185-203. https://doi.org/10.1007/s10798-015-9307-2.

Kaloti-Hallak, F., M. Armoni, and M. Ben-Ari. 2015. "Students' Attitudes and Motivation During Robotics Activities." In *Proceedings of the Workshop in Primary and Secondary Computing Education*, 102-110. ACM. https://doi.org/10.1145/2818314.2818317.

Kert, S. B., M. F. ErkoÇ, and S. Yeni. 2020. "The Effect of Robotics on Six Graders' Academic Achievement, Computational Thinking Skills and Conceptual Knowledge Levels." *Thinking Skills and Creativity* 38: 100-714. https://doi.org/10.1016/j.tsc.2020.100714.

Kim, S., and C. Lee. 2016. "Effects of Robot for Teaching Geometry to Fourth Graders." *International Journal of Innovation in Science and Mathematics Education* 24, no. 2: 52-70.

Klepsch, M., F. Schmitz, and T. Seufert. 2017. "Development and Validation of Two Instruments Measuring Intrinsic, Extraneous, and Germane Cognitive Load." *Frontiers in Psychology* 8: 1997. https://doi.org/10.3389/fpsyg.2017.01997.

Krathwohl, D. R. 2002. "A Revision of Bloom's Taxonomy: An Overview." *Theory Into Practice* 41, no. 4: 212-218. https://doi.org/10.1207/s15430421tip4104_2.

Leppink, J., F. Paas, C. P. Van der Vleuten, T. Van Gog, and J. J. Merriënboer. 2013. "Development of an Instrument for Measuring Different Types of Cognitive Load." *Behavior Research Methods* 45: 1058-1072. https://doi.org/10.3758/s13428-013-0334-1.

Li, Y., S. Xu, and J. Liu. 2021. "Development and Validation of Computational Thinking Assessment of Chinese Elementary School Students." *Journal of Pacific Rim Psychology* 15: 18. https://doi.org/10.1177/18344909211010240.

Lu, Z., M. M. Chiu, Y. Cui, W. Mao, and H. Lei. 2023. "Effects of Game-Based Learning on Students' Computational Thinking: A Meta-Analysis." *Journal of Educational Computing Research* 61, no. 1: 235-256. https://doi.org/10.1177/07356331221100740.

Lye, S. Y., and J. H. L. Koh. 2014. "Review on Teaching and Learning of Computational Thinking Through Programming: What Is Next for K-12?" *Computers in Human Behavior* 41: 51-61. https://doi.org/10.1016/j.chb.2014.09.012.

Maloney, J., M. Resnick, N. Rusk, B. Silverman, and E. Eastmond. 2010. "The Scratch Programming Language and Environment." *ACM Transactions on Computing Education* 10, no. 4: 1-15. https://doi.org/10.1145/1868358.1868363.

Mari, E. 2022. "Potentialités de La Robotique Pédagogique Pour Le Développement de Connaissances Spatiales à L'école." Thèse de Doctorat, Aix Marseille Université (AMU).

Marsh, H. W., K. T. Hau, C. Artelt, J. Baumert, and J. L. Peschar. 2006. "OECD's Brief Self-Report Measure of Educational Psychology's Most Useful Affective Constructs: Cross-Cultural, Psychometric Comparisons Across 25 Countries." *International Journal of Testing* 6, no. 4: 311-360.

Merkouris, A., and K. Chorianopoulos. 2015. "Introducing Computer Programming to Children Through Robotic and Wearable Devices." In *Proceedings of the Workshop in Primary and Secondary Computing Education*, 69-72. ACM. https://doi.org/10.1145/2818314.2818342.

Muradoglu, M., J. R. Cimpian, and A. Cimpian. 2023. "Mixed-Effects Models for Cognitive Development Researchers." *Journal of Cognition and Development* 24: 1-340. https://doi.org/10.1080/15248372.2023.2176856.

Mutlu-Bayraktar, D., V. Cosgun, and T. Altan. 2019. "Cognitive Load in Multimedia Learning Environments: A Systematic Review." *Computers & Education* 141: 103-618. https://doi.org/10.1016/j.compedu.2019.103618.

Namli, N. A., and B. Aybek. 2022. "An Investigation of the Effect of Block-Based Programming and Unplugged Coding Activities on Fifth Graders' Computational Thinking Skills, Self-Efficacy and Academic Performance." *Contemporary Educational Technology* 14, no. 1: ep341. https://doi.org/10.30935/cedtech/11477.

Noh, J., and J. Lee. 2020. "Effects of Robotics Programming on the Computational Thinking and Creativity of Elementary School Students." *Educational Technology Research and Development* 68: 463-484. https://doi.org/10.1007/s11423-019-09708-w.

Olabe, J. C., M. A. Olabe, X. Basogain, and C. Castaño. 2011. "Programming and Robotics With Scratch in Primary Education." In *Education in a Technological World: Communicating Current and Emerging Research and Technological Efforts*, 356-363. ACM.

Ouahbi, I., F. Kaddari, H. Darhmaoui, A. Elachqar, and S. Lahmine. 2015. "Learning Basic Programming Concepts by Creating Games With Scratch Programming Environment." *Procedia-Social and Behavioral Sciences* 191: 1479-1482.

Paas, F., J. E. Tuovinen, H. Tabbers, and P. W. Van Gerven. 2003. "Cognitive Load Measurement as a Means to Advance Cognitive Load Theory." *Educational Psychologist* 38, no. 1: 63-71. https://doi.org/10.1207/S15326985EP3801_8.

Paas, F. G. 1992. "Training Strategies for Attaining Transfer of Problem-Solving Skill in Statistics: A Cognitive-Load Approach." *Journal of Educational Psychology* 84, no. 4: 429-434. https://doi.org/10.1037/0022-0663.84.4.429.

Paas, F. G., J. J. Van Merriënboer, and J. J. Adam. 1994. "Measurement of Cognitive Load in Instructional Research." *Perceptual and Motor Skills* 79, no. 1: 419-430. https://doi.org/10.2466/pms.1994.79.1.419.

Park, I., D. Kim, J. Oh, Y. Jang, and K. Lim. 2015. "Learning Effects of Pedagogical Robots With Programming in Elementary School Environments in Korea." *Indian Journal of Science and Technology* 8, no. 26: 1-5.

Pellas, N. 2024. "Assessing Computational Thinking, Motivation, and Grit of Undergraduate Students Using Educational Robots." *Journal of Educational Computing Research* 62, no. 2: 620-644. https://doi.org/10.1177/07356331231210946.

Piedade, J., and N. Dorotea. 2022. "Effects of Scratch-Based Activities on 4th-Grade Students' Computational Thinking Skills." *Informatics in Education: 499-523*. https://doi.org/10.15388/infedu.2023.19.

Pisarov, J., and G. Mester. 2019. "Programming the Mbot Robot in School." In *Proceedings of the International Conference and Workshop Mechatronics in Practice and Education*, 45-48. MechEdu.

Psycharis, S., and M. Kallia. 2017. "The Effects of Computer Programming on High School Students' Reasoning Skills and Mathematical Self-Efficacy and Problem Solving." *Instructional Science* 45, no. 5: 583-602. https://doi.org/10.1007/s11251-017-9421-5.

Relkin, E., L. de Ruiter, and M. U. Bers. 2020. "TechCheck: Development and Validation of an Unplugged Assessment of Computational Thinking in Early Childhood Education." *Journal of Science Education and Technology* 29, no. 4: 482-498. https://doi.org/10.1007/s10956-020-09831-x.

Riedo, F., M. Chevalier, S. Magnenat, and F. Mondada. 2013. "Thymio II, a Robot That Grows Wiser With Children." In *2013 IEEE Workshop on Advanced Robotics and Its Social Impacts*, 187-193. IEEE. https://doi.org/10.1109/ARSO.2013.6705527.
Román-González, M. 2015. "Computational Thinking Test: Design Guidelines and Content Validation." In *EDULEARN15 Proceedings*, 2436-2444. IEEE.

Román-González, M., J. C. Pérez-González, and C. Jiménez-Fernández. 2017. "Which Cognitive Abilities Underlie Computational Thinking? Criterion Validity of the Computational Thinking Test." *Computers in Human Behavior* 72: 678-691. https://doi.org/10.1016/j.chb.2016.08.047.

Romero, M., B. Lille, T. Viéville, M. Duflot-Kremer, C. de Smet, and D. Belhassein. 2018. "Analyse Comparative D'une Activité D'apprentissage de La Programmation en Mode Branché Et Débranché." In *Dans Educode-Conférence Internationale Sur L'enseignement Au Numérique Et Par Le Numérique*. IEEE. https://inria.hal.science/hal-01861732/file/Educode%20_UnPlugged-CDS06_soumis.pdf.

Royeen, C. B. 1985. "Adaptation of Likert Scaling for Use With Children." *Occupational Therapy Journal of Research* 5, no. 1: 59-69. https://doi.org/10.1177/153944928500500104.

Sáez-López, J. M., M. Román-González, and E. Vázquez-Cano. 2016. "Visual Programming Languages Integrated Across the Curriculum in Elementary School: A Two Year Case Study Using "Scratch" in Five Schools." *Computers & Education* 97: 129-141. https://doi.org/10.1016/j.compedu.2016.03.003.

Scherer, R., F. Siddiq, and B. S. Viveros. 2020. "A Meta-Analysis of Teaching and Learning Computer Programming: Effective Instructional Approaches and Conditions." *Computers in Human Behavior* 109: 106-349. https://doi.org/10.1016/j.chb.2020.106349.

Seraj, M., E. S. Katterfeldt, K. Bub, S. Autexier, and R. Drechsler. 2019. "Scratch and Google Blockly: How Girls' Programming Skills and Attitudes Are Influenced." In *Proceedings of the 19th Koli Calling International Conference on Computing Education Research*, 1-10. IEEE. https://doi.org/10.1145/3364510.3364515.

Sigayret, K. 2023. "L'impact Des Outils Utilisés Pour L'apprentissage de La Programmation Et Le Développement de La Pensée Informatique Au Cycle 3." Doctoral diss., Université Paul Valéry-Montpellier III.

Sigayret, K., N. Blanc, and A. Tricot. 2022. "L'apprentissage de La Programmation: Quels Outils Pour Évaluer Le Développement de La Pensée Informatique à L'école?" *Enfance* 4, no. 4: 479-500. https://doi.org/10.3917/enf2.224.0479.

Sigayret, K., A. Tricot, and N. Blanc. 2022. "Unplugged or Plugged-In Programming Learning: A Comparative Experimental Study." *Computers & Education* 184: 104-505. https://doi.org/10.1016/j.compedu.2022.104505.

Socratous, C., and A. Ioannou. 2019. "Using Educational Robotics as Tools for Metacognition: An Empirical Study in Elementary Stem Education." In *Immersive Learning Research Network Conference, UK*, 64-75. Springer. https://doi.org/10.3217/978-3-85-125-657-4-11.

Sommet, N., and D. Morselli. 2021. "Keep Calm and Learn Multilevel Linear Modeling: A Three-Step Procedure Using SPSS, Stata, R, and MPlus." *International Review of Social Psychology* 34, no. 1: 203-219. https://doi.org/10.5334/irsp.555.

Stewart, W., and K. Baek. 2023. "Analyzing Computational Thinking Studies in Scratch Programming: A Review of Elementary Education Literature." *International Journal of Computer Science Education in Schools* 6, no. 1: 35-58. https://doi.org/10.21585/ijcses.v6i1.156.

Suárez, C. A. H., A. A. G. Suárez, and R. P. Núñez. 2022. "Computational Thinking and Development of Cognitive Skills Mediated by Unplugged Activities." *Journal of Language and Linguistic Studies* 18, no. 2: 1110-1122.

Sun, L., Z. Guo, and D. Zhou. 2022. "Developing K-12 Students' Programming Ability: A Systematic Literature Review." *Education and Information Technologies* 27, no. 5: 7059-7097. https://doi.org/10.1007/s10639-022-10-891-2.

Sun, L., L. Hu, and D. Zhou. 2021. "Which Way of Design Programming Activities Is More Effective to Promote K-12 Students' Computational Thinking Skills? A Meta-Analysis." *Journal of Computer Assisted Learning* 37, no. 4: 1048-1062. https://doi.org/10.1111/jcal.12545.

Sweller, J., J. J. van Merriënboer, and F. Paas. 2019. "Cognitive Architecture and Instructional Design: 20 Years Later." *Educational Psychology Review* 31: 261-292. https://doi.org/10.1007/s10648-019-09465-5.

Sweller, J., J. J. Van Merriënboer, and F. G. Paas. 1998. "Cognitive Architecture and Instructional Design." *Educational Psychology Review* 10, no. 3: 251-296. https://www.jstor.org/stable/23359412.

Talan, T. 2021. "The Effect of Educational Robotic Applications on Academic Achievement: A Meta-Analysis Study." *International Journal of Technology in Education and Science* 5, no. 4: 512-526. https://doi.org/10.46328/ijtes.2024.

Tang, X., Y. Yin, Q. Lin, R. Hadad, and X. Zhai. 2020. "Assessing Computational Thinking: A Systematic Review of Empirical Studies." *Computers & Education* 148: 103798. https://doi.org/10.1016/j.compedu.2019.103798.

Tchounikine, P. 2017. *Initier Les Élèves À La Pensée Informatique Et À La Programmation Avec Scratch*. Laboratoire D'informatique de Grenoble. https://pdfbib.com/pdf/0690-initier-les-eleves-a-la-programmation-avec-scratch.pdf.

Threekunprapa, A., and P. Yasri. 2020. "Unplugged Coding Using Flowblocks for Promoting Computational Thinking and Programming Among Secondary School Students." *International Journal of Instruction* 13, no. 3: 207-222. https://doi.org/10.29333/iji.2020.13314a.

Tikva, C., and E. Tambouris. 2021. "Mapping Computational Thinking Through Programming in K-12 Education: A Conceptual Model Based on a Systematic Literature Review." *Computers & Education* 162: 104083. https://doi.org/10.1016/j.compedu.2020.104083.

Tsai, C. Y. 2019. "Improving Students' Understanding of Basic Programming Concepts Through Visual Programming Language: The Role of Self-Efficacy." *Computers in Human Behavior* 95: 224-232. https://doi.org/10.1016/j.chb.2018.11.038.

Valls Pou, A., X. Canaleta, and D. Fonseca. 2022. "Computational Thinking and Educational Robotics Integrated Into Project-Based Learning." *Sensors* 22, no. 10: 3746. https://doi.org/10.3390/s22103746.

Wang, C., J. Shen, and J. Chao. 2022. "Integrating Computational Thinking in STEM Education: A Literature Review." *International Journal of Science and Mathematics Education* 20, no. 8: 1949-1972. https://doi.org/10.1007/s10763-021-10-227-5.

Webb, M., N. Davis, T. Bell, et al. 2017. "Computer Science in K-12 School Curricula of the 21st Century: Why, What and When?" *Education and Information Technologies* 22: 445-468. https://doi.org/10.1007/s10639-016-9493-x.

Weintrop, D. 2019. "Block-Based Programming in Computer Science Education." *Communications of the ACM* 62, no. 8: 22-25. https://doi.org/10.1145/3341221.

Weintrop, D., and U. Wilensky. 2015. "To Block or Not to Block, That Is the Question: Students' Perceptions of Blocks-Based Programming." In *Proceedings of the 14th International Conference on Interaction Design and Children*, 199-208. IEEE. https://doi.org/10.1145/2771839.2771860.

Weintrop, D., and U. Wilensky. 2019. "Transitioning From Introductory Block-Based and Text-Based Environments to Professional Programming Languages in High School Computer Science Classrooms." *Computers & Education* 142: 103-646. https://doi.org/10.1016/j.compedu.2019.103646.

Wing, J. M. 2006. "Computational Thinking." *Communications of the ACM* 49, no. 3: 33-35. https://doi.org/10.1145/1118178.1118215.