# Assessing Engagement of Students with Intellectual Disabilities in Educational Robotics Activities

Francesca Coin, Monica Banzato

<figure><img src="image_1.png" /></figure>

## To cite this version:

Francesca Coin, Monica Banzato. Assessing Engagement of Students with Intellectual Disabilities in Educational Robotics Activities. IFIP World Conference on Computers in Education (WCCE), Aug 2022, Hiroshima, Japan. pp.124-136, 10.1007/978-3-031-43393-1_13. hal-05533286

HAL Id: hal-05533286

https://inria.hal.science/hal-05533286v1

Submitted on 2 Mar 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
<figure><img src="image_4.png" /></figure>

# Assessing Engagement of Students with Intellectual Disabilities in Educational Robotics Activities

Francesca Coin and Monica Banzato

Ca' Foscari University, Venice, Italy
{francescacoin.psi,banzato}@unive.it

**Abstract.** Engagement is a multi-componential construct, difficult to measure in its general form as in the typical development but even more in the atypical development. Currently, there are no instruments in the literature to measure engagement in its different dimensions (behavioural, affective, social and cognitive) for students with intellectual disabilities involved in creative robotics activities. With this aim, we tried to apply a survey system based on the triangulation of three ad hoc tools: an observation grid, analysis of verbal productions and a questionnaire. This triple system of application allowed us to understand whether the creative robotics activity proposed to children with intellectual disabilities aroused their interest and involvement, and to what extent.

**Keywords:** Engagement · Student with intellectual disability · Educational Robotics

## 1 Introduction

Following the EU educational policies, the last Education Reform in Italy (2016) introduced computational thinking at all school levels. Regrettably, in the Italian school reality, students with intellectual disabilities (ID) appear to be excluded from programming activities with educational robotics (ER) [1]. The research literature confirmed the same gap: there are very few publications on educational ER programming activities with students with ID [2]; most research is focused on humanoid robotics, the role of assistive devices or the effectiveness of computer-assisted instruction. The range of publications narrows dramatically when examining the age of students: the few existing studies focus mainly on children between 6 and 12, while adolescents aged 15 to 20, in high school, appear decidedly under-represented [2]. Therefore, this exploratory study aimed to measure the engagement of 8 high school students with ID, aged between 16 and 22, in ER programming activities (with Arduino as base), in a school in the North-East of Italy. In this work, considering the target students, we considered very simple activities of programming and coding: for example, students lifting flashcards (paper or digital) to make the robot perform the specific action displayed in the flashcards.
Engagement is considered a key construct to identify the educational potential of ER activities, at behavioral, cognitive, affective, and social levels. Although there is a conspicuous amount of research on this construct in the educational field for typical development, the study of engagement for atypical development is still limited to the use of robotics in therapeutic and clinical settings (particularly with the use of humanoid robotics). Thus, there is a gap in the literature on the educational dimension of engagement of ER programming activities for students with ID [2]. To measure the engagement of these students, we constructed and applied three ad hoc research instruments: an observation instrument, a verbal interaction analysis instrument and a questionnaire. In the next paragraphs, the definition and measurement of engagement will be analyzed, the tools of our proposal will be described, and finally the tools will be applied to a case study of 8 students with ID engaged in basic robotics activities.

## 2 Engagement and Its Measurement

The meaning and measurement of engagement has evolved over time “to represent increasingly complex understandings of the relationships between desired outcomes of college and the amount of time and effort students invest in their studies and other educationally purposeful activities” (pp.683) [3]. Engagement is generally described as a multi-componential construct and over time several models have been constructed to define it. For example, Finn’s [4] model includes two components: behavioural (participation) and affective (identification, belonging, evaluation). The same categories are considered by Skinner et al. [5] in their comparison between Engagement vs. Disaffection, both involving a behavioural and an affective aspect. Other authors add a further component, cognitive, linked to self-regulation goals and the amount of investment [6]. Gunuc & Kuzu [7] recall the importance of the social aspect, which is mutually dependent with engagement, especially in school and learning contexts.

The measurement of engagement levels is one of the most challenging issues for researchers in the field. Until the late 1990s this issue had been placed on the back burner [8]. However, over the last two decades a number of scholars have focused on the variety of data collection techniques most suitable for measuring engagement. The most common are self-report measures. Among the earliest are Jacques’ [9] 13-item interview containing questions about attention, perceived time, motivation, needs, control, attitudes and general engagement and Webster and Ho’s [10] seven-item questionnaire with questions about attention, challenge, intrinsic interest, and variety. They are followed by more recent instruments such as the User Engagement Scale by O’Brien & Toms [11], the Motivation and Engagement Scale by Liem & Martin [12] and the UTAUT model by Heerink et al. [13].

Despite the increase in research on this subject, there seem to be considerable difficulties in measuring cognitive and psychological engagement. Frequently, the same questions can be used to represent different subtypes of engagement, while the subtypes are often examined separately, precluding further levels of comparison [6].

Some researchers have used performance indicators, not as direct measures of engagement, but as its correlates. For example, Konradt and Sultz [14] used pre- and post-activity measures to examine changes in users’ affective and cognitive states during
interaction with an educational application. However, even metrics such as interaction analysis, number of eye fixations, heart rate, etc., have limitations: they explain what is happening during a user's interaction with a given activity, but do not address the cognitive or emotional state of users, which are crucial for engagement [10].

There is also the problem of generalizability of the results: often a tool is valid for the precise domain or activity it was constructed for, but these results are not applicable to other domains. Moreover, the elements contained in each of these tools represent a small part of the engagement attributes indicated so far.

Champion [15] hypothesized that multiple measures (self-report, participant observation, biometrics) are needed to study engagement, but did not offer any empirical results or describe how these measures could be triangulated to say anything meaningful about engagement. Therefore, for our purposes, we decided to build ad hoc tools, to measure engagement of student with cognitive disability, which will be outlined in the following sections.

# 3 Tools Developed to Measure Engagement

## 3.1 Observation Grid

The observation grid included the analysis of non-verbal expressive behaviour. We were inspired by the work of Mehrabian [16], making appropriate modifications for our activity, as we will explain below. The behaviours were divided into three categories: Posture and stance, and Movements and Facial expressions. The first category includes: Touch, Distance, Stretching, Eye contact, Body orientation. Since most of the robotics activities were carried out in a sitting position, only touch and eye contact were analysed.

In the second category are: Trunk movements, Swinging, Nodding, Gesturing, Self-manipulation, Foot and leg movements. For the same issue of seated positioning, some categories were excluded; we selected nodding, gesturing and self-manipulations or stimming (i.e., repeatedly touching a part of one's own body, finger-flicking and so on). In relation to trunk and arm movements, the asymmetrical position of the arms is interpreted as a sign of relaxation.

The third category is devoted to Facial expressions and is divided into Positive and Negative expressions. We chose to analyse the most frequent and explicit expressions: smiling and laughter as positive interactions, and yawning indicating boredom and disengagement.

The following criteria were used to compile the observation grid:

1. *Duration of eye contact* with the different objects: duration is expressed in seconds of fixations towards the interlocutor and the objects selected for each activity: robot, paper and digital flashcards with arrow, carpet, interactive whiteboard.

2. *Number of eye contacts* with different objects: the number of direct fixations of eye contact towards the interlocutor and the objects selected for each task are counted: robot, paper and digital flashcards with arrow, carpet, interactive whiteboard (IWB).

3. *Number and duration of facial expressions:* smiling, laughing, yawning, etc. are counted.
4. *Number and duration of body movements:* gesturing, nodding, self-manipulations, standing up and sitting, relaxation, stimming are counted.

5. *Number and duration of physical distance from objects:* spontaneous touching, touching on command, hesitation to respond are counted.

In relation to the above classifications, it can be noted that eye contact refers to the behavioural aspect of engagement, as it involves physical movement, therefore it is related to the factors Time on task [6], Focused attention [8], Persistence [12], Use [13] and Effort [5].

The *number of ocular occurrences* signals interest and attraction and is therefore related to the factors Novelty [8] and Interest [5].

Facial expressions represent the emotions felt towards the object or activity; therefore, they are related to the factors Attitude and Perceived enjoyment [13], Felt Involvement [8] and Enjoyment [5].

Body movements can be interpreted as emotional signals of nervousness or relaxation; therefore, they are related to Anxiety [13], Uncertain control [12], Self-regulation [6] and Perceived usability [8].

The *physical distance from the objects*, in particular in the meaning of *spontaneous touching*, indicates Perceived ease of use [13], Participation [6], Self-efficacy [12], Aesthetic [8], Enthusiasm and Exertion [5].

Time indicates the duration of each work phase, including some, not many, moments of relaxation and those dedicated to the actual activity. The category *Eye contact* reports a continuous event, as it indicates the duration of eye fixation and is expressed in seconds. The other categories indicate punctual events and are expressed in number of times per minute.

## 3.2 Verbal Expressions

Also, via video recording, it was possible to transcribe the verbal comments of the pupils. They were classified into two categories according to the extent of verbal production: *short interjections* and *complete words or sentences*. Subsequently, the spontaneity of these productions was assessed: “*Completely spontaneous*”, “*In response to a question*” or “*Completion/repetition of portions of sentences*” produced by the interlocutor. Spontaneous productions indicate a greater involvement and state of well-being on the part of the pupil, while responses to questions and completion of sentences indicate a high degree of participation but less ease in the situation. In addition to these, the category “*Avoidance of answer*” was added to indicate the condition of discomfort and low willingness to participate [17].

## 3.3 The Questionnaire

The questionnaire (see Table 1) was composed of the most frequent factors mentioned in the literature. In particular, we selected questions with a direct correspondence in the classifications of the five mentioned authors. The aim was to create an instrument lean and quick to complete (19 items), with questions presented in lexically and syntactically simple formulations. The questions are posed in the affirmative form and the participants
are asked to mark their agreement with the proposed statements in a 5-level Likert scale. The answers are accompanied by a verbal label (5 = Completely agree, 4 = Agree, 3 = Uncertain, 2 = Disagree, 1 = Do not agree at all) and a graphic representation (a smiley more or less smiling).

<table><caption>**Table 1.** RE engagement questionnaire for students with ID</caption><thead><tr><th>Item</th><th>Heerink [13]</th><th>Appelton et al. [6]</th><th>Liem et al. [12]</th><th>O'Brien et al. [8]</th><th>Skinner et al. [5]</th></tr></thead><tbody><tr><td>It's nice to use the robot</td><td>Attitude</td><td>Participation</td><td>Self-efficacy</td><td>Aesthetic</td><td>Enthusiasm</td></tr><tr><td>I like working with the robot</td><td>Perceived enjoyment</td><td>Participation</td><td>Self-efficacy</td><td>Felt involvement</td><td>Enjoyment</td></tr><tr><td>I like giving commands to the robot</td><td>Perceived Enjoyment</td><td>Participation</td><td>Self-efficacy</td><td>Felt Involvement</td><td>Enjoyment</td></tr><tr><td>I want to know more about robots</td><td>Perceived usefulness</td><td>Aspiration</td><td>Mastery orientation</td><td>Novelty</td><td>Interest</td></tr><tr><td>I want to continue the activity now</td><td>Intention to use</td><td>Time on task</td><td>Persistence</td><td>Endurability</td><td>Persistence</td></tr><tr><td>I am afraid of making mistakes*</td><td>Anxiety</td><td>Self-regulating</td><td>Anxiety</td><td>Perceived Usability</td><td>Exertion</td></tr><tr><td>I'm afraid of breaking the robot*</td><td>Anxiety</td><td>Self-regulating</td><td>Anxiety</td><td>Perceived Usability</td><td>Exertion</td></tr><tr><td>It is easy to command the robot</td><td>Perceived ease to use</td><td>Strategizing</td><td>Self-efficacy</td><td>Perceived Usability</td><td>Exertion</td></tr><tr><td>I'm scared of the robot*</td><td>Anxiety</td><td>Self-regulating</td><td>Anxiety</td><td>Perceived Usability</td><td>Enthusiasm</td></tr><tr><td>The robot is difficult to use*</td><td>Perceived ease to use</td><td>Strategizing</td><td>Uncertain control</td><td>Perceived Usability</td><td>Exertion</td></tr><tr><td>Papers flashcards with arrow are easy to use</td><td>Facilitating Condition</td><td>Strategizing</td><td>Self-efficacy</td><td>Perceived Usability</td><td>Effort</td></tr></tbody></table>

(continued)
<table><caption>**Table 1.** (continued)</caption><thead><tr><th>Item</th><th>Heerink [13]</th><th>Appelton et al. [6]</th><th>Liem et al. [12]</th><th>O'Brien et al. [8]</th><th>Skinner et al. [5]</th></tr></thead><tbody><tr><td>Digital flashcards with arrow are easy to use</td><td>Facilitating Condition</td><td>Strategizing</td><td>Self-efficacy</td><td>Perceived Usability</td><td>Effort</td></tr><tr><td>I'd spend my free time playing with robots</td><td>Intention to use</td><td>Value</td><td>Persistence</td><td>Endurability</td><td>Interest</td></tr><tr><td>I'm not interested in this activity with robots*</td><td>Perceived usefulness</td><td>Value</td><td>Disengagement</td><td>Felt Involvement</td><td>Interest</td></tr><tr><td>I got bored*</td><td>Attitude</td><td>Participation</td><td>Disengagement</td><td>Felt Involvement</td><td>Interest</td></tr><tr><td>It made me angry*</td><td>Attitude</td><td>Self-regulating</td><td>Disengagement</td><td>Felt Involvement</td><td>Enjoyment</td></tr><tr><td>I want to continue using the robot at school</td><td>Intention to use</td><td>Completion</td><td>Persistence</td><td>Endurability</td><td>Interest</td></tr><tr><td>I want to take the robot home</td><td>Intention to use</td><td>Completion</td><td>Persistence</td><td>Endurability</td><td>Interest</td></tr><tr><td>I want to use it with my schoolmate</td><td>Social influence</td><td>Participation</td><td>Valuing</td><td>Endurability</td><td>Enthusiasm</td></tr></tbody></table>

There are 13 questions concerning positive conditions (e.g., “It is easy to control the robot”) and 6 questions are reported in the opposite form, marked with asterisks (*) in the table (e.g., “The robot is difficult to use”). Despite the adjustments and simplifications, several pupils were not able to read and fill in the questionnaire independently. It was therefore necessary to read the questionnaire and simplify the way of answering. The questionnaire took the form of a structured interview. The adaptations necessary show that the questionnaire is not the easiest or most reliable tool to use in ID contexts. Hence, we suggest turning it into a more interactive conversation and to accompany it with observation.
# 4 Case Study

## 4.1 The Educational Activity with Creative Robotics

The classroom activity was carried out individually by the pupils. After a brief presentation, a researcher began with a recited tale of Aesop's The Mouse and the Frog, to introduce the protagonists of the activity. Then the robot, called Rospino© [18], was presented and the pupils were asked to complete it by assembling some of its parts, in-volving them in the simplest stages of its construction. The robot represented the frog in the story and it had to replicate some of the character's actions or movements (e.g., reaching his friend the mouse, entering the pond, reaching the den while avoiding the kite, etc.), moving on a decorated carpet.

The aim was to program the robot's movements (coding), so it would reach the target object or character, through four paths of increasing difficulty. The movement could be programmed in three ways with increasing levels of abstraction: low, physically moving oneself and the robot along the path; intermediate, selecting the directional paper flashcards with arrow, representing arrows, needing to show the robot the path; and high, clicking the digital flashcards with arrows directly through the interactive whiteboard (IWB).

Not knowing the abilities of the pupils, who had very different intellectual levels and coding experiences, it was decided to start the activities at the intermediate level. If the pupil proved to be particularly good at selecting the instructions on paper, he or she went directly to programming software; if he or she had difficulties, the activity could be simplified further with manipulable materials. The total duration of the activity was about 45 min.

## 4.2 The Participants

Eight young students aged between 15 and 20 years, attending a high school in north-eastern Italy, participated in the case study. Among them were seven males and one female. All of them presented certified ID of various degrees, from mild to moderate. The nature of the disability was different: Down's Syndrome, Autism Spectrum Disorder, premature birth, and other undefined aetiologies. Only children whose parents gave their permission and willingness to accompany them during extracurricular hours participated. The students were asked if they were interested in participating in the activity. They were given time to see the setting and the materials. They could stop the activity at any time according to their wishes and needs. The pupil's support teacher was present at the activities.

## 4.3 The Method

The entire activity was video-recorded, and the observation grid was applied retrospectively, using the ELAN coding software to note the number and duration of the behaviours observed. Some indices were transformed into percentages and number of events per minute to make them comparable despite the different duration of the overall recording.
The questionnaire was submitted at the end of the activity, reading support was provided, or the structured interview method was used. It was not possible to administer the instrument to a pupil who, due to excessive fatigue, did not show his willingness. The small number of participants made it impossible to apply inferential statistical analyses, nor to compare the responses of males and females.

# 5 Results

## 5.1 Questionnaire and Interview

The pupils declared their involvement in the ER activity, in fact the questions “Is it nice to use the robot?” and “Do I like working with the robot?” obtained the highest score (5) among all respondents, as well as their reciprocal in the negative “Did I get bored with during the activity with the robot?” and “Did this activity with the robot make me angry?” obtained the lowest score (0).

A slight difficulty in the use of the tools (e.g., paper, and digital flashcards) was perceived, as the related questions received slightly lower average scores “Do you like giving commands to the robot?” (4.86), “Is it easy to command the robot?” (4.57), “Is it difficult to use the robot?” (4.43), “Are the digital flashcards with arrows easy to use?” (4.50), but not “Are paper flashcards with arrows easy to use?” which gets full marks (5). The question is explained by the fears that emerge regarding possible mistakes, “Are you afraid of making mistakes?” (3.86) and damage, “Are you afraid of breaking the robot? (4.43).

The boys declared that they wanted to continue the activity immediately (4.29), in their free time (4.57), at school (4.86) and at home (4.43). Some more doubts arose about the possible involvement of their companions in the activity (3.86).

## 5.2 Observation Grid

The eye contact showed that the pupils were able to focus their attention correctly on the object of each activity: during the narration phase, they observed the interlocutor 67.27% of the time; during the presentation and assembly phase of the robot, their gaze was directed 59.14% of the time to the robot; during the coding phase with paper flashcards, attention was directed to the arrow flashcards for an average of 62.6% of the time among the four proposed exercises; the interactive whiteboard (IWB) received attention 44.72% of the time it was used; and their gaze returned to the robot 51.08% of the time it was moving on the carpet (see Table 2).

The glances directed at other objects are, however, quite relevant to the activity: during the narration phase 11.42% of the time, they directed their gaze at the paper flashcards representing the characters in the story.

During the different phases, eye contact with the interlocutor remained constant, on average 13.71% of the time, as he continued to give explanations or phrases of encouragement to the pupil.

The other object that seems to have attracted their attention, even when it was not at the centre of the activity, is the robot, at 12.63% of the time on average, which is a good sign for an educational robotics workshop.
<table><caption>**Table 2.** Eye Contact: occurrences in % per activity sections (stage)</caption><thead><tr><th>Stage</th><th>Robot</th><th>Flashcard</th><th>Speaker</th><th>IWB</th></tr></thead><tbody><tr><td>Narration</td><td>02.80</td><td>11.42</td><td>67.27</td><td>00.00</td></tr><tr><td>Robot</td><td>59.14</td><td>00.29</td><td>10.70</td><td>00.00</td></tr><tr><td>Paper Flashcard 1</td><td>16.24</td><td>58.78</td><td>09.94</td><td>00.00</td></tr><tr><td>Paper Flashcard 2</td><td>16.27</td><td>66.77</td><td>10.54</td><td>01.10</td></tr><tr><td>Paper Flashcard 3</td><td>16.83</td><td>62.48</td><td>12.54</td><td>00.00</td></tr><tr><td>Paper Flashcard 4</td><td>09.98</td><td>62.43</td><td>22.83</td><td>00.00</td></tr><tr><td>Digital Flashcard (IWB)</td><td>14.39</td><td>07.97</td><td>16.97</td><td>44.72</td></tr><tr><td>Movement (carpet)</td><td>51.08</td><td>00.16</td><td>12.32</td><td>00.85</td></tr></tbody></table>

The moment in which eye contact appeared less focused, i.e., the only phase in which attention was directed to the key object less than half the time, was the programming phase on the interactive whiteboard, as the pupils frequently directed their eyes to the digital flashcard and the robot to remember the sequence of commands to be entered.

As regards the occurrences, i.e., the number of glances directed at each object, the pupils shifted the direction of their gaze on average 10 times per phase. Most of the glances were directed at the interlocutor (10.63/min) alternating with the target object of the activity of the moment (see Table 3).

<table><caption>**Table 3.** Eye Contact: occurrences of the number of glances in minute</caption><thead><tr><th>Stage</th><th>Robot</th><th>Flashcard</th><th>Speaker</th><th>IWB</th></tr></thead><tbody><tr><td>Narration</td><td>01.93</td><td>01.73</td><td>06.28</td><td>00.00</td></tr><tr><td>Robot</td><td>03.75</td><td>00.13</td><td>10.42</td><td>00.00</td></tr><tr><td>Paper Flashcard 1</td><td>03.54</td><td>05.29</td><td>09.66</td><td>00.00</td></tr><tr><td>Paper Flashcard 2</td><td>02.74</td><td>04.87</td><td>09.71</td><td>00.68</td></tr><tr><td>Paper Flashcard 3</td><td>02.44</td><td>04.86</td><td>11.77</td><td>00.08</td></tr><tr><td>Paper Flashcard 4</td><td>02.08</td><td>04.57</td><td>19.33</td><td>00.00</td></tr><tr><td>Digital Flashcard (IWB)</td><td>01.61</td><td>01.35</td><td>05.92</td><td>04.14</td></tr><tr><td>Movement (carpet)</td><td>03.89</td><td>00.09</td><td>11.96</td><td>00.18</td></tr></tbody></table>

Concerning Facial expressions, smiling was the most frequent, occurring an average of 5 times per phase, approximately 1.4 times per minute. The phases that produced the most smiles were the narration (2.12/min) and the assembly of the robot (2.37/min). The phases that produced fewer smiles were the initial programming activities with the paper flashcards (between 1.16 and 1/min), which probably required more concentration.

Real laughter arose only during the comic moments of the narration (0.32/min), rare in the moments of work with robots (0.09/min) and the interactive whiteboard (0.10/min).
Laughter was completely absent in the phases of work with flashcards which, as already noted, required more effort.

Yawns were only produced by one boy, who left the activity prematurely, stating that he did not feel well due to irregular sleep the night before. Other expressions, including expressions of perplexity, thought and concentration or looks of agreement were produced by only three pupils, the most sociable and least shy of the group.

With regard to Movements, the most frequent was the movement of the head indicating consent, i.e., nodding (2.56/min), followed by significantly less frequent gestures indicating participation in the dialogue and attempts at explanation (1.1/min). Self-handling gestures indicating discomfort were scarce (0.73/min) and body movements to stand up or sit down when not requested were very rare (0.23/min).

Concerning Physical contact with objects, the pupils often tried to touch the objects of the activity spontaneously, on average 4.41 times per phase, or 1.4 times per minute. This occurred especially in the last phases of programming with the digital flashcards (2.5/min), a sign that they had become familiar with the activity, and during the movement of the robot (1.57/min). There were practically no Hesitations in responding (0.15/min), except the few that occurred during the programming phase with the interactive whiteboard (0.30/min), in fact it was the only phase in which it was necessary to give more precise commands on what to touch (0.85/min). The interactive whiteboard is a tool that pupils have in class and daily use. It is possible, however, that during curricular activities teachers explicitly limit contact between pupils and the interactive whiteboard for fear of damage and that this attitude has inhibited pupils from using it spontaneously.

## 5.3 Verbal Expressions

The pupils produced an average of 14.72 verbal comments per phase, i.e., approximately 6.27 comments per minute. Most of them were whole words or sentences (78%) and only a minority were short interjections (22%). Of these, 51% were produced spontaneously, 37.25% in response to a question and 11.43% in completion or repetition of parts of the interlocutor's sentence. As is normal, there were some boys who were more talkative than others: the maximum verbal exchange was reached by a boy who produced 11.10 comments per minute, 56% of which were whole sentences and 43% short interjections. Of these, 60% were spontaneous, 30% in response and 11% in completion. The smallest interaction was with the girl, evidently more shy, who produced 1.74 comments per minute, of which, however, 98% were words and 2% interjections, thus demonstrating a certain communicative effort. Only 27% were related to spontaneous productions, while 50% were provided in response to questions and 21% to complete sentences.

Response avoidance was almost nil. Even in the rare cases in which the verbal response was late in arriving at the interlocutor's input, it was replaced by eye contact and facial expressions.

## 6 Discussion

The results that emerged from observation matched the answers in the interviews.
The pupils declared a high involvement in the ER activity and in fact the direction of their gaze, directed more than half of the time to-wards the key object of the activity, confirmed this finding. Facial expressions and body posture also showed conditions of ease and well-being, while the frequency of spontaneous tactile contact with the objects demonstrated their confidence and interest. Expressions of boredom or discomfort were minimal.

The pupils expressed a slight difficulty in using the tools, particularly in the initial phases of programming with the paper flashcards: this was detected by the pupils' constant glances at the interlocutor in search of confirmation, the number of gestures to implement their explanations and the difference in the frequency of smiling expressions and tactile contact with the objects. This clumsiness often tended to diminish in the later stages, a sign that the children were now familiar with the tool. As pointed out, the most difficult device to use was the interactive whiteboard, which required more direct instructions (e.g. "to take a step forward, click on this icon with the forward arrow"). The problem turned out to be related to the fear of making mistakes, breaking something, or causing damage.

The pupils declared they wanted to continue the activity, and the behavioural feedback to this statement can be seen by counting the small number of times the participants tried to get up or sit down to move away from the activity in progress. The fact that they were focused on maintaining visual and tactile contact with objects and inter-locutors is a clear indication of their involvement. It is of course not possible to verify the truth regarding their intentions to continue the activity at other times of the day.

Eye contact, number of occurrences and facial expressions were found to be the most useful indicators: relatively easy to measure, applicable in any context and strongly predictive of the degree of interest and involvement. Particular attention should be paid to the subcategories of facial expressions. Only the more extroverted and higher cognitive level pupils indulged in more complex expressions; for the others the range of expressions was limited to selected basic expressions.

Body movements are more difficult to assess, as they are limited by the type of setting and activity performed. In the design phase of the instruments, it was decided to select only some of the categories listed by Mehrabian [16]. Distance, Stretching and Body Movements were not assessable in a sitting position, as well as Trunk Movements, Swinging and Foot and Leg Movements, which may, however, be interesting to measure in another work context. Also, the category Relaxation/Arm symmetry, although included in the measurement, proved difficult to assess in a sitting position and not very significant with respect to the real perception of well-being of the children. Motor stereotypes were only performed by the boy with autism spectrum syndrome at the end of the activity, indicating emerging fatigue.

Verbal interactions showed a high degree of participation and comfort in the workshop situation, as most of the comments were spontaneously produced and composed of whole words and/or sentences. There was almost no response avoidance. The analysis of verbal comments, however, turned out to be a partial instrument for the measurement of engagement: it is too subject to pupil's differences, linked to cognitive level and pathology. Moreover, it is extremely limited if not combined with the recording of facial expressions and gestures accompanying verbal production.
# 7 Limits and Conclusions

The triangulation of the instruments proved to be an indispensable method of detection. Each of the three devices proved to be useful for measuring the level of engagement but limited when taken individually.

A major limitation remains the impossibility of establishing standardised parameters that can be used as thresholds to indicate a sufficient range of attention. Eye contact maintained for more than half of the time on the object of interest is a clear sign of engagement in the activity in progress. However, this, and especially the other parameters are very sensitive to personal, setting- and activity-related variations. For example, a high percentage of spontaneously produced words and sentences is a good indicator of social engagement, but a more introverted person will tend to produce fewer verbal comments than a more sociable person, without their engagement being lower. This variability increases further in the presence of syndromes or diseases of different aetiology, as in this case.

Replicating the research in different contexts, it will be possible to refine the tools in order to make them each time more suitable to the setting (e.g., sitting or standing), to the activity (with more or fewer objects used in sequence or at the same time), as well as to different syndromes causing disability (different intellectual levels, socialisation disorders etc.).

From the data, it can be considered that the triad of tools used showed characteristics of completeness, as it covers the four components of engagement; and of ductility as it can be easily modified to make it more suitable to one's own setting; and of practicality, as it can be applied to various contexts, even by teachers.

Note: for reasons of national assessment of Italian university research, the authors must declare which sections each has written, in spite of the fact that work is entirely the result of continuous and intensive collaboration. Sections 2, 3, 4 and 7 are by F. Coin. Sections 1, 5 and 6 are by M. Banzato. Our thanks to P. Tosato, G. Riello and M. Hoffman.

# References

1. Indire: Gli snodi dell'inclusione. Monitoraggio dei centri territoriali di supporto (2020). http://www.indire.it. Last accessed 17 Aug 2022

2. Pivetti, M., Di Battista, S., Agatolio, F., Simaku, B., Moro, M., Menegatti, E.: Educational robotics for children with neurodevelopmental disorders: a systematic review. Heliyon **6**(10), e05160 (2020)

3. Kuh, G.D.: What student affairs professionals need to know about student engagement. J. Coll. Stud. Dev. **50**(6), 683-706 (2009)

4. Finn, J.D.: Withdrawing from school. Rev. Educ. Res. **59**(2), 117-142 (1989)

5. Skinner, E.A., Kindermann, T.A., Furrer, C.J.: A motivational perspective on engagement and disaffection: conceptualization and assessment of children's behavioral and emotional participation in academic activities in the classroom. Educ. Psychol. Measur. **69**(3), 493-525 (2009)

6. Appleton, J.J., Christenson, S.L., Kim, D., Reschly, A.L.: Measuring cognitive and psychological engagement: validation of the student engagement instrument. J. Sch. Psychol. **44**(5), 427-445 (2006)
7. Gunuc, S., Kuzu, A.: Student engagement scale: development, reliability and validity. Assess. Eval. High. Educ. **40**(4), 587–610 (2015)

8. O'Brien, H.L., Toms, E.G.: The development and evaluation of a survey to measure user engagement. J. Am. Soc. Inform. Sci. Technol. **61**(1), 50–69 (2010)

9. Jacques, R.D.: The nature of engagement and its role in hypermedia evaluation and design. Unpublished doctoral dissertation, South Bank University, London (1996)

10. Webster, J., Ho, H.: Audience engagement in multimedia presentations. ACM SIGMIS database: the DATABASE for Adv. Inf. Syst. **28**(2), 63–77 (1997)

11. O'Brien, H.L., Toms, E.G.: What is user engagement? A conceptual framework for defining user engagement with technology. J. Am. Soc. Inform. Sci. Technol. **59**(6), 938–955 (2008)

12. Liem, G.A.D., Martin, A.J.: The motivation and engagement scale: theoretical framework, psychometric properties, and applied yields. Aust. Psychol. **47**(1), 3–13 (2012)

13. Heerink, M., Krose, B., Evers, V., Wielinga, B.: Measuring acceptance of an assistive social robot: a suggested toolkit. In: RO-MAN 2009-The 18th IEEE International Symposium on Robot and Human Interactive Communication, pp. 528–533. IEEE, Cambridge (2009)

14. Konradt, U., Sulz, K.: The experience of flow in interacting with a hypermedia learning environment. J. Educ. Multimedia Hypermedia **10**(1), 69–84 (2001)

15. Champion, E.: Applying game design theory to virtual heritage environments. In: Proceedings of the 1st International Conference on Computer Graphics and Interactive Techniques in Australasia and Southeast Asia, pp. 273–274. ACM, New York (2003)

16. Mehrabian, A.: Some referents and measures of nonverbal behavior. Behav. Res. Meth. Instru. **1**(6), 203–207 (1968)

17. Flanders, N.: Analyzing Teacher Behavior. Addison-Wesley, New York (1970)

18. Rospino© Homepage, http://www.projectschool.it. Last accessed 17 Aug 2022