# Introducing Gestural Interaction on the Shop Floor:
Empirical Evaluations

Salvatore Andolina, Paolo Ariano, Davide Brunetti, Nicolò Celadon, Guido Coppo, Alain Favetto, Cristina Gena, Sebastiano Giordano, Fabiana Vernero

<figure><img src="image_1.png" /></figure>

## To cite this version:

Salvatore Andolina, Paolo Ariano, Davide Brunetti, Nicolò Celadon, Guido Coppo, et al. Introducing Gestural Interaction on the Shop Floor: Empirical Evaluations. 18th IFIP Conference on Human-Computer Interaction (INTERACT), Aug 2021, Bari, Italy. pp.451-455, 10.1007/978-3-030-85607-6_56. hal-04291260

HAL Id: hal-04291260

https://inria.hal.science/hal-04291260v1

Submitted on 17 Nov 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Introducing gestural interaction on the shop floor: empirical evaluations*

Salvatore Andolina<fnref n="1" />, Paolo Ariano<fnref n="2" />, Davide Brunetti<fnref n="3" />, Nicolò Celadon<fnref n="2" />, Guido Coppo<fnref n="1" />, Alain Favetto<fnref n="2" />, Cristina Gena<fnref n="3" />, Sebastiano Giordano<fnref n="2" />, and Fabiana Venero<fnref n="3" />

**Abstract.** This paper describes two empirical evaluations we carried out in the context of an Industry 4.0 project, where we explored the use of touchless gestural interaction on the shop floor.

**Keywords:** Gestural interaction · Empirical evaluation · Industry 4.0.

## 1 Introduction

In the “smart factory” vision, industries are production-oriented cyberphysical systems [4] able to exploit emerging information technologies to guarantee flexible, efficient, environment-friendly, high-quality and low-cost production processes [5]. In this context, however, designing appropriate modalities to communicate, analyze and interact with all the information related to production processes can be challenging, especially in consideration of its many environmental and safety-related constraints. For example, keyboards or touch devices may be inconvenient for workers on the shop floor, who must wear protective gloves, while voice-based interaction may be impractical due to the background noise.

Guided by the requirements expressed by our industrial partners, in the HOME regional project<fnref n="4" /> on Industry 4.0 we explored the use of large displays in combination with touchless gestural interaction. Based on previous work in the neuromotor rehabilitation field [2], the project consortium developed a smart armband which allows to detect gestures from movement and muscle biosignals, while a machine learning library allows to calibrate and recognize task-specific gestures. Notice that this *ad hoc* solution was preferred to the use of motion sensing input devices, in that these can raise a number of issues related to privacy and perceived surveillance. The definition of an appropriate set of gestures

<footnote n="1">¹ Synarea, Torino, Italy
surname@synarea.com</footnote>

<footnote n="2">² IIT, Torino, Italy
Name.Surname@iit.it</footnote>

<footnote n="3">³ Università degli Studi di Torino, Corso Svizzera 185, 10149 Torino, Italy
name.surname@unito.it</footnote>

<footnote>* This work was supported by HOME (Hierarchical Open Manufacturing Europe) Regional Project (POR FESR, code 319-50).</footnote>

<footnote n="4">⁴ https://www.home-opensystem.org/</footnote>
has undergone several steps, including a guessability study [1], in an attempt to
balance the sometimes competing needs for gesture intuitiveness (end users) and
recognizability (project consortium), combined with a preference for minimum
calibration, due to the fact that the armband can be shared by several workers.

In this paper, we describe our final empirical evaluations with our industrial
partner Galeasso s.r.l.<fnref n="5" />, which specializes in sheet metal fabrication and is going
to adopt the armband on its shop floor.

<figure><img src="image_4.png" /><figcaption>**Fig. 1.** The grid-like (a) and CAD-like (b) interfaces.</figcaption></figure>

2 Empirical evaluations

Two evaluations were carried out with the future users of the armband to assess
its ease of use and elicit comments and suggestions on gestural interaction at the
workplace. We focused on a specific scenario suggested by our partner Galeasso
s.r.l., where welders frequently switch between their workbench and a nearby
desktop computer to browse the tasks they have been assigned and visualize
3D models of the final product. Thus, we developed a prototype web application
(Figure 1) consisting in: 1) a grid-like interface to simulate selection tasks and 2)
a CAD-like environment showing 3D objects. The application can be controlled
through the armband and is to be shown on large displays installed on the shop
floor. The main actions allowed by the application are point and select, which
can be enacted through arm movements and the “close fist” gesture, respectively.

## 2.1 First empirical evaluation

The first empirical evaluation was carried out in June 2019. Participants were
two skilled welders. One of them (P1, 32 y/o) used computers and other digital
devices for more than 14 hours per week and was already familiar with gestural
interfaces (Wii, X-Box), while the other (P2, 47 y/o) was less technology-savvy.
Participants were asked to perform two tasks:

<footnote n="5">⁵ https://www.galeasso.com/</footnote>
1. **Task 1 (grid-like interface):** select a series of tiles. Each user was assigned three sequences of four tiles each.

2. **Task 2 (CAD-like interface):** view a cube with differently-coloured facets; select the cube and rotate it, so as to make the blue facet visible.

We manually recorded the number of errors made by participants, as well as their comments, elicited through the thinking aloud protocol. After each task, participants were also invited to assess their user experience through a short survey, consisting of 4 questions which could be answered through a 1-4 scale<fnref n="6" />.

**Results.** In the first task, both participants made two errors each, which consisted in selecting a certain tile twice. Such errors were probably due to the sensitivity setting of the armband, which might have caused small involuntary movements to be treated as selection gestures. Figure 2 reports our results for user experience evaluations. While P1, the participant having previous experience with gestural interaction, consistently expressed positive evaluations for both tasks, P2 was less positive, especially for the second task, where he had experienced most difficulties (see the free comments section for details).

<figure><img src="image_5.png" /><figcaption>**Fig. 2.** User experience assessments for the first (a) and second (b) task.</figcaption></figure>

**Free comments.** P1 experienced no particular difficulties and only suggested that the application should cope with involuntary movements such as little tremors. On the whole, he looked forward to using gestural interaction to carry out his everyday activities at the workplace. P2 found the task of rotating objects especially difficult and pointed out some additional problems due to the specific implementation of the test interface. He was quite satisfied about both the comfort and the sensitivity settings of the armband, but he also observed that arms might get tired after prolonged use. On the whole, he was convinced that using gestural interaction might significantly improve his work experience.

<footnote n="6">⁶ While quantitative measures are not significant due to the small number of participants, they were deemed to be useful to better interpret and understand free comments.</footnote>
## 2.2 Second empirical evaluation

Based on the results of the first evaluation, the project consortium worked on an improved version of the armband, focusing in particular on its sensitivity setting. Taking into account the delays caused by the COVID-19 pandemic, a second and final empirical evaluation was carried out in October 2020, where we used the same procedure, tasks and measures as in the previous evaluation. The participant (P3, 30 y/o) was an accomplished 3D designer, with some welding skills and similar demographics as P1 and P2 (notice that we were unable to recruit the very same participants due to shift management issues). In particular, similarly to P1, he was also a heavy user of digital devices and had had previous experience with gestural interaction (Kinect in particular).

**Results.** User experience evaluations are reported in Figure 2. Although he made no errors, P3 expressed a slightly less positive evaluation for the first task, in comparison with P1 and P2. In particular, he complained that the interaction was too slow. His evaluation, however, was notably higher for the second task.

**Free comments.** P3 observed that he had had some difficulties in pointing to the desired part of the interface and in having his gestures recognized, which was probably due both to his initial lack of familiarity with the armband and to persisting problems in sensitivity settings. Coherently with his low evaluation for the “speed” dimension in the first task, he pointed out that his experience would have been more satisfying, had the application been more responsive.

## 2.3 Conclusion

Our results offer some insight on the adoption of gestural interaction in smart factories. Most importantly, all the participants were favourable to our solution and willing to use it in their everyday work activities. User experience evaluations were generally positive, even if free comments highlighted the importance of fine tuning sensitivity settings: in the second evaluation, there were no errors due to involuntary movements, but the application appeared less responsive.

An open issue is the evaluation of the long-term user experience, which might be negatively affected by a natural decrease in motivation, as the initial enthusiasm starts to fade, as well as by the insurgence of arm fatigue, as hypothesized by one of the participants. While we are quite confident that the identified gestures avoid highly fatiguing arm positions which might cause the well-known “gorilla arm syndrome” [3], these aspects should be assessed empirically in future studies.

Finally, we must point out that one of the limitations of the here presented work is the small number of participants. On the other hand, however, we were able to involve actual representatives of our end users and to carry out the evaluations in a natural setting, i.e., on the shop floor, thus gaining good ecological validity.

## References

1. Andolina, S., Ariano, P., Brunetti, D., Celadon, N., Coppo, G., Favetto, A., Gena, C., Giordano, S., Vernero, F.: Experimenting with large displays and
gestural interaction in the smart factory. In: 2019 IEEE International Conference on Systems, Man and Cybernetics, SMC 2019, Bari, Italy, October 6-9, 2019. pp. 2864-2869. IEEE (2019). https://doi.org/10.1109/SMC.2019.8913900, https://doi.org/10.1109/SMC.2019.8913900

2. Di Girolamo, M., Celadon, N., Appendino, S., Turolla, A., Ariano, P.: Emg-based biofeedback system for motor rehabilitation: A pilot study. In: 2017 IEEE Biomedical Circuits and Systems Conference (BioCAS). pp. 1-4 (Oct 2017). https://doi.org/10.1109/BIOCAS.2017.8325086

3. Hansberger, J.T., Peng, C., Mathis, S.L., Areyur Shanthakumar, V., Meacham, S.C., Cao, L., Blakely, V.R.: Dispelling the gorilla arm syndrome: The viability of prolonged gesture interactions. In: Lackey, S., Chen, J. (eds.) Virtual, Augmented and Mixed Reality. pp. 505-520. Springer International Publishing, Cham (2017)

4. Riedl, M., Zipper, H., Meier, M., Diedrich, C.: Cyber-physical systems alter automation architectures. Annual Reviews in Control **38**, 123-133 (2014)

5. Wang, S., Wan, J., Li, D., Zhang, C.: Implementing smart factory of industrie 4.0: An outlook. International Journal of Distributed Sensor Networks **12**(1), 3159805 (2016). https://doi.org/10.1155/2016/3159805, https://doi.org/10.1155/2016/3159805