# Augmented reality, artificial intelligence and machine learning in Industry 4.0: case studies at SI-Lab

Antonio Bruno, Simone Coscetti, Giuseppe Leone, Danila Germanese, Massimo Magrini, Massimo Martinelli, Davide Moroni, Maria Pascali, Gabriele Pieri, Marco Reggiannini, et al.

<figure><img src="image_1.png" /></figure>

## To cite this version:

Antonio Bruno, Simone Coscetti, Giuseppe Leone, Danila Germanese, Massimo Magrini, et al. Augmented reality, artificial intelligence and machine learning in Industry 4.0: case studies at SI-Lab. ITAL-IA, Feb 2022, Turin, Italy. hal-03593925

HAL Id: hal-03593925

https://hal.science/hal-03593925

Submitted on 2 Mar 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Augmented reality, artificial intelligence and machine learning in Industry 4.0: case studies at SI-Lab

Antonio Bruno, Simone Coscetti, Giuseppe Riccardo Leone, Danila Germanese, Massimo Magrini, Massimo Martinelli, Davide Moroni, Maria Antonietta Pascali, Gabriele Pieri, Marco Reggiannini, Marco Tampucci

Signals and Images Lab, Istituto di Scienza e Tecnologie dell'Informazione "A. Faedo", ISTI-CNR
{name.surname}@isti.cnr.it

## Abstract

In recent years, the impressive advances in artificial intelligence, computer vision, pervasive computing, and augmented reality made them rise to pillars of the fourth industrial revolution. This short paper aims to provide a brief survey of current use cases in factory applications and industrial inspection under active development at the Signals and Images Lab, ISTI-CNR, Pisa.

## 1 Introduction

Industry 4.0 is demanding new solutions to manage the multiple resources available in a production plant integrating and orchestrating all the flows that should be performed. Besides the general production—which is often assumed to be the default mode— maintenance is of key importance since, in principle, it has to be performed so as not to interfere with production, keeping productivity at optimal levels and guaranteeing the expected quantities of processed product. In this context, technological solutions that can improve such aspects should also be sought in consideration of the request for superior safety levels and the presence of human operators.

Industry 4.0 pillars are firmly planted on a vast ground of technological advances, including horizontal and vertical integration, additive manufacturing, internet of things, simulation, augmented reality and big data and analytics. Notwithstanding this variety, artificial intelligence is a ubiquitous ingredient that can corroborate these pillars.

In this paper, we aim at providing some ideas of the possibilities opened by the convergence of multiple technologies by presenting research topics under development at the Signals and Images lab, a multidisciplinary research team, and describing related projects and case studies in Industry 4.0.

## 2 Research topics

### 2.1 Pervasive systems and augmented reality

In order to be efficient, modern production lines should guarantee a smooth operation without any machine stop. Monitoring the process and knowing the status of subsystems and components in real-time is thus of crucial importance. The

Industrial Internet of Things (IoT) offers answers in this direction thanks to integrated solutions for sensing, transmitting and aggregating data. At the SI-Lab, a great interest is played by approaches based on pervasive vision, where sensors equipped with cameras and integrated logics can provide quantitative data about the process they are observing. Anomalies in the form of precursor signs of possible faults can then be either recorded in the Supervisory Control and Data Acquisition (SCADA) system or directly notified to operators. For instance, anomaly detection in a video stream detected by a camera can be used to warn an operator, requiring intervention in a segment of the line. In this context, an integration with Augmented Reality (AR) interfaces for troubleshooting is beneficial for the actual deployment of the solutions in real scenarios. To this end, in the project IRIDE, an integrated system based on pervasive vision and AR has been proposed and then tested in the tissue converting scenario. A network of cameras is capable of collecting data from the converting line. Each video stream is analyzed by embedded components capable of analyzing and detecting anomalies. When an event occurs, cameras automatically notify the SCADA, also collecting a video chunk documenting the circumstances. Through an adapter, the notification reaches the operators on their mobile devices (e.g. rugged phones or tablets). Operators can see the generated alarm and explore the line using augment content. In particular, an AR experience is offered where the operational parameters of the line in real-time are made available as tags. Interactive troubleshooting is also provided to help the operator find a solution or start the necessary maintenance of the line. Tests of the AR system have shown encouraging results [Coscetti et al., 2019; Coscetti et al., 2020] and a demonstrator has been implemented [Moroni et al., 2022].

### 2.2 AI and acoustic for predictive maintenance and adaptive control

It is common to say that an engine or machinery sings, meaning that the sound they emit is representative of perfect operating behaviours. Noise and vibrations are hints correlated to the maintenance status of non-visible or not otherwise appreciable parts and components of machinery, such as bearings, or the process the machinery is performing, such as cutting and grinding a product. Thanks to artificial intelligence, acoustic and vibrational analysis in the factory environment
might provide relevant information about a plant during its
operation. For instance, information collected by acceleromo-
tric sensors might be used to assess the Remaining Useful
Life (RUL) of a bearing. At the same time, acoustic data ob-
tained by directional microphones permits the collection of
precise contactless information from the area of interest of a
machine. Ongoing experimentation shows that acoustic AI
might be an ally of industrial IoT applications, allowing pre-
dictive maintenance and adaptive control of production pro-
cesses.

## 2.3 AI and localisation services

In the factory environment, indoor localisation technologies
can support the exploration, monitoring and maintenance of
the plants, for instance, in conjunction with the methods re-
ported in Section 2.1. There are different generic technolo-
gies to achieve localisation based on radio frequency systems,
either active or passive, or vision methods. Within the latter,
the images acquired by a camera are used to locate oneself by
performing appropriate triangulation procedures with respect
to feature points identified in the images themselves. It is en-
visaged that indoor localisation technologies, such as Ultra-
Wide Band (UWB), might provide context-dependent infor-
mation and support employees in their work. An additional
dimension is investigated in the Smart Converting 4.0 project:
safety. Indeed, the combination of localisation services and
artificial intelligence can bring advanced safety features, es-
pecially in the presence of collaborative robotics, such as Au-
tomatic Guided Vehicles (AGV) or non-segregated robotic
arms. Trajectories of operators, robots and devices can be
tracked and analysed on a suitable multi-level architecture.
When a real-time response is needed, processing can be done
by *ad hoc* embedded devices directly connected to robots and
vehicles. In such a way, their behaviours can be promptly
modified, for instance, reducing the speed of a robotic arm or
avoiding a collision between an AGV and an operator. Track-
ing objects such as Personal Protective Equipment (PPE) can
also assure that operators are wearing adequate protection by
checking the correlation between human and PPEs trajecto-
ries. In general, pattern analysis on the set of trajectories and
business logic allow defining and applying adaptive safety
policies (see e.g. [Calabrò et al., 2019]). Collection of the
trajectory data and pattern analysis permits a better insight
into historical data, understanding the behaviour of opera-
tors, identifying areas where most of their effort is spent and
possible bottlenecks, and devising improvements for a more
efficient process. A demonstrator in the tissue converting sce-
narios is under active development.

## 2.4 Infrastructure monitoring

Monitoring critical infrastructures such as buildings, plants,
pipelines, powerlines and railways is a relevant application of
industrial and societal interest. The need for pervasive, more
accurate or continuous monitoring is increasing for improved
safety and more reliable access to services and resources. In-
telligent systems might help automate most tasks and per-
form extensive data analysis. At the SI-Lab, this is achieved
mainly through the introduction of smart cameras, to be de-
ployed as fixed nodes in a network for in situ observation,

or on vehicles, either Unmanned Aerial Vehicles (UAV) or
Underwater Autonomous Vehicles (UAV - see, e.g. [Allotta
et al., 2015]). Specifically, an application of artificial intelli-
gence at the edge has been showcased using UAVs equipped
with smart cameras in the domain of power line inspection.
An ad hoc deep learning model has been trained to detect
fault components in towers such as broken or corroded insu-
lators. Cameras on UAVs have also been used to assess the
health of buildings, for instance, monitoring the evolution of
cracks by using photogrammetric techniques [Germanese et
al., 2018]. In this case, the approach was integrated into an
IoT system for Structural Health Monitoring (SHM), featur-
ing a Virtual Reality (VR) interface for real-time interaction
with collected data [Bacco et al., 2020]. In railway monitor-
ing, edge computing and computer vision have also been used
to detect fast failures in remote areas [Fantini et al., 2015;
Leone et al., 2016] and analyse vehicular traffic around level
crossings in cities [Magrini et al., 2015].

## 2.5 Thermographic assessment

In industry 4.0, computer vision might be used for analyz-
ing images beyond the visible spectrum. At the SI-Lab, in-
frared imaging has been used to characterize the status of
conductors in powerlines. First, the location of cables was
detected through standard RGB images. Then, the condition
of wires was analyzed in the co-registered infrared images,
permitting to identify hot spots that are very likely repre-
sentative of damage [Jalil et al., 2019]. Other applications
include non-destructive testing and characterization of novel
electronic and photonic components [Hussain et al., 2020].

## 3 Conclusions

In this paper, we have presented a brief survey of research
directions in Industry 4.0 based on the convergence and in-
terplay between artificial intelligence, computer vision, per-
vasive computing and augmented reality. The presented case
studies show a wide application range of the proposed tech-
nologies in a variety of plants where the reliability of pro-
cesses and efficiency play a relevant role, both to guaran-
tee just in time production as constancy in the provision of
essential services. Proactive and predictive systems, inte-
grated with intelligent user interfaces for displaying context-
adaptive content, are seen as key in smooth interactions be-
tween humans and machines. Additionally, the theme of
safety is nowadays more and more relevant. We believe that
intelligent systems favouring a high level of safety, beyond
procedural prescriptions, should be encouraged and required
in modern production environments to allow for the coexis-
tence between humans and robots and drive the transition to
collaborative environments.

## References

[Allotta et al., 2015] Benedetto Allotta, Steven Baines, Fabio Bartolini, Fabio Bellavia, Carlo Colombo, Roberto Conti, Riccardo Costanzi, Can Dede, Marco Fanfani, Jonathan Gelli, et al. Design of a modular autonomous underwater vehicle for archaeological investigations. In *OCEANS 2015-Genova*, pages 1–5. IEEE, 2015.
[Bacco et al., 2020] Manlio Bacco, Paolo Barsocchi, Pietro Cassará, Danila Germanese, Alberto Gotta, Giuseppe Riccardo Leone, Davide Moroni, Maria Antonietta Pascali, e Marco Tampucci. Monitoring ancient buildings: real deployment of an iot system enhanced by uavs and virtual reality. *IEEE Access*, 8:50131-50148, 2020.

[Calabrò et al., 2019] Antonello Calabrò, Eda Marchetti, Davide Moroni, e Gabriele Pieri. A dynamic and scalable solution for improving daily life safety. In *Proceedings of the 2nd International Conference on Applications of Intelligent Systems*, pages 1-6, 2019.

[Coscetti et al., 2019] Simone Coscetti, Davide Moroni, Gabriele Pieri, e Marco Tampucci. Augmented reality for tissue converting maintenance. In *2019 15th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS)*, pages 585-590. IEEE, 2019.

[Coscetti et al., 2020] Simone Coscetti, Davide Moroni, Gabriele Pieri, e Marco Tampucci. Factory maintenance application using augmented reality. In *Proceedings of the 3rd International Conference on Applications of Intelligent Systems*, pages 1-6, 2020.

[Fantini et al., 2015] Andrea Fantini, Massimo Magrini, Salvatore Martino, Davide Moroni, Gabriele Pieri, Alberto Prestininzi, e Ovidio Salvetti. Experimenting an embedded-sensor network for early warning of natural risks due to fast failures along railways. In *IMTA*, pages 85-91, 2015.

[Germanese et al., 2018] Danila Germanese, Giuseppe Riccardo Leone, Davide Moroni, Maria Antonietta Pascali, e Marco Tampucci. Long-term monitoring of crack patterns in historic structures using uavs and planar markers: a preliminary study. *Journal of Imaging*, 4(8):99, 2018.

[Hussain et al., 2020] Bilal Hussain, Bushra Jalil, Maria Antonietta Pascali, Muhammad Imran, Giovanni Serafino, Davide Moroni, e Paolo Ghelfi. Thermal vulnerability detection in integrated electronic and photonic circuits using infrared thermography. *Applied Optics*, 59(17):E97-E106, 2020.

[Jalil et al., 2019] B Jalil, MA Pascali, GR Leone, M Martinelli, D Moroni, O Salvetti, e A Berton. Visible and infrared imaging based inspection of power installation. *Pattern Recognition and Image Analysis*, 29(1):35-41, 2019.

[Leone et al., 2016] Giuseppe Riccardo Leone, Massimo Magrini, Davide Moroni, Gabriele Pieri, Ovidio Salvetti, e Marco Tampucci. A smart device for monitoring railway tracks in remote areas. In *2016 International Workshop on Computational Intelligence for Multimedia Understanding (IWCIM)*, pages 1-5. IEEE, 2016.

[Magrini et al., 2015] Massimo Magrini, Davide Moroni, Giovanni Palazzese, Gabriele Pieri, Ovidio Salvetti, Daniele Azzarelli, e Antonio Spada. An infrastructure for integrated management of urban railway crossing areas. In *2015 IEEE 18th International Conference on Intelligent Transportation Systems*, pages 42-47. IEEE, 2015.

[Moroni et al., 2022] Davide Moroni, Gabriele Pieri, Marco Tampucci, e Daniele Masini. ARTiCo - AR in tissue

converting. In *PERCOM 2022, accepted for publication*, 2022.