# Supporting Resilient Operator 5.0: An Augmented Softbot Approach

Lara P. Zambiasi, Ricardo J. Rabelo, Saulo P. Zambiasi, Rafael Lizot

<figure><img src="image_1.png" /></figure>

## To cite this version:

Lara P. Zambiasi, Ricardo J. Rabelo, Saulo P. Zambiasi, Rafael Lizot. Supporting Resilient Operator 5.0: An Augmented Softbot Approach. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2022, Gyeongju, South Korea. pp.494-502, 10.1007/978-3-031-16411-8_57. hal-04768956

HAL Id: hal-04768956

https://inria.hal.science/hal-04768956v1

Submitted on 6 Nov 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Supporting Resilient Operator 5.0:
an Augmented Softbot Approach

Lara P. Zambiasi<fnref n="1" />[0000-0003-4644-3184], Ricardo J. Rabelo<fnref n="2" />[0000-0002-5488-3692],
Saulo P. Zambiasi<fnref n="3" />[0000-0002-7694-4871], Rafael Lizot<fnref n="1" />

**Abstract.** Industry 4.0 and 5.0 have been posing new challenges to industries. From a focus on supporting resilience at a corporate level, there is a need to do it at a more operational level, enabling people to act with resilience as well. The *resilient operator 5.0* is a new concept emerged from this need. It has the aim of providing more intuitive, symbiotic, human-centered, and cognitive working computing environments to enhance human adaptation capabilities, productivity, and mental health. In this direction, this paper presents an approach that combines softbots and augmented reality, called ‘*augmented softbot*’. Looking at a specific company, a software prototype has been implemented to evaluate how this approach can be useful and feasible for preventive maintenance. Three scenarios have been devised for that, and they are summarized in the paper. The achieved results are discussed, showing the high potential of the approach.

**Keywords:** Industry 5.0, Industry 4.0, Softbots, Virtual Assistants, Augmented Reality, Resilient Operator 5.0, Operator 4.0, Maintenance.

# 1 Introduction

A key goal in the Industry 4.0 model refers to building smart resilient manufacturing systems [1]. In this context, resilience relates to “*the ability of manufacturing activities to withstand or quickly recover from operational disruptions that pose as threats to the continuity of manufacturing operations at the desired level*” [2].

Industry 5.0 also includes being resilient, but it extends it to the companies' work-force, via a socially and cognitively sustainable environment [1], putting people in the (control) loop. This means transforming people into protagonists of several activities, making them to act more collaboratively, enabled by more symbiotic human-machine interactions, and strongly supported by advanced technologies [1, 4].

At shop floor level, literature has presented different approaches to support this higher symbiosis. This work focuses on the use of softbots, a kind of virtual assistant software that helps humans in the execution of tasks, and whose potentials in manufacturing have been demonstrated in some works [e.g., 3]. More recently, augmented reality has arisen as another powerful technology to increase that symbiosis in manufacturing [4]. The union of these two technologies – to what we call as ‘*augmented softbot*’

<footnote n="1">¹ IFSC - Federal Institute of Santa Catarina, Chapecó, Brazil
lara.oberderfer@gmail.com; rafalizot10@gmail.com</footnote>

<footnote n="2">² UFSC - Federal University of Santa Catarina, Florianópolis, Brazil
ricardo.rabelo@ufsc.br</footnote>

<footnote n="3">³ UNISUL - University of Southern Santa Catarina, Florianópolis, Brazil
saulopz@gmail.com</footnote>
- can provide an environment where operators can be guided or interact with cyber-physical systems via voice and real-life-like images in the execution of tasks. This can decrease operation errors and time as well as increase productivity, continuous learning and improvement, and human satisfaction [1, 3, 4]. Despite this potential, no works have been found out in the literature or in commercial products with this combination in the industrial context.

This paper has the goal of showing some industrial shop floor scenarios where softbots and augmented reality are combined to create a working cognitive environment for the so-called "Resilient Operator 5.0" [1], extending the concept of 'augmented operator' [5]. In terms of domain problem, this work has focused on preventive maintenance, one of the most relevant issues being dealt by industries in their chasing for higher operational efficiency, zero error, and lower production costs [6]. The importance of maintenance operators to be assisted by such technologies relies on an increasingly complexity in their daily activities, which includes: continuous training as new products and versions are developed; manual maintenance in several and very different equipment; planning activities and lots of information to be constantly checked about each equipment; the different levels of expertise and operators experience, leading to longer maintenance times and more errors; among others [6, 7].

This work has adopted the *Action-Research* methodology, which aimed at interactively and incrementally developing a Proof-of-Concept (PoC) software prototype to demonstrate the envisaged concept for a particular company; in this case, on how softbots and augmented reality can be a suitable approach for supporting the resilient operator 5.0 in the maintenance of industrial equipment.

This company is a medium-sized enterprise placed in the State of Santa Catarina, Brazil, specialized in producing customized industrial automated assembling solutions for the food sector, attending customers from Brazil and other countries. A machine called "hamburger cartoning automated system" has been chosen for the PoC. This machine is responsible for receiving ready pieces of raw hamburgers and packing them into ready-to-deliver paper boxes. This PoC is one of the company's initiatives towards developing *smart machines* to the market.

This article is organized as follows: section 1 highlights the problem paper's goal. Section 2 summarizes the main theoretical fundamentals related to the proposed approach as well as related works. Section 3 describes the experimental setup, implementation cases, and results. Section 4 presents the preliminary findings of this research and its next short-term steps.

## 2 Basic Foundation and Related Works

The term Resilient Operator 5.0 is defined as "*a versatile and resourceful worker who uses creativity, skill and innovation enhanced by information and technology as a way to break paradigms in search of generating economic innovations that ensure the sustainability and well-being of operation tasks in the face of possible unexpected or difficult conditions that may encounter*" [1].

Augmented Reality (AR) can be defined as a human-machine interaction technology to supplement a real-world environment with computer-generated inputs with the aim of enhancing users' perception of reality and enriching the provided information content. It overlays computer-generated information on the real-world environment, and
can be described as a set of three key features: combination of real and virtual objects in a real environment; real-time interaction with the system, able to react to user's inputs; geometrical alignment of virtual objects to real ones in the real world [8].

A softbot can be defined as "a virtual system deployed in a given computing environment that automates and helps humans in the execution of tasks by combining capabilities of conversation-like interaction, system intelligence, autonomy, proactivity, and process automation" [3]. It is the one that interfaces humans and a target system. A softbot can represent, act on behalf of, or respond to one or several users, computing systems, digital twins, and industrial equipment [9].

In terms of related works, a systematic literature review (SLR) has been carried out over the most important scientific databases looking for papers written in English that have addressed the equivalent problem and technologies in the last ten years.

More than 100 papers have been found out, but all of them covered only partial aspects of the envisaged approach. For example, papers combining softbots with maintenance; maintenance with AR; AR as a powerful technology for the operator 4.0 and 5.0; etc. Yet, most of the papers are conceptual, without computing implementations. Actually, and from the best of our knowledge, only one work [6] has suggested that combination, but “only” proposing a comprehensive conceptual architecture for that.

From a broader perspective on AR, Egger et al. [10] evaluated that its interfaces should allow two-way communication in order to be more effective. Wang et al. [11] seems to be the first one who proposed an AR-based system to support hands-free actions. Majewski et al. [12] have introduced speech recognition in AR-based systems. Palmarini et al. [7] found out, in their literature review on AR applications for manufacturing maintenance, that 2D and 3D visualizations combined with text or audio is the most common approach. It could be also observed from the SLR that the implemented AR mechanisms are basically reactive, responding to users as they request information. The approach with *augmented softbots* includes automatic and proactive actions to warn and guide users before and during their maintenance operation.

Some works can be mentioned here to briefly illustrate how AR or softbots have been viewed as useful technologies to help operators at industrial shop floors.

Fite-Georgel [13] and Mourtzis et al. [4] have discussed the potential of AR and other technologies to better support workers in their production environments. Hořejši et al. [14] evaluated that the available smart glasses present some hardware limitations, but advocate that they will be a strong candidate tool in future smart factories, combined with the use of voice commands. In this line, König et al. [15] implemented a prototype system using *Microsoft HoloLens* with AR and data glove, in which an inexperienced operator has been guided in an assembly process. Almeida et al. [16] proposed an approach where operators are provided with voice-enacted real-time reports and logs about the actions they should do and did as a means to increase operational efficiency and to decrease error rates.

The research of Rabelo et al. [2, 4] showed the use of softbots to help in creating more symbiotic human-machine interfaces close to industrial cyber-physical systems (CPS) and CPS's digital twin as well as a means to support CPS's autonomy and inter-CPS interactions.
# 3 Implementation Scenarios and Results

The first part of the work was related to identifying the needs and requirements in terms of how to do maintenance using the potentials of the proposed approach. After meetings with the company, a cartoning system module called “Cam” has been selected given its complexity and large number of parts (44), which require professionals with large experience to make maintenance on it.

The main professional responsible for doing the maintenance in this equipment helped to refine requirements and needs. This made possible the identification of the modeling needs, the dynamics of the softbot-AR model, and the ergonomics of the user interface and dialogs. Scenarios were devised based on this, being further implemented.

## 3.1 Implementation Resources and PoC Setup

The prototype has been developed as a 3D AR model. The cartoning system's parts had already been designed by the company in a CAD system (Solid Works). After being converted into a proper AR format, colors and textures were added to the parts using the *Blender* software. The whole environment was implemented within the *Unity* framework/tool, *Vuforia* software was used to get the modeled parts and to load them in the Unity environment. Once there, all the parts' animations and choreographies, the design of buttons and heads-up displaying, functionalities programming (via scripts coded in C#), etc., can be implemented.

This prototype was devised as an App to run on a smartphone, although the development environment supports other devices, such as tablets and smart glasses.

In what the softbot implementation is concerned, the *ARISA Nest* web platform [3] has been used. It allows the derivation of particular instances of service-oriented bots for given domains and their deployment in the cloud. The interaction between the derived softbot with the App is done through a web service developed in the *GO* programming language using the *REST* standard.

In general, when the softbot needs something (previously designed) from the App, a message is sent out to it, and vice-versa. This message can be sent due to a direct user's request, or be automatically and internally sent between the softbot and the App depending on the softbot's behavior associated with the desired action (see next section). Some description on how messages are modeled and processed can be found in [3] as well as how the softbot interacts with the equipment. Figure 1 shows the general architecture implemented in this PoC.

<figure><img src="image_4.png" /><figcaption>**Fig. 1.** General systems architecture</figcaption></figure>
The operator uses the App for visualizations and interactions, and the softbot for voice and text interactions. The App's *engine* module centralizes all the general actions and reasoning. At the right side of fig. 1 the main AR's GUI is showed (in Portuguese). The operator can ask for some information [“*informações*”] (e.g., see fig. 4). The option *verification* [“*verificações*”] allows checking the most important maintenance operations to be done. The option *disassembly* [“*desmontar*”] graphically shows how the *Cam* should be correctly disassembled for maintenance purposes. The option *simulation* [“*simulação*”] shows how the *Cam* can be assembled. Finally, the option *maintenance* [“*manutenção*”] shows a step-by-step process how to do the maintenance itself.

## 3.2 Implementation Scenarios

There are many possible scenarios related to how maintenance operators can be assisted by AR with or without softbots, as discussed in e.g. [1, 3, 4, 5, 7].

Considering the main goals of this PoC for the target company, three scenarios have been devised and implemented having the Resilient Operator 5.0 [1] in mind<fnref n="1" />.

These scenarios also tried to exploit some of the *Arisa Nest*'s potentials, where softbots can be configured to have three behaviors: (i) *reactive*, allowing the softbot to act in response to direct user's requests via chatting or voice (e.g., to ask about some specific information from a given equipment); (ii) *planned*, allowing the softbot to act in response to (predefined) scheduled tasks, bringing their results to users after their execution (e.g., to generate consolidated equipment performance reports after every working shift); and (iii) *pro-active*, allowing the softbot to perform predefined tasks autonomously on behalf of users or of the equipment(s) it represents, bringing their results to users - if needed (e.g., to continuously check sensors' information, to anticipate problems and to promptly take measures to solve them, to send warnings and alarms, etc.).

**Proactive Behavior Scenario.** This scenario refers to when maintenance operators leave the maintenance office and go to the shop floor for their daily routine after knowing which machines should be checked and maintained. During their walk the machine's softbot warns them that the respective machine has an issue after receiving some information from sensors. This warning is sent out in an AR way (Fig. 2) when the softbot detects that a maintenance operator is close by the machine (this automatic 'detection' has been so far implemented only in a simulated way), or via sending a SMS message to the operator's smartphone.

<figure><img src="image_5.png" /><figcaption>**Fig. 2.** Proactive case, warning about the machine's situation</figcaption></figure>

<footnote n="1">¹ A partial demo can be visualized (menus are written in Portuguese) at https://youtu.be/V6SmVK4-R9Y</footnote>
**Reactive Behavior Scenario.** This scenario refers to when the operator is doing the maintenance, starts an interaction with the softbot, and has some doubts about what or how to do, whether experienced or beginner operators. A very simple ‘expert system’ inside the softbot is implemented, guiding the operator during the operation. All interactions happen via the AR environment, running on the operator’s smartphone (Fig. 3 left side), and they can be made both via voice and text.

In the case the operator wants more information about some specific aspect, an auxiliary interface pops up; in this case, the operator needs information about the ‘casquilho’ part and the explanation is showed (Fig. 3 right side).

<figure><img src="image_6.png" /><figcaption>Fig. 3. Reactive case, assisting the operator during a maintenance</figcaption></figure>

**Planned Behavior Scenario.** This scenario refers to when the operator is about to end his shift and wants to check the current status of the maintenance activities against what was planned. Reports are made available via the App and are automatically generated by the softbot, as planned. As Arisa Nest does not support *per se* graphical interfaces, the reports are stored and accessed at the derived softbot’s web area (fig. 4).

## 3.3 Implementation Evaluation

Being a PoC tested in a controlled environment, more formal quantitative performance indicators could not be applied and measured (for example, to measure the decrease of number of errors, the decrease in the maintenance time, etc.).

The designed functionalities worked as planned. The main limitation of this PoC was that the operator had to hold the mobile phone with the hands to do the maintenance.

From a more qualitative and general preliminary evaluation, some operators stated that this creates a more symbiotic operation environment. It can provide more confidence when they do the maintenance as they would be guided, warned, or aided by a correct and up to date maintenance information as machines are many and different. This may decrease their mental stress and perhaps decrease training time and costs.

In the case of the particular manufacturer industry which the PoC was developed for, this could decrease enormously the need for the company’s employees to travel to customers for maintenance support as the customers’ operators would be supported by the software. This means saving global maintenance and waiting times as well as costs from both sides when customers have problems in some equipment.
<figure><img src="image_7.png" /><figcaption>Fig. 4. Planned case, generating reports automatically</figcaption></figure>

## 4 Conclusions

This article has presented a PoC implementation about how augmented softbots can be used as a viable approach to support the *Resilient Operator 5.0* concept at shop floor.

Three scenarios have been devised to evaluate the approach. The preliminary results confirmed its potential, in which the integration of a softbot with AR could help overcoming some of the challenges operators used to face in maintenance activities on industrial equipment. This includes the possibility to do some maintenance via voice and AR without using their hands (although this was not implemented), and without being very expert on given industrial machines. For example, the conversation with the softbot provided step-by-step instructions to the operator while showing the exact location of the actions for doing preventive maintenance.

Solutions based on the proposed approach can be helpful to train operators, both experienced and beginners, decreasing training time and hence costs. Besides this, they decrease the enormous effort and mental stress of maintenance operators to keep very up to date about every different equipment as new products, versions, automation devices, etc., are often introduced in the industry and so in the way maintenance is done.

Modeling the diverse parts of a single equipment in a CAD system and dealing with the design itself and implementation of the AR application takes time and requires specialized people for doing that. For example, the implementation of this prototype (with all the parts being previously designed in a CAD system) took about 150 hours of work. Once deployed, and even having a good level of intuitiveness, solutions like that require training of operators in “using diverse tools” in their daily maintenance activities, which may be an additional point of stress.

The main next steps of this research are twofold. Firstly, there is a need to make more tests with AR-related technologies and products in wider scenarios. Secondly, to add machine learning techniques in the softbot so that it can adapt its answers and actions according to operators’ experience as well as it can generate performance operation reports for identifying training needs.

**Acknowledgments.** This work has been partially supported by CAPES Brazilian Agency for Higher Education, project PrInt “Automation 4.0”.
References

1. Romero D, Stahre J (2021) Towards the Resilient Operator 5.0: The Future of Work in Smart Resilient Manufacturing Systems. Procedia CIRP 104:1089-1094.
https://doi.org/10.1016/j.procir.2021.11.183

2. Kusiak A (2020) Open manufacturing: a design-for-resilience approach. Int J Prod Res 58:4647-4658. https://doi.org/10.1080/00207543.2020.1770894

3. Rabelo R J, Romero D, Zambiasi SP (2018) Softbots supporting the operator 4.0 at smart factory environments. IFIP Adv Inf Commun Technol 536:456-464.
https://doi.org/10.1007/978-3-319-99707-0_57

4. Mourtzis D, Angelopoulos J, Panopoulos N (2022) Operator 5.0: A Survey on Enabling Technologies and a Framework for Digital Manufacturing Based on Extended Reality. J Mach Eng 22:43-69. https://doi.org/10.36897/jme/147160

5. Romero D, Stahre J, Wuest T, et al (2016) Towards an operator 4.0 typology: A human-centric perspective on the fourth industrial revolution technologies. In: CIE 2016: 46th International Conf. on Computers and Industrial Engineering. Tianjin, China, pp 1-11

6. Wellsandt S, Klein K, Hribernik K, et al (2022) Hybrid-augmented intelligence in predictive maintenance with digital intelligent assistants. Annu Rev Control. https://doi.org/10.1016/j.arcontrol.2022.04.001

7. Palmarini R, Erkoyuncu JA, Roy R, Torabmostaedi H (2018) A systematic review of augmented reality applications in maintenance. Robot Comput Integr Manuf 49:215-228.
https://doi.org/10.1016/j.rcim.2017.06.002

8. Dini G, Mura MD (2015) Application of Augmented Reality Techniques in Through-life Engineering Services. In: Procedia CIRP. pp 14-23

9. Rabelo R J, Romero D, Zambiasi SP, Magalhães LC (2021) When Softbots Meet Digital Twins: Towards Supporting the Cognitive Operator 4.0. IFIP Adv Inf Commun Technol 634:37-47. https://doi.org/10.1007/978-3-030-85914-5_5

10. Egger J, Masood T (2020) Augmented reality in support of intelligent manufacturing – A systematic literature review. Comput Ind Eng 140:.
https://doi.org/10.1016/j.cie.2019.106195

11. Wang X, Ong SK, Nee AYC (2016) Multi-modal augmented-reality assembly guidance based on bare-hand interface. Adv Eng Informatics 3:406-421.
https://doi.org/10.1016/j.aei.2016.05.004

12. Majewski M, Kacalak W (2016) Human-machine speech-based interfaces with augmented reality and interactive systems for controlling mobile cranes. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 9812 LNCS:89-98.
https://doi.org/10.1007/978-3-319-43955-6_12

13. Fite-Georgel P (2011) Is there a reality in Industrial Augmented Reality? 2011 10th IEEE Int Symp Mix Augment Reality, ISMAR 2011 201-210.
https://doi.org/10.1109/ISMAR.2011.6092387

14. Hořejší P, Novikov K, Šimon M (2020) A Smart Factory in a Smart City: Virtual and Augmented Reality in a Smart Assembly Line. IEEE Access 8:94330-94340.
https://doi.org/10.1109/ACCESS.2020.2994650

15. Konig M, Stadlmaier M, Rusch T, et al (2019) MA2RA - Manual Assembly Augmented Reality Assistant. In: IEEE International Conference on Industrial Engineering and Engineering Management. pp 501-505

16. Almeida B, Bolas JM, Santos RJ, et al (2022) Application of Assisted Reality At Maintenance Activities: E-Redes' Pilot Project Results. 1260-1264.
https://doi.org/10.1049/icp.2021.2090