# Live Video Assistance Systems for Assembly Processes

Christian Kittl, Markus Streibl

► To cite this version:

Christian Kittl, Markus Streibl. Live Video Assistance Systems for Assembly Processes. 9th International Precision Assembly Seminar (IPAS), Dec 2020, Held virtually, Unknown Region. pp.324-331, 10.1007/978-3-030-72632-4_24. hal-03520399

HAL Id: hal-03520399

https://inria.hal.science/hal-03520399

Submitted on 11 Jan 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Live Video Assistance Systems for Assembly Processes

Christian Kittl¹, Markus Streibl²

¹ evolaris next level GmbH

² Know-Center GmbH, Research Center for Data-Driven Business & Big Data Analytics

christian.kittl@evolaris.net, mstreibl@know-center.at

**Abstract.**

Work processes and assembly processes are increasingly gaining in complexity in the industrial context and demand a wealth of knowledge from assembly employees, as well as from service and maintenance personnel. The article describes a system developed in order to support assembly workers using a live video assistance system in combination with "wearables" - in particular smart glasses - in complex assembly processes by experts and reports findings from an acceptance analysis study.

**Keywords:** Assembly Processes, Digital Assistance System, Wearables, Video Support System.

## 1 Introduction and motivation

Production processes, which significantly contribute to the company's success and the competitiveness of a company, have become more complex and, not least due to digitization in industry, are demanding a change in the training and qualification of employees. Digital assistance systems provide an interface for human-machine interaction and are designed to support employees in complex tasks. The Graz innovation center EVOLARIS is developing, among other things as part of the MMASSIST II research project, a system that supports the shop floor worker in such a way that, when complex installation or maintenance work occurs, experts use live video to monitor and guide the work process, thereby being able to successfully manage high complexity of the task. This on one hand should lead to decreased time that an expert is required to be on site to fix a problem resulting especially in reduced time-to-fix, availability, travel costs, and CO2 footprint, and on the other hand improve overall process quality. Systems like WhatsApp or Skype can be used on some cases, but often production companies require dedicated solutions, which work with industry grade hardware including smart glasses, have a specialised set of functionalities that are really required by professional workers and easy to use in stress situations, where data can be stored on serviced on premise or in a local data center, and/or the solutions can be personalized/white labelled. Since such systems are still new and there is little experience in practice with regard to their acceptance in the target group and the impact on process efficiency they have, a study was conducted in a survey of users who have already had several months of experience with such a system during a pilot phase.
## 2 Digital Assistance Systems in Production Processes

Since the eighteenth century, the industry has been undergoing constant change, which has currently reached its peak with the fourth industrial revolution, the networking of intelligent technical systems known as cyber-physical systems (CPS). The basic idea behind networked production was to make it more efficient and flexible by means of forecasts and forecasts. From the data obtained from assembly processes and the associated systems, decision-critical information can be obtained for the company with the appropriate data quality. Digital assistance systems apply this information to assist workers in their actions and their extended use in human-machine interaction, with humans increasingly taking on a supervisory role (Haase et al., 2015). In parallel to the increasing human-machine inter-action, however, the need for rapid knowledge transfer between specialists is also increasing, in order to be able to quickly clarify special questions in the use of increasingly complex machines. The innovation center EVOLARIS is developing such an assistance system, which supports the strategy of "networked" employees via live video remote support.

### 2.1 Live Video Assistance System EVOCALL

Live Video Assistance Systems enable real-time communication between people in different locations, using cameras in parallel with the audio channel and transmitting their images in real-time to the remote station. Often, smart glasses are used in which information is displayed in the field of view of the user, and a camera is integrated to record and transmit the view range which is currently being focused on to the communication partner (cmp. Fig. 1)
<figure><img src="image_2.png" /><figcaption>Fig. 1. Smart glasses as part of industrial live video assistance systems</figcaption></figure>

EVOLARIS' live video remote system EVOCALL uses an open source real-time communication technology called Web Real-Time Communication (WebRTC), which is designed to work in conjunction with the most popular consumer devices. The data transfer takes place by means of the DTLS protocol (Datagram Transport Layer Security), which makes a "man in the middle" attack on the open channel, or a manipulation of the data impossible. As can be seen in Fig. 1, the actual WEBRTC server is only needed for the initial connection setup between the individual subscribers. The actual communication, which is additionally secured by SRTP (secure real-time protocol) video and audio encryption, takes place via a separate media server.
<figure><img src="image_3.png" /><figcaption>Fig. 2. System architecture of the EVOLARIS live video support system using WebRTC protocol</figcaption></figure>

## 2.2 Live Video Assistance Systems in Assembly Processes

In the field of production, there is an increasing complexity of the systems working in the collaborative system, as well as their influence on the work to be carried out and the necessary training and qualification of the service / maintenance staff and assembly employees (Spöttl 2017). Streibl and Brandl (2016) cite the potential application of Live Video Assistance Systems for unscheduled repair with high complexity, thereby restoring system availability in the shortest possible time. The assembly or service or maintenance personnel set up an audio-visual connection with experts with the aid of a smart glasses (cmp Fig. 1) with the digital system EVOCALL as a basic system. These subject experts receive the live stream of the employees on site from a "first-person view" on the respective terminal. The system now offers the ability to not only describe and guide complex issues using an audio channel, but to capture them in a still image that can be generated from the live video

## 2.3 Live Video Assistance Systems for Shadow-Working

In this context, the system can be used by new employees or leasing workers who, for example, have to carry out a complex conversion process for the first time. The machine operator, who is positioned at a different location, has the same machine in his line and now has the possibility, using the Live Video Remote System, to carry out the necessary steps and to transfer these to the data glasses of the employees. The "first-person view"
and the additional audio stream enable employees to imitate the activities to be per-
formed. By changing the streaming direction, whereby the video transmitted by the
machine expert is no longer displayed in the data glasses of the employees, but now the
employees transmit the live video, new possibilities open up in the context of the
"shadow-working" process, which is also used in the Field of assembly processes
(Frohberg et al 2009, Pimmer & Pachler 2014).

3 Acceptance of Live Video Remote Support Systems

This section presents the results of an online survey on system acceptance, which was
conducted as part of a master thesis accompanying the system development. The survey
was aimed at employees in the areas of production, service and maintenance from 15
well-known Austrian, internationally operating companies on their experience with the
use of the EVOCALL live video remote support system. The companies were selected
according to the criteria of the necessary knowledge transfer, as well as an on-site pres-
ence of experts to fulfil the service offered. The participating companies fulfilled both
criteria to a high degree and where given access to the EVOCALL system for the dura-
tion of 6 months in a trial phase. Of the 200 target people addressed, 50 (25%) partici-
pated in the survey. The findings presented here relate in a first step to the dimension
of the service in terms of the usefulness and usability of the system, but also in relation
to the efficiency and the effectiveness of the processes when using such a digital assis-
tance system. As can be seen in Fig. 3, of more than 70% of the participants, the di-
mension of perceived usefulness is rated on a five-point scale, where 1 means "applies
strongly" and 5 "does not apply at all", to 1 or 2. The points of view of perceived
usefulness include a comparison between the experiences made and the expectations
(Question 1), the usefulness of the system (Question 2), the simplicity of the system
(Question 3), and the usability of the system (Question 4).

These results suggest a very high perceived benefit and high acceptance of the live
video assistance system by the participants.
<figure><img src="image_4.png" /><figcaption>Fig. 3. Perceived Usefulness of the EVOCALL remote video support system</figcaption></figure>

The next two questions related to the degree of goal achievement, the result compared to the original goal (effectiveness) and the degree of efficiency, as well as the result versus effort (efficiency) (Fig. 4). More than 90% of respondents believe that using EVOCALL can increase their effectiveness. More than 80% of the respondents that EVOCALL can increase efficiency. The effectiveness in this context refers to problems that exceeded the abilities of the employees and could still be solved through the use of EVOCALL via the assistance of colleagues. The efficiency refers to an accelerated problem-solving cycle and the resulting increase in plant availability.
<figure><img src="image_5.png" /><figcaption>Fig. 4.: Efficiency and effectiveness of the EVOCALL remote video support system</figcaption></figure>

Further results of the online survey showed that service and maintenance work is characterised by complex and varied activities and that the employees require extensive training. This complexity of the plant requires distinctive communication on different channels when assistance is needed. These channels, however, have limitations with regard to delayed or unclearly assignable responses due to asynchronous communication channels (e-mail or WhatsApp), or more difficult problem identification with regard to problem description due to lack of qualifications by local staff and language barriers (telephone). This results in an increased presence of experts on site, which is reflected in the duration of the repair time and thus in the availability of the system.

As can be seen from the answers reflected in the two figures above, the effectiveness and efficiency of the work processes can be positively influenced by the use of EVOCALL. Synchronous audio-visual communication enables the employees to carry out more highly qualified work processes under the guidance of experts via EVOCALL, which can lead to a reduced on-site presence of experts according to the results of the surveys.

## 4 Summary and conclusion

The change in the industry and its effects on prevailing assembly processes as well as on the changed qualification needs of employees requires the use of new methods and technologies such as digital assistance systems as an interface for a human-machine interaction or as an interface between employees. The analysis of the live video remote support system EVOCALL as such a digital assistance system has shown that the use
of production and service processes can be positively influenced. The results from companies that have already tested EVOCALL indicate that live video remote support systems have a high perceived value for users and have a positive impact on the efficiency and effectiveness of task completion. Summarizing the collected results, they clearly indicate that EVOCALL has a positive impact on the current service and maintenance processes, improving communication, reducing the need for on-site presence of experts, and having a positive impact on the problem-solving cycle and the problem-solving skills of service and maintenance personnel. The survey provides new insights by quantifying the extent to which users, who had the opportunity to experience a live video remote support system in practice over a longer period in time (6-monthly trial).

However, due to the relatively small number of cases (n = 50) and the specificity of the tested system, the findings can not be generalized without further notice. Much depends in practice on the actual implementation of a live video remote support system and how easy it is for the users to use or what features are actually supported and how the integration into the general enterprise IT takes place. Through studies with larger case numbers and through the comparative analysis of various systems, a more general and even better reliable statement about the benefits of such digital assistance systems in production is to be made possible in the future, or the existing findings will be verified.

## 5 Acknowledgements

This work was created with support from the 'Production of the Future' program of the Austria Research Promotion Agency (FFG) in the course of the lighthouse project MMASSIST II and the Know-Center within the COMET K2 - Competence Center for Excellent Technologies program of the Austrian Ministry for Transport, Innovation and Technology (bmvit), the Austrian Ministry for Digital and Economic Affairs (bmdw), the Austria Research Promotion Agency and the Styrian Research Promotion Agency (SFG).

## References

Abicht, L., & Spöttl, G. (2012). *Qualifikationsentwicklung durch das Internet der Dinge (Development of Qualification through the Internet of Things)*. Bielefeld: W.Bertelsmann Verlag GmbH & Co. KG.

Brandl, P., Aschbacher, H. & Hösch, S., (2015). *Mobiles Wissensmanagement in der Industrie 4.0 (Mobile Knowledge Management in the Industry 4.0)*. Mensch und Computer 2015 – Workhopband. Berlin: De Gruyter Oldenbourg, S. 225-232.

Frohberg, D., Goth, C. & Schwabe, G. (2009). *Mobile learning projects: a critical analysis of the state of the art*. Journal of Computer Assisted Learning, 25, 307-331.
Haase, T., Termath, W., & Schumann, M. (2015). *Integrierte Lern- und Assistenzsysteme für die Produktion von morgen (Integrated learning and assistance systems for the production of tomorrow)*. Lehren und Lernen für die moderne Arbeitswelt. Berlin, 183-207.

Pimmer, C. & Pachler, N. (2014). *Mobile Learning in the Workplace: Unlocking the Value of Mobile Technology for Work-Based Education*. Increasing Access through Mobile Learning, Athabasca University, 2014, pp. 193-203.

Reichel, J., Müller, G., & Mandelartz, J. (2009). *Betriebliche Instandhaltung (Operational Maintenance)*. Berlin Heidelberg: Springer Verlag

Streibl, M. (2017). *Einsatz eines Live-Video-Assistenzsystem im Problemlösungszyklus von Service- und Instandhaltungspersonal (Application of a live video assistance system in the problem solving cycle of service and mainentance personnel)*. Master thesis, University of Applied Sciences Campus02, Graz

Spöttl, G. (2017). *Industrie 4.0 - Herausforderung für Lehrbetriebe (Industry 4.0 – Challenges for training companies)*. In M. Frenz, C. Schlick, & T. Unger (Hrsg.), Wandel der Erwerbsarbeit (S. 60–75). LIT Verlag Münster.