# Consistent Maintenance Order Processing with Digital Assistance Systems

Robert Rost, Axel Friedewald

► To cite this version:

Robert Rost, Axel Friedewald. Consistent Maintenance Order Processing with Digital Assistance Systems. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2021, Nantes, France. pp.102-111, 10.1007/978-3-030-85906-0_12 . hal-04022116

HAL Id: hal-04022116

https://inria.hal.science/hal-04022116

Submitted on 9 Mar 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Consistent Maintenance Order Processing with Digital Assistance Systems

Robert Rost [0000-0002-4290-5794] and Axel Friedewald

Hamburg University of Technology, Hamburg, Germany
{robert.rost, friedewald}@tuhh.de

**Abstract.** Augmented reality-based digital assistance systems have proven their potential to support with instructions considering numerous application scenarios in the product life cycle. For the maintenance of machines and plants, however, their use becomes particularly efficient if the diagnosis at the operator's site can be effectively supported, so that the service provider and his experts can make accurate diagnoses for this maintenance job, plan the execution including the spare parts logistics precisely, and document as well as billing the maintenance case with low effort. For this purpose, a modular digital assistance system is presented that provides support adapted to the process phases.

**Keywords:** Augmented Reality / Digital Assistance Systems / Maintenance

## 1 Introduction and Motivation

Performing maintenance and repair work with efficient processes and tools is a key success driver for their providers and users. For ship maintenance, for example, there are usually only short time intervals available when the ship is in port for loading and unloading. If not all the required spare parts are available in this narrow time window, a further journey by the service technician becomes necessary, as delays lead to increasing costs and decreasing customer satisfaction with the service [1]. An early assessment of the defect enables the service technician to plan the work in detail in advance and to schedule the necessary spare parts. In addition, all the relevant information must be readily available on site and subsequently documented in a structured way with minimum effort. This increases transparency for the customer and enables internal evaluations to improve service quality control loops.

Digital assistance systems represent an established means of process optimization in the described field. A special form of assistance systems are augmented reality-based systems, which are being intensively developed in research both for assembly [e.g. 2, 3, 4] and for maintenance [e.g. 5, 6]. Remote support receives a particular emphasis in numerous industrial applications, as it supports inexperienced users through contact with experts in the control center, leveraging visual cues and audio instructions during maintenance execution [e.g. 7, 8].

The following section presents a more comprehensive end-to-end approach that supports the entire order processing, from diagnosis to work documentation, and is based
on the assumption that it will be carried out by experienced service technicians, thus
setting different priorities compared with "classic" remote support solutions. The hard-
ware primarily used for this purpose are tablets and smartphones, as they are familiar
to potential users and in this case offer greater potential than hardware found in the
hands-free environment. The article firstly clarifies the individual phases of the mainte-
nance process using a ship engine as example and presents the necessary system archi-
tecture to support them. Subsequently, the implementation for the individual phases is
explained and the economic potential is demonstrated.

## 2 System Architecture

Different stakeholders on the customer and the service provider side are involved in
processing a maintenance order and they must be supported in different ways according
to their activities in the overall process. Those activities therefore give rise to require-
ments for the development and design of a holistic assistance environment, which
should be characterized by precise issue localization and continuous information flows
without media discontinuities.

### 2.1 Maintenance Order Processing and its Requirements

Figure 1 shows the exemplary processing of a maintenance order. The order is triggered by the occurrence of a defect on the maintenance object, in this example a marine engine. To do this, the service provider must first be supplied with symptoms presented as extensively as possible.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Maintenance Order Processing</figcaption></figure>

Then a diagnosis must be carried out so that corrective measures can be determined
correctly. At the same time, a maintenance order is generated to reference processing
and billing down the line. The service provider's technicians then organize the spare
part logistics and travel arrangements. Using the customer's symptom survey and the
maintenance history, the service technician carries out the case-specific work plan for
the maintenance task and documents working times, work results and special features.
Based on this, an activity log is created and confirmed by the customer as an elementary
part of the invoicing for the order and further case evaluation by the service specialist.
Numerous process reviews with service technicians and interviews with operations
managers have resulted in the following requirements with respect to the process above:

* **Symptom recording.** The less experienced onboard technician requires extensive support in symptom recording by means of a predefined symptom hierarchy, if necessary with example images or sounds. Furthermore, it must be possible to locate the symptoms for unequivocal identification.

* **Guidance.** If this support is not sufficient, support of the onboard technician in the form of remote support by experts from the service provider may be necessary. Navigation instructions and guidance are required, e.g., to perform certain measurements.

* **Expert dialog.** For the preparation of the maintenance execution, the location-specific knowledge of the symptoms is helpful. In difficult cases, the definition of remedial activities is already organized today via internal groups in relevant messengers. This expert dialog can be further optimized by a joint case review in the 3D model.

* **Connection to spare parts catalog and maintenance history.** The integration of the assistance system into the corresponding corporate IT (in particular the spare parts catalog and the maintenance history) is indispensable for end-to-end order processing.

* **Automated documentation and report generation.** Today, the effort required to log maintenance work is characterized by handwritten notes, photos and subsequent the time-consuming compilation of the report. At the same time, this leads to high expenses and a high error potential when preparing future maintenance measures on the same machine. Hence, a reduction in effort must be achieved through partial automation and structuring.

## 2.2 Platform Components

Augmented reality (AR) can be used for a location-based representation of work in-
structions, leveraging a digital assistance system that overlays real environments with
virtual content while maintaining a constant positional reference between those two and
allowing the user to interact with this virtual content in real-time [9]. The main reason
for using augmented reality in this context is to provide an intuitive and highly produc-
tive [4] environment for on-board technicians to perform diagnostic activities. How-
ever, for holistic process support, the system needs to be integrated into the corporate
IT, and needs to be optimized regarding the user interaction for the specific sub-pro-
cesses of maintenance. This applies in particular a diagnosis assistant, communication
via cooperation modules and extensive automation of reporting and documentation.
The overall system (Figure 2) for end-to-end maintenance assistance can be configured
in three main components for this purpose [10]:
* **Creator.** A desktop-based authoring system for effortless creation of augmented reality-capable maintenance instructions. The creation of textual information is based on templates, geometrical animations can be created automatically by the system but as well individually with the help of an additional virtual reality (VR) module [11].

* **Visualizer.** An augmented reality-based diagnostic assistant for on-board technicians to easily capture symptoms. Furthermore it serves as a digital service document for service technicians to provide information during the execution as well as automatic documentation for later reporting. At the same time it performs as a communication tool for live collaboration with a remote expert.

* **Director.** The server backend for the storage, provision and processing (especially of collaborative real-time sessions) of both diagnostic and maintenance runs, that serves as a connector to the company's internal spare parts management, maintenance history and business order processing. It provides extended functions for the conversion of existing documents as well as for the preparation of 3D models for the Creator and Visualizer (especially VR / AR specific).

<figure><img src="image_4.png" /><figcaption>Fig. 2. DIA Platform [10]</figcaption></figure>

## 3 Diagnostic Assistant for On-Board Technicians

In order to provide the service technician with a precise view of the maintenance task as comprehensively as possible in advance, the on-board personnel must be enabled to describe the problem with the highest feasible accuracy. Therefore the diagnostic assistant (Figure 3 right) can be supplied directly with the corresponding component. A classic aid for this purpose are fault or symptom catalogs [12], which can be linked to the underlying geometry as possible component attributes for the assistance system and can then be selected in the assistance system via selection menus, if necessary enriched with example images or typical noise samples. In addition, it is possible to create photos or sound recordings of the respective case using the assistance system, whereby the exact location of the recordings / measurements / photos are documented due to the (AR) tracking of the system. This diagnosis from the on-board personnel can be supported through predefined work instructions displayed by the assistance system in the
AR display. If this form of support is not sufficient, it is possible to switch to the coop-
eration mode of the assistance system, in which the supervising expert sees the infor-
mation recorded by the on-board personnel on his terminal and can provide further in-
structions. These can be navigation instructions (e.g. supported by directional arrows)
for finding specific measurement points or further supplementary instructions for action
that can be generated ad-hoc. Since the target group is usually familiar with
smartphones (also from the private sector), an adapted smartphone interface was also
created in analogy to [13], which is particularly contributory to acceptance. This can
further simplify symptom recording and make diagnosis by the expert correspondingly
more precise.

<figure><img src="image_5.png" /><figcaption>Fig. 3. Localized Diagnostic Measurements</figcaption></figure>

4 Assistant for Service Technicians

On the service technician side, the process is divided into the three phases of prepa-
ration, execution and documentation. The following sections describe the approach for
each phase.

4.1 Preparation

After the maintenance order has been placed, the symptom report generated by the
customer (if necessary with the help of an export from the service provider) can be
assigned to the order in the assistance system environment, thus ensuring the link to the
service provider's billing systems. This is done using the corresponding functions of the
Director module, which provides the interfaces to financial accounting, the spare parts
catalog and the maintenance history of the defective device, where it is managed as an additional entry or case. The history includes and combines all maintenance events in a chronological order (starting from the past) as well as the detailed maintenance logs recorded by means of the digital assistance system. The maintenance tasks can then be prepared either on the PC or on the tablet, since both systems generate the same data format for later use on tablet or smartphone. The result is an instruction manual adapted to the specific case [14], which is generated from available templates. If the service expert wants to consult with his colleagues on difficult diagnoses, this is also supported by the system's collaboration mode, as the entire scope of information from the assistance system is then immediately available to the colleagues and does not have to be posted in a messenger group at great expense. Both direct communication and time-delayed annotations can be used here.

## 4.2 Execution

The execution of maintenance work is usually characterized by the use of the CAD view of the assistance system as well as the supplementary case-specific text information such as, for example, tightening torques of individual screws that deviate from the standard [15]. The reason lies in the service technician's basic familiarity with the subject matter, as observation of numerous cases of use has shown. The acknowledgement of individual steps (and the recording of new settings made during maintenance) (Figure 4) also makes it easy to document how the work was carried out. Supplementary photos are captured by the assistance system (again, correctly located) and automatically assigned to the work step.

<figure><img src="image_6.png" /><figcaption>Fig. 4. Visualizer UI for Service Technicians [15]</figcaption></figure>

## 4.3 Documentation

With the selection of a work step the recording of the start time takes place and with the finished message the total duration is recorded (Figure 5). This log report helps in
the subsequent calculation of the time spent on the work as well as the optimization of internal cost calculations. At the same time, the scope and type of activities are also recorded via the work step recording.

If consumables or spare parts are stored for a work step, these are also booked with the confirmation of a work step and listed in the session log for billing later on.

In the previous report generation, both the formulation of handwritten notes and the naming and assignment of photos take up a lot of time. In order to avoid this high effort of a manual report creation, the assistance system was extended by an automatic reporting feature. The entries described in 4.2 are automatically summarized in a digital maintenance log. This includes all work performed, tools used and associated photos or sound recordings. This data record can then be formatted and forwarded depending on its intended use.

<figure><img src="image_7.png" /><figcaption>Fig. 5. Automatic Maintenance Report</figcaption></figure>

## 5 Economic Efficiency and Acceptance

The benefits of the use of assistance systems were extensively investigated at several maritime component manufacturers which also provide service, with regard to their productivity increase and user acceptance. Exemplary for one engine manufacturer, savings potentials for the creation of work instructions of 51% [6] were found. The test took place in a laboratory environment where the test subjects had to create the same maintenance instructions with a conventional system and the DIA Creator solution.
When it comes to the processing of the maintenance order savings the visualizer solu-
tion was used on a test basis in a two-day maintenance operation on control cabinets
(see Figure 6). In order to examine the productivity increase, the working process was
recorded to find the distribution of working time across the different phases of order
processing. Afterwards the key users estimated the savings per phase. Especially in the
phases of condition analysis (50%), function check (20%) and action documentation
(40%) a high potential for increasing productivity could be demonstrated in this way.
Supplementary expert interviews in the field of engine maintenance have even shown
that the documentation phase could be shortened by up to 80%.

<figure><img src="image_8.png" /><figcaption>Fig. 6. Economic Efficiency [15]</figcaption></figure>

In addition, a second evaluation of the visualizer component was carried out together
with service technicians from the engine repair division, which performed an exchange
of a gas valve. Time was measured for the execution both with the help of conventional
paper documents and the visualizer. It has been shown that the proportion of time spent
on information-related activities could be reduced by 35% [16].
Since profitability and customer satisfaction for the system operator depend directly on
accurate symptom recordings, the acceptance of a tablet or smartphone application (al-
ways at hand) is crucial for their (on-board) personnel, which was rated positively in
another survey [16].

6 Outlook

It is precisely the integration into the overall process with diverse support for sub-
tasks characterized by different challenges that demonstrates the benefits of digital AR-
based assistance systems. Nevertheless, implementation in real processes is a major
challenge due to the large number of interfaces to company IT systems. In particular,
use across company boundaries makes operation and use of the solution more difficult. For further dissemination in other holistic process chains with several companies involved, further development into a web-based solution with reduced deployment effort can open up new potential uses.

**Acknowledgements.**

The presented work was done in cooperation with maritime suppliers within the research project WASSER (Wartung und Service von Schiffen mit erweiterter Realität / Maintenance and Service of Ships with Augmented Reality), funded by the German Federal Ministry of Economics and Energy (Bundesministerium für Wirtschaft und Energie – BMWi) due to a decision of the German Bundestag.
References

1. Kühne, R.: Visualisierung der Prozeßqualität. In: Nedeß, Chr.; Friedewald, A. (Ed.): Wissen schafft Qualität, Band III – Servicemanagement in der neuen Fabrik, FQS-DGQ-Band 96-08, Beuth, Berlin 1997, pp. 121-131

2. Bottani, E., Vignali, G.: Augmented reality technology in the manufacturing industry: A review of the last decade. IISE Transactions **51** (3), 284–310 (2019), https://doi.org/10.1080/24725854.2018.1493244.

3. Halata, P.-S.: Augmented-Reality-gestützte Informationsbereitstellung für die Unikat-produktion, PhD-Thesis, TU Hamburg 2018

4. Friedewald, A., Halata, P.S., Meluzov, N., Lödding, H.: The productivity impact of augmented reality in one-off manufacturing. In: Schlick, C. M. (ed.): Megatrend Digitalisierung - Potenziale der Arbeits- und Betriebsorganisation, pp. 141-162, GITO, Berlin (2016)

5. Petersen, P.: Enhancing the Quality and reliability of Maintenance Operations using Mixed Reality. In: Bertram, V. (Ed.): 19th International Conference on Computer and IT Applications in the Maritime Industries (COMPIT '20), Proceedings, Pontignano 2020, pp. 38-47

6. Meluzov, N.; Friedewald, A.; Mundt, C.; Lödding, H.: Produktivitätssteigerung in der Instandhaltung durch digitale Assistenzsysteme. In: Biedermann, H. (Ed.): Digitalisierte Instandhaltung. Stand und Perspektiven. 33. Instandhaltungsforum (Ed.. ÖVIA) Proceedings, TÜV-Verlag, Köln 2019, pp. 181-197

7. PTC Accounces Vuforia Chalk App for Augmented Reality Communications. https://www.ptc.com/en/news/2017/vuforia-chalk, last retrieved 07.04.2021

8. Features Made for a Complete New Way of Remote Collaboration. https://oculavis.de/en/product/features/, last retrieved 07.04.2021

9. Azuma, R.T.: A survey of augmented reality. Presence: Teleoperators and Virtual Environments 6(4), 355–385 (1997), https://doi.org/10.1162/pres.1997.6.4.355.

10. Meluzov, N.; Friedewald, A.; Elzalabany, A.; Lödding, H.: Aufwandsarme Erstellung von Augmented-Reality-Anleitungen für die maritime Instandhaltung. In: Lukas, U. v. et al. (Ed.): Go-3D 2019 "Mit 3D Richtung Maritim 4.0". Proceedings, Rostock 2019, pp. 31-44

11. Meluzov, N.; Friedewald, A.: Modular Authoring of Augmented Reality Based Service Instructions. In: Bertram, V. (Ed.): 17th International Conference on Computer and IT Applications in the Maritime Industries (COMPIT '18). Proceedings, Pavone 2018, pp. 176-188

12. Plog, J.: Expertensystemunterstützte Qualitätssicherung bei der Fertigung von Elastomerteilen, PhD-Thesis, TU Hamburg-Harburg 1989

13. Jahn, N.; Friedewald, A.; Lödding, H.: Using the Smartphone as an Augmented Reality Device in ETO Industry. In: Lalic, B.; Majstorovic, V.; Marjanovic, U.; Cieminski, G. v.; Romero, D. (Ed.): Advances in Production Management Systems: Towards Smart and Digital Manufacturing: IFIP WG 5.7 International Conference, APMS 2020, Novi Sad, Proceedings, Part II. Springer International Publishing, Cham 2020, pp. 538–546

14. Friedewald, A.; Meluzov, N.; Rost, R.: Potentials of Augmented Reality in Aircraft Assembly and MRO. In: Estorff, O. v.; Thielecke, F. (Ed.): 6th International Workshop on Aircraft System Technologies (AST 2017). Proceedings, Shaker, Aachen 2017, pp. 295-306

15. Meluzov, N.; Rost, R.; Friedewald, A.: Holistic Maintenance Support with a digital As-sistance, in: Bertram, V. (Ed.): 19th International Conference on Computer and IT Applications in the Maritime Industries (COMPIT '20), Proceedings, Pavone 2020, pp. 48-62

16. Lödding, H.; Friedewald, A.; Meluzov, N.; Rost, R.: Wartung und Service von Schiffen mit erweiterter Realität (WASSER) – Grundlagen (GRUND). Final Report, TU Hamburg 2021