# Integrated Production and Maintenance Scheduling Through Machine Monitoring and Augmented Reality: An Industry 4.0 Approach

Dimitris Mourtzis, Ekaterini Vlachou, Vasilios Zogopoulos, Xanthi Fotini

► To cite this version:

Dimitris Mourtzis, Ekaterini Vlachou, Vasilios Zogopoulos, Xanthi Fotini. Integrated Production and Maintenance Scheduling Through Machine Monitoring and Augmented Reality: An Industry 4.0 Approach. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2017, Hamburg, Germany. pp.354-362, 10.1007/978-3-319-66923-6_42. hal-01666188

HAL Id: hal-01666188

https://inria.hal.science/hal-01666188

Submitted on 18 Dec 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Integrated production and maintenance scheduling through machine monitoring and augmented reality: an Industry 4.0 approach

Dimitris Mourtzis<sup>a*</sup>, Ekaterini Vlachou<sup>a</sup>, Vasilios Zogopoulos<sup>a</sup>, Xanthi Fotini<sup>a</sup>

<sup>a*</sup>Laboratory for Manufacturing Systems and Automation (LMS), Department of Mechanical Engineering and Aeronautics, University of Patras, Rio Patras, Greece

*mourtzis@lms.mech.upatras.gr

**Abstract.** Maintenance tasks are a frequent part of shop floor machines' schedule, varying in complexity, and as a result in required time and effort, from simple cutting tool replacement to time consuming procedures. Nowadays, these procedures are usually called by the machine operator or shop floor technicians, based on their expertise or machine failures, commonly without flagging the shop floor scheduling. Newer approaches promote mobile devices and wearables as a mean of communication among the shop floor operators and other departments, to quickly notify for similar incidents. Shop floor scheduling is frequently highly influenced by maintenance tasks, thus the need to include them into the machine schedule has arisen. Moreover, production is highly disturbed by unexpected failures. As a result, the last few years through the industry 4.0 paradigm, production line machinery is more and more equipped with monitoring software, so as to flag the technicians before a maintenance task is required. Towards that end, an integrated system is developed, under the Industry 4.0 concept, consisted of a machine tool monitoring tool and an augmented reality mobile application, which are interfaced with a shop-floor scheduling tool. The mobile application allows the operator to monitor the status of the machine based on the data from the monitoring tool and decide on immediately calling AR remote maintenance or scheduling maintenance tasks for later. The application retrieves the machine schedule, providing the available windows for maintenance planning and also notifies the schedule for the added task. The application is tested on a CNC milling machine.

**Keywords:** Maintenance; Scheduling; Augmented Reality; Machine monitoring; Industry 4.0;

## 1 Introduction

As modern manufacturing marches towards Industry 4.0, unified solutions that integrate Information and Communication technology (ICT) and sensors while remaining agile

adfa, p. 1, 2011.

© Springer-Verlag Berlin Heidelberg 2011
and adaptive to environment conditions are being developed, updating physical systems to cyber physical systems [1]. Currently, production scheduling software packages are used by the majority of manufacturing companies. Though capable of supporting the production in an adequate level, they fail to integrate production line disturbances, thus creating inaccuracies in high- altering systems [2]. One of the main causes of changes in the schedule is maintenance tasks, which, though they are a common part of the machine's lifecycle, are not reported back for rescheduling. Event-driven rescheduling could dictate a more flexible and close to the real system method of tackling this issue [3, 4].

Maintenance is an important part of products' lifecycle, accounting for a high percentage of its total cost [5] and having a high impact on the utilization period of the machine. In the last few years, manufacturing companies have started adopting monitoring solutions in order to increase system reliability through reducing the unplanned breakdowns. Machine condition monitoring systems are being implemented in modern machinery informing human operators for the majority of the upcoming maintenance tasks, based on Wireless Sensor Network (WSN) input [6], defining the remaining operating time between failure of the machine tools [7]. Moreover, the last few years, new technologies are starting to be implemented in maintenance processes in order to increase its efficiency and facilitate the maintenance support service provision. Augmented Reality is one of the upcoming technologies in manufacturing [8] with numerous applications aiming to provide remote maintenance support [9,10], in an effort to facilitate technical knowledge distribution in a digitalized and easy-to-perceive way. Additionally, it poses a digitalized way of bridging the gap between the CAD files the maintenance instructions manuals. To facilitate system communication, cloud platforms have emerged as a viable solution in manufacturing systems, as they enable the seamless data and knowledge exchange from different systems embedded in the shop-floor and its users [11]. Moreover, high security of these systems makes them a viable solution to combine with different wireless sensor networks, providing unified solutions of monitoring and acting on the production line by bringing together different systems and stakeholders [7].

The literature review makes apparent that although there is a lot of progress in implementing isolated solutions for machine monitoring, scheduling and maintenance support, there are limited unified approaches integrating communication between operational planning and maintenance planning [12]. Aiming to create an integrated framework that, following the concept of Industry 4.0, will bring together these systems, an application that will support production and maintenance scheduling through machine monitoring and Augmented Reality maintenance instructions is developed. The developed application integrates all the necessary connections that will allow the machine operator to monitor the status of the machine, schedule maintenance tasks and communicate with a maintenance expert so as to receive AR maintenance instructions remotely. The developed application is applied in a CNC milling machine use case.

## 2 Integrated Production and Maintenance scheduling via machine monitoring and AR technology

This work proposes an integrated solution for combined production and maintenance scheduling, by receiving input from a machine monitoring system. The framework combines three major functionalities: (i) machine health status visualization, including the remaining operating time between failures (ROTBF), (ii) maintenance scheduling based on current production schedule and (iii) remote maintenance support supported by Augmented Reality. All the functionalities are bundled in a mobile device application that is
used by the machine operator. The workflow that shows all the actions and the sub-systems of the framework is presented below in Fig.1.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Workflow of the proposed framework</figcaption></figure>

The first step of using the developed application is the status monitoring of the machine. The machine is equipped with a data acquisition (DAQ) device that monitors the status of the manufacturing resources and especially machine tools [13]. The monitoring device, through current and voltage sensors, gathers real time sensor data and transmits them through a wireless sensor network to a central database where the status of the machine (down, busy or idle) is detected and the actual machining time (AMT), the machine tool utilization and availability are calculated. Since the mean time between failures (MTBF) per machine is known as a technical characteristic, using the following Equation, the remaining operating time between failures (ROTBF) is calculated:

$$ROTBF = MTBF - AMT \qquad (1)$$
To connect the visualized results with each machine, an AR marker is placed on each one. The operator, using the developed application, scans the AR marker and calls the status information of the machine and the ROTBF in the GUI of the application. Then, based on the ROTBF of the machine, the operator may plan ahead the maintenance task. Whenever the ROTBF is below a preselected threshold *k* (e.g. 10%), the operator is notified that a maintenance task will be needed shortly. At this point, the operator calls, through the application, the machine schedule and the mean time to repair for the required process. The mean time to repair is either extracted by past maintenance tasks, if the same task has re-occurred or, by implementing sophisticated algorithms that may provide a trustworthy estimation, such as Case-based Reasoning (CBR) [14]. Based on the estimated duration of the maintenance task and the machine schedule, the operator may either, if possible, plan the task for a timeslot when the machine is idle or plan it whenever the operator thinks is necessary despite of the current schedule; this feature is highly important, especially in cases of unexpected breakdowns. In both cases, the information related to the maintenance planning is returned back to the scheduling system. In the present approach, the remaining tasks of the machine are shifted right on the schedule, always checking if a task has to be shifted to another working shift or even another day.

Moreover, the developed application supports the operator in dealing with maintenance procedures that the operator may have limited experience on. The operator, through the same application, receives the maintenance instructions by a maintenance expert that can be remote located, in a digitalized form, exploiting Augmented Reality [15]. The technician may connect to a cloud database that includes all the past AR maintenance instructions, organized per machine and named in an easy-to-perceive way, and download the required procedure. In case the malfunction is not listed in the database or the operator is not aware of the cause, a malfunction report is constituted by the operator and sent to the maintenance expert. The application allows the operator to write explanatory text, take photos and sound recordings that will facilitate the expert to diagnose the malfunction cause. This expert may either be from the machine manufacturer, an external specialized technician or even from the central maintenance department of the company. The expert diagnoses the malfunction cause and creates the AR maintenance instructions, which are sent back to the on-spot technician through the cloud database. To support this process, a series of pre-created part animations, GUIs and scripts have been developed. Moreover, an algorithm that breaks down the assembly CAD to assembly/disassembly steps has been implemented; this way the instructions for the assembly processes of the maintenance task can be easily created, saving a lot of time since they account for a large part of the maintenance tasks [15]. The operator receives the AR instructions in the mobile device, performs the maintenance and validates that the machine is operational again.

# 3 System Implementation

In order to implement the machine monitoring, a data acquisition device was developed. It utilizes split-core current transformers as current sensors, a closed loop hall current sensor, voltage sensors, as well as a camera. A WSN is established so as to send the data from the DAQ for processing and to make them accessible by the integrated application. The WSN is facilitated with the use of DIGI XBee ZigBee RF module [16]. ZigBee is a specification of the IEEE 802.15.4 standard. The selection of ZigBee over other wireless standards is due to its support to various network topologies and encryption algorithms, and its robust operation with functionalities, such as collision avoidance, retries, and acknowledgments performed in the hardware. In order to derive the status of
the machine, as well as the AMT, an information fusion technique consists of the Ana-
lytical Hierarchy Process (AHP) and the Dempster–Shafer (DS) theory of evidence [13]
was applied. To develop the integrated application and also to create the AR mainte-
nance instructions, two commercial software tools were used: Unity [18] and Vuforia
[17]. The first one supports the creation of high quality and usability GUIs that guide the
operator through the available functionalities and supports seamless connection with the
cloud databases, while also allowing the application to be directly built as a mobile de-
vice app for Android OS. The latter, used as an add-on on the first, is used to add Aug-
mented Reality features into the maintenance instructions and the monitoring applica-
tion. Moreover, in an effort to support and partially automate the AR maintenance in-
structions generation process, an algorithm that breaks down the assembly CAD into
steps of assembly and disassembly is implemented [15].

<figure><img src="image_3.png" /><figcaption>Fig. 2. Interfaces between the developed application and other tools and databases</figcaption></figure>

Furthermore, the algorithm detects the axis and direction of assembly/ disassembly for each part thus allowing quick implementation of part animations in the AR scenes. An important part of the proposed framework is the integration and communication with various other platforms. The developed application connects to applications and cloud platforms via internet so as to send and receive data, creating a set of interconnected tools that function together; the data exchange between those tools can be seen in Fig. 2. The application receives the AMT as an input from the monitoring system through an xml file. When a maintenance task needs to be planned, the MTTR is either extracted by past maintenance tasks, if the same task has re-occurred or is estimated using similarity mechanisms and algorithms. Then, the machine schedule is called, updated with the task and returned back to the scheduling; both data exchanges are performed via xml. When sending a maintenance report, the operator sends text, photo and sound files and receives a sequence of AR scenes that can be directly loaded by the application.

4 Case study and Results

The proposed applications are validated in a case study including a three axis CNC
machine tool XYZ SMX SLV. In the current procedure, the operator of the machine tool
plans a maintenance task based on a breakdown that may occur, without considering any data from the machine tool. The maintenance department is informed on the maintenance issue and deploys the maintenance expert to go on spot to diagnose the issue and physically perform the maintenance task in the production plant. In an effort to eliminate this need and reduce maintenance time and cost, the proposed system is applied. The developed monitoring system considering the data acquisition device and the WSN was set up. The monitoring system gathered all the necessary data to identify the machine tools status, the actual machining time, as well as the ROTBF.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Application of the developed framework</figcaption></figure>

Using the developed application, the technician could foresee the required maintenance task and plan it ahead in an available timeslot in the machine schedule. In this use case, an unexcepted maintenance task is occurred, where the cutting tool and the tool holder needed replacement due to tool breakage. The operator directly connected to the AR maintenance instructions database, downloaded the corresponding maintenance in-
structions on how to change the cutting tool and the tool holder performing the mainte-
nance task. Moreover, using the same application, the operator could plan the task for
a later timeslot, and update the machine schedule. The sequence of actions followed by
the technician are visualized in Fig. 3. By implementing this application on the shop-
floor, the operator is capable of monitoring the status of the machine, pre-identifying
malfunction and scheduling their repair, while updating the machine schedule in a way
that is more efficient and easier to use than the currently existing methods, bundled into
one software. In case of an unexpected maintenance task the operator through the pro-
posed system calls the AR maintenance instructions and performs the maintenance
tasks easily, quickly and with low cost. Based on the first implementation of the proposed
tools, it is estimated that the awareness of machine tools’ condition will be increased
over 30% and the response time to changes will be decreased over 25%. Moving also
towards Industry 4.0, the proposed approach enables the integration of different systems
supporting integrated planning and control, increasing systems interoperability and effi-
ciency.

5 Conclusion and Future work

The proposed work presents an integrated way of production and maintenance sched-
uling supported by augmented reality technology and real-time machine monitoring,
aiming to move towards industry 4.0. The developed system exploits sensor data from
production machines to perform condition- based maintenance and integrated schedul-
ing and AR supported maintenance, implemented as a mobile device application. The
system includes all the required connections that allow the application to receive data
from the sensors so as to monitor the remaining operating time between failure, plan a
maintenance task based on the available timeslots, update the machine schedule based
on the maintenance task’s duration and the connect to a cloud database of AR mainte-
nance instructions. Moreover, in order to improve the AR maintenance instructions gen-
eration and secure the high quality of the generated result, the maintenance expert is
supported by a set of functionalities that facilitate and accelerate the instructions gener-
ation process, based on an algorithm that breaks down the assembly tasks and pre-
created GUIs and animations. The proposed system increases system’s interoperability,
efficiency, and communication, providing useful data from the monitoring system which
can be further analyzed and transform isolated planning and control systems into adap-
tive. As a step towards Industry 4.0, the proposed system integrates data from different
sources (scheduling system, monitoring system) and supports the human operator
through a mobile device to plan and perform maintenance tasks easily and efficiently,
increasing system’s productivity. Future work will be focused on enhancing the existing
approach and predicting the ROTBF based on the sensory data. Moreover, it is im-
portant to connect the developed application to other mobile devices, such as smart-
watches, in the production line keeping in mind the projected marching towards Industry
4.0. Finally, it is important to study the implementation of the system to other, more
complex use cases.

References

1. Kadiri, S.E., Grabot, B., Thoben, K.D., Hribernik, K., Emmanouilidis C., Cieminski, G., Kiritsis, D. Current trends on ICT technologies for enterprise information systems. *Computers in Industry*, 79, pp.14-33 (2016)
2. Mourtzis, D., Doukas, M. The evolution of manufacturing systems: From craftsmanship to the era of customisation. Design and Management of Lean Production Systems, V. Modrak, P. Semanco (Eds.), IGI Global (2014)

3. Mehta, S. Predictable scheduling of a single machine subject to breakdowns, International Journal of Computer Integrated Manufacturing, 12(1), pp. 15-38 (1999)

4. Chryssolouris G, Manufacturing Systems: Theory and Practice, 2nd ed. New York: Springer-Verlag (2006).

5. Dhilon, B. S., 2006, Maintenability, Maintenance, and Reliability for Engineers, Taylor and Francis, ISBN: 9780849372438

6. Giannoulis, S., Koulamas, C., Emmanouilidis, C., Pistofidis, P., Karamatzakis, D. Wireless sensor network technologies for condition monitoring of industrial assets, *IFIP International Conference on Advances in Production Management Systems*, pp. 33-40, Springer (2012)

7. Mourtzis, D., Vlachou, E., Milas, N., Xanthopoulos, N. A cloud-based approach for maintenance of machine tools and equipment based on shop-floor monitoring, 48th CIRP Conference on Manufacturing Systems, pp. 655-660 (2015)

8. Nee AYC, Ong SK, Chryssolouris G, Mourtzis D. Augmented Reality applications in design and manufacturing, CIRP Annals- Manufacturing Technology 2012;61, pp. 657-679.

9. Mori, M., Fujishima, M., Komatsu, M., Bingyan Z., Yadong L. Development of Remote Monitoring and Maintenance System for Machine Tools, CIRP Annals, 57(1), (2008)

10. Platonov, J., Heibel, H., Meier, P., Grollmann, B. A mobile markerless AR system for maintenance and repair, Proceedings of the 5th IEEE and ACM International Symposium on Mixed and Augmented Reality, pp 105–108 (2006)

11. Xu, X. From cloud computing to cloud manufacturing, Robot and Computer-Integrated Manufacturing, 28(1): pp. 75-86 (2014)

12. Papakostas, N., Papachatzakis, P., Xanthakis, V., Mourtzis, D., & Chryssolouris, G. An approach to operational aircraft maintenance planning. *Decision Support Systems*, 48(4), pp. 604-612 (2010).

13. Mourtzis, D., Vlachou, E., Doukas, M., Kanakis, N., Xanthopoulos, N., Koutoupes, A., Cloud-Based Adaptive Shop-Floor Scheduling Considering Machine Tool Availability. ASME International Mechanical Engineering Congress and Exposition, pp. V015T19A017 (2015)

14. Mourtzis, D., Vlachou, E., Milas, N., Dimitrakopoulos, G. Energy consumption estimation for machining processes based on real-time shop floor monitoring via wireless sensor networks. *Procedia CIRP*, 57, pp. 637-642 (2016).

15. Mourtzis, D., Vlachou, A., & Zogopoulos, V. Cloud-Based Augmented Reality Remote Maintenance through Shop-Floor Monitoring: A Product-Service System Approach. *Journal of Manufacturing Science and Engineering*, 139(6), pp. 061011 (2017).

16. Zigbee, http://www.zigbee.org/

17. Vuforia, https://www.vuforia.com/

18. Unity3D, https://unity3d.com/