# Detailed Performance Diagnosis Based on Production Timestamps: A Case Study

Johannes Man, Felix Mannhardt

► To cite this version:

Johannes Man, Felix Mannhardt. Detailed Performance Diagnosis Based on Production Timestamps: A Case Study. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2019, Austin, TX, United States. pp.708-715, 10.1007/978-3-030-30000-5_86. hal-02419240

HAL Id: hal-02419240

https://inria.hal.science/hal-02419240

Submitted on 19 Dec 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Detailed performance diagnosis based on production timestamps: A case study

Johannes Cornelis de Man¹[0000-0003-2220-9086] and Felix Mannhardt¹[0000-0003-1733-777X]

¹ SINTEF Digital, S.P. Andersens veg 5, 7031 Trondheim, Norway.
{hans.deman, felix.mannhardt}@sintef.no

**Abstract.** This paper demonstrates a detailed performance diagnosis of a production process. With limited investment power for new technologies, managers want to diagnose the reason for system underperformance, i.e. diagnosing performance gaps. This paper found detailed performance measures for specific production orders by using event log data, i.e. a set of timestamps that denote the occurrence of an atomic event in production. Sequential time registrations for each production order give detailed insights in how the production process is behaving. The reported case study gave managers a web application that lets them zoom in and out of different characteristics to get an understanding how their production process results in a certain performance. Based on the background and case, a framework and way forward are proposed on how to perform detailed diagnosis to explain performance gaps in production.

**Keywords:** Data-driven production management, Event Logs, Problem Diagnosis, Performance Measurement.

## 1 Introduction

While computing power and data science are developing at high speeds, production companies often rely on self-constructed spreadsheet solutions for data-driven decision taking for planning and control [1]. This is not wrong and has been proven useful over the last 20 years to measure along the performance of resources, output and flexibility [2].

Companies rely on less advanced data sources as they are not matured into digital factories. Using sensors and buying expensive systems to support them is neither feasible nor the core business of most companies. This does not mean that there is nothing to be won from the data stored in existing systems; with little capital investment data analysis tools are available to give performance insights along the production process, at a more detailed level than established single number key-performance indicators (KPI) used in traditional production control methods. This enables the possibility to diagnose problems occurring in production at a more detailed level.

In this paper we present a web application for detailed performance diagnosis in production based on event logs, i.e. a set of timestamps that denote the occurrence of atomic events. The background describes the current challenges with translating per-
formance measurements into relevant diagnoses first. Second, a background on the use of event logs is sketched, coming from the domain of process mining. Then we describe a method and case where event logs were used for diagnosis, discuss the results and a framework for data-driven performance diagnosis, before concluding the paper.

# 2 Background

## 2.1 From performance measure to diagnosis

Performance measurement systems (PMS) in production measure along the lines of resources (cost containment and efficiency), output (customer service) and flexibility (ability to respond to a changing environment) [2]. Having a performance measure, however, does not ensure that corrective action will result in the desired outcome. It is therefore unclear whether users of PMS outperform those that do not [3]. So while it seems imperative that “if you don’t measure progress toward an objective, you cannot manage and improve it” [4], organizations can perform with or without measuring it, as long as they learn how to perform [5]. Learning how to perform or how to solve problems, however, requires diagnosis.

Wagner [6] frames diagnosis as “the analysis of a present condition or present state of the system” [6], differentiating between causal diagnosis in which one tries to determine a cause to an issue, and situation understanding, in which no deviations from a desired state are researched. A single aggregated performance measure does not give a causal analysis or a situational understanding and more information would be required to explain why performance is as it is.

There is a large theoretical and practical domain for diagnosis or problem analysis, e.g. root-cause analysis, fishbone diagrams, and 5-whys. Wagner [6] summarizes how diagnosis should be undertaken:

1. Have an indication that there is a performance gap, i.e. the performance is different from the desired situation.

2. Description of the system under investigation.

3. Listing potential causes.

4. Planning the search, i.e. how to identify problem sources.

5. Analysis and model building; identifying what is wrong by eliminating all the causes and explanations that are not supported by evidence.

This leads to today's challenges in translating a performance measure to a diagnosis. Many manufacturers want to achieve a combination of on-time performance, low work-in-progress (WIP) and related low throughput times, low inventory, and high output [2]. The basis for improving these performance measures could start with the theory-of-constraints approach [7] or a lean approach. To see the development of these performance measures over time, one could use the throughput diagrams from load-oriented manufacturing control [8]. To be able to diagnose a performance gap, however, the buildup of a KPI must be understood. Unlike other theories, this paper
considers the possibilities of modern data-analytics to analyze multiple factors that
influence the KPI, arguing against aggregated KPIs.

## 2.2 The use of event logs and data-driven diagnosis

When production data is collected, it often comes with a timestamp, i.e. a time and date associated with an atomic event that indicates an activity or action has occurred. Moreover, it is often possible to attribute events to an instance of a certain process, e.g., the handling of a specific Production Order (PO). One data-driven method to apply on such timestamped data that can be grouped according to process instances, which are also denoted as event logs, is process mining. Based on event logs, process mining method can discover a model of how processes are actually executed [9]. Such discovered models are then used to analyze deviations from the planned process both to investigate compliance (quality, regulations) and performance (bottlenecks, re-work) related questions [10].

In a production context, some applications of process mining have been reported. For example, in [11] workload in a production process is analyzed using process mining methods, in [12] production shutdowns are analyzed, and in [13] process mining is used to predict the cost of a manufacturing process. In addition to process mining, there is a large domain of other data analysis methods reaching from data-collection technologies, data processing and various methods for data visualization.

# 3 Methodology and case description

This paper builds a new method for diagnosing the performance of production. The method is built from a pragmatic perspective using a real-life dataset and existing theory as described in the background, advancing the use of data analytics for performance diagnosis, but at the same time considering the limitations of a real case.

The case company is a metal parts producer in the Netherlands. Production orders (POs) are prepared to customer specification. A PO leads to the production of a batch of the same part. The factory floor is organized in machining centers for each operation. Each machining center has multiple machines with different specifications, e.g. the bending center has multiple machines with different capabilities.

Capacity is calculated by the maximum revenue the factory can generate. POs are scheduled backwards from the order due date with five days buffer, one day per planned operation and seven days for preparing the order for production. This gives a two-week lead-time for a PO that needs to be processed in two machine centers.

The cutting operation is the first operation performed on each PO and is controlled for by optimizing the nesting of sheet metal. The other operations are organized based on earliest due date, machine availability and machine changeover time. This process is controlled by the supervisors of the different machining centers and is not formalized.

The case company has one enterprise resource planning system (ERP). For each PO, the system registers start and end times for all its operations which results in an
event log. In addition, each logged event contains information on the end-customer, planned production dates, and the kind of operation performed. The registration of start and end times is a manual process by scanning the order slip barcode. For each PO production operation, a machine is assigned. This assigned machine satisfies the minimal capabilities to perform the required operation. During execution, POs can be operated on a machine with a better specification, but this is not registered in the ERP system. Wrong time registration and variations in machine use can skew the data.

Using the start and stop timestamps of all the operations of the POs, a detailed record of all POs flowing through the factory can be constructed from the moment an order is finished at first machining center until finishing its last operation, i.e. each PO is a process instance of possible processes in the factory.

The data analysis was performed with the R statistical software package. This resulted in a web-application that builds up performance measures and graphs from the available event log. Table 1 summarizes the different performance measures that can be calculated, and the possibility to select specific PO timestamps in the event log based on selection criteria by filtering the data. Figure 1 demonstrates how for all selected POs throughput time and waiting time between cutting and other operations is visualized in a graph, such that variation and outliers become immediately apparent.

<table><caption>**Table 1.** Performance measures per focus area, based on selection possibilities</caption><thead><tr><th>Performance measures</th><th>Selection possibilities for these performance measures by filtering data</th></tr></thead><tbody><tr><td><strong>Total process:</strong></td><td></td></tr><tr><td>Average throughput time (days)</td><td>Selection of specific POs</td></tr><tr><td>Standard deviation throughput-time (days)</td><td># of operations performed</td></tr><tr><td>Average value added (hours)</td><td>Specific customer</td></tr><tr><td>Average waiting days for outbound transport</td><td>On time delivery (yes/no)</td></tr><tr><td>Average days late</td><td>Min/max throughput time</td></tr><tr><td>On time percentage</td><td>Order type (customer and stock orders)</td></tr><tr><td># of POs delivered early</td><td>Time range (all orders started within a time range)</td></tr><tr><td># of POs delivered late</td><td></td></tr><tr><td><strong>Between cutting and other machine centers:</strong></td><td></td></tr><tr><td># of POs waiting for next operation</td><td>Selection of specific POs</td></tr><tr><td>Average waiting time until next operation</td><td>Machining center following after cutting</td></tr><tr><td>Standard deviation in waiting time until next operation</td><td></td></tr><tr><td><strong>WIP:</strong></td><td></td></tr><tr><td># of POs in production over time</td><td>--</td></tr></tbody></table>

For example, one can select the time range 1 January 2017 – 21 February 2017, for a specific customer and for POs that had between 3 to 6 operations. For this selection of POs in the event log, all performance measures and graphs are calculated instantly.
<figure><img src="image_2.png" /><figcaption>Fig. 1. Visualizing the individual POs to show variation and outliers for the filtered data</figcaption></figure>

## 4 Diagnosis results

Based on perceived performance gaps, diagnoses were performed by using the web-application. Before addressing the perceived performance gaps, trends over the last two years are discussed. From the first half year in 2017 to the last year in 2018, the throughput time on average increased by 1.5 days and the delivery performance dropped by 20% points. At every moment in time between 50 and 225 POs were on the shop floor as there is no limit on the amount of orders that can be released into production.

For each perceived performance gap, the performed diagnosis is described:

1. A general perception of no flow in the factory, POs queuing at various points, especially after the first operation for each PO, metal cutting.

Using the web application, it was tested whether orders had a throughput-time equal to the planned throughput time. The planned throughput times were exceeded significantly mainly due to an average waiting time of over 3 days with a standard deviation of more than 6 days between cutting and the subsequent machining operation.

This indicates that there is no connection between the planning of the first operation and the subsequent operations, overloading some machining centers after cutting the metal.

2. The different machining centers focus on optimizing the utilization of their bottleneck machines and less on delivery dates, leading to bad delivery performance.

The analysis found a great spread in delivery performance and throughput times. This leads to potential explanations: Many POs run on bottleneck machines, while other POs flow through without delaying queues. A second explanation is that optimizing the usage of machining leads to unnecessary delays for certain POs.
### 3. Perceived low delivery performance.

This could be confirmed and visualized by plotting the earliness or lateness of each order in a two-year period, seeing the variation in delivery performance, also seeing that many orders were delivered early. In addition, some customers were favored over others, having a much higher delivery performance (difference of 20% points to the average), while this is not official policy. Orders that were delivered on time, on average took 2.9 days less to produce than orders that were not on time.

The diagnosis confirmed several challenges:

* The planned days to perform an operation are underestimated and throughput times have increased significantly in the last two years. The planning method based on revenue capacity does not account for bottleneck operations. As the number of orders keep on growing, this will become more visible.

* The order release for the first operation is not linked to the availability of machines in subsequent operations, leading to long and variable waiting times after cutting.

* There are most likely bottleneck machines that are not planned for.

* Delivery performance has drastically decreased in the time-period of the provided dataset.

## 5 Discussion

The diagnosis in the last section lead to potential explanations of several performance gaps under investigation. It also gives the possibility to formulate hypotheses and start testing improvements in practice, as it gave a detailed image of a production environment up to the machine center level. In this also lies a risk: The data does not contain all information and other methods might be necessary to diagnose the performance gap. For the case described in the last paragraph, this applies for bottleneck machines and a recommendation would be to start registering timestamps for the use of specific machines instead of machining centers.

Figure 2 illustrates a framework that integrates decision-making from production control with data available from past performance. From production, timestamp data is recorded that is used to calculate performance measures. For each performance measure a threshold must be set, such that a performance gap can be detected. Once a performance gap is detected, the diagnosis process starts. The case in this paper used performance measure calculations based on timestamp data. The diagnosis should lead to potential causes to explain the performance gap. The case study did not go further than this activity.

Based on the potential causes, hypotheses can be formulated to describe the effect between the potential cause and the performance gap. Testing the hypothesis with the available data can be a difficult exercise. Instead we propose to introduce improvements that reduce the effect of the potential cause on the performance and monitor the effect of the outcome.
<figure><img src="image_3.png" /><figcaption>Fig. 2. Framework for detailed performance diagnosis using an event log</figcaption></figure>

# 6 Conclusion and future work

In this paper, we discussed a realistic way forward for detailed performance diagnosis using an event log. For our case we demonstrated that traditional performance measures for production management can be specified for a specific case selection, depending on the quality and granularity of events recorded in an event log

The use of timestamped data for each process instance lead to a process perspective where we could calculate performance measures for a specific dataset that complies with certain conditions. We related this method to problem diagnosis to come up with a method for detailed performance diagnosis in production.

Compared to traditional methods, the filtering of an available event log on characteristics such as time-period, number of operations performed, customer and delivery performance, gives managers a method to quickly identify causes for certain performance gaps.

This research can be extended in multiple ways:

1. An integration of process mining techniques and performance measurement to facilitate better performance gap diagnosis within production. This specifically holds for visualizing production processes in process models and enriching them with performance measures.

2. Using atomic event production data in existing methods for visualizing workload control (see [8]).

3. The use of different types of data in production and data-analysis techniques to facilitate the diagnosis of performance gaps.

4. The use of new technologies to facilitate better data and easier, faster or more thorough diagnosis. One way to overcome data quality issues is automatic data regis-
tration, e.g., based on activity recognition using sensorised environments and
wearables in a smart manufacturing environment.

Even though factories are not fully digitalized, valuable information can often be
extracted from existing systems to diagnose the performance gaps that factories are
experiencing. General available software can be used to gain new insights, avoiding a
technology push to facilitate the hype around IoT, machine learning and artificial
intelligence. Data-driven production management should not be a solution, but an
increasing powerful set of tools to facilitate diagnosis and problem solving. Connect-
ing these toolsets to performance measurement within supply chains and general
problem-solving techniques is as equally important as further developing technologies
under the digitalization or Industry 4.0 paradigms.

References

1. de Man, J.C. and J.O. Strandhagen, Spreadsheet Application still dominates Enterprise Resource Planning and Advanced Planning Systems. IFAC-PapersOnLine, 2018. 51(11): p. 1224-1229.

2. Beamon, B.M., Measuring supply chain performance. International Journal of Operations and Production Management, 1999. 19(3): p. 275-292.

3. Koufteros, X., A.J. Verghese, and L. Lucianetti, The effect of performance measurement systems on firm performance: A cross-sectional and a longitudinal study. Journal of Operations Management, 2014. 32(6): p. 313-336.

4. Kaplan, R.S. and D.P. Norton, Mastering the management system. Harvard business review, 2008. 86(1): p. 62.

5. Bititci, U., et al., Performance Measurement: Challenges for Tomorrow. International Journal of Management Reviews, 2012. 14(3): p. 305-327.

6. Wagner, C., Problem solving and diagnosis. Omega, 1993. 21(6): p. 645-656.

7. Goldratt, E.M., Theory of constraints. 1990: North River Croton-on-Hudson.

8. Bechte, W., Theory and practice of load-oriented manufacturing control. The International Journal of Production Research, 1988. 26(3): p. 375-395.

9. van der Aalst, W.M.P., Process Mining - Data Science in Action, Second Edition. 2016: Springer.

10. van der Aalst, W.M.P., A. Adriansyah, and B.F. van Dongen, Replaying History on Process Models for Conformance Checking and Performance Analysis. WIREs Data Min Knowl Discovery, 2012. 2(2): p. 182-192.

11. Park, M., et al. Workload and Delay Analysis in Manufacturing Process Using Process Mining. in {AP-BPM}. 2015. Springer.

12. Bettacchi, A., A. Polzonetti, and B. Re, Understanding Production Chain Business Process Using Process Mining: A Case Study in the Manufacturing Scenario, in CAiSE 2016. 2016, Springer International Publishing. p. 193-203.

13. Tu, T.B.H. and M. Song. Analysis and Prediction Cost of Manufacturing Process Based on Process Mining. in 2016 International Conference on Industrial Engineering, Management Science and Application ([ICIMSA]). 2016. {IEEE}.