# A Framework for Defining Customised KPI in Manufacturing Systems

Pascal André, Virginie Goepp

► To cite this version:

Pascal André, Virginie Goepp. A Framework for Defining Customised KPI in Manufacturing Systems. Borangiu, T., Trentesaux, D., Leitão, P., Berrah, L., Jimenez, JF. (eds). Service Oriented, Holonic and Multi-Agent Manufacturing Systems for Industry of the Future, 1136, Springer Nature Switzerland, pp.309-320, 2024, Studies in Computational Intelligence, 978-3-031-53444-7. 10.1007/978-3-031-53445-4_26. hal-04440557

HAL Id: hal-04440557

https://hal.science/hal-04440557v1

Submitted on 6 Feb 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Framework for defining customised KPI in Manufacturing Systems

Pascal André and Virginie Goepp

**Abstract** Key-Performance Indicators (KPIs) and their implementation are crucial for the continuous improvement of manufacturing systems (MSs). In this article, we work out a KPI framework as a means to implement KPI computations in a convenient way. The idea is to go beyond the simple, standard definition of the KPIs by linking them explicitly with their instrumentation that it to say the data required to calculate them. To do so, we present a set of meta-models: a KPI meta-model, a KPI-capture meta-model and a KPI-instrumentation meta-model. These are defined independently from any MS implementation but are generic enough to be implemented in various Manufacturing Execution Systems. The use of the framework is illustrated on the *Lampex* learning factory from Strasbourg University.

**Key words:** Manufacturing Systems, KPI, Metrics, Metamodels, Instrumentation

## 1 Introduction

Manufacturing systems (MSs) face several challenges such as maximising productivity, ensuring high product quality, and reducing the production time while minimising the production cost simultaneously [28] requiring continuous improvement. In this context, Key-Performance Indicators (KPIs) and their implementation within production systems through e.g. a MES (Manufacturing Execution System) become crucial. Beyond the choice of the KPIs themselves, the link with the instrumentation required is an open question. Indeed, the way to get the data needed to calculate the KPI values is seldom treated in a manner that enables to make easily the link with

<footnote>Pascal André
LS2N - Nantes University, CNRS 2, rue de la Houssinière F-44322 Nantes Cedex, France
e-mail: name.surname@ls2n.fr</footnote>

<footnote>Virginie Goepp
ICube - INSA Strasbourg, 24 bd de la Victoire, 67084 Strasbourg cedex, France
e-mail: name.surname@insa-strasbourg.fr</footnote>
the MS. Indeed, existing KPI models such as KPI-ML<fnref n="1" /> based on the ISO 22400 standard [9, 10] are very coarse and abstract harming their applicability.

In this article, we explore models and instrumentation as a means to organise KPIs in a convenient way. We propose a manageable KPI model, which is defined independently from any MS implementation but is generic enough to be implemented in various MES. KPIs and MES stay fully independent and an instrumentation enables to add observers (sensors, timers, counters) that fill KPI measures in order to compute KPI values.

The paper is organised as follows. Section 2 overviews the role of KPIs in MSs, existing models and issues related to applicability. In Section 3 we propose a structured definition of KPIs down to measures and we design a concrete instrumentation in MSs in Section 4 and implementation in Section 5. We illustrate the approach on a simple case study in Section 6. Section 7 points out relevant information on the applicability of the approach to other situations. In conclusion, we draw perspectives for a larger integration and development of the framework.

## 2 Background and related works

This section discusses the role of KPIs in Manufacturing Systems (MSs), the different KPI models found in the literature and related issues for MSs.

### KPIs role

Generally, "KPIs represent a set of measures focusing on those aspects of organisational performance that are the most critical for the current and the future success of the organisation" [21] In recent years, the research on KPIs and their management has grown exponentially giving rise to various works including those on KPIs taxonomies. The one proposed in [6] has 21 different aspects grouped into five main dimensions: (1) What is measured by a KPI? (2) What is a KPI measured for? (3) What features are considered in the specification of KPIs? (4) What artifacts are used for KPI design and specification? (5) What are the characteristics of a management approach? This taxonomy intends to frame a complete view on KPI management and emphasises the complex nature of this activity leading to so called Performance Management Systems (PMS). They have to integrate all levels of an organisation from the strategic one until the operational one. That is why KPIs are critical for Manufacturing Operation Management (MOM) and continuous improvement [12]. In this context, they reflect the operation performance of a MS such as efficiency, throughput or availability in regard to a given target.

The KPIs rely on measures retrieved manually or automatically from a given MS. A MS includes the hardware and software involved in a runnable production system. In the context of Industry 4.0, a Cyber-Physical Production System is a way to implement a MS. A Manufacturing Execution System (MES) is the information system layer between the Enterprise Resource Planning (ERP) layer and the pro-

<footnote n="1">¹ https://github.com/MESAInternational/KPI-ML</footnote>
duction system. As an example of MES, a Holonic Manufacturing System (HMS)
is a distributed way (Multi-Agent System) to design the MES using holons.

KPIs in Manufacturing Systems and the ISO 22400 standard

Whatever the manufacturing context is, the works dealing with KPIs for MSs are nu-
merous. One can note a set of works based on the ISO 22400 standard [9, 10]. This
standard aims to specify “an industry-neutral framework for defining, composing,
exchanging, and using key performance indicators (KPIs) for manufacturing oper-
ations management [...] for batch, continuous and discrete industries”. It defines
a set of 34 KPIs for MOM that are grouped according to their purpose, as follows:
(1) cost, time, quality, flexibility and sustainability; (2) lead and lag; (3) quantitative
and qualitative; (4) maintenance, production, inventory and quality; (5) resources
(i.e. personnel, material or equipment); (6) process, product and production.

Bauer et al. [3] exploit the standard to interface scheduling with control. To do
so, the authors use the KPIs involving both the scheduled and actual times. The
approach is illustrated for the KPIs ‘effectiveness’ and ‘energy effectiveness’ in a
amino acid production. In [22] a part of the ISO 22400 KPIs are implemented
and visualised in a web-based application enabling the monitoring and control of
discrete manufacturing systems at the run-time. In [15] Kikolski uses the standard
and proposes a methodology to determine them through simulation models.

Other works tend to improve the standard as it is very general and descriptive.
[29] identifies and analyses the gaps between the standard and the process indus-
trial needs. Consequently, they refine the standard by redefining some KPIs to fit
to this specific context. [26] works out a classification model enabling to define
three possible application scopes - work orders, work units and production orders
- of the standard. [12] deals with the intrinsic relationships between the KPIs that
are not considered in the standard. A hierarchical structure of the KPIs (basic and
comprehensive KPIs) is proposed to explore and set these relationships. [16] de-
velops a information model named KEI (KPI Element Information Model) and a
procedure to implement the ISO 22400 standard [9, 10] at the ISA 95 [8] levels
1 (sensing the production process) and 2 (monitoring the production process) of
the production system. The procedure is very coarse and only applied to the KPI
'worker efficiency'.

These works show the relevance of the standard to provide a set of KPIs but also
emphasise the need to develop operational approaches enabling to implement KPIs.

KPIs meta-models

When looking for approaches enabling to implement KPIs, one can note that they
are mainly based on meta-models. Meta-models seem to be a way to have a shared
and comprehensive view on KPIs and support their integration in various fields.

In [17] a model-based methodology named TADAM is developed for semi-
automatic generation of analytical models in manufacturing. A meta-model for the
manufacturing process is provided as well as a structuring of the manufacturing into
four layers going from the physical systems (SCADA, controller) until the busi-
ness intelligence analytics layer dedicated to control the physical system thanks to dashboard. Here the KPIs are not modelled explicitly.

In [27] the BPMN meta-model is extended with a KPI view to improve business processes. Each class of the BPMN meta-model is related to a class of the KPI extension enabling to integrate performance improvement and business process management. This is interesting as it enables to connect KPIs and the system in which they have to be implemented.

In [20] a meta-model for IT performance management is proposed. More recently, [18] provides a general view on performance management. The link between strategic goals and the KPIs that have to be deployed is addressed through an ontology-driven approach. The authors provide a meta-model linking goals to indicators. The indicator class is a generalisation of the KPI, measure and Key-Result-Indicator classes.

Last but not least, KPI-ML<fnref n="2" /> is an XML implementation of the ISO 22400 standard [23, 19, 4]. It consists of a set of XML schemas written using the World Wide Web Consortium's XML Schema language (XSD) that implement the ISO 22400 data models. Implementation issues are discussed in [24, 22]: The main issues of

<figure><img src="image_2.png" /><figcaption>Fig. 1 KPIML Object Model</figcaption></figure>

KPIML Object Model of Fig. 1 are: (1) redundant information between definitions and instances of KPI, (2) missing pertinent information to sort the collections of values for an instance and the collections of instances for a definition, (3) over-aggregated notions of properties enabling confusion between elementary data and values, (4) KPI instances are related to *resources only* while other manufacturing concepts *e.g.* production, would be of interest, (5) The way KPI values are fed (*e.g.* timed values) is out of the scope. KPIML remains at a documentation level (*e.g.* the *formula*

<footnote n="2">² https://github.com/MESAInternational/KPI-ML</footnote>
field is a textual information) and requires much software development inside the MES. The KPIML object model is a centralised solution where all data information are stored in a database but this is not a resilient solution when changing the production system (reconfiguration, technical debt, new frameworks...). All the acquisition system is to be rewritten.

Next section we propose a meta-model that answers these issues.

# 3 A KPI MetaModel for Manufacturing systems

We propose in Fig. 2 a simpler version of the KPI metamodel that includes data acquisition in order to be more applicable in MES. A KPI is an instance of a KPI type

<figure><img src="image_3.png" /><figcaption>Fig. 2 KPI metamodel</figcaption></figure>

which is a KPI Definition (from ISO 22400). The latter enables to stay consistent with ERP representations and provide interoperability. The main difference with ISO is to put forward the tree based calculation of KPIs by aggregating KPIs from smaller KPIs until reaching atomic KPIs<fnref n="3" />. In KPIML, formulas are meaningless strings while we need explicit expressions to compute the KPI, the kpi-expression property is defined via the KPI code key qualifier (for each qualifier there is only one KPI). Atomic KPI are computed similarly on Measures, which store time stamped Measures-values. Again this point was under specified by KPIML where KPI Values are left to MES implementors. Note that KPI and Measure are Indicator but our definition is again more precise than the one of [18] since values are stored in time intervals. Next step is to link the KPI metamodel with instrumentation.

<footnote n="3">³ Aggregation has been studied in details in [1].</footnote>
# 4 KPI Instrumentation

To instrument the measuring process, we need to connect measures to the MS. As mentioned in Section 3, KPI-ML associates KPI instances to resources, and we mentioned that it can be associated to other concepts. However, tight direct connections between the KPI model (from enterprise vision) and the MS model (from the production vision) make those layers mutually dependent. In Fig. 3, we define

<figure><img src="image_4.png" /><figcaption>Fig. 3 KPI metamodel Measure Instrumentation</figcaption></figure>

a KPI capture layer to weave both layers without intrusion. Consequently, we can replace the KPI model or the manufacturing model without regression. KPI capture plays the role of instrumenting the MS to observe it. This is non-standard in manufacturing applications where the events are handled by the manufacturing control. A CapturingDevice can be a counter, a timer, any sensor or IoT device; it can be triggered manually or automatically.

This instrumentation is then plugged to the manufacturing model. Until this stage we do not set strong assumptions on the MES organisation or guiding reference [11] but we will illustrate with Holonic MS. For example, in Fig. 4, we connect it to the PROSA holons [25]; this can be done with other MES reference architectures. Basically a CapturingDevice is connected to one holon. High level ISO KPIs are mainly related to product holons or resource holons but orders provide the actual state of the manufacturing process and give values to low-level KPIs. For example, the Actual unit DOwn Time (ADOT) or the Worker efficiency are associated to resources ; the Scrap Quantity (SQ) is associated to products, and Actual Queuing Time (AQT) or Planned Order Time (POET) are associated to product orders.

# 5 KPI Implementation

Recall from Section 2 that KPIML is a centralised data information solution. We promote modularity to reuse integrated (MS+KPI) parts in other production systems and to mitigate the risk of regression when the system evolves. To implement modular software systems, we make use of Software design patterns that are quality solutions to design software proposed by Gamma et al. [7]. Design patterns will provide an adequate API to build, organise and manage the above-mentioned aggregation patterns.
<figure><img src="image_5.png" /><figcaption>Fig. 4 KPI metamodel Measure Instrumentation</figcaption></figure>

For sake of space, in this section, we only list the software design patterns that we consider to be pertinent to implement KPI instrumentation in a maintainable way. We identified the following solutions:

* Among Creator patterns, the Factory Builder patterns enable to separate type and instance management and to customise the instantiating process especially when the composition part applies to aggregation patterns of KPIs and not single classes. In addition, the Prototype pattern simplifies this process by duplicating instances.

* Among Structural patterns, we identified of course the Composite for the recursive aggregation structures e.g. Fig. 2, Adapter pattern to implement a Capturing device by a proxy e.g. plugged to the MES for specific measures, the Facade pattern for the pool management of capturing devices and the Proxy pattern for measurement delegation.

* Among Behavioural patterns, we identified the Command pattern for orders, the ChainOfResponsabilities for KPI computation through the organisational units and product hierarchical constraints, the Iterator pattern for orders process KPIs, Visitor to broadcast events on various holons, Observer to describe query interfaces on complex data e.g. on the Capturing device and their target, Mediator for collaborative measurement, Strategy for planning and actual management and staff decisions.

This looks like organising the manufacturing metrics as a metrics manufacture. Additional work is needed to step forward intuitions and structure the instrumentation software by combining design patterns.

The communication issues for exchanging values between physical devices, MES and KPI management are not discussed here but is similar to those discussed in [2].
# 6 Application on the Lampex Case Study

In this section, we provide the application of the framework to the *Lampex* case study. We illustrate the situation with the *Lampex* production system used as a project case study at Strasbourg University.

## Case Study Description

The *Lampex* production system is a school Factory from Strasbourg University used for training purposes. It enables to assemble small dynamo flashlights dedicated to promotional purposes (see Fig. 5). The components of the flashlights are purchased from subcontractors in South East Asia. There are 17 components including the packaging and the product information sheet. For sake of place we do not detail them here. The finished product assembled differs only in the colour of the flashlights' housing (blue, green or red).

<figure><img src="image_6.png" /><figcaption>**Fig. 5** One type of dynamo flash-light produced within the Lampex MS</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Fig. 6** One possible configuration of the Lampex MS and its workstations</figcaption></figure>

The production system consists of two identical production lines. These are flow shops with five manual assembly workstations each (cf. Fig. 6). The assembly flow goes through workstation 1 until workstation 5. The assembly time of each workstation is known. The layout of the workstations can be changed and rearranged to optimise the functioning of the MS. There is one operator and one trolley per workstation enabling to transfer the semi-products from one workstation to the next one. The average utilisation rate of the assembly process is 80%. The size of the transfer lot is 20 semi-products. There is also one common automatic stock HANEL for the storage of the finished products and purchased components. Each operator picks-up from this stock the components required to perform the assembly steps assigned to its workstation.

The objective of the practical works the students have to perform is to optimise one of the line whose layout is pre-defined. To do that they have at their disposal a
set of data related to: (1) a set of objectives: the target volume of sales, the target quality level, the target inventory level, (2) the main defects in the assembly process, (3) a list of dysfunctions in the assembly line organisation.

## KPI and Measure Instrumentation

The objective of this study was to identify the data and KPIs we are able to get. To do so, we base our work on the structure of ISO 22400 standard [10] that describes first the elements used in the KPI description and then the KPIs themselves. The elements (time, quality, logistical) detail the data required to calculate the KPIs. We go through each of them to determine if they are available or not. As a result we get the following list of elements and KPIs for our case:

* Estimation elements: to be instrumented by values: (1) PRI (Planned Run Time per item) it can be calculated thanks to the time of each assembly operation and depends on the way the students assign the assembly operations to the workstations ; (2) PBT (Planned Busy Time) for each workstation one per resource x order. (3) POQ (Planned Order Quantity) per order ; (4) PSQ (Planned Scrap Quantity) per order.

* Time elements (measure): to be instrumented by timers: (1) APT (Actual Production Time) for each workstation ; one per resource, starts when production operation is launched, stopped either ; (2) AOET (Actual order Execution Time); one per order, starts when the production starts with the first product instance, stopped when the last product instance is manufactured ; (3) ATT (Actual Transportation Time) one per transportation resource ; (4) AUBT (Actual Unit Busy Time) one per resource x order ; (5) AQT (Actual Queuing Time); theoretically AQT is associated to "material" according to ISO 22400. For sake of simplicity and according to Fig. 4 of ISO 22400, $AQT = AOET - \Sigma(AUBT) - \Sigma(ATT)$.

* Logistical elements (measure): to be instrumented by values or counters: (1) GQ (Good Quantity) per order ; (2) RQ (Rework Quantity) per order ; (3) SQ (Scrap Quantity) per order ; (4) PQ (Produced Quantity) per order = GQ + RQ + SQ (5) RMI (Raw Material Inventory) per component-type ; (6) FGI (Finished Goods Inventory) per (finished) product-type.

* Quality elements: to be instrumented by counters: (1) GP (Good Part) per product-type ; (2) IP (Inspected part) per product-type ; (3) FPY (first pass yield) = GP / IP.

* KPIs: to be computed per period: (1) Throughput rate = PQ/AOET per order ; (2) Quality ratio = GQ/PQ per order ; (3) Availability is set at the out set to 80% in the As-Is situation of the MS = APT / PBT ; (4) Allocation efficiency = AUBT / PBT ; (5) Effectiveness = PRI * PQ/APT ; (6) Overall Equipment Effectiveness Index: OEE index = Availability * Effectiveness * Quality ratio.

## KPI Implementation

All the timers and counters were triggered manually by the students. For example, the Scrapped Quantity per workstation was a button pushed every time a part did not
meet the quality requirements (visual test, functional test, failing material). The in-
ventory counters (RMI, FGI) are handled by the automatic stock resource HANEL.
Only manual KPI computation has been processed by students. Implementating the
instrumentation and the computation processes are to be automated in the next step.

7 Discussion

This section points out relevant information on the applicability of the approach.

The main objective of the ISO 22400 standard or KPIML is to structure KPI information for databases but leaves data acquisition to the MES provider. As far as we know, KPI systems assume reading available production events of the MES to collect information to populate KPI values. However, leaving implementation principle to programmers will lead to free interpretation and heterogeneous detail design. For example, implementation issues are discussed in [22] and details are provided in [24]: KPIs are associated to equipment (resources) only. The orchestra- tor engine subscribes to all events and this is no clear how to select pertinent ones. The KPI Implementation components implements KPI formulas using the incoming data from the knowledge-based system and receives notifications of events from the Manufacturing Plant at runtime. Again this remains abstract and postpones the diffi- culties to MES programming. The approach we defend here is different: we propose to handle the KPI point of view as an orthogonal concern where instrumentation is a first class issue. This is important to provide a modular independent layer for KPI instrumentation in order to avoid regression when the production system changes (workshop reconfiguration, MES change...). We already shared that point of view for the communication concern in [2].

Managing the KPI is a main activity when handling production processes, and it
must not only include the KPI definitions but also study the way the measures that
feed the KPI values are made real in the MSs, as we illustrated in the *Lampex* case
study. By separating KPI model, measurement and instrumentation we should cover
a wide range of implementations. For example, when the MES already includes
measuring information (e.g. the trace of all events happening during the produc-
tion) the instrumentation is implemented by analysing these events to feed the time
stamped measures (cf. Fig. 4). A *CapturingDevice* is a logical unit that can be imple-
mented by a physical device or a numeric one ; it can be part of the MS (e.g. a
robot resource provides data information through its API) or just observe the MS
(e.g. a RFID reader or a video camera that captures product information). Chang-
ing a resource may change the implementation of just the related capturing devices.
Separating the KPI management from the MES improves modularity and thereafter
reusability and evolution. Note that our proposition is generic and not specific to a
given evolution context contrary to [14, 15] where the author focus on facility layout
design and a restricted subset of ISO 22440 KPIs. Moreover with our framework it
becomes possible to link KPI, measurement and instrumentation with a high level
vision and customise the business objectives. One track is to exploit the proposi-
tions of [13, 5] in which the ISO 22400 KPIs are not considered in isolation but
their relationships defined in the form of matrix and trees.
## 8 Conclusion

Every real MS is evaluated through KPIs in order to manage long-term production of factories, however KPIs computation is usually left to software developers and every change requires costly software developments. In this paper, we propose a manageable KPI model, which is defined independently from any MS implementation but is generic enough to be implemented in various MES. An instrumentation layer enables to add observers (sensors, timers, counters) to connect the KPI system to the MES and therefore to fill KPI measures in order to compute KPI values. KPI and MES can evolve independently, the modified KPI or MES features are the starting point to modify the instrumentation layer and avoid regression or mismatches.

Next step will be to connect this KPI definition and instrumentation layers to a reconfigurable manufacturing system (RMS) model and then to implement the observers that compute automatically the KPI values. This model will also include a Enterprise Architecture layer enabling to link MS operation to tactical and strategical decisions. The instrumentation layer requires deeper design and experimentation to build a reusable framework that could be plugged to other MSs. We plan to implement the KPI chain for a Reconfigurable Manufacturing System evaluation tool which is designed to compare reconfiguration scenarios and help operators to choose the configuration to implement.

## Acknowledgments

Authors thank financial support from the French National Research Agency (ANR) under the RODIC project, grant number ANR-21-CE10-0017.

## References

1. André, P., Cardin, O.: Aggregation patterns in holonic manufacturing systems. In: T. Borangiu, D. Trentesaux, P. Leitão, O. Cardin, L. Joblot (eds.) Proceedings of SOHOMA 2021, pp. 3–15. Springer International Publishing (2022)

2. André, P., Cardin, O., Azzi, F.: Multi-protocol communication tool for virtualized cyber manufacturing systems. In: T. Borangiu, D. Trentesaux, P. Leitão, O. Cardin, S. Lamouri (eds.) Proceedings of SOHOMA 2020, *Studies in Computational Intelligence*, vol. 952, pp. 385–397. Springer (2020)

3. Bauer, M., Lucke, M., Johnsson, C., Harjunkoski, I., Schlake, J.C.: Kpis as the interface between scheduling and control. IFAC-PapersOnLine 49(7), 687–692 (2016), cited by: 18; All Open Access, Bronze Open Access

4. Brandl, D.L., Brandl, D.: Kpi exchanges in smart manufacturing using kpi-ml. IFAC-PapersOnLine 51(11), 31–35 (2018), 16th IFAC Symposium on Information Control Problems in Manufacturing INCOM 2018

5. Bumba, A., Gomes, M., de Jesus, C., Lima, R.: Kpi tree - a hierarchical relationship structure of kpi for value streams. Production Engineering Archives 29, 175–185 (2023)

6. Domínguez, E., Pérez, B., Ángel L. Rubio, Zapata, M.A.: A taxonomy for key performance indicators management. Computer Standards & Interfaces 64, 24–40 (2019)

7. Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley Longman Publishing Co., Inc., USA (1995)

8. ISA: Isa95 iec/iso62264 enterprise - control system integration part1-part2 (2010)

9. ISO: ISO22400: 1-Automation Systems and Integration- KPIs for Manufacturing Operations Management—Part. 1: Overview, Concepts and Terminology (2014)
10. ISO: Iso22400: 2-automation systems and integration-key performance indicators (kpis) for manufacturing operations management-part. 2: Definitions and descriptions (2014)

11. Kaiser, J., McFarlane, D., Hawkridge, G., André, P., Leitão, P.: A review of reference architectures for digital manufacturing: Classification, applicability and open issues. Computers in Industry **149**, 103,923 (2023)

12. Kang, N., Zhao, C., Li, J., Horst, J.A.: A hierarchical structure of key performance indicators for operation management and continuous improvement in production systems. International journal of production research **54**(21), 6333-6350 (2016)

13. Kang, N., Zhao, C., Li, J., Horst, J.A.: A hierarchical structure of key performance indicators for operation management and continuous improvement in production systems. International Journal of Production Research **54**(21), 6333-6350 (2016), pMID: 29398722

14. Kikolski, M.: Sample measurement of the iso 22400 standard key performance indicators with the use of simulation models. In: 2019 IEEE Technology & Engineering Management Conference (TEMSCON), pp. 1-5 (2019)

15. Kikolski, M.: Determination of ISO 22400 Key Performance Indicators using Simulation Models: The Concept and Methodology. In: Proceedings of the 8th International Conference Modelsward, pp. 92-99. SCITEPRESS, Valletta, Malta (2020)

16. Kumagai, K., Fujishima, M., Yoneda, H., Chino, S., Ueda, S., Ito, A., Ono, T., Yoshida, H., Machida, H.: Kpi element information model (kei model) for iso22400 using opc ua, fdt, plcopen and automationml. In: 2017 56th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE), pp. 602-604 (2017)

17. Lechevalier, D., Narayanan, A., Rachuri, S., Foufou, S.: A methodology for the semi-automatic generation of analytical models in manufacturing. CIM **95**, 54-67 (2018)

18. del Mar Roldán-García, M., García-Nieto, J., Maté, A., Trujillo, J., Aldana-Montes, J.F.: Ontology-driven approach for KPI meta-modelling, selection and reasoning. International Journal of Information Management **58**, 102,018 (2021)

19. Muhammad, U., Ferrer, B.R., Mohammed, W.M., Lastra, J.L.M.: An approach for implementing key performance indicators of a discrete manufacturing simulator based on the iso 22400 standard. In: 2018 IEEE Industrial Cyber-Physical Systems (ICPS), pp. 629-636 (2018)

20. Pajic, A., Pantelic, O., Stanojevic, B.: Representing IT Performance Management as Meta-model. International Journal of Computers Communications & Control **9**(6), 758-767 (2014)

21. Parmenter, D.: Key performance indicators: developing, implementing, and using winning KPIs. John Wiley & Sons (2015)

22. Ramis Ferrer, B., Muhammad, U., Mohammed, W.M., Martínez Lastra, J.L.: Implementing and Visualizing ISO 22400 Key Performance Indicators for Monitoring Discrete Manufacturing Systems. Machines **6**(3), 39 (2018)

23. Tahir, M.A., Mahmoodpour, M., Lobov, A.: Kpi-ml based integration of industrial information systems. In: 2019 IEEE 17th International Conference on Industrial Informatics (INDIN), vol. 1, pp. 93-99 (2019)

24. Usman, M.: An Implementation of KPI-ML to a Multi-Robot Line Simulator. Master's thesis, Teknisten tieteiden tiedekunta, Tampere University (2018), accepted: 2017-12-20T13:09:54Z

25. Van Brussel, H., Wyns, J., Valckenaers, P., Bongaerts, L., Peeters, P.: Reference architecture for holonic manufacturing systems: Prosa. Computers in Industry **37**(3), 255-274 (1998)

26. Varisco, M., Johnsson, C., Mejvik, J., Schiraldi, M.M., Zhu, L.: KPIs for Manufacturing Operations Management: driving the ISO22400 standard towards practical applicability. IFAC-PapersOnLine **51**(11), 7-12 (2018)

27. Wannes, A., Ghannouchi, S.A.: KPI-Based Approach for Business Process Improvement. Procedia Computer Science **164**, 265-270 (2019)

28. Zhou, J., Wang, Y., Chua, Y.Q.: Machine oee monitoring and analysis for a complex manufacturing environment. In: 2020 15th IEEE Conference on Industrial Electronics and Applications (ICIEA), pp. 1413-1418 (2020)

29. Zhu, L., Johnsson, C., Varisco, M., Schiraldi, M.M.: Key performance indicators for manufacturing operations management - gap analysis between process industrial needs and iso 22400 standard. Procedia Manufacturing **25**, 82-88 (2018)