# An Ontology-Based Model for Training Evaluation and Skill Classification in an Industry 4.0 Environment

Stefano Perini, Damiano Arena, Dimitris Kiritsis, Marco Taisch

► To cite this version:

Stefano Perini, Damiano Arena, Dimitris Kiritsis, Marco Taisch. An Ontology-Based Model for Training Evaluation and Skill Classification in an Industry 4.0 Environment. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2017, Hamburg, Germany. pp.314-321, 10.1007/978-3-319-66923-6_37. hal-01666223

HAL Id: hal-01666223

https://inria.hal.science/hal-01666223

Submitted on 18 Dec 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# An ontology-based model for training evaluation and skill classification in an Industry 4.0 environment

Stefano Perini¹, Damiano Arena², Dimitris Kiritsis², Marco Taisch¹

¹ Department of Management, Economics and Industrial Engineering (DIG),
Politecnico di Milano, Italy

² École Polytechnique Fédérale de Lausanne, Switzerland
{stefano.perini, marco.taisch}@polimi.it
{damiano.arena, dimitris.kiritsis}@epfl.ch

**Abstract.** The recent advancements of manufacturing towards the Industry 4.0 paradigm should be supported by the effective training of industrial workers in order to align their skills to the new requirements of companies. Therefore, the evaluation of the training is becoming in this context increasingly important, given also the possibility of exploiting a huge amount of data from the shop floor about the workers' activities. These data – indeed – can be properly collected and analysed so as to provide real-time indications about the workers' performances and an evolving classification of their skills. In order to pursue this objective, a solution can be represented by the integration of semantic technologies with training evaluation models. For this reason, the paper aims at presenting a Training Data Evaluation Tool (TDET), which is based on the integration of a Training Evaluation Ontology (TEO) with a Training Analytics Model (TAM) for the definition of the skill levels of the workers. The main components and features of the TDET are provided in order to show its suitability towards the collection of data from the shop floor and their subsequent elaboration in summary indicators to be used by the management of the company. Finally, the implications and next steps of the research are discussed.

**Keywords:** Industry 4.0, Ontologies, Training, Skill, Evaluation.

## 1 Introduction

The introduction of the new Industry 4.0 paradigm has recently allowed the increasingly high dissemination of advanced technologies for the improvement of the industrial processes [1]. However, the technological advancements that are taking place should go hand in hand with a rapid effective training of the industrial workers so that this change can be supported by a proper alignment of the skills to the new needs of the companies [2]. In this context, the issue of the evaluation of the training provided to the workers is increasingly critical, since more and more complex data have to be collected and properly analyzed in order to define the skill level of the workers and introduce in case the necessary corrective actions [3]. For this reason, the identification and measurement of data directly from the shop floor becomes extremely im-
portant. In fact, this data can provide real-time indications about the workers' performances following training interventions implemented by the company. The problem is hence twofold. On one hand, it is necessary to understand the approach that is needed to collect and manage a potentially huge amount of data from the field, in order to make their elaboration automatic. On the other hand, the development of models that can provide a summary evaluation of the evolution over time of the worker skill levels on the basis of the available data is certainly a non-trivial task.
An answer to this problem can be represented by the integration of the so-called semantic technologies with training evaluation models. The former are in their turn one of the most relevant enabler of the Industry 4.0 paradigm and can provide the structure that is necessary to model, enrich and make interoperable the data collected from the shop floor, while the latter can address the concurrent creation and elaboration of summary indications on the skill levels that can be easily interpreted and used by the managers of the company. In the light of this final objective, the paper aims at presenting the Training Data Evaluation Tool (TDET), i.e. an extension of the previously developed SatisFactory Ontology (SFO) with an analytical model for the definition of the skill levels of the workers. In particular, in Section 2 the SFO is briefly introduced, as well as a literature review about semantic technologies and the models for training evaluation. In Section 3, the extension of the SFO, i.e. the Training Evaluation Ontology (TEO), is presented. In Section 4, the Training Analytics Model (TAM) for the evaluation of training and skills is introduced. Finally, in Section 5, the conclusions and the next steps of the research are reported.

# 2 Relation to existing theories and work

## 2.1 Semantic technologies and the SFO

Nowadays, the areas of knowledge representation (KR) and knowledge management (KM) are gaining from the use of semantic technologies over conventional approaches. Semantic models, such as ontologies, play an important role for many knowledge-intensive applications since they provide a formal representations of domain knowledge [4]. Organizations that use the languages and standards of the semantic Web, i.e. RDF, RDFS, OWL, SPARQL/RIF, aim to integrate existing information assets, using the best practices of linked data and the open world assumption, aiming to enhance their knowledge management system. However, despite the common belief that semantic technologies might be limited to cloud computing and big data, they are equivalently useful to private or proprietary data. Ontologies provide, indeed, a formal and ubiquitous information artefact aiming to make all the elements of a domain and their relations explicit [5].

The SatisFactory Ontology (SFO) presented in this work has been developed in the framework of the H2020 funded SatisFactory project. The SFO has an upper structure developed to gather and manage manufacturing knowledge, mainly focused on processes and assets at shop floor level. Therefore, the model is specialized in two different directions (clepsydra-like shape, see Fig. 1): i) the data structure-oriented level that enhances with semantics the xml schemas used to exchange data within the Sat-
isFactory ecosystem; ii) the *shop floor-oriented* level that models the elements and terms characterizing the specific manufacturing environment.

<figure><img src="image_2.png" /><figcaption>Fig. 1. SatisFactory network of ontologies</figcaption></figure>

Lastly, the use of semantic technologies, and in particular the exploitation of the pre-
sented model, is not meant to be a replacement for existing information technologies,
but rather an added layer that can leverage those assets for semantic interoperability.

## 2.2 Models for training evaluation

Different models for training evaluation have been developed and used so far. The
most widespread and commonly accepted is the Kirkpatrick's [6] four-levels model
[7]. The four levels of the model are Reaction, Learning, Employee Behaviour and
Organizational Results. The first level evaluates the reaction of the trainees on the
training activity. The second level evaluates the knowledge and skills acquired by the
trainees. The third level evaluates the transfer of the knowledge and skills on the job.
The fourth level evaluates the overall impact on the company/business unit in terms of
economic and /or organizational performances. In particular, the two last levels
should be considered as the most important for a company/business unit [8].

Over time, various changes to the Kirkpatrick's model have been proposed. For ex-
ample, Hamblin [9] proposed five levels, by splitting the two last levels of Kirkpat-
rick's model in Organization and Ultimate value. Guskey [10] also presented a five
levels model where the first two levels correspond to Kirkpatrick's ones, while the
other three are Organizational support and Learning, Participant use of knowledge
and skills, and Participant learning outcomes. The Nine outcomes model [11] present-
ed a list of nine items (Reaction, Satisfaction, Knowledge, Skills, Attitude, Behaviour,
Results, Return on investment, Psychological capital) to be checked in order to evalu-
ate the overall impact of training activities, where all the concepts of Kirkpatrick's
model are included.

The Kirkpatrick's model, together with the variations described above, represent con-
ceptual approaches to be followed in a general training evaluation. From an analytical
perspective, the few contributions available are focused on the estimation of the ROI
as in the case of Phillips [12], therefore leaving to the single intervention/company the
specific measurement of the impact, which is usually unstructured and targeting only
the first two levels of Kirkpatrick's model. For this reason, the TDET described in the
following section aims at providing a mechanism to collect and reuse the data from
the shop floor in order to provide analytical indications about the impact of the train-
ing and the skill level reached by the trainees involved in the program.

3 Training Data Evaluation Tool (TDET)

The TEO should be perceived as a further enrichment of the actual SatisFactory on-
tology. In particular, the lowest level (or data structure-oriented) semantic model is
extended with concepts that support the management and analysis of training data.
This is due to the need of enriching with semantics training data coming from the
shop floor, which are collected from heterogeneous sources (Fig. 2)

<figure><img src="image_3.png" /><figcaption>Fig. 2. Overall architecture</figcaption></figure>

The TDET aims at producing a semantics-driven classification and evaluation of the trainees' expertise. As a first step, the so-called Training Evaluation Ontology (TEO) drives the semantic enrichment of the training activities and their evaluation data coming from the shop floor. Then, the model underpinning the quantitative evaluation is addressed by the Training Analytics Model (TAM) that exploits such semantically enriched data.

**3.1 Training Evaluation Ontology (TEO)**

The data structure-oriented ontology model, as conceived in the SFO, has been ex-
tended with the following concepts: i) Training activity; ii) KPI, iii) KPI_Category, iii) KPI_Score: iv) KPI Type; v) KPI_Focus. These, together with the specification of their inter-links, represent the pillars of the trainee expertise evaluation (see Fig. 3 ).
<figure><img src="image_4.png" /><figcaption>Fig. 3. Training Evaluation Ontology</figcaption></figure>

The evaluation of the worker's expertise level exploits the semantic structure presented in Fig 4. In particular each instance of the KIP_Score should be perceived according to the following statement: *One trainer may have several KPI scores. The latter ubiquitously describes (refers to) one kind of KPI and one specific activity. The KPI score has a (xsd:double) value.*

<figure><img src="image_5.png" /><figcaption>Fig. 4. Semantic structure of the KPI score</figcaption></figure>

## 3.2 Training Analytics Model (TAM)

The quantitative evaluation of the training and the skill classification is addressed by the TAM, which partially stems from the work of Kiritsis et al. [13], who proposed the ActionPlanT Industrial Learning (IL) methodology for the implementation and evaluation of IL actions specifically addressing the last developments of ICT for manufacturing. It is in this framework that an evaluation approach based on Kirkpatrick's training evaluation model was presented. In particular, the first three levels
were targeted, namely Reaction, Learning and Behaviour, and a weighted sum model (WSM) to quantify the impact of the training was proposed. However, the Action-PlanT IL WSM had some limitations:

* The performance indicators (KPIs) identified for each level were based only on answers to questionnaires and not on data retrieved from the shop floor

* The KPIs were only at the level of the single trainee, with no KPIs to evaluate the performances of the team (i.e. Organizational level)

* No mechanism to evaluate the skill level of the trainee according to a pre-defined target and the performances of the other trainees was provided

Therefore, on this basis a set of KPIs based on data to be retrieved automatically from the shop floor was developed for each level of the Kirkpatrick's training evaluation model (i.e. Reaction, Learning, Behaviour, and Organization). Furthermore, a mechanism able to assign each trainee to a given skill level (Low, Medium, and High) based on the comparison with both a target level and the performances of the other trainees was designed.

In particular, on the basis of the KPIs identified, a summary indicator summarizing the overall performance of the trainee *i* at time *j* for the KPI *k* was defined:

$$P_{ijk} = \left[ \left( \frac{V_{ijk}}{AV_{jk}} \right) * WR_{jk} + \left( \frac{V_{ijk}}{TV_{jk}} \right) * WA_{jk} \right] * 100. \quad (1)$$

$V_{ijk}/AV_{jk}$ is the ratio between the value of KPI *k* for employee *i* at time *j* ($V_{ijk}$) and the average of the values of KPI *k* of the *n* employees at time *j* ($AV_{jk}$). If this ratio is higher than 1 it means that the trainee *i* at time *j* for the KPI *k* is performing better than the average, if it is lower than 1 that is performing worse. $V_{ijk}/TV_{jk}$ is the ratio between the value of KPI *k* for employee *i* at time *j* ($V_{ijk}$) and the target value of KPI *k* at time *j* ($TV_{jk}$) established by the management. If this ratio is higher than 1 it means that the trainee *i* at time *j* for the KPI *k* is performing better than the target value, if it is lower than 1 that is performing worse. $WR_{jk}$ is the weight related to the Value/Average ratio while $WA_{jk}$ is the weight related to the Value/Target ratio. These two weights can be balanced in order to give each time more importance to the performance of the single trainee compared with the average performance of the group or to the same performance compared with the pre-defined target value.

According to the value of $P_{ijk}$, each trainee *i* can be classified according to three different skill levels, namely Low, Medium and High. Considering as a reference all the values of KPI *k* of the *n* employees at time *j*, the trainee's skill level will be classified as Low if included between the zero and first quartile, as Medium if included between the first and third quartile and as High if included between the third and fourth quartile.

In order to aggregate the values of more KPIs for a given trainee *i*, standardized values of the single $P_{ijk}$ should be used, in order to take into account different average and target values. As a consequence, for each $P_{ijk}$ the following formula should be used:
$$P_{std_{ijk}} = (P_{ijk} - AV_{jk})/SD_{jk}. \qquad (2)$$

, where SD<sub>jk</sub> is the standard deviation of the values of KPI k of the n employees at time j. On this basis, the following aggregated performance of the trainee i at time j can be formulated:

$$AP_{ij} = \sum_{kj} \alpha_{kj} * P_{std_{ijk}}. \qquad (3)$$

, where α<sub>k</sub> is the weight related to the KPI k at time j. This way it is possible to evaluate the overall skill level of the trainee i at time j, by using the same approach described above. Finally, the overall performances of the single team/group can be easily obtained by means of the averages previously calculated for the different evaluation and temporal levels.

## 4 Conclusions and next steps

With the introduction of the Industry 4.0 paradigm and of the related technologies, a huge effort is nowadays requested to manufacturing companies in order to continuously update the skill of the workers. For this reason, the relevance of training is increasing as well as the need to properly evaluate its effectiveness. In particular, thanks to the huge amount of data that can be retrieved from the shop floor through smart devices, sensors and HMIs, the evaluation of the training and skill levels can be now potentially made automatic with the provision of summary performance indicators to the management as an output. In order to pursue this final objective, the paper has presented the results of a preliminary work aiming at the integration of semantic technologies with training evaluation models in the Training Data Evaluation Tool (TDET), which put together the Training Evaluation Ontology (TEO) with the Training Analytics Model (TAM) for the summary evaluation of training and skills.

The TDET is suitable to the automatic elaboration of data retrieved from the shop floor and to the provision of a general evaluation of the given training activity by means of the aggregation of different KPIs developed ad hoc, which are related to all the four levels of the well-known Kirkpatrick's model. In addition, the overall performance of a trainee can be computed, as well as his skill level (Low, Medium, High) according to a comparison with the performances of the other trainees. The overall performance and the skill level can be obtained for both a given time j or as a summary elaboration of the results over a longer time span.

The next steps of the research will include the deployment of the TDET through existing tools for ontologies and smart data management, and the evaluation of this framework on different industrial use cases in order to provide empirical evidence of the effectiveness of the presented approach.

**Acknowledgements.** This work was partially supported by the EU funded SATISFACTORY IA project (H2020-636302).
References

1. M. Hermann, T. Pentek e B. Otto, «Design principles for industrie 4.0 scenarios,» in Proceedings of the Annual Hawaii International Conference on System Sciences, Grand Hyatt on the Island of KauaiKoloa, 2016.

2. J. Posada, C. Toro, I. Barandiaran, D. Oyarzun, D. Stricker, R. De Amicis, E. Pinto, P. Eisert, J. Dollner e I. Vallarino, «Visual Computing as a Key Enabling Technology for Industrie 4.0 and Industrial Internet,» IEEE Computer Graphics and Applications, vol. 35, n. 2, pp. 26-40, 2015.

3. M. Teimourikia e M. Fugini, «Ontology development for run-time safety management methodology in Smart Work Environments using ambient knowledge,» Future Generation Computer Systems, vol. 68, n. 1, pp. 428-441, 2017.

4. S. El Kadiri e D. Kiritsis, «Ontologies in the context of product lifecycle management: state of the art literature review,» International Journal of Production Research, vol. 53, n. 18, pp. 5657-5668, 2015.

5. T. Walter, F. Parreiras e S. Staab, «An ontology-based framework for domain-specific modeling,» Software and Systems Modeling, vol. 13, n. 1, pp. 83-108, 2014.

6. D. Kirkpatrick, «Evaluation of training,» in Training and development handbook: A guide to human resource development, New York, McGraw-Hill, 1976, pp. 301-319.

7. R. E. DeRouin, B. A. Fritzsche e E. Salas, «e-Learning in Organizations,» Journal of Management, vol. 31, n. 6, pp. 920-940, 2005.

8. P. Tate e R. Klein-Collins, «Establishing High-Performance Workforce Development Organizations,» in Linking training to performance - A guide for workforce development professionals, Washington, DC, Community College Press, 2004, pp. 15-24.

9. Hamblin, Evaluation and control of training, McGraw-Hill, 1974.

10. T. Guskey, «Does it make a difference? Evaluating professional development,» Educational leadership, vol. 59, n. 6, pp. 45-51, 2002.

11. P. Donovan e J. Townsend, Training evaluation pocketbook, Management Pock-
etbooks Ltd, 2004.

12. J. Phillips, Return on Investment in Training and Performance Improvement Pro-
grams, Butterworth-Heinemann, 2003.

13. D. Kiritsis, A. Bufardi, D. Mavrikios, T. Knothe, H. Szigeti e A. Majumdar, «A competence-based industrial learning approach for factories of the future,» Education and Information Technologies, vol. 18, n. 2, pp. 331-350, 2013.