# Towards a Novel Comparison Framework of Digital Maturity Assessment Models

B. Cognet, J.-P. Pernot, L. Rivest, C. Danjou, T. Wuest, H. Kärkkäinen, M. Lafleur

► To cite this version:

B. Cognet, J.-P. Pernot, L. Rivest, C. Danjou, T. Wuest, et al. Towards a Novel Comparison Framework of Digital Maturity Assessment Models. 16th IFIP International Conference on Product Lifecycle Management (PLM), Jul 2019, Moscow, Russia. pp.58-71, 10.1007/978-3-030-42250-9_6. hal-04021877

HAL Id: hal-04021877

https://hal.science/hal-04021877

Submitted on 9 Mar 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Towards a novel comparison framework of digital maturity assessment models

B. Cognet¹, J-P. Pernot¹,², L. Rivest¹, C. Danjou³, T. Wuest⁴, H. Kärkkäinen⁵, M. Lafleur⁶

¹ École de Technologie Supérieure, Département de Génie des Systèmes, Montréal, Canada
brunot.cognet.1@ens.etsmtl.ca, louis.rivest@etsmtl.ca

² Arts et Métiers, LISPEN EA 7515, HeSam, Aix-en-Provence, France
jean-philippe.pernot@ensam.eu

³ École Polytechnique, Département de Mathématiques et Génie Industriel, Montréal, Canada
christophe.danjou@polymtl.ca

⁴ West Virginia University, Department of Industrial and Management Systems
Engineering, Morgantown, USA
thwuest@mail.wvu.edu

⁵Tampere University of Technology, Laboratory of Industrial and Information Management,
Finland
hannu.karkkainen@tuni.fi

⁶ AéroMontréal, Montréal, Canada
martin.lafleur@aeromontreal.ca

**Abstract.** The fourth industrial revolution is forcing companies to rethink their status quo – creating a need to assess their digital maturity as a basis for improvements. As a result, there is a variety of maturity models available in the literature. This paper introduces a novel comparison framework designed to compare different digital maturity assessment models. Our framework has several steps: reverse engineering of criteria from existing models, criteria matching analysis, as well as computation of the coverage and spread ratios. These two metrics characterize respectively the similarity of two maturity models, and the spread between them. We tested the proposed approach with two well-known maturity self-assessment approaches, namely the IMPULS and PwC methods. From our analysis, we were able to derive several insights that will help to develop a new maturity model specifically dedicated to support SMEs in the aerospace industry and manufacturing sector.

**Keywords:** Industry 4.0, smart manufacturing, digitalization, maturity models, comparison framework, coverage and spread ratios, assessment.

## 1 Introduction

Manufacturing companies feel increasing pressure to adopt the Industry 4.0 paradigm to evolve and remain competitive on the market worldwide [1]. To support Small and Medium-sized Enterprises (SMEs) with their digital transformation, seen as a pillar of the Industry 4.0, several maturity models have been developed that allow evaluating
the level of digital maturity of individual companies [2,3]. The results of these evalua-
tions are then used to design and setup a digital transformation plan. However, there
are many different approaches to evaluate the digital maturity of a company available
today and it can be difficult to identify the most appropriate option as they do not always
focus on the same set of criteria.

The current literature shows that each maturity model has its specific benefits and challenges. Therefore, a suitable solution may lie in the partial exploitation of some of their core advantages. It has to be noted that most digital maturity models do indeed share some common characteristics and goals. For example, most of them are using a set of questions to evaluate certain criteria which are grouped in dimensions, and possibly sub-dimensions. However, when they have been formalized, criteria are not always evaluated in the same way. This is in some cases performed by asking the user to self-assess the levels or by using some black box mechanisms to compute a mark from multiple user-specified answers. Regarding the questionnaires, some approaches are referring to self-assessment whereas some others are focused on guided-assessment.

This paper is a first attempt towards understanding and identifying the possible key
criteria to be used when developing a digital maturity model. Actually, when evaluating
the digital maturity of a company, the so-called criteria can be considered as Key Per-
formance Indicators (KPIs) used to monitor the status of the company; this is the ter-
minology that is used in this document. To reach this objective, a novel digital maturity
assessment models comparison framework is introduced. At its core the framework
works at the level of the KPIs, rather than at the level of the questions/answers or di-
mensions/subdimensions. We consider that focusing on the KPIs is a good tradeoff as
it reduces the subjectivity and increases the objectivity when comparing different mod-
els. In most cases, questions are directly linked to certain KPIs, making them explicit
for the target end-users, i.e., the ones who will answer the questionnaire. The alterna-
tive, i.e. working at the level of the dimensions is considered too high-level, which in
turn would make the comparison significantly more difficult.

The framework counts several steps: reverse engineering of KPIs from existing mod-
els, KPIs matching analysis, as well as computation of the so-called coverage and
spread ratios. These two metrics characterize respectively the similarity between two
maturity models overlap and the spread between them. The proposed approach has been
tested with two well-known maturity self-assessment approaches, namely the IMPULS
[4] and PwC ones [5]. The contribution is threefold: (i) a three-step comparison frame-
work, (ii) new quantitative metrics to characterize the coverage and spread of two ma-
turity models, and (iii) a summary of the key findings obtained when analyzing two
maturity self-assessment approaches.

This paper is organized as follows. The proposed comparison framework is intro-
duced in section 2 together with the newly defined coverage and spread ratios. Section
3 discusses the results obtained following the proposed framework to compare two ma-
turity self-assessment approaches (IMPULS, PwC). The last section concludes this pa-
per and discusses the next steps.
# 2 Overall comparison framework

## 2.1 Background literature

In general, the term “maturity” refers to a “state of being complete, perfect, or ready” [6]. Maturity models provide a structured approach to initiate and accompany short-term operational projects, as well as medium- term tactical changes and long-term strategic change [7]. Currently, there is a variety of available digital maturity models to support companies in their digitalization activities. Their common goal is to assess the digital maturity level of an organization, providing an indication of required activities to increase the maturity level. Existing studies have reviewed most common maturity models, in general [8], as well as digital maturity models in particular [9]. According to the above and other related studies, common features for maturity models include their incorporation of maturity dimensions (usually 3-7 dimensions descriptive of the maturity to be assessed by means of maturity models, which are often divided into more detailed maturity criteria, descriptive of the related maturity dimensions), maturity levels and related maturity descriptions. Maturity dimensions, in general, can be divided into three broader categories: maturity of people/culture (e.g. skills, capabilities), processes/structures, and objects/technology (such as ICT tools). A recent literature review-based conceptual paper related to the broad concept of digital maturity [9] demonstrates that in current digital maturity studies, digital maturity has included aspects of digital maturity that can be divided into eight capability dimensions (i.e. broad digitalization related maturity categories): strategy, leadership, business and operating model, people, culture, governance, and technology. Prior research [10], in addition, presumes that the development of a specific set of the above types of digital capabilities leads to higher digital maturity, and moreover, the higher degree of digital maturity can lead to superior corporate performance. However, such maturity models can be very different in terms of their structure, scope and industry focus [11]. Furthermore, currently, in current research, there has been conceptual unclarity and fragmented views about the concept and the measurement frameworks for digital maturity [9], while Rossman’s recent study has been among the first to bring a more unified conceptualizations for the topic. It is also our attempt to clarify the topic and the concept of digital maturity, through our framework designed to compare different digital maturity assessment models.

## 2.2 Comparison framework

To ease the description of the proposed comparison framework of digital maturity models, and to start generalizing the approach, a proper formalization is introduced. A maturity model $M^κ$ (with $κ \in \{IMPULS, PwC, ADN, ...\}$) contains $N_c^κ$ criteria denoted $C_i^κ$ (with $i \in [1..N_c^κ]$) and grouped in $N_a^κ$ dimensions denoted $D_j^κ$ (with $j \in [1..N_a^κ]$). The $j^{th}$ dimension $D_j^κ$ contains $N_{c,j}^κ$ criteria, which start at index $s_j^κ$ and end at index $e_j^κ$. The criteria can be gathered together in the list $L_j^κ = \{C_i^κ, i \in [s_j^κ..e_j^κ]\}$. The following rules apply:
$$N_c^\kappa = \sum_{j=1}^{N_d^\kappa} N_{c,j}^\kappa \qquad (1)$$

$$s_1^\kappa = 1, \text{ and } \forall j \in [2..N_d^\kappa], s_j^\kappa = s_{j-1}^\kappa + N_{c,j-1}^\kappa \qquad (2)$$

$$\forall j \in [1..N_d^\kappa], e_j^\kappa = \sum_{k=1}^{j} N_{c,k}^\kappa \qquad (3)$$

In the rest of the document, the so-called criterion $E_i^\kappa$ (with $i \in [1..N_c^\kappa]$) of a maturity model $M^\kappa$ will be considered as a KPI. The overall comparison framework of two maturity models $M^{\kappa_1}$ and $M^{\kappa_2}$ is composed of three main steps which are further detailed in the next subsections (figure 1):

- **Reverse Engineering (RE in figure 1)**: when the maturity models to be compared do not explicitly formulate the adopted KPIs, they are reverse engineered before starting the matching phase;

- **Matching of the KPIs**: the KPIs of the two compared maturity models are cross-checked and systematically compared in pairs so as to evaluate the levels of matching, which are captured in the so-called matching matrix;

- **Coverage and spread ratios computation**: quantitative metrics are computed from the matching matrix to further analyze the coverage and spread ratios of the KPIs against the models and their dimensions.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Overall comparison framework of two maturity models $M^{\kappa_1}$ and $M^{\kappa_2}$</figcaption></figure>

## 2.3 Reverse Engineering of KPIs

This step is only required for maturity models that do not provide enough information on the KPIs used to assess the maturity levels. More specifically, this happens in cases when the considered maturity models are not sufficiently detailed and internal assessment mechanisms resemble black boxes.

This step aims at extracting and formalizing the list of KPIs that best characterize the criteria adopted by a given method to assess the maturity levels by using all available resources describing the considered maturity model (e.g., online self-assessment tools, questionnaires, benchmarking reports, articles). The output list of KPIs results from consensual exchanges involving a pool of experts in the domain. During the evaluation, experts are requested to focus on the explicitly available information rather than on more implicit data whose interpretation could be questionable. Following this process, the risk of bias due to reinterpretations is reduced, but cannot be fully disregarded.
## 2.4 Matching of KPIs

The comparison of the maturity models is performed at the level of the KPIs. To characterize the ‘KPIs match’, three levels are introduced: *Strong match*, *Partial match*, and *No match*. Two KPIs are considered a Strong match if the experts involved in this process identify sufficient similarity between the two. Conversely, if the two KPIs do not share any similar features, a No match is considered. In between, when the KPIs share some similar features, but also have dissimilarities, a Partial match is assigned. Such a three-level matching analysis presents a good tradeoff between an under-segmentation, which would lead to a coarse analysis, and an over-segmentation that would complexify the comparison making it cumbersome and not practical.

Therefore, the matching function $CCmat$, evaluating the matching level of two KPIs $C_{i_1}^{\kappa_1}$ and $C_{i_2}^{\kappa_2}$ of two maturity models $M^{\kappa_1}$ and $M^{\kappa_2}$, is defined as follows:

$$CCmat(C_{i_1}^{\kappa_1}, C_{i_2}^{\kappa_2}) = \begin{cases} \text{Strong if } C_{i_1}^{\kappa_1} \text{ and } C_{i_2}^{\kappa_2} \text{ strongly match} \\ \text{Partial if } C_{i_1}^{\kappa_1} \text{ and } C_{i_2}^{\kappa_2} \text{ partially match} \\ \text{No otherwise} \end{cases} \quad (4)$$

This matching function is then called to fill in the matching matrix $MMat$ containing the $(N_c^{\kappa_1} \times N_c^{\kappa_2})$ values returned by the function when composed with the KPIs of $M^{\kappa_1}$ and $M^{\kappa_2}$. Clearly, due to the adopted procedure, one can notice that the matching function $CCmat$ is symmetric, i.e., it returns the same matching level no matter the order of the arguments.

Here again, the assessment of the matching levels results from consensual exchanges meetings involving a pool of experts. In a first individual phase, experts are asked to suggest a matching level for each KPIs couple. Then, during a consensus phase, experts exchange on their classifications and further discuss the matching levels for which there are discrepancies. When the discussion fails to reach an adequate consensus, a simple majority rule can be used, while weighting differently the choice of the most experienced experts. Ultimately, an additional expert is to be considered to solve the residual conflicts. Thus, the matching process final results strongly rely on the exchanges between the involved experts, and consequently on their knowledge and experience in the domain. Clearly, similar results could hardly be obtained using simple text-based similarity analysis tools. This is further discussed in the conclusion.

## 2.5 Coverage and spread ratios computation

The computation of the coverage and spread ratios is directly based on the counting of the number of KPIs assigned to the three previously introduced matching levels, as well as on the overall number of KPIs and dimensions of the compared maturity models. Thus, two counting functions are first introduced to track the number of matched KPIs of a certain level within the overall matching matrix. The first function, $CMLcount$, counts the number of times a KPI of the first maturity model is matched to the KPIs of the second maturity model with a given matching level. The second function,
CDLcount, performs a similar search but on a particular dimension of the second maturity model. They are expressed as:

$$ \text{CMLcount}(C_{i_1}^{\kappa_1}, \mathcal{M}^{\kappa_2}, \text{level}) = \sum_{i_2=1}^{N_c^{\kappa_2}} [\text{CCmat}(C_{i_1}^{\kappa_1}, C_{i_2}^{\kappa_2}) == \text{level}] \quad (5) $$

$$ \text{CDLcount}(C_{i_1}^{\kappa_1}, D_{j_2}^{\kappa_2}, \text{level}) = \sum_{i_2=s_{j_2}^{\kappa_2}}^{e_{j_2}^{\kappa_2}} [\text{CCmat}(C_{i_1}^{\kappa_1}, C_{i_2}^{\kappa_2}) == \text{level}] \quad (6) $$

where “level” corresponds to one of the previously introduced levels, i.e. Strong match, Partial match or No match. The equality test (==) returns 1 in case the two compared levels are the same, and 0 otherwise. Capital letters used in the name of the functions help understanding the type of processed data, i.e. C for criteria, M for model, L for level, D for dimension, S for strong, P for partial and N for no. This naming strategy is adopted for each newly introduced function.

Based on those definitions, the two following functions can be defined so as to consider both Partial and Strong matching levels at the same time:

$$ \text{CMcount}(C_{i_1}^{\kappa_1}, \mathcal{M}^{\kappa_2}) = \text{CMLcount}(C_{i_1}^{\kappa_1}, \mathcal{M}^{\kappa_2}, \text{Strong}) + \text{CMLcount}(C_{i_1}^{\kappa_1}, \mathcal{M}^{\kappa_2}, \text{Partial}) \quad (7) $$

$$ \text{CDcount}(C_{i_1}^{\kappa_1}, D_{j_2}^{\kappa_2}) = \text{CDLcount}(C_{i_1}^{\kappa_1}, D_{j_2}^{\kappa_2}, \text{Strong}) + \text{CDLcount}(C_{i_1}^{\kappa_1}, D_{j_2}^{\kappa_2}, \text{Partial}) \quad (8) $$

**Coverage ratios.** These percentages characterize how much two maturity models overlap. The four ratios are computed while evaluating the number of strongly, partially, strongly-and-partially, and not matched KPIs of a maturity model $\mathcal{M}^{\kappa_1}$ when compared to the KPIs of another maturity model $\mathcal{M}^{\kappa_2}$. Thus, the four following functions make use of the previously introduced counting functions:

$$ \text{SMMcover}(\mathcal{M}^{\kappa_1}, \mathcal{M}^{\kappa_2}) = \frac{1}{N_c^{\kappa_1}} \times \sum_{i_1=1}^{N_c^{\kappa_1}} [\text{CMLcount}(C_{i_1}^{\kappa_1}, \mathcal{M}^{\kappa_2}, \text{Strong}) \geq 1] \quad (9) $$

$$ \text{PMMcover}(\mathcal{M}^{\kappa_1}, \mathcal{M}^{\kappa_2}) = \frac{1}{N_c^{\kappa_1}} \times \sum_{i_1=1}^{N_c^{\kappa_1}} \left[ \left( \text{CMLcount}(C_{i_1}^{\kappa_1}, \mathcal{M}^{\kappa_2}, \text{Partial}) \geq 1 \right) \text{AND} \left( \text{CMLcount}(C_{i_1}^{\kappa_1}, \mathcal{M}^{\kappa_2}, \text{Strong}) == 0 \right) \right] \quad (10) $$

$$ \text{SPMMcover}(\mathcal{M}^{\kappa_1}, \mathcal{M}^{\kappa_2}) = \text{SMMcover}(\mathcal{M}^{\kappa_1}, \mathcal{M}^{\kappa_2}) + \text{PMMcover}(\mathcal{M}^{\kappa_1}, \mathcal{M}^{\kappa_2}) \quad (11) $$

$$ \text{NMMcover}(\mathcal{M}^{\kappa_1}, \mathcal{M}^{\kappa_2}) = 1 - \text{SPMMcover}(\mathcal{M}^{\kappa_1}, \mathcal{M}^{\kappa_2}) \quad (12) $$

where the inequality (≥) and AND tests return 1 if true and 0 otherwise. From equation (10), one can see that a KPI of $\mathcal{M}^{\kappa_1}$ is considered as partially covering the KPIs of $\mathcal{M}^{\kappa_2}$, if it matches to at least one KPI of $\mathcal{M}^{\kappa_2}$ at a Partial level, and if there is however no Strong match. Indeed, a Strong match absorbs a Partial match. Of course, the above functions are no more symmetric and are to be evaluated in both directions, i.e. coverage of $\mathcal{M}^{\kappa_1}$ when compared to $\mathcal{M}^{\kappa_2}$, and from $\mathcal{M}^{\kappa_2}$ to $\mathcal{M}^{\kappa_1}$.
**Spread ratios.** These percentages characterize how much a KPI of a maturity model spreads over another maturity model, i.e. how much a KPI is interlaced inside a given maturity model. Thus, the spread ratios are KPI-dependent and are to be evaluated for each KPI of each maturity model. Two spread ratios can be distinguished. The first radio (SPCMspread) evaluates the number of matched KPIs between a given KPI and all the KPIs of a maturity model, when compared to the overall number of KPIs of that maturity model. The second ratio (SPCDspread) evaluates the number of dimensions of a maturity model to which a KPI is matched, when compared to the overall number of dimensions of that maturity model. Here, Strong and Partial matching levels are considered all together using both CMcount and CDcount counting functions:

$$ \text{SPCMspread}(C_{i_1}^{\kappa_1}, \mathcal{M}^{\kappa_2}) = \frac{1}{N_c^{\kappa_2}} \times \text{CMcount}(C_{i_1}^{\kappa_1}, \mathcal{M}^{\kappa_2}) \quad (13) $$

$$ \text{SPCDspread}(C_{i_1}^{\kappa_1}, \mathcal{M}^{\kappa_2}) = \frac{1}{N_d^{\kappa_2}} \times \sum_{j_2=1}^{N_d^{\kappa_2}} [\text{CDcount}(C_{i_1}^{\kappa_1}, D_{j_2}^{\kappa_2}) \ge 1] \quad (14) $$

where it is assumed that the test of inequality returns 1 if true and 0 otherwise. The names of the functions follow the previously introduced naming strategy which makes use of capital letters to specify the type of data manipulated. For instance, SPCMspread refers to the computation of the spread ratio of a criterion (C) over the KPIs of a maturity model (M), when considering both strong and partial (SP) matching.

# 3 Results and discussion

Even though there is a wide variety of different maturity and assessment models available in literature, the proposed comparison framework has been tested and validated with two first maturity models: IMPULS and PwC. Those two maturity models have been selected because they both are digital maturity self-assessment tools easily available online, and the number of questions and the number of dimensions are quite similar and reasonably low for a first testing phase of our novel comparison framework. Of course, the proposed approach is to be tested and validated with other available maturity models and this is further discussed in the conclusion.

## 3.1 IMPULS and PwC maturity models

Considering the formalization introduced in section 2, the two maturity models can be quantitatively characterized by the values gathered together in table 1.

<table><caption>**Table 1.** Numerical characteristics of the compared maturity models</caption><thead><tr><th>κ</th><th>κ<sub>1</sub> = IMPULS</th><th>κ<sub>2</sub> = PwC</th></tr></thead><tbody><tr><td>N<sup>κ</sup><sub>d</sub></td><td>6</td><td>6</td></tr><tr><td>N<sup>κ</sup><sub>c</sub></td><td>19</td><td>33</td></tr><tr><td>{N<sup>κ</sup><sub>c,j</sub>, j ∈ [1..N<sup>κ</sup><sub>d</sub>]}</td><td>{4,4,4,2,3,2}</td><td>{6,6,5,6,6,4}</td></tr></tbody></table>
From this table, one can clearly see that the two maturity models have the same number of dimensions, whose description has been reported in table 2. Clearly, this is a particular case as there is no obvious reason to have $N_d^{IMPULS} = N_d^{PwC}$. It is also clear from this initial analysis that each dimension is not evaluated with the same number of KPIs. For instance, the first dimension of IMPULS includes four KPIs whereas its fourth dimension has only two KPIs. This might be a good indicator to stress implicitly how important the dimensions are in the overall maturity assessment, independently of additional weights that could also be more explicitly used.

<table><caption>**Table 2. Dimensions of the compared maturity models**</caption><thead><tr><th>j<sub>1</sub></th><th>Dimension D<sub>j<sub>1</sub></sub><sup>IMPULS</sup></th><th>j<sub>2</sub></th><th>Dimension D<sub>j<sub>2</sub></sub><sup>PwC</sup></th></tr></thead><tbody><tr><td>1</td><td>Strategy & organization</td><td>1</td><td>Business models, product & service portfolio</td></tr><tr><td>2</td><td>Smart factory</td><td>2</td><td>Market & customer access</td></tr><tr><td>3</td><td>Smart operations</td><td>3</td><td>Value chains & processes</td></tr><tr><td>4</td><td>Smart products</td><td>4</td><td>IT architecture</td></tr><tr><td>5</td><td>Data-driven services</td><td>5</td><td>Compliance, legal, risk, security & tax</td></tr><tr><td>6</td><td>Employees</td><td>6</td><td>Organization & culture</td></tr></tbody></table>

The first part of the proposed framework aims at reverse engineering the KPIs of the maturity models to be compared. Actually, for IMPULS, it has been decided to keep the available and already formalized criteria, even though they are sometimes quite generic without considering the underlying dimensions and corresponding questions (table 3). Thus, only the KPIs of PwC have been reverse engineered through a consensus workshop involving four experts (table 4). Starting from the available online self-assessment tool of PwC, each question and possible answers have been carefully analyzed and discussed to come out with a consensual formalization of the KPIs. This step is not straightforward and required several in-depth discussions to achieve a consensus. The main difficulty was to avoid over-interpretation of the online questionnaire and to remain as objective and factual as possible.

<table><caption>**Table 3. KPIs from IMPULS maturity model**</caption><thead><tr><th>i<sub>1</sub></th><th>KPI c<sub>i<sub>1</sub></sub><sup>IMPULS</sup></th><th></th><th></th></tr></thead><tbody><tr><td>1</td><td>Implementation status of I4.0 strategy</td><td>10</td><td>Cloud usage</td></tr><tr><td>2</td><td>Operationalization and review of I4.0 strategy through a system of indicators</td><td>11</td><td>IT security</td></tr><tr><td>3</td><td>Investment activity relating to I4.0</td><td>12</td><td>Autonomous processes</td></tr><tr><td>4</td><td>Use of technology and innovation management</td><td>13</td><td>Add-on functionalities</td></tr><tr><td>5</td><td>Digital modeling through the collection, storage and processing of data</td><td>14</td><td>Data collection and use</td></tr><tr><td>6</td><td>Functionalities of the equipment infrastructure</td><td>15</td><td>Availability of data-driven services</td></tr><tr><td>7</td><td>Data usage</td><td>16</td><td>Share of revenues derived from data-driven services</td></tr><tr><td>8</td><td>IT system</td><td>17</td><td>Share of data used</td></tr><tr><td>9</td><td>Information sharing</td><td>18</td><td>Skill levels</td></tr><tr><td></td><td></td><td>19</td><td>Effort of the company to acquire new skills</td></tr></tbody></table>
<table><caption>Table 4. KPIs reverse engineered from PwC maturity model</caption><thead><tr><th>i<sub>2</sub></th><th>Reverse engineered KPI C<sup>PwC</sup><sub>i2</sub></th><th></th></tr></thead><tbody><tr><td>1</td><td>Contribution of digital features, products and services to the overall value creation of the organization's portfolio</td><td>17</td></tr><tr><td>2</td><td>Degree of digitalization of the organization's products and/or services</td><td>18</td></tr><tr><td>3</td><td>Products customization possibilities by the customers</td><td>19</td></tr><tr><td>4</td><td>Degree of digitalization of the products life cycle phases</td><td>20</td></tr><tr><td>5</td><td>Importance of data usage and analysis for the organization's business model</td><td>21</td></tr><tr><td>6</td><td>Intensity of the collaboration with external partners and clients for the development of products and services</td><td>22</td></tr><tr><td>7</td><td>Level of integrated sales channels used to sell the organization's products</td><td>23</td></tr><tr><td>8</td><td>Level of integration of communication channels for customer interactions</td><td>24</td></tr><tr><td>9</td><td>Degree of digital enablement of the organization's sales forces</td><td>25</td></tr><tr><td>10</td><td>Degree of dynamic customization of the prices based on customer's willingness to pay</td><td>26</td></tr><tr><td>11</td><td>Degree of customer data analysis to increase customer insight</td><td>27</td></tr><tr><td>12</td><td>Level of collaboration with partners regarding customers access approach</td><td>28</td></tr><tr><td>13</td><td>Degree of vertical value chain digitalization from product development to production</td><td>29</td></tr><tr><td>14</td><td>Capability to monitor production and to dynamically respond to changes in demand</td><td>30</td></tr><tr><td>15</td><td>Degree of integration of the end-to-end IT enabled planning and steering process over the entire value chain</td><td>31</td></tr><tr><td>16</td><td>Degree of digitalization of the production equipment up to a virtual representation of the factory</td><td>32</td></tr><tr><td></td><td></td><td>33</td></tr></tbody></table><table><tbody><tr><td></td><td>Degree of horizontal value chain digitalization from customer order to service</td><td></td></tr><tr><td></td><td>Degree of consideration of the digitalization and I4.0 requirements in IT architecture</td><td></td></tr><tr><td></td><td>Level of use of a Manufacturing Execution System (MES) or similar to control the manufacturing process</td><td></td></tr><tr><td></td><td>Level of maturity of the IT and data architecture to gather, aggregate and interpret real-time manufacturing, product and client data</td><td></td></tr><tr><td></td><td>Importance of new technologies (social media, mobility, analytics and cloud computing) to enable business operations</td><td></td></tr><tr><td></td><td>Ability of the IT organization to fulfill business requirements in the requested time, quality and cost</td><td></td></tr><tr><td></td><td>Level of IT integration with customers and partners</td><td></td></tr><tr><td></td><td>Degree of sophistication of the digital compliance policy</td><td></td></tr><tr><td></td><td>Levels of the organization's IP protection and of the external IP consideration</td><td></td></tr><tr><td></td><td>Level of consideration of the digital product portfolio and production factory in the risk management</td><td></td></tr><tr><td></td><td>Level of management of the digital components of the organization's value chain with respect to tax related topics</td><td></td></tr><tr><td></td><td>Level of consideration of the production in the organization's IT security concept</td><td></td></tr><tr><td></td><td>Level of consideration the service partners or customers into the organization's compliance and risk management</td><td></td></tr><tr><td></td><td>Capability to create value from data so as to optimize operations and foster new business models</td><td></td></tr><tr><td></td><td>Level of the organization's capabilities and resources related to I4.0</td><td></td></tr><tr><td></td><td>Level of involvement, support and expertise of the organization's managers with regards to I4.0</td><td></td></tr><tr><td></td><td>Level of collaboration of the organization with external partners (e.g. academia, industry, suppliers, customers) on I4.0 topics</td><td></td></tr></tbody></table>
<figure><img src="image_3.png" /><figcaption>**Table 5.** Matching matrix MMmat of the considered maturity models (IMPULS, PwC) wherein Green cells correspond to Strong matches, and yellow cells to Partial matches.</figcaption></figure>

## 3.2 Matching matrix MMmat

Following the proposed comparison framework, the matching matrix MMmat then had to be filled out while evaluating the matching levels between the KPIs of the two maturity models. This step involved six experts who took part in a two-step evaluation process as discussed in section 2. Here again, during the consensus phase, particular attention has to be paid in order to avoid over-interpretation of what the KPIs are supposed to assess. The individual assessment phase revealed several conflicts due to multiple possible interpretations of the IMPULS's KPIs. Clearly, those original KPIs (table 3) are not sufficiently detailed and are very much linked to the underlying dimensions and questions. As a consequence, to proceed with those issues, the experts decided to come back to the dimensions and questions in order to better integrate the context in which the KPIs are supposed to be assessed. This has clearly shown the requirement for developing self-understandable KPIs which would directly embed the context within their formulation.

The matching matrix resulting from the consensus phase is shown in table 5. Green colors correspond to Strong matches between two KPIs, and yellow colors to Partial matches, whereas no color indicates No match. For instance, one can observe that five KPIs from each maturity model had a strong match. We can also observe, for instance, that KPI 4 from IMPULS partly matches with four KPIs from PwC.
## 3.3 Coverage and spread ratios

The coverage ratios can be computed using the formula introduced in section 2. They are expressed as percentages. They evaluate in both directions and through their KPIs how much $M^{IMPULS}$ covers $M^{PwC}$, and reversely how much $M^{PwC}$ covers $M^{IMPULS}$. Table 6 gathers together the results obtained using equations (9) to (12), which consider four levels: No, Partial, Strong and Strong-and-Partial overall coverages.

<table><caption>**Table 6.** Coverage ratios computed from MMmat in both directions</caption><thead><tr><th>Direction</th><th>No</th><th>Partial</th><th>Strong</th><th>Overall</th></tr></thead><tbody><tr><td>Coverage from <i>M</i><sup>IMPULS</sup> to <i>M</i><sup>PwC</sup> (%)</td><td>16</td><td>58</td><td>26</td><td>84</td></tr><tr><td>Coverage from <i>M</i><sup>PwC</sup> to <i>M</i><sup>IMPULS</sup> (%)</td><td>24</td><td>61</td><td>15</td><td>76</td></tr></tbody></table>

Overall, when considering the strongly and partially matching KPIs, the coverage is quite high in both directions (84% and 76%). Here, it is important to stress that a very high coverage ratio could be reached even though the coverage matrix has very few colors and a lot of white cells. For instance, the matching matrix of IMPULS compared to itself would be a square matrix with only green cells on its diagonal and having 100% of KPIs strongly matched.

Furthermore, strongly matching KPIs can be clearly distinguished from the others (five KPIs for IMPULS and five for PwC). Indeed, they match so strongly that a common formulation could be thought and the shortlist of newly formulated KPIs could be considered as a common kernel of the two maturity assessment models. This is an important finding for the development of our own maturity model in the next stage. Similarly, the KPIs which do not match at all can be considered specific to a particular maturity model and no common formulation is suggested.

Furthermore, the spread ratios of each KPI can then be evaluated using the equations (13) and (14). The results of those evaluations are gathered together in tables 7 and 8, depending on whether the spread ratios are considered from IMPULS to PwC, or reversely from PwC to IMPULS.

<table><caption>**Table 7.** Spread ratios of IMPULS's KPIs when compared to the overall list of KPIs and dimensions of PwC. Results are sorted according to the overall spread ratios.</caption><thead><tr><th>Rank</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th><th>9</th><th>10</th><th>11</th><th>12</th><th>13</th><th>14</th><th>15</th><th>16</th><th>17</th><th>18</th><th>19</th></tr></thead><tbody><tr><td>KPI</td><td>1</td><td>2</td><td>3</td><td>10</td><td>12</td><td>16</td><td>18</td><td>6</td><td>11</td><td>13</td><td>19</td><td>7</td><td>4</td><td>17</td><td>8</td><td>15</td><td>5</td><td>14</td><td>9</td></tr><tr><td>Overall (%)</td><td>0</td><td>0</td><td>0</td><td>3</td><td>3</td><td>3</td><td>3</td><td>6</td><td>6</td><td>9</td><td>9</td><td>12</td><td>12</td><td>12</td><td>15</td><td>15</td><td>15</td><td>15</td><td>18</td></tr><tr><td>Dimension (%)</td><td>0</td><td>0</td><td>0</td><td>17</td><td>17</td><td>17</td><td>17</td><td>33</td><td>33</td><td>17</td><td>33</td><td>67</td><td>50</td><td>50</td><td>38</td><td>50</td><td>67</td><td>67</td><td>67</td></tr></tbody></table>

<table><caption>**Table 8.** Spread ratios of PwC's KPIs when compared to the overall list of KPIs and dimensions of IMPULS. Results are sorted according to the overall spread ratios.</caption><thead><tr><th>Rank</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th><th>9</th><th>10</th><th>11</th><th>12</th><th>13</th><th>14</th><th>15</th><th>16</th><th>17</th></tr></thead><tbody><tr><td>KPI</td><td>7</td><td>9</td><td>10</td><td>22</td><td>24</td><td>27</td><td>29</td><td>32</td><td>3</td><td>6</td><td>8</td><td>18</td><td>19</td><td>25</td><td>26</td><td>28</td><td>33</td></tr><tr><td>Overall (%)</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>5</td><td>5</td><td>5</td><td>5</td><td>5</td><td>5</td><td>5</td><td>5</td><td>5</td></tr><tr><td>Dimension (%)</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>17</td><td>17</td><td>17</td><td>17</td><td>17</td><td>17</td><td>17</td><td>17</td><td>17</td></tr></tbody></table>
<table><thead><tr><th>Rank</th><th>18</th><th>19</th><th>20</th><th>21</th><th>22</th><th>23</th><th>24</th><th>25</th><th>26</th><th>27</th><th>28</th><th>29</th><th>30</th><th>31</th><th>32</th><th>33</th></tr></thead><tbody><tr><td>KPI</td><td>23</td><td>31</td><td>2</td><td>12</td><td>13</td><td>14</td><td>15</td><td>17</td><td>21</td><td>1</td><td>4</td><td>11</td><td>16</td><td>20</td><td>30</td><td>5</td></tr><tr><td>Overall (%)</td><td>11</td><td>11</td><td>11</td><td>11</td><td>11</td><td>11</td><td>11</td><td>11</td><td>11</td><td>16</td><td>16</td><td>16</td><td>16</td><td>21</td><td>21</td><td>26</td></tr><tr><td>Dimension (%)</td><td>17</td><td>17</td><td>33</td><td>33</td><td>33</td><td>33</td><td>33</td><td>33</td><td>33</td><td>33</td><td>33</td><td>33</td><td>33</td><td>50</td><td>50</td><td>67</td></tr></tbody></table>

Both lists are sorted according to the overall spread ratios. Vertical lines split the tables to group the KPIs which have the same overall spread ratios, and consequently the same number of matched KPIs. For instance, table 7 shows that KPI 9 of IMPULS has the greatest overall spread ratio (rank = 19) since it spreads on six out of 33 KPIs of PwC (6/33 ≈ 18%). It also has the greatest spread ratio over the dimensions since it spreads on 4 out of 6 dimensions of PwC (4/6 ≈ 67%). One can also see that several KPIs do not spread at all (values of 0% in tables 7 and 8), which reveals the specific KPIs of each maturity model. Spreading over too many KPIs or dimension can be confusing. As already highlighted, this can be due to the fact that some KPIs of IMPULS are certainly too generic and can therefore be matched to several of the PwC model's KPIs. From those values, one can see that some strongly matched KPIs spread on a single KPI and on a single dimension (e.g. KPIs 12 and 16 of IMPULS over the KPIs and dimensions of PwC). Again, this configuration is suitable to circumvent the action level of the considered KPIs. Furthermore, the analysis reveals that the two maturity models are not organized in the same way for some reasons not yet been fully identified.

As introduced in section 2, the spread ratios somehow characterize how much a given KPI is interlaced with the KPIs and dimensions of another maturity model. Such an understanding can be very beneficial to split existing KPIs in lower-level KPIs that would evaluate more circumscribed criteria. Furthermore, this can also be interesting to define new KPIs and assign new dimensions so as to limit the spread over several KPIs and dimensions. Those improvements can certainly help to better rationalize the evaluation, and consequently limit the duplications and misunderstandings when performing a digital maturity self-assessment.

## 4 Conclusion and future works

Digital maturity models help identifying the maturity level of SMEs with respect to specific KPIs and dimensions, and consequently they provide important inputs to better design and setup the digital transformation plans. Today, many countries and consulting firms have been engaged in the development of their own model, and it is therefore required to understand the positioning of each model with respect to the others. This paper has introduced a framework to compare two digital maturity models within the Industry 4.0 paradigm. The new comparison framework consists of three successive steps: 1) reverse engineering of the KPIs when not explicitly available; 2) matching of the KPIs to identify the Strong matches, Partial matches, and No matches; and 3) computation of the coverage and spread ratios to further characterize the overlap and interlace of the two maturity models assessed. The proposed framework has been tested and validated with two maturity models, namely from IMPULS and PwC.
Our results show that the proposed approach is capable to successfully capture the
similarities and differences between the KPIs of two maturity models. The reverse en-
gineering and matching steps could have hardly been performed with some automatic
text-based or corpus-based similarity evaluation tools. Thus, the pool of experts has
played a key role. Of course, the work now needs to be extended and tested with addi-
tional maturity models. For instance, this will certainly help defining a strong, common
kernel of KPIs, i.e., those identified as strongly matching across the board. This will be
very helpful to specify a new maturity model, together with its KPIs and dimensions.

Through the analysis, some limitations clearly appeared. First, mitigation measures
had to be set-up to avoid over-interpreting the KPIs. In this sense, experts were asked
to focus on explicit and tangible information rather than on implicit ones whose inter-
pretation can be discussed endlessly. Second, KPIs should be as much self-explanatory
as possible in order to avoid going back to the dimensions or questions to clearly un-
derstand the context of use. Finally, to avoid working on too much interlaced KPIs,
criteria should be decomposed in low-level KPIs.

References

1. Taisch, M.; Arena, D.; Gorobtcova, P.; Kiritsis, D.; Luglietti, R.; May, G.; Morin, T.; Wuest, T., World Manufacturing Forum 2018 Report, World Manufacturing Forum, (2018).

2. Scremin, L.; Armellini, F.; Brun, A.; Solar-Pelletier, L.; Beaudry, C., Towards a framework to assess the maturity for Industry 4.0 adoption in manufacturing companies, in: Brunet-Thornton, R.; Martínez, F. (eds.) Analyzing the Impacts of Industry 4.0 in Modern Business Environments, IGI-Global, chapter 12, pp. 224-254, (2018).

3. Mittal, S.; Khan, M.; Romero, D.; Wuest, T., A Critical Review of Smart Manufacturing & Industry 4.0 Maturity Models: Implications for Small and Medium-sized Enterprises (SMEs). Journal of Manufacturing Systems, vol.49, pp. 194-214, (2018).

4. Lichtblau, K.; Stich, V.; Bertenrath, R.; Blum, M.; Bleider, M.; Millack, A.; Schmitt, K.; Schmitz, E.; Schröter, M., "IMPULS - Industrie 4.0-Readiness", VDMA, (2015).

5. Geissbauer, R.; Vedso, J.; Schrauf, S., Industry 4.0: building the digital enterprise, 2016 global industry 4.0 survey, (2016).

6. Simpson JA, Weiner ES C (1989) The Oxford English dictionary. Oxford University Press.

7. Felch, V., Asdecker, B. and Sucky, E. (2019) Maturity Models in the Age of Industry 4.0 – Do the Available Models Correspond to the Needs of Business Practice? Proceedings of the 52nd Hawaii International Conference on System Sciences, 2019.

8. Wendler, R., 2012. The maturity of maturity model research: A systematic mapping study. Information and Software Technology, 54(12), pp.1317–1339.

9. Rossman, A. (2018) Digital Maturity: Conceptualization and Measurement Model. Proceedings of Thirty Ninth International Conference on Information Systems, 2018.

10. Westerman, G., Bonnet, D., and McAfee, A. (2014) Leading Digital: Turning Technology into Business Transformation. Harvard Business Review Press.

11. Schwer, K., Hitz, C., Wyss, R., Wirz, D. and Minonne, C. (2018) Digital maturity variables and their impact on the enterprise architecture layers. Problems and Perspectives in Management, Vol. 16, Iss. 4.