# How to Measure Digitalization? A Critical Evaluation of Digital Maturity Models

Tristan Thordsen, Matthias Murawski, Markus Bick

► To cite this version:

Tristan Thordsen, Matthias Murawski, Markus Bick. How to Measure Digitalization? A Critical Evaluation of Digital Maturity Models. 19th Conference on e-Business, e-Services and e-Society (I3E), Apr 2020, Skukuza, South Africa. pp.358-369, 10.1007/978-3-030-44999-5_30. hal-03222863

HAL Id: hal-03222863

https://inria.hal.science/hal-03222863

Submitted on 10 May 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# How to measure digitalization?
## A critical evaluation of Digital Maturity Models

Tristan Thordsen¹, Matthias Murawski¹ and Markus Bick¹

¹ESCP Business School Berlin, Berlin, Germany
{tthordsen,mmurawski,mbick}@escpeurope.eu

**Abstract.** To preserve competitive advantage in a more and more digitalized environment, today's organizations seek to assess their level of digital maturity. Given this particular practical relevance, a plethora of digital maturity models, designed to assess a company's digital status quo, has emerged over the past few years. Largely developed and published by practitioners, the academic value of these models remains obviously unclear. To shed light on their value in a broader sense, in this paper we critically evaluate 17 existing digital maturity models – identified through a systematic literature search (2011-2019) – with regard to their validity of measurement. We base our evaluation on established academic criteria, such as generalizability or theory-based interpretation, that we apply in a qualitative content analysis to these models. Our analysis shows that most of the identified models do not conform to the established evaluation criteria. Based on these insights, we derive a detailed research agenda and suggest respective research questions and strategies.

**Keywords:** Digital Maturity Models, Measurement, Research Agenda

## 1 Introduction

If we believe in the press, consultancies, and the majority of researchers, being ‘digital’ is paramount for companies to stay competitive in today’s business world. The goal is to seize the opportunities of the ongoing digital transformation [1]. To evaluate the status quo of a company’s digitalization and to provide guidance for future investments, latest IS literature has established the term digital maturity. Following Chanias and Hess, digital maturity is “the status of a company’s digital transformation” – it describes “what a company has already achieved with regard to transformation efforts” [2]. Here, efforts include implemented changes from an operational point of view as well as acquired capabilities regarding the mastering of the transformation process. In this context, given the high practical relevance of the topic, a great variety of so-called digital maturity models (DMMs) has emerged in the past few years. According to Mettler [3] this trend will prevail as the demand for maturity models will further increase.

Largely developed and published by consultancies in a practical setting, DMMs on the one hand aim at measuring the current level of a company’s digitalization, on the other hand at providing a model path to digital maturity. However, so far, due to their practical nature and the lack of external quality assessments such as peer reviews, the
generalizability and consistency of DMMs remains largely unclear. The constant pub-
lication of new maturity models for similar fields of application suggests a certain ar-
bitrariness and does not contribute to the clarification of this case.

An analysis of corresponding literature of this field shows that the academic com-
munity also does not yet provide comprehensive answers to these questions. Although
there are scholars attempting to measure digitalization based on more concrete metrics
(e.g. [4]), these metrics are however neither comprehensive nor generalizable.

To provide not only practical but also substantial theoretical value, DMMs should conform to several academic requirements such as a common understanding of their underlying concepts. However, although the terms digital and digitalization of a company exist for many years already, they still remain abstract. Related concepts are context-dependent (e.g., society, company, individual) and vary according to the perspectives on the topic (e.g., human, process, technology). This lack of a common and concrete definition leads to a certain ambiguity already when it comes to the basic task of DMMs: measuring a company's level of digitalization. In this context, questions arise such as:

What are relevant variables to measure digitalization? How can they be quantified?
How can a certain comparability between companies be ensured? How can we investi-
gate if a certain level of digitalization affects the performance of a company?

Unfortunately, the authors of existing models only rarely reveal their motivation be-
hind the procedures and results of their measurement [5].

Given their high practical relevance and popularity, we have set out to evaluate ex-
istng DMMs with regard to their conformity to above mentioned quality standards and
thus theoretical value. Determining the status quo of digitalization is the main goal of
DMMs. Without such an assessment, DMMs cannot provide a comprehensive path to
higher levels of maturity. We thus focus in our analysis on the quality of measurement
of a company's digitalization carried out by the single models. While doing so, we
follow the request for further scholarly inquiry on digital maturity, in order to contribute
to a more profound understanding of the ongoing sociotechnical phenomenon of digital
transformation [6]. Furthermore, this work is in line with existing works who are calling
for a deeper understanding, analysis and differentiation of IS maturity models and their
usefulness for practitioners [7, 8]. With our undertaking we also answer the call to iden-
tify and remedy shortcomings of existing maturity models [5]. The research question
we have set out to answer is:

*To what extent do DMMs respect quality standards in their measurement of a com-
pany's degree of digitalization?*

To answer the research question, we have selected established measurement criteria as basis for the evaluation of the DMMs [9]. In this paper, we will first provide a defi-
nition and background information regarding DMMs. Then, we identify quality criteria for the process of measurement and derive a catalogue of requirements from established literature. Based on this catalogue, we will evaluate and compare relevant DMMs iden-
tified through a systematic literature search. Finally, after a discussion of this evalua-
tion, we propose a research agenda to remedy shortcomings of existing maturity models and provide guidance for future research.
# 2 Theoretical Background

## 2.1 Digital Maturity Models

The term maturity can be defined as “the ability to respond to the environment in an appropriate manner through management practices” [10]. According to Scott and Bruce maturity with regards to an organization is a “reflection of the appropriateness of its measurement and management practices in the context of its strategic objectives and in response to environmental change” [11]. Rosemann and de Bruin describe maturity more pragmatically as “a measure to evaluate the capabilities of an organization in regard to a certain discipline” [12, p. 2]. Scholars agree that the organization’s reaction to its external environment is generally learned rather than instinctive. However, the maturity of an organization does not necessarily relate to its age.

The evaluation of a firm's maturity marks one of the key points in the process of achieving a higher level of organizational performance [11]. Defining and assessing the maturity of different types of organizational resources allows organizations to evaluate their capabilities with respect to different business areas. In this context, for example, the maturity of certain organizational processes, objects or technologies can be the focus of such a reflection. Based on this concept, maturity models are multistage frameworks to describe a typical path in the development of organizational capabilities [12]. Here, the different levels are conceptualized in terms of evolutionary stages [13]. Maturity models can be described as normative reference models. In general, they are applied in organizations in order to assess the current status quo of a set of capabilities, to derive measures for improvement and prioritize them accordingly [1]. Through a set of benchmark variables and indicators for each stage, the matrix frameworks offer a possibility to make the progress of an object of interest towards a desired state tangible [14]. The relationship between maturity and organizational performance is properly understood, e.g. higher maturity leads to higher performance [15].

Given the simplistic character of maturity models and their practical value, several frameworks have emerged over the last decades across all disciplines [8]. Most recently, in the context of digitization, researchers and practitioners have set out to assess the digital maturity of organizations [2].

Despite their relevance and popularity in various management disciplines, the development and application of maturity models still bears a number of shortcomings [16]. The most prominent point of criticism concerns the poor theoretical basis and empirical evidence of maturity models [17]. Furthermore, de Bruin et al. claim that particularly limited documentation on the development of the maturity model may contribute to the lack of validity and rigor [19]. In response, IS researchers have increasingly set out to determine guidelines for the development of maturity models that are intended to bolster more rigorous design processes [18]. However, regarding the measurement procedures of maturity models, existing academic literature does not yet provide specific quality criteria. In the following, we have thus set out to derive a set of requirements for that purpose based on established literature in the field.
## 2.2 Criteria of a valid measurement process

The procedure of measurement can be considered as one of the main components of the research process itself [19]. Ferris provides a comprehensive definition of the term measurement: "It is an empirical process, using an instrument, effecting a rigorous and objective mapping of an observable into a category in a model of the observable that meaningfully distinguishes the manifestation from other possible and distinguishable manifestations [22, p. 101]."

The quality of measurement is determined by its validity. The concept of validity refers to the substance, the proximity to the truth, of inferences made when justifying and interpreting observed scores. The validity of measurement is evaluated through the underlying assertions, building a complex net of arguments to back up the findings. The judgement about validity is thus made based on so-called validity arguments [19]. According to the Standards for Educational and Psychological Testing "validity is the most important consideration in test evaluation" [20]. However, Kane points out that like many virtues, it is more honored than practised [24]. Therefore, Kane et al. have developed a set of requirements designed to warrant a certain level of validity in performance measurement [9].

The five requirements are: 1) Observation, 2) Generalizability, 3) Theory-based Interpretation, 4) Exploration, and 5) Implication [9, 21, 22]. Brühl offers an updated and comprehensive overview of these criteria. After having consulted additional literature in this field, as suggested by Becker et al. [4], we have chosen the five guidelines defined by Kane et al. and Brühl as theoretical foundation and thus as the basis for our argument [25,[19]. The requirements catalogue for a valid measurement process in DMMs is the following:

**Observation:** This argument ensures that the measurement of a single indicator is in line with the predefined measurement procedure [19]. In other words, this requirement makes sure that on the one hand, the appropriate indicator for the subject matter is measured and on the other hand, that it is measured correctly. In this context, the target domain needs to be defined first. The target domain designates the full range of performances actually observed and included in the interpretation. It differs from a wider range of observations that is not subject to interpretation. A target's expected score over all possible performances in the target domain is defined as the target score.

To determine the target domain and lay the foundation for the target score, the definition of the subject matter to be interpreted should be as specific as possible [9]. As we have already pointed out, the target domain of interest with regard to a company's digitalization tends to be very broad. Central points of evaluation for the DMMs derived from the argument of observation here are thus:

* Definition of the phenomenon

* Definition of target domain

* Predefined measurement procedure to determine target scores

**Generalizability:** This requirement constitutes in a statistical generalization. Researchers conclude from the observed target score based on actual performance on all
expected scores of similar tasks of the target domain. In this context, it is assumed that observed scores are “based on random or at least representative samples” [9]. It is thus implied that a certain measurement is valid at any point of time; all context-specific factors are abandoned [19]. For this procedure, the measurement needs to involve a sample of performances from the target domain. Generalizability increases as the sample of independent observations increases. Also, standardization of the assessment procedure tends to improve the generalizability of the scores. This argument leads us to a set of two evaluation criteria for the DMMs:

* Measurement approach

* Sample size of independent observations

* Degree of standardization in the measurement procedure

**Theory-based Interpretation:** This argument stresses that measurement requires a sound theoretical framework. This means that every single indicator should have a 'theoretical' connection to the overall construct [19]. Assumptions that are made need to be backed up by corresponding theory. Given the practical setting the DMMs largely stem from, we will focus in our evaluation on a single key point:

* Theoretical basis of the model/ measurement procedure

**Extrapolation:** This concept is important for scenarios in which the measured construct is put into relation with other constructs [19]. An example here could be that the level of digital maturity identified by the DMM is put in relation with the performance or competitiveness of a company. In this context, especially the plausibility of inferences is of importance to guarantee the validity of the measurement. With regards to extrapolation we thus focus on:

* Assumed connections between maturity level and other constructs

* Plausibility of inferences to argument for such a relationship

**Implications:** This criterion is relevant if decisions are to be made based on the measured construct [19]. Here, again the transparency and plausibility of inferences made to justify these decisions are key. A thorough chain of arguments is necessary to back up the path from the initial status quo of digitalization that is measured towards higher levels of digital maturity. For this argument, we thus focus on:

* Justification of steps on the path towards maturity

With regards to the design of our evaluation and comparison of the DMMs we follow Becker et al.'s established work in the field of IT maturity models [5]. We will take these criteria as the basis for evaluating the DMMs identified through the next step (see 3.2).
# 3 Research Design

## 3.1 Literature Review for selecting Digital Maturity Models

Vom Brocke et al. state that the searching process of the literature review “must be comprehensively described” so that “readers can assess the exhaustiveness of the review and other scholars can more confidently (re)use the results in their own research [23].” In this context, researchers of the IS field suggest a systematic and structured way of identifying and reviewing the literature [24].

We searched through a 8-year period (2011 to 2019) in ten leading IS journals, five major IS conferences and two additional databases (Business Source Premier and Google Scholar). The chosen timeframe appears to be especially relevant as in 2011 the first so called DMM was published by a consultancy. The outlets and databases have been selected based on the experience reported by [12] who have investigated existing IS maturity models.

The search terms used in our systematic review were constructed based on the PICO criteria (Population, Intervention, Comparison and Outcomes). These criteria are usually applied as guidelines in the medical field to frame the research question by identifying keywords and formulating search strings. Kitchenham and Charters deem the PICO criteria as particularly suitable when conducting a systematic review in the academic context of Information Systems [25]. In addition to deriving major search terms through PICO, we identified synonyms and alternative spellings for these keywords by consulting both experts and literature of this field [26]. As a result the following keywords were compiled: “maturity model”, “stages of growth model”. “stage model”, “change model”, “transformation model”, and “grid”. In addition, we framed the literature search using “Information Technology” and “Digital*”. Lastly, when constructing the search strings, we used both the Boolean AND and OR to on the one hand link the major terms derived in the first step and on the other hand to incorporate these synonyms and alternative spellings in our systematic literature search. We first applied the search strings to the IS journals and conferences to check for the accuracy of the search terms. As a result, we added “grid” as search term and applied the search string to the electronic databases.

<table><caption>Table 1. Identified Digital Maturity Models</caption><thead><tr><td>ID</td><td>Reference</td><td>Name of DMM</td><td>Year</td></tr></thead><tbody><tr><td>1</td><td>[27]</td><td>Industry Digitization Index</td><td>2011</td></tr><tr><td>2</td><td>[28]</td><td>Digital Transformation Maturity</td><td>2011</td></tr><tr><td>3</td><td>[29]</td><td>Digital Maturity Matrix</td><td>2012</td></tr><tr><td>4</td><td>[30]</td><td>Status of Digitalization</td><td>2013</td></tr><tr><td>5</td><td>[31]</td><td>Digital Quotient</td><td>2015</td></tr><tr><td>6</td><td>[32]</td><td>Digital Transformation Index</td><td>2015</td></tr><tr><td>7</td><td>[33]</td><td>Digitale Reife</td><td>2016</td></tr><tr><td>8</td><td>[6]</td><td>Stages in Digital Business Transformation</td><td>2016</td></tr><tr><td>9</td><td>[34]</td><td>Digital maturity & transformation report</td><td>2016</td></tr><tr><td>10</td><td>[35]</td><td>Digital maturity model 4.0</td><td>2016</td></tr><tr><td>11</td><td>[36]</td><td>Digital maturity model for telecom.</td><td>2016</td></tr><tr><td>12</td><td>[37]</td><td>Industry 4.0 readiness</td><td>2017</td></tr><tr><td>13</td><td>[1]</td><td>Digital Maturity in Traditional Industries</td><td>2017</td></tr><tr><td>14</td><td>[38]</td><td>Maturity assessment for Industry 4.0</td><td>2018</td></tr><tr><td>15</td><td>[39]</td><td>Digitalisierungsindex Mittelstand 2018</td><td>2018</td></tr><tr><td>16</td><td>[40]</td><td>Strategic factors enabling digital maturity</td><td>2019</td></tr><tr><td>17</td><td>[41]</td><td>Maturity Model of Digital Transformation</td><td>2019</td></tr></tbody></table>
## 3.2 Content analysis design

All selected DMMs (see Table 1) were subject to a content analysis. The overarching objective was to identify whether they fulfill the quality criteria defined in section 2.2. We conducted a deductive category assignment in which we considered the corresponding sub-dimensions and assigned one of the following “codes” (see Table 2): A black (‘filled’) Harvey ball indicates that the respective sub-dimension is completely fulfilled. A black white (‘partly filled’) Harvey ball means that the sub-dimension is only fulfilled to some extent. A white (‘blank’) Harvey ball was assigned in case the sub-dimension is not fulfilled at all or not applicable to the model.

For example (see Table 2): DMM 2 has provided a definition for “digital transformation”, but not for “digital transformation maturity”. Therefore, the subdimension 1a) *Definition of the phenomenon* has only partially been fulfilled and marked with a black white Harvey ball.

Following established criteria of qualitative research, this procedure has been done by two researchers independently. They agreed in 94% of the cases. Remaining cases have been discussed among the authors until consensus was reached.

## 4 Findings

The findings of our analysis are summarized in Table 2. Only three of the 17 analyzed models provide definitions for digital maturity. Five models omit giving any definition of the concept of digital maturity itself or of related concepts such as digital transformation or digitization. However, all of the models offer a number of different dimensions to describe the target domain. Dimensions of the target domain are for example: customer experience, operational processes, business models and digital capabilities. Furthermore, the vast majority of the DMMs have a predefined measurement approach as well as a fairly large sample. A large stake of the models does not provide any information about the degree of standardization in the measurement undertaken. Nine of the analyzed DMMs fail to provide a theoretical basis for their work. In most cases, when assumptions with regards to connections between maturity levels and other constructs are made, these were not backed by plausible argumentation. Also, four of the models fail to justify the order of the consecutive steps towards digital maturity. In total, six models are peer reviewed and published in academic outlets.
<table><caption>**Table 2.** Analysis of Digital Maturity Models - Overview</caption><thead><tr><th rowspan="2">ID</th><th colspan="3">1. Observation</th><th colspan="3">2. Generalizability</th></tr><tr><th>a) Definition of the phenomenon</th><th>b) Definition of target domain</th><th>c) Predefined measurement procedure</th><th>a) Measurement approach</th><th>b) Sample size of independent observations</th><th>c) Degree of standardization</th></tr></thead><tbody><tr><td>1</td><td>●</td><td>●</td><td>●</td><td>○</td><td>●</td><td>●</td></tr><tr><td>2</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td><td>○</td></tr><tr><td>3</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>4</td><td>○</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>5</td><td>○</td><td>●</td><td>●</td><td>○</td><td>●</td><td>○</td></tr><tr><td>6</td><td>●</td><td>●</td><td>○</td><td>●</td><td>●</td><td>○</td></tr><tr><td>7</td><td>○</td><td>●</td><td>●</td><td>●</td><td>●</td><td>○</td></tr><tr><td>8</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>9</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>10</td><td>○</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>11</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td><td>○</td></tr><tr><td>12</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>13</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>14</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>15</td><td>○</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>16</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>17</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><th rowspan="2">ID</th><th colspan="2">3. Theory-based Interpretation</th><th colspan="2">4. Extrapolation</th><th colspan="2">5. Implications</th><th rowspan="2">Peer reviewed</th></tr><tr><th>a) Theoretical basis of the model</th><th>a) Assumed connections between maturity level and other constructs</th><th>b) Plausibility of inferences to argument for such relationship</th><th>a) Justification of steps on the path towards maturity</th><th>published in a journal</th></tr><tr><td>1</td><td>○</td><td>●</td><td>●</td><td>○</td><td>○</td><td>No</td></tr><tr><td>2</td><td>○</td><td>○</td><td>○</td><td>○</td><td>●</td><td>No</td></tr><tr><td>3</td><td>○</td><td>●</td><td>●</td><td>●</td><td>●</td><td>No</td></tr><tr><td>4</td><td>○</td><td>○</td><td>○</td><td>○</td><td>○</td><td>No</td></tr><tr><td>5</td><td>○</td><td>●</td><td>●</td><td>●</td><td>○</td><td>No</td></tr><tr><td>6</td><td>○</td><td>●</td><td>●</td><td>●</td><td>●</td><td>No</td></tr><tr><td>7</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td><td>No</td></tr><tr><td>8</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td><td>Yes</td></tr><tr><td>9</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td><td>No</td></tr><tr><td>10</td><td>○</td><td>●</td><td>●</td><td>●</td><td>●</td><td>No</td></tr><tr><td>11</td><td>●</td><td>○</td><td>○</td><td>●</td><td>●</td><td>Yes</td></tr><tr><td>12</td><td>○</td><td>○</td><td>○</td><td>●</td><td>●</td><td>No</td></tr><tr><td>13</td><td>●</td><td>○</td><td>○</td><td>●</td><td>●</td><td>Yes</td></tr><tr><td>14</td><td>●</td><td>●</td><td>○</td><td>●</td><td>●</td><td>Yes</td></tr><tr><td>15</td><td>○</td><td>○</td><td>○</td><td>○</td><td>○</td><td>No</td></tr><tr><td>16</td><td>●</td><td>○</td><td>○</td><td>●</td><td>●</td><td>Yes</td></tr><tr><td>17</td><td>●</td><td>○</td><td>○</td><td>●</td><td>●</td><td>Yes</td></tr></tbody></table>
# 5 Research agenda

Based on our analysis, we suggest a first draft of a future research agenda for the field of DMMs (see Table 2). The research agenda is the result of the discussion on our findings and corresponding potential avenues for further research.

The main challenge researchers should focus on is developing precise conceptual definitions of DMM-related terms. The current situation with multiple understandings and definitions of the core terms hinders the development of a measurement model [42]. Thus, scholars should put effort in defining digital maturity first. We consider this conceptual work as the most important step for improving the research field around DMMs as it lays the foundation for subsequent tasks. Particularly, definitions form the core of theoretical frameworks [43] that have to be developed to increase the quality of DMM research.

Having discussed required conceptual definitions and theoretical frameworks, we can now take a closer look at the empirical aspects. Most existing DMMs are tested through real data, but the quality of the methods and approaches applied largely differs or cannot be evaluated at all. For example, in most of the analyzed papers, the data collection procedure is not transparently explained. Thus, our first call with regards to empirical aspects is to increase transparency of both sampling and data collection. Furthermore, depending on the focus of the model, it could be beneficial to concentrate on specific data groups such as certain company sizes or certain cultures. The second call concerns the methods applied to analyse the data. In most cases, DMMs can be considered as quantitative in nature, which is particularly useful when the goal is for example a comparison between different companies. However, a qualitative approach could contribute to the measurement quality of a company's digital maturity. Qualitative research settings have proven to be especially useful in the process of DMM development [6].

<table><caption>**Table 3.** Suggested Research Agenda</caption><thead><tr><th>Category</th><th>Selected research problems/questions</th><th>Proposed research strategies/methods</th></tr></thead><tbody><tr><td rowspan="2">Definitions and measurement procedure</td><td>How to define digital maturity?</td><td>Conceptual research</td></tr><tr><td>What are the dimensions of DM?</td><td>Content validity checks (e.g., interviews)</td></tr><tr><td rowspan="2">Theoretical framework</td><td>How to design a measurement procedure for DM?</td><td>Construct development</td></tr><tr><td>How to link / position DMM to existing theories?</td><td>Conceptual research</td></tr><tr><td rowspan="2">Empirical aspects</td><td>Is there a need for a new DMM theory?</td><td>Explorative (theory development)</td></tr><tr><td>How could such a theory look like?</td><td>Grounded theory</td></tr><tr><td rowspan="2"></td><td>How to ensure generalizability?</td><td>Large data sets, cross-company analysis (multiple case study design)</td></tr><tr><td>Evolution of DM over time?</td><td>Longitudinal studies</td></tr></tbody></table>

The scope of our suggested research agenda clearly shows that there is a number of significant shortcomings with regards to the measurement validity of existing DMMs. Correspondingly, the overall quality and theoretical value of DMMs is put to the test.
With our research agenda we contribute on a theoretical level by identifying and
remedying shortcomings of existing maturity models in the field of IS [5]. On a practi-
cal level, we provide managers with a much needed critical evaluation of these popular
tools [8]. We acknowledge that due to the practical nature of DMMs, the exhaustiveness
of the literature review is questionable. Future research could thus include a systematic
literature review of additional practitioners’ outlets and search terms to provide a more
comprehensive overview. We hope that this research agenda will inspire researchers to
further investigate the topic of DMMs which will ultimately lead to a higher practical
and theoretical value of these useful models.

References

1. Remane, G., Hanelt, A., Wiesboeck, F., Kolbe, L.: Digital Maturity in Traditional Industries–An Exploratory Analysis. In: Proceedings of the 25th European Conference on Information Systems (ECIS) (2017)

2. Chanias, S., Hess, T.: How digital are we? Maturity models for the assessment of a company’s status in the digital transformation. Management Report/Institut für Wirtschaftsinformatik und Neue Medien, 1–14 (2016)

3. Mettler, T.: Thinking in terms of design decisions when developing maturity mod-
els. International Journal of Strategic Decision Sciences (IJSDS) 1, 76–87 (2010)

4. Kotarba, M.: Measuring digitalization–key metrics. Foundations of Management 9, 123–138 (2017)

5. Becker, J., Knackstedt, R., Pöppelbuß, J.: Developing maturity models for IT man-
agement. Business & Information Systems Engineering 1, 213–222 (2009)

6. Berghaus, S., Back, A.: Stages in Digital Business Transformation: Results of an Empirical Maturity Study. In: MCIS, p. 22 (2016)

7. Proença, D., Borbinha, J.: Maturity models for information systems-A state of the art. Procedia Computer Science 100, 1042–1049 (2016)

8. Becker, J., Niehaves, B., Pöppelbuß, J., Simons, A.: Maturity Models in IS Re-
search. In: 18th European Conference on Information Systems, ECIS2010-0320
(2010)

9. Kane, M., Crooks, T., Cohen, A.: Validating measures of performance. Educational measurement: issues and practice 18, 5–17 (1999)

10. Bititci, U.S., Garengo, P., Ates, A., Nudurupati, S.S.: Value of maturity models in performance measurement. International journal of production research 53, 3062–3085 (2015)

11. Pedrini, C.N., Frederico, G.F.: Information Technology Maturity Evaluation in a Large Brazilian Cosmetics Industry. International Journal of Business Administration 9, 15 (2018)

12. Poeppelbuss, J., Niehaves, B., Simons, A., Becker, J.: Maturity models in infor-
mation systems research: Literature search and analysis. CAIS 29, 1–15 (2011)

13. Rosemann, M., de Bruin, T.: Towards a Business Process Managment Maturity Model. ECIS 2005 Proceedings, 37 (2005)
14. Lahrmann, G., Marx, F., Winter, R., Wortmann, F.: Business intelligence maturity: Development and evaluation of a theoretical model. In: System Sciences (HICSS), 2011 44th Hawaii International Conference on, pp. 1–10. IEEE (2011)

15. Dooley, K., Subra, A., Anderson, J.: Maturity and its impact on new product de- velopment project performance. Research in Engineering Design 13, 23–29 (2001)

16. Mettler, T., Blondiau, A.: HCMM-A maturity model for measuring and assessing the quality of cooperation between and within hospitals. In: Computer-Based Med- ical Systems (CBMS), 2012 25th International Symposium on, pp. 1–6. IEEE (2012)

17. Carvalho, J.V., Rocha, Á., van de Wetering, R., Abreu, A.: A Maturity model for hospital information systems. Journal of Business Research (2017)

18. Solli-Sæther, H., Gottschalk, P.: The modeling process for stage models. Journal of Organizational Computing and Electronic Commerce 20, 279–293 (2010)

19. Brühl, R.: Wie Wissenschaft Wissen schafft – Wissenschaftstheorie für Sozial- und Wirtschaftswissenschaften. UVK Verlagsgesellschaft/Lucius & Lucius, Konstanz (2015)

20. APA, A.: NCME: Standards for educational and psychological testing. Washington, DC, American Psychological Association, American Educational Research Association, National Council on Measurement in Education (1985)

21. Kane, M.T.: Validation. Educational measurement 4, 17–64 (2006)

22. Kane, M.T.: An argument-based approach to validity. Psychological bulletin 112, 527 (1992)

23. Vom Brocke, J., Simons, A., Niehaves, B., Riemer, K., Plattfaut, R., Cleven, A.: Reconstructing the giant: On the importance of rigour in documenting the literature search process. In: Ecis, 9, pp. 2206–2217 (2009)

24. Bandara, W., Miskon, S., Fielt, E.: A systematic, tool-supported method for con- ducting literature reviews in information systems. In: Proceedings of the19th Eu- ropean Conference on Information Systems (ECIS 2011) (2011)

25. Kitchenham, B., Charters, S.: Guidelines for performing systematic literature re- views in software engineering (2007)

26. Lasrado, L.A., Vatrapu, R., Andersen, K.N. (eds.): Maturity models development in is research: a literature review (2015)

27. Friedrich, R., Gröne, F., Koster, A. and Le Merle, M.: Measuring industry digiti- zation: Leaders and laggards in the digital economy, https://www.strate- gyand.pwc.com/report/measuring-industry-digitization-leaders-laggards (2011)

28. Westerman, G., Calméjane, C., Bonnet, D., Ferraris, P., McAfee, A.: Digital Trans- formation: A roadmap for billion-dollar organizations. MIT Center for Digital Business and Capgemini Consulting, 1–68 (2011)

29. Westerman, G., Tannou, M., Bonnet, D., Ferraris, P., McAfee, A.: The Digital Ad- vantage: How digital leaders outperform their peers in every industry. MITSloan Management and Capgemini Consulting, MA 2, 2–23 (2012)

30. Becker, W., Ulrich, P., Vogt, M.: Digitalisierung im Mittelstand-Ergebnisbericht einer Online-Umfrage. Univ., Lehrstuhl für Betriebswirtschaftslehre, insbes. Unternehmensführung und Controlling (2013)

31. Catlin, T., Scanlan, J., Willmott, P.: Raising your digital quotient. McKinsey Quar- terly, 1–14 (2015)
32. Roland Berger: The digital transformation of industry, www.rolandberger.com/publications/publication pdf/roland_berger_digital_transformation_of _industry_20150315.pdf (2015)

33. Arreola González, A., Becker, K., Cheng, C., Döricht, V., Duchon, M., Fehling, M., Grolman, H. von, Hallensleben, S., Hopf, S., Ivandic, N.: Digitale Transformation: Wie Informations-und Kommunikationstechnologie etablierte Branchen grundlegend verändert–Der Reifegrad von Automobilindustrie, Maschinenbau und Logistik im internationalen Vergleich. Abschlussbericht des vom Bundesministerium für Wirtschaft und Technologie geförderten Verbundvorhabens „IKT-Wandel “(Steuerkreis: Grolman von, H., Krcmar, H., Kuhn, K.-J., Picot, A., & Schätz, B.) (2016)

34. Berghaus, S., Back, A. and Kaltenrieder, B.: Digital maturity & transformation report 2016, https://crosswalk.ch/dmtr2016-delivery (2016)

35. Gill, M., VanBoskirk, S.: Digital Maturity Model 4.0. Benchmarks: Digital Transformation Playbook (2016)

36. Valdez-de-Leon, O.: A digital maturity model for telecommunications service providers. Technology Innovation Management Review 6 (2016)

37. Lichtblau, K., Stich, V., Bertenrath, R., Blum, M., Bleider, M., Millack, A., Schmitt, K., Schmitz, E. and Schröter, M.: Studie: Industrie 4.0 Readiness, http://www.impuls-stiftung.de/documents/3581372/4875835/Industrie+4.0+Readiness+IMPULS+Studie+Oktober+2015.pdf/447a6187-9759-4f25-b186-b0f5eac69974 (2017)

38. Colli, M., Madsen, O., Berger, U., Møller, C., Wæhrens, B.V., Bockholt, M.: Contextualizing the outcome of a maturity assessment for Industry 4.0. Ifac-papersonline 51, 1347–1352 (2018)

39. Deutsche Telekom, A.G.: Digitalisierungsindex Mittelstand 2018. Der digitale Status Quo des deutschen Mittelstands, 2–3 (2018)

40. Salviotti, G., Gaur, A., Pennarola, F.: STRATEGIC FACTORS ENABLING DIGITAL MATURITY: AN EXTENDED SURVEY (2019)

41. Ifenthaler, D., Egloffstein, M.: Development and Implementation of a Maturity Model of Digital Transformation. TechTrends, 1–8 (2019)

42. MacKenzie, S.B., Podsakoff, P.M., Podsakoff, N.P.: Construct measurement and validation procedures in MIS and behavioral research: Integrating new and existing techniques. MIS quarterly 35, 293–334 (2011)

43. Whetten, D.A.: What constitutes a theoretical contribution? Academy of management Review 14, 490–495 (1989)