# A Qualitative Research Approach to Obtain Insight in Business Process Modelling Methods in Practice

Céline Décosse, Wolfgang A. Molnar, Henderik A. Proper

► To cite this version:

Céline Décosse, Wolfgang A. Molnar, Henderik A. Proper. A Qualitative Research Approach to Obtain Insight in Business Process Modelling Methods in Practice. 6th The Practice of Entrprise Modeling (PoEM), Nov 2013, Riga, Latvia. pp.161-175, 10.1007/978-3-642-41641-5_12. hal-01474781

HAL Id: hal-01474781

https://inria.hal.science/hal-01474781

Submitted on 23 Feb 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Qualitative Research Approach to Obtain Insight in Business Process Modelling Methods in Practice

Céline Décosse¹, Wolfgang A. Molnar¹, Henderik A. Proper¹,²

¹ Public Research Center Henri Tudor, 29 Kennedy avenue, L-1855 Luxembourg, Luxembourg
¹ Radboud University, Comeniuslaan 4, 6525 HP Nijmegen, The Netherlands
{celine.decosse, wolfgang.molnar, erik.proper}@tudor.lu

**Abstract.** In this paper we are concerned with the development of an observational research approach to gain insights into the performance of Business Process Modelling Methods (BPMMs) in practice. In developing this observational approach, we have adopted an interpretive research approach. More specifically, this involved the design of a questionnaire to conduct semi-structured interviews to collect qualitative research data about the performance of BPMMs. Since a BPMM is a designed artefact, we also investigated Design Science Research literature to identify criteria to appreciate the performance of BPMMs in practice. As a result, the questionnaire that was used to guide the interview is based on a subset of criteria of progress for information systems theories, while the observational research approach we adopted involves the collection of qualitative data from multiple stakeholder types. As a next step, the resulting questionnaire was used to evaluate the performance of an actual BPMM in practical use; the DEMO method. Though the analysis of the collected qualitative data of the DEMO case has not been fully performed yet, we already foresee that part of the information we collected provides new insights compared to existing studies about DEMO, as is the fact that a variety of types of stakeholders have been approached to observe the use of DEMO.

**Key words:** enterprise modelling in practice; information systems method evaluation criteria; qualitative research approach

## 1 Introduction

In this paper we are concerned with the development of an observational research approach to obtain more insights into the actual performance of Business Process Modelling Methods (BPMM) in practice. Our initial research goal was to gain more insight into the performance of the DEMO (Design and Engineering Methodology for Organisations) method, which is a specific BPMM that is employed by enterprise architects to design concise models of organizations processes. Indeed, we knew that several projects had been performed with DEMO [1–12]. For some of these projects, the application of DEMO seems to be very promising, e.g. DEMO helped to “*con-struct and analyse more models in a shorter period of time*” (p.10)[2]. Therefore, we were curious about the performance of DEMO *in practice*. In addition, we had access
to practitioners who have used DEMO in their projects, and who would agree to have
these projects investigated by researchers.

However, rather than limiting ourselves to DEMO only, we decided to generalize
our effort to BPMMs in general. In other words, rather than developing an observa-
tional approach to only observe the performance of DEMO in practice, we decided to
develop an approach to observe the practical performance of BPMMs in general (still
using DEMO as a specific case). Having insights about a BPMM in practice is valua-
ble because it is easier to select, promote, improve or even better use a BPMM when
knowing what can be expected when using it in practice. In doing so, we were also
inspired by Winter (p.471)[13]: "Not every artefact construction, however, is design
research. 'Research' implies that problem solutions should be generic to some extent,
i.e., applicable to a set of problem situations", where in our case the constructed arte-
fact is the observation approach for the performance of BPMM in practice.

In developing the observation approach, we chose the interpretive research ap-
proach as a starting point, where qualitative research data pertaining to the use and
performance of the specific BPMM, will be collected with semi-structured interviews.
The original contribution of this paper is the way in which we defined the themes and
questions of these interviews: we selected a subset of criteria of progress for infor-
mation systems theories proposed by Aier and Fischer [14]. We then performed the
interviews to investigate the use of DEMO in practice in several projects and contexts
by several types of stakeholders acquainted with DEMO. Although the analysis of the
data has not been completely performed, we nevertheless already present some first
insights about DEMO in practice.

The paper is structured as follows: in the introduction we defined the problem to be
addressed. Section 2 introduces definitions and positions DEMO as a specific BPMM
in these definitions. In section 3, we reviewed the literature about the evaluation of
DEMO in practice. Section 4 is the core of the paper: it presents the proposed re-
search approach for getting insights about a BPMM in practice. Section 5 deals about
the validity of this proposed observation approach by applying it to the use of DEMO
in practice as a case study. A few first insights we gained about the use of DEMO in
practice, based on the conducted interviews, are briefly presented in section 6. We
then underline some limitations of our work and conclude in section 7.

## 2 DEMO as a method

*Method.* Unlike process definitions, method definitions often refer to a modelling
language and to "underlying concepts" [16], "hidden assumptions" [17] or "way of
thinking" [18]. In the information technology domain, March and Smith define a
method as "a set of steps (...) used to perform a task." [19]. The definition proposed
by Rescher [20] and adopted by Moody [21] is more general: as methods define
"ways of doing things, methods are a type of human knowledge (the "knowledge
how")". Mettler and Rohner (p.2)[16] bring a prescriptive flavour and an ideal goal to
method definition: "methods (...) focus on the specification of activities to reach the
ideal solution (how)". They distinguish methods "key activities" (that aim to reach a
business goal) from their "underlying concepts" (that is "the conceptual view on the
world that underlies the performance of the activities") [16]. This distinction seems to
be consistent with the view of Seligman et al. [18] on information systems methodologies, that they characterise with “5 ways”:

- the “way of thinking”, defined as “hidden assumptions” that are “used to look at organisations and information systems” [17, 18],

- the “way of working” (how to do things),

- the “way of controlling” (how to manage things),

- the “way of modelling”, which they define as the “network of [the method’s] models, i.e. the models, their interrelationships and, if present, a detailed description of the model components and the formal rules to check them”,

- the “way of supporting”, which is about the tools supporting the method.

In the current paper, we want to have an insight about methods that are used as BPMMs in practice. The method definitions above are a way for the researcher to establish a set of themes to interview stakeholders about without forgetting aspects of methods that are recurrent in the literature.

*Method and modelling language.* When modelling languages are mentioned in the methods definitions above, they are presented as being part of methods: March and Smith explain that “*although they may not be explicitly articulated, representations of tasks and results are intrinsic to methods.*” (p.257)[19]. For Seligman et al. [18], the way of modelling is one of the features of a method, as is the “*way of working*”.

*DEMO, language or method? Primarily, an artefact!*

Whereas authors argue that DEMO is a modelling language [22], almost all other recent publications about DEMO usually consider it as a method [23, 24]. Winter et al. argue that a method and recommendations concerning the representation of a model can be seen as aspects of an artefact, and then propose the following artefact definition: “*A generic artefact consists of language aspects (construct), aspects referring to result recommendations (model), and aspects referring to activity recommendations (method) as well as instantiations thereof (instantiation).*” (p.12)[25].

## 3 Literature review on the evaluation of DEMO in practice

This literature review aims at investigating whether DEMO has been evaluated in practice and how. Many papers [1–12] deal with case studies in which DEMO has been used to design situational DEMO based methods or to propose ontologies. Besides, qualities of DEMO models have been studied in several evaluations [22, 26]. We found two papers dealing with a partial evaluation of DEMO in practice across several cases. The first one [27] studies the adoption of DEMO by DEMO professionals in practice, in order to improve this adoption (Table 1). This study is restricted to the adoption of DEMO in practice so the use of DEMO in practice is not the core of the study [27]. The second one [11] investigates DEMO as a means of reflecting upon the Language/Action perspective (LAP) (Table 2). The DEMO related part of this paper aims at finding out how the actual application of DEMO differs from its intended application. Besides, only DEMO professionals were asked to answer the survey so the study only reflects why DEMO certified practitioners adopted (or not) DEMO,
it provides less insights about people aware of the existence of DEMO who are not DEMO professionals. Dumay et al. focused on how the professional application of DEMO differs from its intended application so only practitioners have been involved in their study [11].

<table><caption>**Table 1.** Khavas, 2010 [24] - Master thesis: the adoption of DEMO in practice</caption><tr><td>Source</td><td>Khavas, 2010 [24] - Master thesis</td></tr><tr><td>Subject</td><td>The adoption of DEMO in practice</td></tr><tr><td>Motivation</td><td>Ensure the adoption of DEMO in practical fields. This problem has been introduced in [24].</td></tr><tr><td>Research questions</td><td>1 What is the adoption rate of DEMO among DEMO Professionals in practice?<br/>2 What are the factors that can influence the decision of a DEMO Professional to adopt or ignore DEMO? (p.2)</td></tr><tr><td>Sample</td><td>DEMO professionals [24]</td></tr><tr><td>Approach</td><td>- White-box approach: to define questions, DEMO is first thoroughly studied through a literature review. Then two surveys have been performed.<br/>- A researcher who understands DEMO asks people who master DEMO about adoption matters.</td></tr><tr><td>Methods</td><td>Based on literature review about what DEMO is and how it works on the one hand and about method adoption on the other hand, Khavas elaborated first a quantitative survey and later a qualitative analysis based on semi-structured interviews.</td></tr><tr><td>Results</td><td>- Identification of several levels of adoption in an organization: individual, project, unit or organization<br/>- Identification of factors that influence adoption<br/>- Recommendations to DEMO professionals to ease the adoption or DEMO</td></tr></table>

<table><caption>**Table 2.** Dumay et al., 2005, [11] - Professional versus intended application of DEMO</caption><tr><td>Source</td><td>Dumay et al., 2005, [11] - Conference Proceedings</td></tr><tr><td>Subject</td><td>Subject of the DEMO evaluation included in the paper: find out how the professional application of DEMO differs from its intended application.</td></tr><tr><td>Motivation</td><td>"Devise several recommendations on how the Language/Action Perspective (LAP) can improve its footprint in the community of Information Systems Development practice." (p.78) LAP is an approach for the design of Information Systems.</td></tr><tr><td>Research questions</td><td>1. What is the relationship between DEMO theory and its intended application?<br/>2. How does the professional application of DEMO differ from its intended application?<br/>3. Can LAP unify the apparent incompatible social and technical perspectives present in Information Systems Development practice? (p.78)</td></tr><tr><td>Sample</td><td>DEMO practitioners</td></tr><tr><td>Approach</td><td>DEMO evaluation by practitioners is a means of evaluating in practice the LAP. The idea is to study DEMO theory, then identify the proposed</td></tr></table>
<table><tr><td></td><td>applications of DEMO; study DEMO applications by practitioners; establish the relationship between DEMO theory and practice; compare LAP theory and DEMO theory to draw conclusions about LAP thanks to DEMO analysis.</td></tr><tr><td>Methods</td><td><ul><li>To study DEMO theory: the framework proposed by Mingers and Brockles [28] to analyse methodologies has been used.</li><li>To study DEMO in practice: a survey has been sent by email to practitioners about DEMO application contexts (domain, duration, projects). Then a 4-hour workshop has been organised with the 19 practitioners amongst the survey respondents willing and able to attend. The subject of the workshop was DEMO areas of application.</li></ul></td></tr><tr><td>Results</td><td>The DEMO in practice evaluation part of this study "answers the question how the professional application of DEMO differs from its intended application." (p.80)</td></tr></table>

To the best of our knowledge, no study has been performed with the aim of giving a holistic view on the use of DEMO in practice both by approaching a variety of themes and a diversity of stakeholder's profiles. The subject or the current paper is to define an observational approach to do so.

# 4 Observational approach for BPMMs performance in practice

## 4.1 Observational approach overview: an interpretive approach

This section discusses the set up of the observational research approach that is to be used to gain insights about the performance of a BPMM in practice. These insights are provided by exploring stakeholders' views about the use of a BPMM in practice. For this exploratory purpose, we adopted a qualitative research approach, because it is aimed at understanding phenomena and provides modes and methods for analysing text [29]. Qualitative research can be positivist, interpretive, or critical [29]. We adopted an interpretive approach because it allows to produce "an understanding of the context of the information system, and the process whereby the information system influences and is influenced by the context" [30](p.4-5). "Interpretive methods of research adopt the position that our knowledge of reality is a social construction by human actors" [31]. We found it suitable for exploring the use of a method, because a method is an artefact that is designed, performed and evaluated by human people.

To collect these qualitative data, we selected the semi-structured interview technique. Qualitative interviews are one of "the most important data gathering tools in qualitative research" (p.2)[32]. The reason is that it is "permitting us to see that which is not ordinarily on view and examine that which is looked at but seldom seen" (p.vii)[33]. We created a questionnaire to be used as a guideline by the interviewer during the interviews to discover what can be expected from DEMO in practice.

Whether being a method or a language, DEMO is a designed artefact; so are BPMMs. We considered that to gain insights about what can be expected from a designed artefact, we could use artefacts evaluation criteria. We selected those criteria by reasoning on criteria of progress for information systems theories proposed by
Aier and Fischer [14]. We based the questions of the questionnaire on the artefacts
evaluation criteria we selected, and then gathered these questions in themes to ease
the interviewing process. We then obtained an interview questionnaire about IS (In-
formation Systems) DS (Design Science) artefact evaluation, to which we added a
few complementary questions that are specific to DEMO.

The scope of the current paper does not involve the mode of analysis of the col-
lected data; it is concerned with the identification of the observational approach for
gaining insights about a BPMM in practice and with the first insights about DEMO
provided by the interviews that have been performed.

## 4.2 Interview guideline setup

Identification of the interview themes

The themes of the questionnaire used as an interview guideline have been identi-
fied according to the goal of the questionnaire: gain insights about DEMO in practice.
Themes have been identified from the design science literature regarding artefact and
method evaluation, so that the core of the questionnaire might be used as a guideline
for any IS method evaluation; from literature about DEMO, regarding stakeholders
feed-back about DEMO; during a brainstorming with fellow researchers to complete
the above points.

In 2010, Aier and Fischer proposed a set of Criteria of Progress for Information
Systems Design Theories [14]. We call this set of criteria CriProISDT<fnref n="1" />. Aier and
Fischer based their reflection, amongst other elements, on “evaluation criteria for IS
DSR artefacts” (“IS DSR” stands for “Information Systems Design Science Re-
search”) and reviewed the literature about that. In particular, they used March and
Smith [19] set of criteria for IS DSR artefacts evaluation<fnref n="2" />. Then, Aier and Fischer
focused on IS DSR artefacts evaluation criteria that they considered as being inde-
pendent of any particular artefact type (method, model, construct, instantiation),
which is interesting for our purpose. They established a table of comparison of the
evaluation criteria for IS DSR artefacts by March and Smith with CriProISDT. Adopt-
ing Aier and Fischer’s position that “evaluation criteria for IS DSR artefacts should be
strongly related to those for IS design theories”[14], we choose to use this comparison
the other way round: we selected amongst CriProISDT the criteria that we thought
may be applicable to artefacts (called “CriProISDT subset”), and we then used the
“comparison table” to retrieve the “matching” IS DSR artefacts evaluation criteria. By
doing that we obtained a set of artefact evaluation criteria (called AEC) that are: ge-
neric to all types of artefact evaluation; based on a recent literature review; in line
with one of the most well-known set of criteria for IS DSR artefacts evaluation

<footnote n="1">¹ Aier and Fischer CriProISDT: Utility, Internal Consistency, External consistency, Broad purpose and scope, Simplicity, Fruitfulness of further research [14].</footnote>

<footnote n="2">² March and Smith set of criteria for IS DSR *artefacts* evaluation: Completeness, Ease of use, Effectiveness, Efficiency, Elegance, Fidelity with real world phenomena, Generality, Impact on the environment and on the artefacts’ users, Internal consistency, Level of detail, Operationality, Robustness, Simplicity, Understandability [19].</footnote>
(March and Smith's) although refining it. Criteria we selected to be included in CriProISDT subset are: Utility, External consistency, Broad purpose and scope.

*Utility (usefulness).*

The reason for selecting utility is that DSR literature emphasizes that DSR products “*are assessed against criteria of value or utility*” (p.253)[19]. Aier and Fischer define “utility” in (p.158)[14] and “usefulness” in [34] as “*the artefact’s ability to fulfil its purpose if the purpose itself is useful. The purpose of an artefact is only useful if it is relevant for business.*” Following the comparison table, “matching” IS DSR artefacts evaluation criteria are: ease of use, effectiveness, efficiency, impact on the environment and on the artefacts’ users and operationality. We call this list the “utility list”.

*External consistency, Broad purpose and scope.*

Many authors underline the interdependence between a DSR artefact and its performance environment; that evaluation criteria and results are environment dependent. [14, 19, 35, 36]. So we selected the criteria of “external consistency” (“*fidelity with real world phenomena*” [19]) and “broad purpose and scope” because they are related to the performance environment of an artefact. Following the comparison table, “matching” IS DSR artefacts evaluation criteria are: fidelity with real world phenomena, generality. We call this list the “context list”. We remove from this list the “robustness” criteria, which is mainly aimed at algorithmic artefact evaluation.

*IS DSR artefact evaluation criteria we used.*

By aggregating the “utility list” and the “context list”, we obtain a list of IS DSR artefact evaluation criteria, with the definitions adopted or proposed by Aier and Fischer in [14] when they can be applied to artefacts: ease of use, effectiveness, efficiency, impact on the environment and on the artefacts’ users, operationality, fidelity with real world phenomena (external consistency), generality.

This list actually extends the list of criteria for methods evaluation proposed by March and Smith: ease of use, efficiency, operationality, generality [19]. We can see that because of the systematic way of collecting IS DSR criteria we adopted, the following criteria are not included: completeness, elegance, internal consistency, level of detail, robustness, simplicity, understandability [19].

- ease of use: the artefact shall be easily usable [14];

- effectiveness: the degree to which the artefact meets its goal and achieve its desired benefit in practice [37]. So questions about the method under evaluation added value were included in the questionnaire

- efficiency: the degree to which the modelling process utilises resources such as time and people [19]; a quotient of output and input [14]. The notion of “Return on Modelling effort” conveys the same idea. “*If an artefact resulting from a design theory is used very often, its efficiency might be the best criterion for measuring its utility.*” (p.149)[14]

- impact on the environment and on the artefacts’ users: a side effect. “*Side effects can increase or decrease utility*” (p.164)[14]. “*A critical challenge in building an*
artefact is anticipating the potential side effects of its use, and insuring that unwanted side effects are avoided" (p.254)[19]

- operationality: "the ability to perform the intended task or the ability of humans to effectively use the method if it is not algorithmic" [14, 19]

- fidelity with real world phenomena (external consistency): Questions about to what extent the constructs of the method under evaluation reflect business concepts that stakeholders have an interest to model with BPMMs.

- generality: the same as "broad purpose and scope" (p.164)[14]. Questions about the possibility to tailor a BPMM to specific business context are included in the questionnaire. Besides, as DSR artefacts address classes of problems [35, 38], questions about the "kind" of problems for which it is interesting to use the method under evaluation are included in the questionnaire.

### Themes of the interview guideline.

For the fluidity of the interview, the questions have been gathered in themes C to G). Themes A and B complement these themes.

- A- Interview situation - Location, date, duration, language of the interview

- B- Interview context - Actual context of use of DEMO in a particular project, also allows to determine how much the person remembers about the project

- C- Typical context of use of the Method: recommendations, factors of influence

- D- Use of the Method in practice

- E- Organisation fit: Necessary skills to apply the Method and satisfaction about the Method

- F- Method chunks identification

- G- Method construction (only for the designer stakeholder type)

### Stakeholder types.

Stakeholders are part of the "impact on the environment and on the artefacts' users" criteria. Because different stakeholders have different purposes and because utility definition is related to a purpose, we defined several stakeholders types [38]. "The utility of an artefact is multi-dimensional: one dimension for each stakeholder type" (p.10)[39]. Stakeholder types we *a priori* thought of were related to their role regarding the method under evaluation (here, DEMO):

- Designers: they took part in the creation or evolution of DEMO,

- Sponsor: owner of the engineering effort, this stakeholder pays or is financially responsible for the project in which DEMO has been applied,

- Manager of the engineering effort: project manager for example,

- Modeller: this stakeholder created DEMO models,

- Final beneficiaries: they benefit from the use of the method.

Themes are common to all stakeholder types except the designers: a theme (G) dedicated only to the latter ones has been included in the questionnaire.

In themes A and B, many questions are about the stakeholders, so that new stakeholder types may emerge from the analysis of the interviews. Indeed, the semi-structured interview technique provides a guidance to help collect data on themes the researcher is interested in, but the kind and scope of answers are not predefined. Ac-
tually, surprising answers are an asset provided they relate to the goal of the study: they may enable the researcher to reconsider themes, sampling, questions and research approach.

*With regard to the question if DEMO is considered as a language or a method.*

As our goal is to have an insight about DEMO in practice, we asked the interviewees what they would call DEMO and whether they would consider it as being prescriptive or descriptive. Besides, we ensured that interviews were not exclusively “method oriented” by selecting evaluation criteria that are common to all types of designed artefacts. Winter et al. argue that a method and a model can be seen as aspects of an artefact (p.12) [25]. This is especially convenient in the case of asking questions about DEMO. For this reason and because of the double definition of DEMO, we added questions that are specifically related to methods as defined in Section 2, and to languages as being part of methods.

## Overview of the questionnaire structure

Each set of questions is related to a stakeholder type, a theme, and is aimed at collecting data exclusively either about stakeholders' intentions or stakeholders' experience. The purpose of this is to collect stakeholders' a priori views when intending to do something ("What were your original intentions/expectations when you...?" ) and a posteriori views when they had experienced this something ("What is your experience about ...? ")<fnref n="3" />. The resulting questionnaire structure is depicted in **Table 3**.

Questions are actually often similar between stakeholder types: such a structure to design a questionnaire is only a tool for the researcher to think of many types of questions related to the goal of the study. Themes A and B are about the knowledge of the context and about the stakeholders.

<table><caption>**Table 3.** Structure of the questionnaire used as a semi-structured interview guideline</caption><thead><tr><td>Themes</td><td>Designer</td><td>Sponsor</td><td>...</td></tr></thead><tbody><tr><td>A</td><td>Questions</td><td>Questions</td><td>Questions</td></tr><tr><td>B</td><td>Questions</td><td>Questions</td><td>Questions</td></tr><tr><td>C</td><td>Intention and experience questions</td><td>Intention and experience questions</td><td>Intention and experience questions</td></tr><tr><td>D</td><td>Intention and experience questions</td><td>Intention and experience questions</td><td>Intention and experience questions</td></tr><tr><td>...</td><td>Intention and experience questions</td><td>Intention and experience questions</td><td>Intention and experience questions</td></tr></tbody></table>

<footnote n="3">³ “Theories seek to predict or explain phenomena that occur with respect to the artefact’s use (intention to use), perceived usefulness, and impact on individuals and organizations (net benefits) depending on system, service, and information quality (DeLone and McLean 1992; Seddon 1997; DeLone and McLean 2003)”, cited by Hervner et al. (p.77)[35]</footnote>
# 5 Reflections about the proposed research approach validity

The themes of the questionnaire used as a guideline in the semi-structured interview technique influence the answers that are given by the respondents. So, we found it necessary to justify our position for referring to design science literature to define a set of criteria to evaluate a method. These criteria will be used to gain insights on a method in the current study, not to evaluate it.

## *Information Systems literature.*

As BPMMs may not be information technology related, the question arises as to whether information systems literature is relevant to study them [13]. In this paper we assume that as BPMMs are often implemented in the context of IS projects, IS literature is relevant to reflect upon BPMMs.

## *Information Systems Design Sciences literature.*

BPMMs evaluation can be considered as a wicked problem, because it has a critical dependence upon human cognitive and social abilities to produce effective solutions (evaluation depends both on the performance of the method under evaluation and of the evaluation process itself on the other hand) and because it is strongly context dependent [35]. Such wicked problems can be addressed by the iterative nature of design science research [35].

Besides, the design science pa explores the art of building and evaluating artefacts and especially information systems related artefacts with a strong importance given to the behavioural aspects [35]. Is DS, evaluation of an artefact is performed against the criteria of utility, which is a practical perspective. So we can then investigate the design science literature with benefits for approaching the question of how to gain insights about BPMMs in practice. In short, evaluating BPMMs in practice is a practical problem, as such it can be approached with DS literature [40].

## *Design Science literature, Design Science Research literature or both?*

Winter makes the following difference between IS design research and IS design science: “While design research is aimed at creating solutions to specific classes of relevant problems by using a rigorous construction and evaluation process, design science reflects the design research process and aims at creating standards for its rigour” (p.471)[13]. With this definition, we may rather investigate more DS research literature than DS literature. But on the one hand DSR literature and DS literature are not always “self-labelled” this way and on the other hand reflections and criteria of progress that are applicable to DS can sometimes be related to those of DSR [14], we will investigated both DS literature and DSR literature.

# 6 Early experience report about the DEMO case study

This section exposes how we started to implement the proposed research approach, and some of the preliminary insights we gained about DEMO in practice. The data analysis is still to be performed.
## 6.1 Data collection

Based on the questionnaire used as a guideline [32], for our semi-structured interviews we collected 13 interviews. Multiple stakeholder types were represented. Interviews took place in interviewee's offices except one, which was performed with Skype. Each interviewee was interviewed individually by two researchers: one interviewer to ask the questions and one "shadow" interviewer to complete questions and take notes. Interviewees agreed to be interviewed and that the interviews were recorded. Only one interviewee asked us not to disclose his name. Immediately after each interview, the interviewer and the "shadow" wrote down notes about the interview that involved what had been said or interviewee's reaction to some specific subject for example. In order to capture the actual experience of the individuals in practice with DEMO, interviewers tried to avoid "leading the witness" by: following the interview guideline, asking questions in which the answer is not included, not giving their own opinions. They attempted to reduce their role to information collectors, influencing as less as possible the content of the collected data and encouraging interviewees to keep talking. One of the interviewers was a DEMO expert, the other one was a business analyst whose knowledge about DEMO could be summarised in a few short lines. The questionnaire used as a guideline contained about 100 questions, but as not every question was meant at every stakeholder types, around 60 questions were asked to each interviewee – or not, in case interviewee spontaneously provided the information while answering another question. Average time of interviews is one hour an thirty minutes. A total amount of twenty hours of recording have been collected.

## 6.2 Interview data analysis and initial insights about DEMO in practice

Interviews have been transcribed by interviewers. Scripts have been coded [41]. The full analysis still has to be performed, so only first insights can be presented here.

DEMO is seen as a way of thinking that comes with a way of modelling. Interviewees mentioned a set of concepts helping enterprise engineer analysts to analyse organizations and reveals what is actually going on when it is about responsibility, authority, role, transaction. According to the interviewees, DEMO seems to be suited for complex problems. Interviewees often mention that DEMO models were implementation independent and that to apply DEMO to produce DEMO models, abstract thinking is required, but reading DEMO models seem to require only a few hours of training. When interviewees were asked about DEMO return on modelling effort, they all were very positive about it, sometimes adding "provided it is used by trained people". Several interviewees deplore a lack of interfaces with other methods. Still, all interviewees would use DEMO again if they would have to work in their project again.

# 7 Conclusion and further work

The current paper is about setting up an observational research approach for an exploratory goal: having insights about BPMMs in practice. We adopted a qualitative research approach with semi-structured interviews for collecting research data. To
define the interview guideline, we relied on criteria of progress for information sys-
tems design theories.

The main criteria against which we could assess the proposed observational re-
search approach may be the appropriateness of the insights we obtained during the
interviews against the purpose of the research effort. Still, interview guideline themes
are only a parameter to ensure this appropriateness: among other things, interviewees
sampling, interviewees background compared to interviewers background, way of
asking question, interviewers' attitude and degree of remembrance of interviewees
regarding the case studies also influence the nature and quality of the collected data.

This paper provides an interview guideline structure that may be adapted from the DEMO interview experience and then potentially used to get insights about other BPMMs. Although interviews analysis has not been performed yet, we may already say that during the interviews performance, no understanding problems between interviewers and interviewees occurred, collected information was actually related to the themes and questions, new information appeared compared to the literature review about DEMO. Besides, the diversity of interviewed stakeholder types allowed the collection of various points of views, sometimes conflicting ones. All interviewees encouraged us both to carry on in DEMO investigation effort and to contact them again in case we would need further information.

The proposed observational research approach has some limitations, amongst
which are the following ones:

- As we defined the list of evaluation criteria with a systematic process (gathering two lists), we should investigate for each criterion we included or not what the implications are, then we may (certainly) integrate again some criteria.

- Aier and Fischer explain that the set of criteria of progress they propose for information systems design theories [14] might not be complete. So, for this reason again we should reflect upon the completeness of the criteria we proposed.

- Criteria we proposed are generic to all types of artefacts, further reflection is required whether add aspects specific criteria (model, constructs, method, instantiation).

- We have not reflected upon the limitations that are inherent to the interview technique to evaluate a method.

- Whatever the research approach to get insights about an artefact, the influence of some parameters should be discussed, namely the amount of knowledge of the interviewers and researchers have about the artefact they want to have insights on.

- Various frameworks have been proposed to evaluate methods in IS literature, e.g. [42]. Though, they have not been taken into account in the current paper because our scope is about having insights about a method, so only method evaluation criteria were used for this purpose.

Some of these limitations may be addressed in future work: we plan to analyse the
interviews, this will generate insights about the application of DEMO in practice and
allow us to reflect upon the practical use of the themes and evaluation criteria that are
proposed in the current paper. According to the findings, we may adapt these criteria
and investigate in case studies BPMMs other than DEMO to have a variety of experi-
ences with the proposed observational research approach.
8 References

1. Dias, D.G., Lapão, L.V., Mira da Silva, M.: Using enterprise ontology for improving emergency management in hospitals. Studies in health technology and informatics. 180, 58–62 (2012).

2. Dias, D.G., Mendes, C., Mira da Silva, M.: Using Enterprise Ontology for Improving the National Health System-Demonstrated in the Case of a Pharmacy and an Emergency Department. In: Filipe, J. and Dietz, J.L.G. (eds.) Proceedings of the International Conference on Knowledge Engineering and Ontology Development (KEOD 2012). pp. 441–451. SciTePress, Barcelona, Spain (2012).

3. Dias, D.G., Mira da Silva, M., Helfert, M., Shuyan, X.: Using Enterprise Ontology Methodology to Assess the Quality of Information Exchange Demonstrated in the case of Emergency Medical Service. Proceedings of the 18th Americas Conference on Information Systems (AMCIS 2012). pp. 1–11. Association for Information Systems, Seattle, Washington, USA (2012).

4. Guerreiro, S., Vasconcelos, A., Tribolet, J.: Enterprise dynamic systems control enforcement of run-time business transactions - Lecture notes. In: Aalst, W. van der, Mylopoulos, J., Rosemann, M., Shaw, M.J., and Szyperski, C. (eds.) Enterprise Engineering VI. pp. 46–60. Springer (2012).

5. Maij, E., Toussaint, P.J., Kalshoven, M., Poerschke, M., Zwetsloot-Schonk, J.H.M.: Use cases and DEMO: aligning functional features of ICT-infrastructure to business processes. International journal of medical informatics. 65, 179–191 (2002).

6. Henriques, M., Tribolet, J., Hoogervorst, J.: Enterprise Governance and DEMO - Guiding enterprise design and operation by addressing DEMO's competence, authority and responsibility notions. 473–476 (2010).

7. Pombinho, J., Aveiro, D., Tribolet, J.: Towards Objective Business Modeling in Enterprise Engineering-Defining Function, Value and Purpose. Advances in Enterprise Engineering VI. (2012).

8. Op 't Land, M., Zwitzer, H., Ensink, P., Lebel, Q.: Towards a fast enterprise ontology based method for post merger integration. In: Shin, Sung, Y. and Ossowski, S. (eds.) Proceedings of the 24th Annual ACM Symposium on Applied Computing (SAC 2009). pp. 245–252. Honolulu, Hawaii, USA (2009).

9. Nagayoshi, S., Liu, Y., Iijima, J.: A study of the patterns for reducing exceptions and improving business process flexibility. Advances in Enterprise Engineering VI. pp. 61–76. Springer (2012).

10. Barjis, J.: A business process modeling and simulation method using DEMO. Enterprise Information Systems. pp. 254–265. Springer (2009).

11. Dumay, M., Dietz, J.L.G., Mulder, J.B.F.: Evaluation of DEMO and the Language / Action Perspective after 10 years of experience. The Language Action Perspective on Communication Modelling, Kiruna, Sweden. pp. 77–105 (2005).

12. Enterprise Engineering Institute website, http://www.demo.nl/publications.

13. Winter, R.: Design science research in Europe. European Journal of Information Systems. 17, 470–475 (2008).

14. Aier, S., Fischer, C.: Criteria of progress for information systems design theories. Information Systems and e-Business Management. 9, 133–172 (2010).

15. Henderson-Sellers, B.: Method Engineering: Theory and Practice. In: Karagiannis, D. and Mayr, H.C. (eds.) Information Systems Technology and Its Applications. 5th International Conference ISTA 2006. p. 84. Lecture Notes in Informatics (LNI) – Proceedings, Klagenfurt, Austria (2006).
16. Mettler, T., Rohner, P.: Situational maturity models as instrumental artifacts for organizational design. Proceedings of the 4th International Conference on Design Science Research in Information Systems and Technology (DESRIST 2009). p. 1. ACM Press, New York, New York, USA (2009).

17. Kensing, F.: Towards Evaluation of Methods for Property Determination: A Framework and a Critique of the Yourdon-DeMarco Approach. Beyond Productivity: Information Systems Development for Organizational Effectiveness. 325–338 (1984).

18. Seligmann, P.S., Wijers, G.M., Sol, H.G.: Analyzing the structure of IS methodologies, an alternative approach. In: Maes, R. (ed.) Proceedings of the First Dutch Conference on Information Systems. pp. 1–28. Amersfoort, The Netherlands (1989).

19. March, S.T., Smith, G.F.G.: Design and natural science research on information technology. Decision Support Systems. 15, 251–266 (1995).

20. Rescher, N.: Methodological pragmatism: A systems-theoretic approach to the theory of knowledge. Blackwell, Oxford (1977).

21. Moody, D.L.: The method evaluation model: a theoretical model for validating information systems design methods. In: Ivan (ed.) Proceedings of the 11th European Conference on Information Systems (ECIS 2003). , Naples, Italy (2003).

22. Hommes, B.-J., van Reijswoud, V.E.: Assessing the quality of business process modelling techniques. Proceedings of the 33rd Hawaii International Conference on System Sciences (HICSS 2000). pp. 1–10 (2000).

23. Weigand, H.: LAP: 10 years in retrospect. The Language Action Perspective on Communication Modelling, Kiruna, Sweden, June 19-20, 2005. pp. 1–8 (2005).

24. Khavas, S.S.: The Adoption of DEMO in Practice - Dissertation for a Master of Science in Computer Science, (2010).

25. Winter, R., Gericke, A., Bucher, T.: Method versus model - two sides of the same coin? Advances in Enterprise Engineering III. 1–15 (2009).

26. Huysmans, P., Ven, K., Verelst, J.: Using the DEMO methodology for modeling open source software development processes. Information and Software Technology. 52, 656–671 (2010).

27. Ven, K., Verelst, J.: The adoption of demo: A research agenda. Advances in Enterprise Engineering III. pp. 157–171. Springer (2009).

28. Mingers, J., Brocklesby, J.: Multimethodology: Towards a framework for mixing methodologies. Omega. 25, 489–509 (1997).

29. Myers, M.D.: Qualitative research in information systems. MISQ Discovery, archival version, June 1997, http://www.misq.org/supplements/. Association for Information Systems (AISWorld) Section on Qualitative Research in Information Systems, updated version, last modified: March 21, 2013 www.qual.auckland.ac.nz. 21:2, 241–242 (1997).

30. Walsham, G.: Interpreting information systems in organizations. John Wiley & Sons, Inc. (1993).

31. Walsham, G.: The Emergence of Interpretivism in IS Research. Information Systems Research. 6, 376–394 (1995).

32. Myers, M.D., Newman, M.: The qualitative interview in IS research: Examining the craft. Information and Organization. 17, 2–26 (2007).

33. Rubin, H.J., Rubin, I.S.: Qualitative interviewing: The art of hearing data. Sage (2011).

34. Aier, S., Fischer, C.: Scientific Progress of Design Research Artefacts. Proceedings of the 17th European Conference on Information Systems (ECIS 2009). (2009).

35. Hevner, A.R., March, S.T., Park, J., Ram, S.: Design Science in Information Systems Research. Information Systems Research. 28, 75–104 (2004).
36. Gregor, S., Jones, D.: The anatomy of a design theory. Journal of the Association for Information Systems. 8, 312–335 (2007).

37. Venable, J.R., Pries-Heje, J., Baskerville, R.: A comprehensive framework for evaluation in design science research. In: Peffers, K., Rothenberger, M., and Kuechler, B. (eds.) Design Science Research in Information Systems. Advances in Theory and Practice. pp. 423–438. Springer-Verlag Berlin Heidelberg (2012).

38. Venable, J.R.: Identifying and Addressing Stakeholder Interests in Design Science Research: An Analysis Using Critical Systems Heuristics. Information Systems-Creativity and Innovation in Small and Medium Enterprises. 93–112 (2009).

39. Aier, S., Fischer, C., Winter, R.: Theoretical Stability of Information Systems Design Theory Evaluations Based upon Habermas’s Discourse Theory. A Journal On The Theory Of Ordered Sets And Its Applications. (2011).

40. Wieringa, R.J.: Design science as nested problem solving. Proceedings of the 4th international conference on design science research in information systems and technology (DESRIST). p. 8 (2009).

41. Miles, M.B., Huberman, A.M.: Qualitative Data Analysis: An Expanded Sourcebook. SAGE (1994).

42. Vavpotic, D., Bajec, M.: An approach for concurrent evaluation of technical and social aspects of software development methodologies. Information and Software Technology. 51, 528–545 (2009).