# Incommensurability and Multi-paradigm Grounding in Design Science Research: Implications for Creating Knowledge

Dirk S. Hovorka

► To cite this version:

Dirk S. Hovorka. Incommensurability and Multi-paradigm Grounding in Design Science Research: Implications for Creating Knowledge. IFIP WG 8.2/8.6 International Working Conference on Human Benefit through the Diffusion of Information Systems Design Science Research, Mar 2010, Perth, Australia. pp.13-27, 10.1007/978-3-642-12113-5_2. hal-01060391

HAL Id: hal-01060391

https://inria.hal.science/hal-01060391

Submitted on 4 Sep 2014

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
2

INCOMMENSURABILITY AND MULTI-
PARADIGM GROUNDING IN DESIGN
SCIENCE RESEARCH: Implications for
Creating Knowledge

Dirk S. Hovorka

Bond University
Gold Coast, Queensland, Australia

**Abstract**

The problem identification–design–build–evaluate–theorize structure of design science research has been proposed as an approach to creating knowledge in information systems and in broader organizational and social domains. Although the approach has merit, the philosophical foundations of two specific components warrant attention. First, the grounding of design theory on potentially incommensurate kernel theories may produce incoherent design theory. In addition, design theory has no strong logical connection to kernel theories, and so cannot be used to test or validate the contributing kernel theories. Second, the philosophical grounding of evaluation may inadvertently shift from functionally based measures of utility and efficiency, to evaluation based on the pragmatic fulfillment of multidimensional human actions as people encounter information systems, resulting in evaluation errors. Although design and evaluation from a single paradigm is not desirable, sufficient, or representative of design science research, multi-paradigm grounding of design and evaluation must be realized and used consciously by the research community if the design science approach is to remain a legitimate approach to knowledge creation.

**Keywords**

Design science research, incommensurability, paradigm, pragmatism, functionalist, kernel theory
# 1 INTRODUCTION

The emergence and influence of design science research (DSR) as a distinct research approach in information systems is gathering significant attention. IS as a discipline has always contained a significant intellectual focus on designing systems for functional goals, and the emergence of DSR lends legitimacy and credibility to the generative aspects of IS. But the suggestion that DSR has "become a new way of creating and studying phenomena where understanding comes from building solutions to solve problems"<fnref n="1" /> has a number of important implications that warrant discussion.

Although the conceptualization of DSR is under discussion and is still evolving, convergence on a number of central tenets and a general structure of “problem identification–build–evaluate–theorize” (Winter 2008) is emerging. The primary design/build-justify/evaluate phases suggested in Hevner et al. (2004) have been expanded by Baskerville et al. (2007) and the potential benefit of interpretive approaches (Niehaves 2007) has been suggested. In addition, the importance of extending design to the user-as-designer (Germonprez et al. 2007; Hovorka and Germonprez 2009) and expanding evaluative criteria of DSR (Baskerville et al. 2007) are gradually influencing the DSR community to incorporate a broader view. The coalescence of a community of researchers with shared problem domains, exemplars, methods, and evaluative criteria has led some to consider DSR to be achieving paradigmatic status (Hevner et al. 2004; van Aken 2004). As such, there is interest in how the DSR approach and structure can be diffused to a wider context of organizational and societal needs to create knowledge by building working solutions to problems. DSR seeks to create knowledge in the form of *technological rules* (Bunge 1967; van Aken 2004), which are composed of explicit prescriptions for building an artifact with an expected performance or outcome in a specific problem domain (Gregor and Jones 2007). Humans have long created technological rules or models intended to achieve goals including artifacts (in a broad sense), social processes, and organizational interventions and structures. Critically, technological rules must be grounded in the natural and behavioral sciences to produce coherent knowledge claims (Goldkuhl 2004a; van Aken 2004).

As DSR becomes reified into a set of guidelines (Hevner et al. 2004), and design theories are evaluated against a particular anatomical structure (Gregor and Jones 2007; March and Smith 1995), the emphasis begins to approach a dominance of method over science (Nietzsche 1968). These guidelines and structures are used to define the differences between DSR and paradigms of knowledge production such as the natural and behavioral sciences (Hevner et al. 2004) and alternative systems design paradigms (Butler and Murphy 2007; Hirschheim and Klein 1989). This paper seeks to shift the focus away from the method of DSR, to a deeper consideration of the philosophical assumptions underlying knowledge claims resulting from the DSR approach.

The goal of this research is to reinvigorate discussion of the philosophical foundations by which the DSR approach creates and evaluates knowledge. By peering

<footnote n="1">¹"Call for Papers," IFIP 8.2 + 8.6 Joint International Working Conference (http://www.ifip.or.at/Cfp/CfP-8286%20Perth.htm, accessed November 29, 2009), emphasis added.</footnote>
underneath the guidelines and structure that are becoming dominant in DSR, in order to
examine foundational concepts in the creation and refinement of scientific knowledge,
researchers will be better prepared to justify their choice and use of kernel theories and
the evaluation of their knowledge claims.

To focus this discussion, this research examines two aspects that have received little
attention.

1. Potential incommensurability in the selection, use, and interactions of kernel theory in DSR, and subsequent implications for kernel theory validation.

2. The shift in our conceptualization of evaluation implied by a *building solutions to solve problems* approach.

The paper begins by examining issues of incommensurability and implications for the choice of kernel theories in DSR. Next, it reframes the nature of DSR as producing knowledge which mediates, rather than solves, problems. In doing so, the research specifically points to the phenomenon of secondary design to demonstrate the potential issues of incommensurate paradigms in evaluation. The paper concludes with a call for greater attentiveness to the philosophical foundations, rather than the method, by which DSR makes knowledge claims.

## 2 KERNEL THEORY SELECTION AND COMMENSURABILITY

Design science has long recognized that design theories are composite theories whose kernel theories (March and Smith 1995) or justificatory knowledge (Gregor and Jones 2007) are derived from reference disciplines. These kernel theories serve a dual purpose: first, they provide the often informal hypotheses that a given design principle will produce the desired phenomenon, and second, they are the target of *extension* and *refinement* rather than disconfirmation through the generate/test cycle of DSR (Kuechler and Vaishnavi 2008). The refinement and extension of kernel theory is claimed as a key contribution of the DSR approach. It has been suggested that DSR is inextricably bound to the inclusion, testing, and improvement of kernel theories (Kuechler and Vaishnavi 2008), and that artifact development relies on kernel theories that are applied, tested, modified, and extended through the creation of artifacts (Hevner et al. 2004). However, little attention has been paid to either the potential problems resulting from selection of kernel theories from paradigmatically distinct origins, or how a new design theory can be used to test a kernel theory upon which it is somehow based. Refinement of kernel theories, and evaluation of the resultant design theory, becomes problematic if the causal contributions and interactions of the kernel theories cannot be compared against a shared measurement. In raising the question of kernel theory incommensurability, it is assumed that theory incommensurability and concept incommensurability are salient (Andersen et al. 2006; Burrell and Morgan 1979; Kuhn 1977) and have not been cast aside.

The principles of design are frequently drawn from multiple disciplines. For example, Germonprez et al. (2007) grounded their research in principles from infor-
mation systems, computer science, and human–computer interaction, in addition to
architecture, music, and cybernetics. Each of these disciplines contributed to the pro-
posed theory of tailorable technology. The theory of organizational memory information
systems (Stein and Zwass 1995) was grounded in organizational-level effectiveness and
a model of individual level memory. As a third example, the theory of learning-oriented
knowledge management systems (Hall et al. 2003) was grounded in Churchman’s (1971)
theory of inquiring systems and Simon’s (1969) intelligence–design–choice model as
kernel theories (Walls et al. 2004). Interestingly, none of these design theories discussed
the appropriateness of combining theories that account for phenomenon at different levels
of analysis (organizational versus individual) or in distinctly different disciplines
(organizational behavior versus psychology; architecture versus HCI). This raises a
question of two potential types of philosophical conflict: combining incommensurate
theory derived from different methodological or ontological assumptions (Kuhn 1977),
and combining incommensurate concepts in which conceptual meaning varies between
disciplines (Andersen et al. 2006).

## 2.1 The Problem of Incommensurability

Incommensurability is a concern in many disciplines with pluralist traditions. The term *incommensurate* refers to a relation between entities, and raises a potential problem for combining paradigms, theories, and concepts. A full discussion of the ongoing debate on incommensurability is beyond the scope of this paper, but a synopsis will provide a perspective on the problem and its relevance to theory grounding in DSR. Incommensurate theories come from epistemic or ontologically distinct paradigms, such that the theories are mutually unintelligible. Two distinct theories representing systems of orientation (e.g., methods, paradigms) are considered incommensurate if they present conflicting perspectives about possible actions or language, and an acceptable reference system from which to evaluate both theories is lacking (Scherer 1998). There exists no common measure by which to determine the appropriateness of each theory, and the result of combining these theories as justificatory knowledge for a new design theory would be incoherent. Thus, a theory based on the symbolic meaning attached to an information system by users and its subsequent use patterns is incommensurate with a theory positing the independent material variables contributing to a dependent variable measured as system performance. It is meaningless to refer to the cognitive sense-making of material variables, and unwarranted to look for a causal-mechanical explanation of human subjective understanding of systems. Suggesting that these are incommensurate as kernels for design theory does not privilege one theory over the other. Each in its own right may provide a foundation for new design theory. But we should focus a critical eye on combining paradigmatically incommensurate kernel theories in DSR, as they constitute entirely different views of the world. Furthermore, to suggest that kernel theories may be incommensurate does not contradict the value of pluralism in research (Mingers 2001), as mixed method studies are intended to discover truths about the world, not to build novel artifacts. The role of theory in discovery research is quite distinct from the role of kernel theory in DSR.
Kernel theory selection is another rarely examined area in which we are faced with the question of which theories will best serve as kernels for DSR. Kuhn (1962) suggests that the evaluation of good theory be based on accuracy, simplicity, scope, consistency, and fruitfulness, but he recognizes the inherently social and practical underpinning of these criteria. There is no objective measure by which to determine which theories would *best* serve as kernels for new designs. The approach used by Kuechler and Vaishnavi (2008) diminishes this problem by drawing kernel theory from experimental results in domains (e.g., cognitive and social psychology and education) closely associated with the problem domain of the designed artifact. But even with such an approach to reducing the potential problems of incommensurability, we are unable to claim that these kernel theories are the *best* theories upon which to base design research. For every set of selected kernel theories, there exist alternative kernel theories from which design knowledge could potentially be developed for the same problem space. This discontinuity between the subjective selection of kernel theory and the desired functionalist evaluation of design theory places a significant burden on the evaluation of all knowledge contributions made by a DSR approach.

Also salient is conceptual or linguistic incommensurability of similar terms drawn from different reference disciplines. Although there are multiple theories of concepts, some consensus suggests that conceptual incommensurability varies in degree and importance, but does occur between cognitively derived human conceptual structures (Andersen et al. 2006). Much of the debate has revolved around conceptual changes over time within a single discipline, but the problem also exists as concepts are imported across disciplinary boundaries. Two potential problems arise here. First, not recognizing differences in concepts is likely to result in an attempt to relate, in a theoretical manner, two ideas that are individually coherent and clear but are not in any way associated. Second, as the evaluation phase attempts to refine kernel theories, the researcher will have lost the ability to distinguish between the concepts and will be unable to resolve the antecedents of the artifact's success on the contributing concepts. For example, the concept of *information* may refer to the mathematical telecommunications concept, to a human psychological construct, to an object that can be stored, transmitted, and retrieved (Buckland 1991), or to a description about, for, or as reality (Borgmann 1999). The term *process* is multi-conceptual depending on context (e.g., process records; process redesign; the system development process). Thus, the seemingly simple combination of *information* and *processing* across references disciplines (*information processing* in computer science versus psychology) refers to different activities and constructs and illustrates the potential difficulties of concept incommensurability.

It should be emphasized that incommensurability does not preclude successful design. Indeed, there are examples of artifacts that *work* without researchers understanding how or why. But the distinction between DSR and design practice is the former's emphasis on the knowledge resulting from design and evaluation versus the latter's desire to simply fulfill a functional goal. An incoherent design theory from DSR is not a knowledge contribution inasmuch as it may result in functional but theoretic instantiations.
## 2.2 Solutions or Mediations: An Evaluative Shift

A significant rhetorical issue stems from the emphasis on DSR as an inherently problem-solving process for creating solutions to problems of interest to practice (Hevner et al. 2004; Kuechler and Vaishnavi 2008). This focus raises the issue of potential epistemic incommensurability in the DSR evaluation phase. If we turn to the definition of *solution*, as “the resolution of a difficulty or the solving of a problem,”<fnref n="2" /> we see that few technologies actually resolve or eliminate a problem at all. For example, a hammer does not solve the problem of building houses or even driving nails. Although it is a tool that allows a carpenter to more easily drive nails, the process of driving nails still needs to be accomplished. So we modify the artifact and produce different types of hammers for different contexts and even embrace the compressed-air nail-gun for greater efficiency and efficacy. But this new technological solution does not work in all circumstances. The nail-driving problem is multidimensional and the larger problem of connecting timbers to build houses still exists. The designed artifact creates a more *useful state of affairs* (Angell and Ilharco 2004) than previously existed, but does not resolve the root problem with a solution.

If we look at other professional disciplines to which IS is often compared, we can see that their activities do not claim to resolve or eliminate root problems. Rather, they provide a means for humans to mediate or reduce the impacts of those problems. Laws and legal procedures do not solve the problems of crime, inequality, or breeches of contract. The legal frameworks do provide mechanisms for managing problems when they arise on a case by case basis. In a similar manner, medicine does not solve the problems of disease, traumatic injury, or pain. To suggest that DSR seeks to design information technology artifacts as a *solution to a problem* implies permanent resolution of the problem that requires no future modification of the tools as designed. But laws, medicine, information technology—and even hammers—undergo large scale revision and a continuous series of localized refinements, modification, and secondary design in the context of their use.

Hard disciplines such as mathematics and physics aside, singular and permanent solutions to problems do not exist in most disciplines. In the social sciences, a solution is a model or representation of the world that works better than other models for achieving a desired outcome or mediating a problem instance within a broad problem domain. Design models are context-dependent knowledge bundles (technological rules) among a set of possible alternative contrast-classes which are expected to achieve an expected outcome relevant to a particular set of requirements derived from a specific problem domain. Models are considered better relative to other models through fulfillment of specific measurement criteria and by the context of the person formulating the problem. Thus a manager may implement a technology that selectively benefits a subset of stakeholders, while at the same time increasing problems for other actors. Design models, therefore, identify the contrast-classes of solutions, and then define the relevance relations (Hovorka et al. 2008; van Fraassen 1980) of subjectively selected criteria of the

<footnote n="2">²The definition of solution as used in this paper is from http://define.com/solution.</footnote>
stakeholders championing the design project. As changes in context, task, or stake-
holders occur, the original alternatives and requirements may expose the opportunity for
secondary design or the creation of work-arounds. Therefore, the technological rule was
not a solution as much as a temporarily better state of continuously changing affairs.

This argument may seem obvious as DSR, like all research, is progressive, and
technological rules at the primary design phase will change over time. But the rhetorical
shift from solution to mediation is a necessary part of understanding the role of
pragmatism as an alternative perspective for evaluation in DSR. The ongoing process of
secondary design suggests that information systems do not solve a problem, but instead
provide mediation of information processes between desired states of being (goals) and
current states. The designed artifact provides a *potential for human action* (Winograd
and Flores 1986), which may include the creation and attachment of meaning, increased
capacity for idea generation, or emancipation from organizational structures, in addition
to purely rational functionalist measures of utility. But the technology itself does not
provide a final solution, or even fulfill necessary or sufficient conditions of a solution.
A successful design may offer a model for *change and human action* toward a more
positive outcome in specific problem domains. This stance is more aligned with a
pragmatist philosophy (Goldkuhl 2004b, 2005) than the DSR rational functionalist
perspective. After implementation of a specific artifact, refined models, which mediate
human action when faced with specific problem instances in the domain, will be offered,
and each one will be modified, redesigned, or worked-around, as contexts, actors, and
tasks change.

This seemingly obvious observation conceals the underlying philosophical shift from
the rational functionalist perspective of DSR, in which success is evaluated in terms of
utility-based goals, to a pragmatic perspective where the information systems may be
evaluated as successful (or unsuccessful) due to unanticipated or intangible effects not
specified in the original design, and on the ability of the system to support human action.
Rational functionalism emphasizes the technology impact as measured by productivity
and effectiveness of work practices, whereas pragmatism considers the rearrangement of
things and people and the way in which artifacts perturb the assemblages of technologies,
people and work processes (Coyne 1995; Latour 1995). However, it is important to
recognize that *pragmatic* is not the same as *utilitarian*. Pragmatism posits the researcher
or the stakeholder requesting the designed artifact has values dependent on their own
interpretation of the relevance and important evaluative measures associated with their
purposes (Goles and Hirschheim 2000).

To examine evaluation in more detail, we must first consider whether we are
evaluating the artifact based upon the criteria determined by the designers, or based upon
how actors actually interact with the built artifacts.

3 ENCOUNTERING DESIGN

One implication of the artifact-centric conceptualization of DSR is the belief that
“people will encounter technology as something that is encountered just as it was
designed, to be appropriated or incorporated into practice” (Dourish 2006, p. 6). This
is stated quite directly in the position that DSR does not attend to the actors using the
technology, nor to the manner in which the technology or work practices are modified
over time (Hevner et al. 2004). Researchers following these guidelines are likely to
privilege the technical artifact over an evaluation of social processes, secondary design,
or emergent benefits in their theorizing. But numerous researchers have noted the
common phenomena of users redesigning technologies and the practices supported by the
technologies as part of their practice (Ciborra 2002; Latour 1995; Robey and Boudreau
1999). Research in human-computer interaction has long recognized that designed
systems often do not match the needs of the people using the system. MacLean et al.
(1990) note that it is impossible to design systems that will fulfill the goals of all users
in all situations. Dourish (2001) suggests that the designers do not share the same model
of the task domain as the users. Unique functions and applications are created as systems
are used in ways that the designers did not anticipate (Winograd and Flores 1986).
Design theories are representations or models of the designer's view of the problem
domain and the artifact that will mediate human action in that domain. If human actions
are over-determined, such that the coupling of the system actions to the situated world
is too rigid or incomplete, by necessity end-users will modify the information process to
complete their realized, *in situ* work. Human agency and learning play a large role in
enactment of technology (Boudreau and Robey 2005). Human actors who tailor
information processes are acting as secondary designers in the ongoing creation and
recreation of information environments. This is fundamental human activity but currently
not recognized in most design theorizing. Although guidelines for design theory fall
short of creating theories that account for the end users' reflections, tinkering, and
subsequent tailoring of information systems in a process of secondary design, an even
larger problem is presented in the evaluation phase of DSR.

4 SECONDARY DESIGN

The belief that artifacts are encountered just as they were designed has resulted in
IS research evaluating workers' deviation from prescribed uses of information systems,
and the creation of workarounds as resistance. Yet the same research calls upon
designers, developers, and managers to develop adaptable and reconfigurable systems
that can accommodate a wider variety of user behaviors and tasks (Ferneley and
Sobreperez 2006). As an increasing number of design models are conceptualized as
information environments, where actors engage in information processes through
reflection and action and engage in secondary design (Germonprez et al. 2007; Hovorka
and Germonprez 2009), the evaluation of the artifact solely by the initial criteria risks
undervaluing innovative system modifications (Ciborra 2002).

Actors tailor systems and practices during use for many reasons. One reason for
secondary design is the actors’ desire that the designed artifact enable them to accomplish
their own goals. But it is unlikely that their goal coincides with the highly functionalist
and rational goals upon which the artifact was designed/built and upon which evaluation
will be based. Few knowledge workers are thinking to themselves how efficient,
profitable, or even how useful the artifact is. Research from phenomenological perspec-
tives (Boland 1985; Introna and Whittaker 2002) and from pragmatic perspectives (Goldkuhl 2005; Goles and Hirschheim 2000) reveal that technology users may be motivated by pragmatic reasons such as “this is the only information systems available,” “this will work if I tailor the system to shortcut three steps,” or “my modified procedure makes more sense to me than the designed process.”

A second reason for secondary design comes from the limitation of designers to fully comprehend the conditions of use. All models and evaluations are based on objects and attributes preselected by the designer. The motivation for the design model, and the rationale for how the designer arrived at that model, is absent from the actual instantiation. When the actor is incapable of achieving desired goals with the technology because the task demands placed on the artifact are different than the original model, a breakdown has occurred and there is no basis for the artifact, as designed, to operate (Winograd and Flores 1986). The only way to generate a new model or representation is from the actor's experience, which is outside the artifact's original design realm. Furthermore, many innovative processes, and the creation of new knowledge, are unexpected consequences of use. In the implementation and secondary design of technologies, many system features and user behaviors emerge that are not within the scope of the original specifications (Ciborra 2002). Evaluation of design success must include the ability to recognize beneficial outcomes that are idiosyncratic, unplanned, and emergent. It is evident that it is impossible for a primary design effort to completely specify all possible system uses *ex ante*.

Therefore, the current conceptualization of DSR brings forth a tension between our desire for a rational and emotionless logic through which information technologies are designed and evaluated, contrasted with real human actors who encounter those technologies in situated and emotion-laden practice. An examination of variable-centered IS research notes that, in most studies, the actors or managers who might benefit from the research are not represented in the study (Ramiller and Pentland 2009). In the same way, design science research neglects the actors who will be using, subjected to, and whose work processes will be evaluated through the rational lens of the technology. The actions themselves, the meaning of the actions attached by the actors for whom the system is designed, and the embodied participation of use, are expunged and not accounted for in the design or evaluation. Although researchers recognize that technology and action are inseparable in information system design (Hevner et al. 2004), the current view of evaluation is hampered by a narrow definition of design, which produces an appliance mentality of design (Lee 2001), and a rational functionalist view of evaluation, which does not account for the secondary design of the system in practice. A critical extension to design science research for both the design and evaluation phases is to incorporate the tendency of people to tinker, tweak, tailor, and otherwise modify the system to fit their particular context (Ciborra 2002; Dourish 2006; Hovorka and Germonprez 2009).

# 5 EVALUATION

DSR explicitly incorporates evaluation as one of the essential guidelines, yet it is an impoverished view of evaluation based upon a narrow functionalist perspective that
defines successful design only in terms of utility, quality, and efficacy of technological artifacts (Hevner et al. 2004) and thin epistemological grounding. Systematic testing is often achieved by treating the model as a black box, and by linking its use to specific outcomes (van Aken 2004). This is a very pragmatic philosophy interested in change and action (Goldkuhl 2005) and is not concerned with causality or the explanatory truth of theories (Gregor 2006; Hovorka et al. 2008). DSR recognizes that models can also be tested scientifically, whereby the functionality and use of the artifact can be explained and predicted. But phenomena such as secondary design (Germonprez et al. 2007; Hovorka and Germonprez 2009), and the separation between a user encountering an artifact and the original design specifications, makes evaluation from a solely functionalist perspective problematic. Difficulties arise in rigorous scientific testing of design theory *in situ* where pragmatic evaluation vies with rational functionalist requirements. If the DSR approach is extended to other areas of research, such as management or organizational studies (Romme 2003; van Aken 2004), for the purpose of creating knowledge, then evaluation becomes a cornerstone of its legitimacy.

To address this evaluation problem, Baskerville et al. (2007) introduce the idea of soft design science research, which includes a multistage evaluation process, but the proposed framework is fully embedded in the rational functionalist paradigm of meeting articulated requirements. The framework does suggest that the determination of success and of failures is complex, and includes multiple perspectives by multiple stakeholders, as well as the attribution of failure to externalities rather than design.

The exposition of an organizational information system case presented in Baskerville et al. (2004) illustrates the difficulties presented by incommensurate perspectives on evaluation. Although the study artifact was originally evaluated to be a success, changes in context (new managers who were not as well known or trusted by upper management) led to subversion of the information system and its eventual removal because it had become socially destructive (Baskerville and Land 2004). It is important to recognize that the philosophy underlying the evaluative criteria shifted during the time period in question. Even as the system became socially destructive, it was still capable of meeting the original functionalist goals of delivering information to senior executives. This suggests that the original requirements were instrumentalist in nature, but the later evaluation emphasized a pragmatic perspective of the ability of the system to support human actions over time. Although it is useful to classify evaluation errors in a typology of errors (Baskerville et al. 2007), it is equally important to recognize the philosophical basis upon which evaluation is based and whether it is commensurate with the design paradigms.

It comes as no surprise that shifting paradigms for evaluation will result in conflicting results, particularly if the context, task, or stakeholders have also changed. Adopting a pragmatic desire to create artifacts that *work* or that have beneficial mediation of human action (Goldkuhl 2004b) conflicts with the functionalist, radical, or critical paradigms under which the systems may have been developed (Hirschheim and Klein 1989). The risk is in not recognizing the paradigm in which the design was created and the paradigm from which we are evaluating a built artifact.

The discussion above reinforces and extends the argument that broadening of evaluation to include interpretative or critical approaches capable of capturing outcomes not included in the original utility-based performance measures necessarily requires a
shift from rational functionalist paradigms to other evaluative approaches. Interpretive (Boland 1978; Niehaves 2007) or phenomenological (Introna and Whittaker 2002) approaches inform initial design and also evaluation by uncovering the ontology of the actual work (Butler and Murphy 2007; Suchman et al. 1999) and viewing the technology through the actor's eyes. Although it is recognized that organizational actors learn and modify processes or technologies to better fit their actual work (Robey and Boudreau 1999), from the functionalist perspective of the artifact this is resistance (Lapointe and Rivard 2005) and a failure of the information system. But from an interpretive or a critical perspective, respectively, it may represent an actor's creation of identity, or liberation from organizational strictures. Our understanding of the philosophical underpinnings of evaluation in DSR can be broadened to recognize and incorporate different, clearly defined criteria that will extend the domains in which DSR is a legitimate approach to knowledge creation.

# 6 CONCLUDING THOUGHTS

This research has sought to clarify three implications of expanding the predominantly functionalist DSR approach to knowledge creation into broader organizational and societal research domains. Whereas the DSR approach has enormous potential for knowledge creation in a variety of domains, care must be taken to comprehend and articulate the philosophical underpinnings of theory building and evaluation to avoid grounding knowledge on unwarranted amalgamations of paradigm-bound concepts and the creation of incoherent design theory. While design theory development may be influenced by more than one paradigm, and can be evaluated from multiple perspectives, awareness of the need for clarity when grounding design theory in multiple kernel theories or potentially incommensurate concepts will strengthen the legitimacy of DSR.

First, design theories of artifacts, be they instantiations, algorithms, managerial programs, constructs, or organizational structures, are all *models* for enabling human action and change. These models are not descriptions or explanations of states of being that exist, but rather are models of “new ways of being that did not previously exist and a framework for action that would not have previously made sense” (Winograd and Flores 1986, p. 177). This aspect of DSR is a strongly pragmatic activity wherein pragmatism is concerned with goal-oriented action. Significantly, the design models exist within a spectrum of alternative models that do not have a verifiable truth-value, but rather can each satisfy a variety of predefined or emergent goals. We can only say that this design is *better* than the alternative models against a background of the particular interpretation of conditions declared as *better* by an individual or community.

Second, by focusing attention of the composite nature of design theories, this research identifies the risk of grounding design theory on disparate explanatory kernel theories which themselves may be based upon distinct philosophical stances. The risk is not that the built artifact would not work. Rather, the risk lies in confounding our understanding of why the design works, as we look to the kernel theories from which the new theory was derived. Ontologically incommensurate assumptions or conceptual conflicts in kernel theories will result in a design that may be pragmatically beneficial but at theoretic. We cannot assume that incommensurate kernel theories are operating in
conjunction or in opposition. In fact, we cannot assume anything about the interactions
of such theories! By concatenating theories with disparate ontological or epistemological
assumptions, we lose coherence of the derived design theory or design principles. The
epistemic distance between the new design theory and the kernel theories upon which it
is grounded precludes any direct refinement, testing, or validation of kernel theory. In
addition, the pragmatist emphasis on change and action, rather than the rationalist
emphasis on truth and explanation, requires considerable discrimination to advance
design knowledge while evaluating the *in situ* use of a new artifact. Whereas knowledge
is often perceived as an increasingly accurate reflection of reality, pragmatism recognizes
that to achieve goals, humans must perceive what features can be afforded practical
action, while often neglecting to invoke basic science (Bunge 1996; Goldkuhl 2005).

Third, the tension between rational functionalist evaluation, based upon utility and efficiency, and the pragmatist emphasis on human action and change contribute to confounding evaluations of artifacts and design theories. The evaluation phase of DSR must be firmly grounded and should not meander between pragmatic, functionalist, critical, and interpretative paradigms. Evaluation may flow from any of these positions, and will result in quite different evaluative outcomes depending on the contrast-class between models of reality and the relevant criteria of the evaluator. The conditions of satisfactory performance or fit are not necessarily determinate in advance, but may emerge during the development of the human- artifact interaction. Secondary design in the context of use and changes in the environment of use itself may further complicate evaluation. But the multiple goals of DSR during the design/build-justify/evaluate process are often at odds and may lead to inconsistent results. Greater rhetorical precision is required to insure that the paradigmatic grounding of design and evaluation phases are clearly articulated.

This research does not attempt to settle the long-standing discussion between those
who would isolate paradigms and pluralists who recommend a diversity of paradigms and
research methods. Most social sciences have accepted that there is a diversity of opinions
about what is knowable and how we can know something exists (Mingers 2001; Scherer
1998; Tadajewski 2008). Nor does this research privilege particular research paradigms.
Rather, it suggests that as DSR is expanded across IS and into other organizational and
social domains as an approach to knowledge creation and evaluation, researchers must
recognize and surface their paradigmatic assumptions, boundaries, and limitations. To
assume away or to simply ignore the significant debate surrounding the production and
validation of knowledge would be a disservice to design science research and reduce its
validity as a process of knowledge creation.

References

Andersen, H., Barker, P., and Chen, X. 2006. *The Cognitive Structure of Scientific Revolutions*, Cambridge, England: Cambridge University Press.

Angell, I. O., and Ilharco, F. M. 2004. "Solution Is the Problem: A Story of Transitions and Opportunities," in *The Social Study of Information and Communication Technology*, C. Avgerou, C. Ciborra, and F. Land (eds.), New York: Oxford University Press, pp. 38-61.
Baskerville, R., and Land, F. 2004. "Socially Self-Destructive Systems," in *The Social Study of Information and Communication*, C. Avgerou, C. Ciborra, and F. Land (eds.), New York: Oxford University Press, pp. 263.285.

Baskerville, R., Pries-Heje, J., and Venable, J. 2007. "Soft Design Science Research: Extending the Boundaries of Evaluation in Design Science Research," in *Proceedings of the Second International Conference on Design Science Research in Information Systems and Technology* (Pasadena, CA, May 13-15), pp. 19-38.

Boland, R. 1978. "The Process and Product of System Design," *Management Science* (28:9), pp. 887-898.

Boland, R. 1985. "Phenomenology: A Preferred Approach to Research in Information Systems," in *Research Methods in Information Systems: Proceedings of the IFIP WG 8.2 Colloquium*, E. Mumford, R. Hirschheim, G. Fitzgerald, and T. Wood-Harper (eds.), Amsterdam: North Holland, pp. 181-190.

Borgmann, A. 1999. *Holding on to Reality: The Nature of Information at the Turn of the Century*, Chicago: University of Chicago Press.

Boudreau, M.-C., and Robey, D. 2005. "Enacting Integrated Information Technology: A Human Agency Perspective," *Organization Science* (16: 1), pp. 3-18.

Buckland, M. K. 1991. "Information as Thing," *Journal of the American Society for Information Science* (42:5), pp. 351-360.

Bunge, M. 1967. *Scientific Research II: The Search for Truth*, Berlin: Springer Verlag.

Bunge, M. 1996. *Finding Philosophy in Social Science*, New Haven, CT: Yale University Press.

Burrell, G., and Morgan, G. 1979. *Sociological Paradigms and Organizational Analysis*, London: Heinemann.

Butler, T., and Murphy, C. 2007. "Understanding the Design of Information Technologies for Knowledge Management in Organizations: A Pragmatic Perspective," *Information Systems Journal* (17:2), pp. 143-163.

Churchman, C. W. 1971. *The Design of Inquiring Systems*, New York: Basic Books, Inc.

Ciborra, C. 2002. *The Labyrinths of Information*, Oxford, England: Oxford University Press.

Coyne, R. 1995. *Designing Information Systems in the Postmodern Age*, Cambridge, MA: MIT Press.

Dourish, P. 2001. *Where the Action Is: The Foundations of Embodied Interaction*, Cambridge, MA: MIT Press.

Dourish, P. 2006. "Implications for Design," in *Proceedings ACM Conference on Human Factors in Computing Systems* (Montreal, Quebec, Canada, April 22-27), New York: ACM, pp. 541-550.

Ferneley, E., and Sobreperez, P. 2006. "Resist, Comply or Workaround? An Examination of Different Facets of User Engagement with Information Systems," *European Journal of Information Systems* (15:4), pp. 345-356.

Germonprez, M., Hovorka, D., and Callopy, F. 2007. "A Theory of Tailorable Technology Design," *Journal of the Association of Information Systems* (8:6), pp. 351-367.

Goldkuhl, G. 2004a. "Design Theories in Information Systems: A Need for Multi-Grounding," *Journal of Information Technology Theory and Application* (6:2), pp. 59-72.

Goldkuhl, G. 2004b. "Meanings of Pragmatism: Ways to Conduct Information Systems Research," in *Proceedings of the Second International Conference on Action in Language, Organizations and Information Systems*, Linköping University, Sweden.

Goldkuhl, G. 2005. "Socio-Instrumental Pragmatism: A Theoretical Synthesis for Pragmatic Conceptualisation in Information Systems," in *Proceedings of the Third International Conference on Action in Language, Organizations and Information Systems*, Limerick, Ireland.
Goles, T., and Hirschheim, R. 2000. "The Paradigm is Dead, the Paradigm is Dead...Long Live the Paradigm: The Legacy of Burrell and Morgan," *Omega* (28), pp. 249-268.

Gregor, S. 2006. "The Nature of Theory in Information Systems," *MIS Quarterly* (30:3), pp. 611-642.

Gregor, S., and Jones, D. 2007. "The Anatomy of a Design Theory," *Journal of the Association of Information Systems* (8:5), pp. 312-335.

Hall, D., Paradice, D., and Courtney, J. 2003. "Building a Theoretical Foundation for a Learning-Oriented Management System," *Journal of Information Technology Theory and Application* (5:2), pp. 63-85.

Hevner, A. R., March, S. T., Park, J., and Ram, S. 2004. "Design Science in IS Research," *MIS Quarterly* (28:1), pp. 75-106.

Hirschheim, R., and Klein, H. K. 1989. "Four Paradigms for Information Systems Development," *Communication of the Association for Information Systems* (32:10), pp. 1199-1216.

Hovorka, D. S., and Germonprez, M. 2009. "Tinkering, Tailoring, and Bricolage: Implications for Theories of Design," in *Proceedings of the 15th Americas Conference on Information Systems*, San Francisco, CA, August 6-9.

Hovorka, D. S., Germonprez, M., and Larsen, K. R. T. 2008. "Explanation in Information Systems," *Information Systems Journal* (18:1), pp. 23-43.

Introna, L. D., and Whittaker, L. 2002. "The Phenomenology of Information Systems Evaluation: Overcoming the Subject Object Dualism," in *Global and Organizational Discourse About Information Technology*, E. H. Wynn, E. A. Whitley, M. D. Myers, and J. I. DeGross (eds.), Boston: Kluwer Academic Publishers, pp. 155-175.

Kuechler, B., and Vaishnavi, V. 2008. "On Theory Development in Design Science Research: Anatomy of a Research Project," *European Journal of Information Systems* (17), pp. 489-504.

Kuhn, T. S. 1962. *The Structure of Scientific Revolutions*, Chicago: University of Chicago Press.

Kuhn, T. S. 1977. "Second Thoughts on Paradigms," in *The Essential Tension*, Chicago: University of Chicago Press, pp. 293-319.

Lapointe, L., and Rivard, S. 2005. "A Multilevel Model of Resistance to Information Technology Implementation," *MIS Quarterly* (29:3), pp. 461-491.

Latour, B. 1995. "A Door Must Be Either Open or Shut: A Little philosophy of Techniques," in *Technology and the Politics of Knowledge*, A. Feenberg and A. Hannay (eds.), Bloomington, IN: Indiana University Press, pp. 272-281.

Lee, A. 2001. "Challenges to Qualitative Researchers in Information Systems," in *Qualitative Research in IS: Issues and Trends*, E. Trauth (ed.), Hershey, PA: Idea Group Publishing, pp. 240-270.

MacLean, A., Carter, K., Lovstrand, L., and Moran, T. 1990. "User-Tailorable Systems: Pressing the Issues with Buttons," in *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Empowering People* (Seattle, WA, April 1-5), J. C. Chew and J. Whiteside (eds.), New York: ACM, pp. 175-182.

March, S. T., and Smith, G. S. 1995. "Design and Natural Science Research on Information Technology," *Decision Support Systems* (15:4), pp. 251-266.

Mingers, J. 2001. "Combining IS Research Methods: Toward a Pluralist Methodology," *Information Systems Research* (12:3), pp. 240-259.

Niehaves, B. 2007. "On Epistemological Diversity in Design Science – New Vistas for a Design-Oriented IS Research?," in *Proceedings of the 28th International Conference on Information Systems*, Montreal, Quebec, Canada, December 10-12.

Nietzsche, F. 1968. *The Will to Power*, New York: Vintage Books.
Ramiller, N. C., and Pentland, B. T. 2009. "Management Implications in Information Systems Research: The Untold Story," *Journal of the Association of Information Systems* (10:6), pp. 474-494.

Robey, D., and Boudreau, M. C. 1999. "Accounting for the Contradictory Organizational Consequences of Information Technology: Theoretical Directions and Methodological Implications," *Information Systems Research* (10:2), pp. 167-185.

Romme, A. G. L. 2003. "Making a Difference: Organization as Design," *Organization Science* (14:5), pp. 558-573.

Scherer, A. G. 1998. "Theory: A Problem in Search of a Solution Pluralism and Incommensurability in Strategic Management and Organization," *Organization* (5), pp. 147-168.

Simon, H. A. 1969. *Sciences of the Artificial*, Cambridge, MA: MIT Press.

Stein, E. W., and Zwass, V. 1995. "Actualizing Organizational Memory with Information Systems," *Information Systems Research* (6:2), pp. 85-117.

Suchman, L., Blomberg, J., Orr, J., and Trigg, R. 1999. "Reconstructing Technologies as Social Practice," *American Behavioral Scientist* (43:3), pp. 392-408.

Tadajewski, M. 2008. "Incommensurable Paradigms, Cognitive Bias and the Politics of Marketing Theory," *Marketing Theory* (8:3), pp. 273-297.

van Aken, J. 2004. "Management Research Based on the Paradigm of the Design Sciences: The Quest for Field-Tested and Grounded Technological Rules," *Journal of Management Studies* (41:2), pp. 219-246.

van Fraassen, B. 1980. *The Scientific Image*, Oxford, England: The Clarendon Press.

Walls, J. G., Widemeyer, G. R., and El Sawy, O. A. 2004. "Assessing Information System Design Theory in Perspective: How Useful Was Our 1992 Initial Rendition?," *Journal of Information Technology Theory and Application* (6:2), pp. 43-58.

Winograd, T., and Flores, F. 1986. *Understanding Computers and Cognition: A New Foundation for Design*, Norwood, NJ: Ablex Publishing Corporation.

Winter, R. 2008. "Design Science Research in Europe," *European Journal of Information Systems* (17:5), pp. 470-475.

## About the Author

Dirk S. Hovorka is an associate professor in Information Systems in the School of Information Technology, Bond University, Queensland, Australia. He attended Williams College (in Massachusetts) for his BA, holds an MS in Geology and an MS in Interdisciplinary Telecommunications, and received his Ph.D. in Information Systems from the University of Colorado in 2006. His research includes the philosophical foundations of IS research, the development of design theory, and the evolving role of information systems in science. He has published research in *European Journal of Information Systems*, *Journal of the AIS*, *Information Systems Journal*, *Communications of the AIS*, and *Decision Support Systems*, and has book chapters published in *Business Agility and Information Technology Diffusion* (IFIP 8.6), *Reframing Humans in Information Systems Development* (forthcoming), and *Perspectives on Information Management: Setting the Scene*. He has presented research at the International Conference on Information Systems, the Australian Conference on Information Systems, the European Conference on Information Systems, IFIP WG 8.2 and 8.6, and the Americas Conference on Information Systems. Dirk can be reached at dhovorka@bond.edu.au.