# A Design Science Oriented Framework for Experimental Research in Information Quality

Mouzhi Ge, Markus Helfert

► To cite this version:

Mouzhi Ge, Markus Helfert. A Design Science Oriented Framework for Experimental Research in Information Quality. 15th International Conference on Informatics and Semiotics in Organisations (ICISO), May 2014, Shanghai, China. pp.145-154, 10.1007/978-3-642-55355-4_15 . hal-01350919

HAL Id: hal-01350919

https://inria.hal.science/hal-01350919

Submitted on 2 Aug 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Design Science Oriented Framework for Experimental Research in Information Quality

Mouzhi Ge<fnref n="1" />, Markus Helfert<fnref n="2" />

mouzhi.ge@unibz.it, markus.helfert@computing.dcu.ie

**Abstract.** Design science has been used as a new research paradigm in information quality research. Within design science, experimental validation has been recognized as one of the most important research methodologies. However, as there is a lack of a coherent framework for conducting experimental research, different information quality studies may produce different results, which can be even conflicting. In order to reduce this ambiguity, we have proposed a framework that is used to (1) refine the experimental methodology in information quality research, (2) advance the rigorous information quality research in design science, and (3) demonstrate an exemplary experimental validation in design science.

**Keywords:** Design science, experimental design, information quality, experimental research methodology, information system framework

## 1 Introduction

Design science research (DSR) focuses on creations of artificial systems. It addresses research through the *building* and *evaluation* of artifacts designed to meet stated objectives [10]. *Building* refers to the process of constructing an artifact for a specific purpose and *evaluation* assesses how well the artifact meets objectives. Evaluation is considered as the centre of DSR focusing on the output of design science research. Although evaluation strategies and guidelines have been proposed, practical evaluation of design artifacts is still challenging, as many approaches are subjective and designed for a small number of application scenarios or specific projects. Limited contributions have addressed the practicalities of evaluating research outputs that are designed within a complex research environment.

Recent methodological research has confirmed that experimental research is an effective way to evaluate artifacts, however, when incorporating experimental research into practical research scenario, we are yet facing the problem of how to conduct the experiment in design science. Therefore in this paper, we have proposed a framework to refine the experimental research in design science in an application level. We demonstrate the framework in information quality research.

Information quality research has been well developed during the last two decades. DeLone and McLean [3] proposed a comprehensive IS success model that considers

<footnote n="1">¹ Free University of Bozen-Bolzano,
I-39100, Bolzano, Italy</footnote>

<footnote n="2">² Dublin City University, Glasnevin
Dublin 9, Ireland</footnote>
information quality and system quality are the influencing factors to IS use and user
satisfaction. In turn they will cause individual and organizational impact. This work
not only lead to a large number of validation research on this model but also bring
further attention of information quality into the IS community. Afterwards, different
information quality research such as information quality dimensions [21, 30],
information quality assessment [5, 9, 31] and information quality management [12,
32] has been conducted.

With the emerging research paradigm of design science, more and more
researchers are using design science to conduct information quality research [13, 33,
34]. Given the nature that information quality research can be conducted with
experimental research methodology in design science, we therefore donate our
demonstration in this research area.

## 2 Design Science

In information system research, researchers distinguished two paradigms: behavioral
science research and design science research [10]. The former is understood as a
"problem understanding paradigm", the latter as a "problem solving paradigm". A key
characteristic of DSR is that it resolves an important, previously unsolved problem,
for a class of businesses or environments, while making a contribution to the
knowledge base [29].

Design researchers investigate the current knowledge and solutions to insure they
do not just replicate past work of others. The value of a new solution may come from
various activities such as solving a known or expected problem, satisfying needs, or
innovating something new. However, the new knowledge comes from "the number of
unknowns in the proposed design which when successfully surmounted provides the
new information that makes the effort research and assures its value" [26]. The
research may involve searching the existing knowledge base, or collecting primary
data through empirical work such as case studies, interviews, experiments or surveys.
Research should stop if the problem has already been solved, or if it is found to be
unimportant for the targeted objectives. Through this research process, the design
science researcher satisfies the relevance condition for DSR in IS [11], while also
addressing generalizability [1]. Characteristic for DSR is that rich phenomena that
emerge from the interaction of people, organizations, and technology may need to be
qualitatively assessed to yield an understanding of the phenomena adequate for theory
development or problem solving [15]. The process of constructing and exercising
innovative IT artifact enable design-science researchers to understand the problem
addressed by the artifact and the feasibility of their approach to its solution [19].

It is generally agreed, that design science research develops knowledge that can be
used by professionals in the field in question to design innovative solutions to their
field problems [25]. To obtain knowledge for innovative solutions, Van de Ven [27]
proposed engaged scholarship as a participative form of design science research. It
accommodates points of views of key stakeholders to understand complex problems.
By exploiting differences between stakeholders, engaged scholarship develops
knowledge that is more penetrating and insightful than when researchers work alone.
Sein et al. [22] propose action design research method to interlink the buil-ding and evaluation phases and thereby emphasising the organisational context. Illust-rating the complexity of developing innovative outputs, Leonard [18] outlines that working across boundaries between disciplines, specializations, or expertise is a key ingredient for most innovative solutions.

Since design is inherently an iterative and incremental activity, the evaluation phase provides essential feedback to the build phase concerning the quality and utility of the design output under development and its design process. Evaluation delivers evidence that an artifact developed achieves the purpose for which it was designed and consequently provides indications for the design process. Experimental research has been recoganized as one the most important methods to evaluate and confirm the artifact.

## 3 Experimental Research in Design Science

Researchers identified a number of methods that can be used for evaluation of design science artifact. Hevner, et al. [10] proposed five classes of evaluation methods: (1) Observational methods include case study and field study. (2) Analytical methods include static analysis, architecture analysis, optimization, and dynamic analysis. (3) Experimental methods include controlled experiment and simulation. (4) Testing methods include functional testing and structural testing. (5) Descriptive methods include informed argument and scenarios.

As a further study, Venable [28] divides evaluation into artificial and naturalistic. Artificial evaluation includes laboratory experiments, field experiments, simulations, criteria-based analysis, theoretical arguments, and mathematical proofs. It evaluates a solution in a contrived and non-realistic way. Naturalistic evaluation explores the performance of a solution in its real environment. By performing evaluation in a real environment (real people, real systems, and real settings [23], naturalistic evaluation embraces all of the complexities of human practice in real organizations. This approach is always empirical, and includes methods such as case studies, field studies, surveys, and action research [29]. While the dominance of the naturalistic paradigm brings to naturalistic DSR evaluation the benefits of stronger internal validity [8], limited research has been done on the artificial evaluation such as laboratory and field experiments.

Experimental research involves directly manipulating a small number of variables and identifying the relationship between these variables. Using quantitative analysis, we can use the analysis results to test hypotheses or validate the artifact. An ideal experiment is designed to control all other possible factors affecting the experimental outcome and show how independent variables affect dependent variables [17]. It has been found that laboratory experiments are an effective methodology in addressing the cause and effect relationship [2, 6, 14], especially in investigating the cause and effect relationship between attributes of the decision environment, characteristics of information system and decision performance [4].

One critical concern in experimental research is the validity. Experiment validity can be divided into internal validity and external validity. The lack of internal validity
means the experimental result is affected by uncontrolled factors. To improve internal validity, Field and Hole [6] proposed eight factors potentially threatening internal validity: group threats, regression to mean, time thread, history, maturation, instrument change, different mortality, reactivity, and experimenter effects. The above threats can be resolved or minimized by experimental controls, such as providing monetary incentive to subjects and selecting appropriate subjects at random. External validity tests how well the research findings generalize to other populations and circumstances. Two threats are associated with external validity: over-use of the special participants and restricted numbers of participants [6]. Considering the two threats, external validity can be increased by carrying out empirical tests across different participants and situations.

## 4 Experimental Research Framework in Design Science

In order to develop the experimental research framework, we have firstly reviewed the methodological issues that may occur in experimental research. Jarvenpaa et al. [14] proposed four open methodological issues in experimental information system research: research strategy, measuring instruments, research design, and experimental task. Research strategy emphasizes that the research program should be performed under a theory, a model or a framework. Two issues are related to the research strategy: a lack of theories for guiding the research [24], and studies which fail to build upon the work of others [14]. Measuring instruments focus on the reliability and validity of the measurements. Research designs concentrate on two issues: the importance of the research and the absence of experimental control [14]. Experimental task refers to a work that is taken by subjects in the experiment. The task is considered inappropriate when it is ambiguous or excessively complex. An ambiguous task might consist of inconsistent, incomplete and incorrect problems. An overly complex task may foster in the subjective influences such as preference, experience and even gambling.

<table><caption>**Table 1.** Methodological issues in experimental research</caption><thead><tr><th colspan="2">Methodological issues in experimental research [14]</th></tr></thead><tbody><tr><td>Research Strategy</td><td><ul><li>Lack of theories for guiding the research</li><li>Studies without building upon other's work</li></ul></td></tr><tr><td>Measuring Instrument</td><td><ul><li>Reliability</li><li>Validity</li></ul></td></tr><tr><td>Research design</td><td><ul><li>The importance of the research</li><li>The lacking of the experimental control</li></ul></td></tr><tr><td>Experimental task</td><td><ul><li>Ambiguous</li><li>Overly complex</li></ul></td></tr></tbody></table>

Considering the methodological issues mentioned in Jarvenpaa et al. [14], we have proposed a framework as shown in Figure 1. This framework consists of three components, which are artifact, experiment and data analysis. Along with each
component, we provided a set of guidelines to deal with the methodological issues in
experimental research.

Given the nature of design science, this paradigm is used to solve the practical
problems, thus artifact should intend to solve a real-world problem. In order to build
the artifact with theoretical basis, certain theory should be used to support for building
this artifact. To highlight the novelty and importance of the artifact, an extensive
literature review needs to be conducted. The artifact building will provide basis for
the experimental design. In the experimental design, a validation of measuring
instrument is needed, this is to intensify the experiment validity. The external factors
that may influence the experiment should be kept under control. From the
participant's perspective, the experiment should be easy to understand and easy to
operate. After collecting the experimental data, we need to firstly understand which
type of data is collected such as nominal data, ordinal data, interval data or ratio data,
based on the data type, the according data analysis can be carried out. In turn, the data
analysis can used to validate, evaluate and improve the artifact.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Experimental Research Framework in Design Science</figcaption></figure>

5 Validation with Information Quality Research

In order to demonstrate the usage of our framework, we have conducted an empirical
information quality research design. In today's organizations, one important factor
concerning information quality is that it directly influences decision-making. Owing
to this, recent information quality research shows an increasing tendency to study the
relationship between information quality and decision-making. Although their
research findings confirmed that making correct decisions is dependent upon high
quality information, exactly how information quality affects decision-making is still
not entirely understood [7].

Case studies concerning poor information quality in decision support system are
frequently documented in recent years and relate to a broad range of domains.
Information quality issues may not only cause errors in business operations but also
potentially impact society and wider aspects. For example, in 1986 NASA lost the space shuttle Challenger with seven astronauts onboard. The Presidential Commission investigated the Challenger accident and found that NASA's decision-making process was based on incomplete and misleading data. Just 2 years later the US Navy Cruiser USS Vincennes shot accidentally an Iranian commercial aircraft with 290 passengers onboard. Officials who investigated the Vincennes accident admitted that poor-quality information was a major factor. Yet not only in the space and military industries but also in our daily decision, certain information quality problems can lead to severe results; for instance, Pirani [20] reported that one piece of wrong biopsy information caused a patient's death in an Australian hospital. Real-world examples such as these illustrate cases in which poor information quality has significant impact on decision-making and may lead to irreversible damages.

From different case studies, we can conclude a real-world problem: *"How to build a decision support system with high quality information?"* To investigate this research question, one key question is to find out how information quality affects decision-making. As we have mentioned in Section 3, experimental research is an effective way to address the cause and effect relationship. We therefore use experiment to conduct this research. As we mainly focus on the experimental research in design science, we in the following only detail the research design related to the experiment part.

To start the design science procedure, first we need to define our artifact. As derived from the practical case study, our artifact is a *decision support system with high quality information*. To build this artifact, we use DeLone and McLean IS success theory to guide the design. It can be seen that high quality information can affect the decision-making (*Use*) and user satisfaction, and it will in turn generate individual impact and organizational impact. Literature across the domain information system, information management and information quality is related to this research work.

The experimental design is based on a well-known management game, the BeerGame. This game is a role-playing simulation, which involves managing supply and demand in a beer supply chain. The concept for this game was first developed at the Massachusetts Institute of Technology in the 1960s. Since then, several extensions and modifications have been suggested. Kaminsky and Simchi-Levi [16] identified several weaknesses in this traditional game and consequently developed the computerized Beer Game.

Based on the computerized Beer Game, we provide various quality levels of marketing and selling information to subjects. Using the given information, subjects are asked to make inventory control decisions. In the experiment, we have adopted a set of validated information quality measurements from [30]. Also we have also considered 10 external factors that may influence the experiment such as task complexity, decision time, expertise, decision strategy, interaction, information overload, information presentation, decision aids, decision model and environment. All the external factors are kept under control. That means keeping the same status of all the external factors for every experimental treatment. By conducting a pilot study, we can find out if the experiment task is clear and easy to operate to the participants.

For this experiment we use a four-component beer supply chain: manufacturer, distributor, retailer and customer. One episode of the experiment includes 10 weeks.
In each week, the order of events is as follows: (1) Manufacturer fills the distributor's demands of last week. (2) Distributor fills the retailer's demands of last week and places an order with manufacturer for next week. (3) Retailer fills the customer's demands of this week and places an order with distributor for next week. If the demands are not catered for, the unsatisfied demands are recorded as back orders. The manufacturer is guaranteed to provide enough products for the distributor. Therefore there is no back order with the manufacturer. At the beginning of the game, there is no back order in each component and the demands of last week are perfectly satisfied.

A software-based system is developed to deliver the experimental scenario (figure 2). Subjects play the role of distributors who place orders to manufacturers and meet demands of retailers. The other three roles are taken over by the computer. To simplify the design of JIT inventory control, no lead time is set between distributors and manufacturers. This is to encourage subjects not to stock any product, accordingly, to achieve zero inventories. In each week, we provide the marketing information and selling history to subjects. According to the given information, subjects are able to make more reliable and reasonable inventory decisions. In one episode, subjects are asked to place 10 orders to manufacturers. Orders which conform to the best decision are recorded as the correct inventory decision. Since the goal of this experiment is to minimize the inventory to zero, the best decision is determined by the order which equals the retailer's need plus existing back orders.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Experiment of Beer Game inventory control</figcaption></figure>

From the experiment, we can then collect the ratio date from the beer game. Therefore, parametric statistical analysis such as ANOVA is used to analyze the data. Afterwards, the data analysis can validate and evaluate our proposed artifact. A list of detailed design is shown in Table 2.
<table><caption>**Table 2.** Demonstration in information quality research</caption><thead><tr><th>Artifact</th><th>Experiment</th><th>Data Analysis</th></tr></thead><tbody><tr><td>Decision support system with high quality information</td><td>Beer Game experiment from MIT</td><td>Parametric statistical analysis</td></tr><tr><td><strong>Practical problem:</strong><br>Different case studies have demonstrated that decision support system has information quality problem.</td><td><strong>Validation of measuring instrument:</strong><br>Adopted validated measuring instrument from [30]</td><td><strong>Data collection:</strong><br>Ratio Data</td></tr><tr><td><strong>Theory:</strong><br>DeLone and McLean IS success theory</td><td><strong>Control of external factors:</strong><br>A total of 10 external factors such as Task Complexity And decision time are controlled</td><td><strong>Data analysis:</strong><br>AVONA and descriptive analysis</td></tr><tr><td><strong>Literature review:</strong><br>Papers in information quality, information management and information system.</td><td><strong>Concise and easy-to-operate experiment:</strong><br>We adopted a computerized beer game and carried out pilot study</td><td></td></tr></tbody></table>

## 6 Conclusion

In this paper, we have proposed a framework that can guide the experimental research in design science. Based on the methodological issues in experimental research pointed by [14], we have proposed an experimental framework, which consists of three components: artifact, experiment, and data analysis. In each of the component, we have taken the methodological issues into consideration and proposed a set of detailed guidelines to design the experiment. In order to demonstrate and validate our proposed framework, we have conducted an empirical study in information quality research by using the framework. Under this framework, we can primitively avoid the possible methodological issues for experimental research in design science. The validation has not only shown that it is feasible to apply our framework in empirical information quality research, but also indicated that the framework can enhance more rigorous and quantitative design science oriented experimental research.

## References

1. Benbasat I, Zmud R (1999) Empirical Research in Information Systems- The Practice of Rel-evance, MIS Quarterly, 23(1), pp. 3-36

2. Campbell, D.T. and Stanley, J. (1963), Experimental and quasi-experimental designs for research. Houghton-Mifflin, Boston, Massachusetts, USA.

3. DeLone, W.H. and McLean, E.R. (1992), Information system success: the quest for dependent variables, Information System Research, 3(1), pp. 60-96.

4. Dickson, G., Senn, J. and Chervany, N. (1977), Research in management information systems: the Minnesota experiments, Management Science, 23(9), pp. 913-923.
5. English, L.P. (1999), Improving data warehouse and business information quality: methods for reducing costs and increasing profits, Publisher: Wiley, New York.

6. Field, A, and Hole, G. (2003), How to design and report experiments, Sage publication.
London, UK.

7. Fisher, C.W., Chengalur-Smith I. and Ballou, D.P. (2003), The impact of experience and time on the use of data quality information in decision making, Information Systems Research, 14(2), pp.170-188.

8. Gummesson E (1988) Qualitative Methods in Management Research. Chart-well-Bratt.
Lund, Sweden

9. Ge, M., & Helfert, M. (2007). A Review of Information Quality Research - Develop a Research Agenda. International Conference on Information Quality, MIT, Cambridge, Massachusetts, USA

10. Hevner A R, March S T, Park J, Ram S (2004) Design Science in Information Systems Research, *MIS Quarterly*, 28(1), pp. 75-105

11. Hevner A R (2007) A Three Cycle View of Design Science, *Scandinavian Journal of Information Systems*, 19(2), pp. 87-92

12. Huang, K.T., Lee, Y., Wang, R.Y. (1999), Quality information and knowledge management, Publisher: Prentice Hall. New Jersey, USA.

13. Helfert M. and O'Brien T. (2011) Sustaining Data Quality - Creating and Sustaining Data Quality within Diverse Enterprise Resource Planning and Information Systems. IFIP WG 8.6 Hamburg 2011 - Governance and Sustainability in Information Systems September 22-24, 2011: 317-324

14. Jarvenpaa, S.L., Dickson, G.W. and DeSanctis, G. (1985), Methodological issues in experimental IS research: experiences and recommendations, MIS Quarterly, 9(2), pp.141-156.

15. Klein H K, Myers M D (1999) A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems, MIS Quarterly, 23(1), pp. 67-93

16. Kaminsky, P. and Simchi-Levi, D. (1998), A new computerized beer game: a tool for teaching the value of integrated supply chain management. In Supply Chain and Technology Management, Hau Lee and Shu Ming Ng, eds., The Production and Operations Management Society, Miami, Florida.

17. Kerlinger, F.N. and Lee, H.B. (2000), Foundation of Behavioral Research, Harcourt College Publisher, Fort Worth, Texas, USA.

18. Leonard-Barton D (1995) Well Springs of Knowledge: Building and Sustaining the Sources of Innovation, Harvard Business School Press, Boston, MA.

19. Nunamaker J F, Dennis A R, Valacich J S, Vogel D, George J F (1991) Electronic Meeting Systems to Support Group Work, Communications of the ACM, 34 (7), pp 40–61

20. Pirani, C. (2004), How safe are you hospital? The Weekend Australia.

21. Pipino, L., Lee, Y., and Wang, R.Y. (2002), Data quality assessment, Communications of the ACM, 45(4), pp. 211-218.

22. Sein M, Henfridsson O, Purao S, Rossi M, Lindgren R (2011) Action Design Research. MIS Quartely, 35(1), pp. 37-56.

23. Sun Y, Kantor P B (2006) Cross-Evaluation: A New Model for Information System Evalua-tion, Journal of the American Society for Information Science and Technology, 57(5), pp. 614-62.

24. Taylor, R.N. and Benbasat, I, (1980), Cognitive styles research and managerial information use: problems and prospects, Joint National Meeting of the Operations Research Society of America and The Institute of Management Sciences, Colorado Springs, Colorado, USA.

25. Van Aken, J E (2005) Management Research as a Design Science: Articulating the Research Products of Mode 2 Knowledge Production in Management, British Journal of Management, 16(1), pp. 19-36
26. Vaishnavi V. K, Kuechler W (2004) Design Science Research in Information Systems, January 20, 2004, last updated September 30, 2011. URL: http://www.desrist.org/desrist

27. Van de Ven A H (2007) Engaged Scholarship: A Guide for Organizational and Social Research, Oxford Univ. Press, Oxford, England

28. Venable J R (2006) A Framework for Design Science Research Activities, In Proceedings of the Information Resource Management Association Conference, , 21-24 May, Washington, DC, USA

29. Venable J R, Pries-Heje J, Baskerville R (2012) A Comprehensive Framework for Evaluation in Design Science Research, In Design Science Research in Information Systems. Advances in Theory and Practice, Lecture Notes in Computer Science (Peffers K, Rothenberger M Marcus, Kuechler B, Eds.) Volume 7286, Springer, Berlin, Heidelberg, pp. 423-438

30. Wang, R.Y. and Strong, D.M. (1996), Beyond accuracy: what data quality means to data consumers. Journal of Management Information Systems, 12(4), pp. 5-34.

31. Wand, Y. and Wang, R.Y. (1996), Anchoring data quality dimensions in ontological foundations, Communications of the ACM, 39(11), pp. 86-95.

32. Wang, R.Y. (1998), A product perspective on total data quality management, Communications of the ACM, 41(2), pp. 58-65.

33. Xie, S. and Helfert, M. (2012) ‘An assessment technique for information quality support in emergency response’, Int. J. Business Continuity and Risk Management, Vol. 3, No. 4, pp.373–393.

34. Xie, S., Helfert M. and Ostrowski L. (2012) Human Involvement in Designing an Information Quality Assessment Technique – Demonstrated in a Healthcare Setting – Accepted for Human-Computer Interaction & Knowledge Discovery at Active Media Technology 2012, Dec. 4-7, Macau SAR, China.