Suzuki, K. & Nemoto, J.(2015.8.18).Is Design-based Research desirable and feasible methodology for educational
media and technology? (Refreed concurrent session paper) presented at ICoME2015 (International Conference
for Media in Education), Northeast Normal University, China, August 17-20, 2015

Is Design-based Research desirable and feasible methodology foreducational media and technology?
Katsuaki Suzuki
ksuzuki@kumamoto-u.ac.jp
Professor
Kumamoto University
Kumamoto, Japan

Junko Nemoto
nemoto.junko.nu@ehime-u.ac.jp
Associate Professor
Ehime University
Matsuyama, Japan

ABSTRACT
Design-based research (DBR) has been accepted as preferable research methodology fromits birth in early 2000’s. DBR has such advantages as avoiding novelty effect, ensuring
practical effectiveness to be created in a long-term educational practices, and contributing to
the advancement of theory base (as design principles), when compared to short-term
experimental designs that produce only statistical significance. On the other hand, DBR
creates challenges to be adopted as the research methodology, especially toward an advanced
degree, due to its long duration, a need for a research team with committed practitioners,
among others. This paper explored the way in which DBR can be adopted in more cases aspreferred methodology in educational media and technology studies.
Keywords: Design-based Research, Research Methodology, Desirability, Feasibility,
Educational Media and Technology

INTRODUCTION
Design-based research (DBR) has been accepted as preferable research methodology from its
birth in early 1990’s. Special issues in major journals, including Educational Researcher，Journal of
the Learning Sciences，Educational Psychologist, and Educational Technology, were issued in
2003-2005. Handbook of Design Research Methods in Education was published in 2008 (Kelly, Lesh,& Baek, 2008), introducing the latest thinking and current examples of DBR in education, whichbrought innovations into real-world practices (as opposed to constrained laboratory contexts) andexamining the impact of those designs on the learning process. Designed prototype applications (e.g.,
instructional methods, technology or materials) and the research findings are then cycled back into thenext iteration of the design innovation in order to build evidence of the particular theories being
researched, and positively impact practice and the diffusion of the innovation.
Some recent works using DBR include Feng, Lu & Yao (2015), where two cycles of DBR wereconducted that resulted in a new professional-task based master-level curriculum for distance educationin China, as well as a revised curriculum development model, which is capable of dealing multiple
roles in the industry with some degrees of prior knowledge and experiences of the target learners. Lay,et al. (2014) designed four solutions to scale up informal learning and tested them out in two different
workplaces (healthcare and building-construction sectors), refining the solutions as well as creating an
integrative model of scaling informal learning. In November 2013, Netherlands Institute forCurriculum Development has compiled an extensive digital book on “educational design research,”
freely available on their Website (http://international.slo.nl/publications/edr/). After an introductory
section, the book has a total of 51 DBR studies, ranging from (pre-)preliminary education, secondary
education, teacher education, and higher education and workplace learning.
Anderson & Shattuck (2012) reported that increasing numbers of studies have been conductedduring the last decade, as shown in Figure 1. According to them, over 2000 papers had been publishedin 10 years, with increasing numbers in recent years, shifting more about educational practices fromtheories. Majority of papers (75%) were published in US, but papers from other countries wereincreasing, with multiple iterations of practice and revisions reported in many papers (65%). In a

Suzuki, K. & Nemoto, J.(2015.8.18).Is Design-based Research desirable and feasible methodology for educational
media and technology? (Refreed concurrent session paper) presented at ICoME2015 (International Conference
for Media in Education), Northeast Normal University, China, August 17-20, 2015

systematic literature review on design-based research from 2004 to 2013, Zheng (2015) reviewed 162
selected studies in the database of 219 social sciences citation index (SSCI). She found more than halfof the reviewed studies were about technological intervention (86%), lasted one year or less (69%),
reported some revision of intervention (74%), but reported only one iteration (50%). The last point, thenumber of iteration, was concluded differently in the two reviews, perhaps due to different selection
procedure of target papers.

Figure 1: Number of articles using or discussing design-based
research (From: Figure 2 of Anderson & Shattuck, 2012)
DBR has such advantages as avoiding novelty effect, ensuring practical effectiveness to becreated in a long-term educational practices, and contributing to the advancement of theory base (asdesign principles), when compared with short-term experimental studies that produce statisticalsignificance, but with little impact in educational practice. On the other hand, DBR creates challenges
to be adopted as the research methodology, especially toward an advanced degree, due to its long
duration, a need for a research team with committed practitioners, among others. This paper will
explore the way in which DBR can be adopted in more cases as preferred methodology in educational
media and technology studies.
.

WHAT IS DESIGN-BASED RESEARCH?
Table 1 summarizes a comparison between traditional experimental design with DBR, as known
as “design experiment,” made by Allen Collins in 1992, one of the pioneers in the movement of DBR.
Whereas a traditional experiment is conducted in a laboratory situation, with a control group, to test
pre-determined hypotheses, DBR is taking place in a naturalistic environment where multiple factors
interact each other. A simultaneous comparison group is not used in DBR; the comparison usually
made with practices in previous years, to check if any improvement has been accomplished bychanging the way the classroom was taught before.

Table 1: Collins’ Comparison between traditional experimental design with design-based research
Traditional experimental design
Design-based research(aka. Design experiment)
Place
Laboratory
Complex situation (e.g., classrooms)
FactorChange one factor
Deal with multiple factors
Experimental Researcher control with purpose
A specific situation with no control
situation
Procedure
Fixed
Flexible with alterations
Relation to
DisconnectedInteractive
society
Research
Validate hypotheses
Develop framework
style
StanceAs an experimenter
As a co-participant in design and analysis
Note: Based on Masukawa, 2011, translated by the first authorSuzuki, K. & Nemoto, J.(2015.8.18).Is Design-based Research desirable and feasible methodology for educational
media and technology? (Refreed concurrent session paper) presented at ICoME2015 (International Conference
for Media in Education), Northeast Normal University, China, August 17-20, 2015

Table 2 summarizes characteristics of DBR to be pragmatic, grounded, interactive/iterative,
integrative, and contextual. It is important to notice that DBR is pragmatic in the sense of refining
both theory and practice. DBR not only addresses complex problems in real contexts incollaboration with practitioners, but it also integrates known and hypothetical design-principles
with technological affordances to render plausible solutions to these complex problems, andconducting rigorous and reflective inquiry to test and refine innovative learning environments aswell as to define new design-principles. That is to say, DBR tries to produce not only better
practices using theory, but also to improve theory through practice.

Table 2: Characteristics of design-based research
Explanations* Design-based research refines both theory and practice.
* The value of theory is appraised by the extent to whichprinciples inform and improve practice.
Grounded* Design is theory-driven and grounded in relevant research,
theory and practice.
* Design is conducted in real-world settings and the design
process is embedded in, and studied through, design-based
research.
Interactive,* Designers are involved in the design processes and work
iterative,
together with and flexible participants.
* Processes are iterative cycle of analysis, design,
implementation, and redesign.
* Initial plan is usually insufficiently detailed so that designers
can make deliberate changes when necessary.
Integrative* Mixed research methods are used to maximize the credibilityof ongoing research.
* Methods vary during different phases as new needs andissues emerge and the focus of the research evolves.
* Rigor is purposefully maintained and discipline applied
appropriate to the development phase.
Contextual* The research process, research findings, and changes fromthe initial plan are documented.
* Research results are connected with the design process andthe setting.
* The content and depth of generated design principles varies.
* Guidance for applying generated principles is needed
Note: From Wang & Hannafin (2005), Table 2, p. 8
Characteristics
Pragmatic

Figure 2 depicts the process of DBR, as compared with predictive (traditional experimental)
research. While traditional predictive research produces refined theory, as a result of hypothesis-driven
experiments with limited impact on educational practices, DBR starts with challenges in educational
practices. Solutions are then sought in collaboration with practitioners using existing theory andhypothesized design-principles. Multiple iterations of cycles of testing are required, as it has no control
group to compare, before a refined set of design-principles can be proposed as a part of
expanded/refined theory. However, as described in Introduction, mixed findings were observed in two
reviews as to how many iterations have been actually reported in previous studies.

Suzuki, K. & Nemoto, J.(2015.8.18).Is Design-based Research desirable and feasible methodology for educational
media and technology? (Refreed concurrent session paper) presented at ICoME2015 (International Conference
for Media in Education), Northeast Normal University, China, August 17-20, 2015

Figure 2: Predictive and Design-based Approaches in Educational Technology Research(Reeves, 2006)

WHAT IS NOT DESIGN-BASED RESEARCH?
DBR is different from traditional experimental approach as we discussed in previous section.
Table 3 summarizes how DBR can be compared with other similar research approaches that have been
adopted in the field of educational media and technology, namely Grounded Theory Approach (GTA),
Action Research, and Formative Evaluation. GTA has been widely employed as a substitute fortraditional experimental approach to produce qualitative explanations of educational practices, but it is
not design-oriented. It focuses on understanding what is happening; not on how to improve the current
practice. Action research, on the other hand, focuses on how to improve own practice, thus considered
to be design-oriented, but not putting much emphasis on theory building. Formative evaluation is also
design-oriented, but not interested in theory building, whereas DBR aims for both improving practiceand refining theory. Although there are some differences in emphasis or aims in each approach, they
can represent variety of alternatives to the traditional experimental studies.

Figure 3 compares DBR with other research approaches by placing it in a spectrum ofreform/individuality description vs. causal/rule establishment, using the researcher’s stance as first,
second and third person. DBR is place in the middle, representing the second person stance, because
DBR is usually carried out in conjunction with practitioners (thus, you and I relationship), whereas
traditional experiment requires objectivity of the researchers (thus, third party view). Action research is
conducted to improve own practice (thus, always subjective using the first person view). The spectrum
also consider whether or not any development occur in the research (vertical comparison). DBR isalways involves creating a new methodology in educational practice, whereas interview is a second
person methodology with no intervention (i.e., just collecting second person data).

Suzuki, K. & Nemoto, J.(2015.8.18).Is Design-based Research desirable and feasible methodology for educational
media and technology? (Refreed concurrent session paper) presented at ICoME2015 (International Conference
for Media in Education), Northeast Normal University, China, August 17-20, 2015

Approaches
PurposeTargetDomainResearch
Process

Characteristics

Table 3: Comparison of Research Approaches
Grounded Theory
Design-based
Action Research
Research
Approach (GTA）
Improvement of
Explanation of
Improvement ofPractice andPhenomena andSpecific Situation＜design-oriented＞
Construction ofConstruction ofTheoryTheory(Not General,＜design-oriented＞
but Domain specific=
Grounded Theory)＜descriptive＞
Educational
Humanities andHumanities andPractice
Social Science inSocial Science ingeneralgeneralTheory-driven
Collection andStatus-quo
Practice→
Analysis of Interview Analysis→
Assessment ofdata
Revision
effect → Revision(Coding)→Concept
suggested basedof Practice
Extraction
on theory→Assessment of→Theoretical→Implementation
effect →Proposal of Sampling and→Assessment
Design Principles(→Consideration
Saturation→Improvement ofof Generalization
Theoryand Limits）･Making both
･Process already･Many studies
practiceestablishedconducted in
improvement andnursing field, with･Many reference
paper submission
ample directions/books available
possiblesuggestions, easy
･Hypothesis finding
to imitate･More and more
･Not apt torecognizedgeneralization

Formative
Evaluation
Improvement ofPractice(Class
Instruction orInstructional
Material)＜design-oriented＞

Educational
Practice
Development
based on theory ormodel→One-on -oneevaluation→Small group
evaluation→Tryout･Pass/fail on
posttest･Necessary step
beforeimplementation,
but ignored orskipped in manyoccasions

Note: From Suzuki & Nemoto (2013), Table 1, translated by the first author

Figure 3: Representative Research Methodologies placed in Methodology Spectrum(From: Kogo, 2013, Figure 11-1, p. 210, translated by the first author)
Note: *Original word is “Design Experiment”

Suzuki, K. & Nemoto, J.(2015.8.18).Is Design-based Research desirable and feasible methodology for educational
media and technology? (Refreed concurrent session paper) presented at ICoME2015 (International Conference
for Media in Education), Northeast Normal University, China, August 17-20, 2015

HOW SHOULD DESIGN-BASED RESEARCH BE CONDUCTED?
Ma & Harmon (2009), when reporting a DBR case study, proposes a set of suggestions as to how
to conduct a DBR study. They subdivided DBR processes proposed by Reeves (2000) shown in Figure
2 (bottom) to make points of emphasis. In Step 1, they consider topic selection is important to avoid
using DBR to a problem limited to a local situation, as DBR takes much effort in aiming at theory
building. In Step 2 of developing solutions, they recommend selecting focused research questions that
fit a particular iteration to avoid overdoing beyond getting answers to the questions. They also suggest
to identify a complex problem with no clear solution available, that calls for collaboration withpractitioners. For Step 3 of evaluation, they caution that deciding evaluation methods itself require
some time for try and error. For Step 4, they recommend to reflect and document the methodology usedin research, in addition to proposing design-principles, as methodology for DBR is still not mature.
Easterday, Lewis, & Gerber (2014) identified that one of the problems arising from theill-definition of DBR was the uncertainty of DBR process, and proposed a design process model,
consisting of 6 iterative phases: focus, understand, define, conceive, build and test. They suggest thatthe design phases are not carried out in a linear sequence but rather iteratively, and rapid iteration is atenet of modern human-centered design. To protect against the risks of designing interventions that areover-budget and behind schedule by quickly testing the designer’s assumptions, they recommend forquickly building low fidelity prototypes, testing them, and re-designing so that gradually evolving theintervention over time, rather than design an entire intervention and discover only at the end that it does
not work.
Herrington, et al. (2007) suggested guidelines for preparing a dissertation proposal for doctoral
students. Using the four phases of DBR by Reeves (2006) illustrated in Figure 2 (bottom), they mappedeach phase against the typical requirements of a research proposal as shown in Table 4. They
recommend that educational researchers who supervise doctoral work are engaged in design-based
research themselves, so that their students begin to play active roles in that research agenda from thefirst day. If the supervisor already be involved in long-term, meaningful coactive engagement withpractitioners, their students will be able to enter as apprentices to this ongoing collaboration, just as
so-called “natural” or “pure” science (such as physics and chemistry) students are not expected to start
their research projects on their own.
Table 4: Phases of design-based research mapped against typical elements of a research proposal
Phase of DBR (Reeves, 2006) Element: The topics/elements that need to be described
PHASE 1: Analysis of practical Statement of problem
problems by researchers andConsultation with researchers and practitioners
practitioners in collaboration
Research questions
Literature review
PHASE 2: Development ofTheoretical framework
solutions informed by existing
Development of draft principles to guide the design of thedesign principles andintervention
technological innovations
Description of proposed intervention
PHASE 3: Iterative cycles of
Implementation of intervention (First iteration)
testing and refinement ofParticipants
solutions in practiceData collection
Data analysis
Implementation of intervention(Second and further iterations)
Participants, Data collection, Data analysis
PHASE 4: Reflection to produce
Design principles“design principles” and enhance
Designed artefact(s)solution implementation
Professional development Methodology
Note: From Table 1 of Herrington, et al. (2007), omitting “Position” column

Suzuki, K. & Nemoto, J.(2015.8.18).Is Design-based Research desirable and feasible methodology for educational
media and technology? (Refreed concurrent session paper) presented at ICoME2015 (International Conference
for Media in Education), Northeast Normal University, China, August 17-20, 2015

CONCLUSIONS AND SUGGESTIONSThis paper tried to collect useful guidelines of DBR so that DBR can be adopted in more cases aspreferred methodology in educational media and technology studies. Although DBR has become awidely accepted research methodology, it requires longer duration and deeper collaboration withpractitioners, which may be a barrier especially for doctoral students. Many reported studies deals with
one iteration only, but with suggested revisions of intervention, which may be regarded as a sign ofchallenge for conducting a longer study, with multiple iterations. However, at the same time, the factthat such DBR studies with only one iteration have been accepted as referred journal paper can be agood news for our students. As we experience ourselves in conducting DBR studies (Nemoto, Shibata& Suzuki, 2011), we realize that DBR requires much time to examine the effects of revisions fromprevious iterations. It requires more time and effort to propose design guidelines in a reliable andgeneralizable manner. We may need to speed up the study by adopting a notion of rapid prototyping toshorten the required duration for DBR studies, as suggested by Easterday, Lewis, & Gerber (2014).
One suggestion that can be drawn from this review is to advise our master’s student engage in aformative evaluation study (see Table 3) by incorporating a model or theory to try to invent a solution
to their own challenges in their setting. It is contrary to the advice that Herrington et al (2007) has
suggested to let them engage in an on-going DBR of their supervisors. Yet, it may be regarded, when
they move up to doctoral level, to be the first iteration of DBR. It is similar to action research, because
they are engaging in from the first person viewpoint (see Figure 3), but by incorporating an existingmodel or theory to solve their own problems, it may be considered as the first step in DBR.
It has been the emphasis in our graduate program to try to “make somebody happy” byconducting a research study; to make the research practical and useful to an educational practice, rather
than learning a “proper way” of conducting a research. We do not want take away an opportunity tothink deeply what study our students should be engaged in as their own research. We may use our own
DBR and ask our students to play an active and increasingly important role, as a part of theircoursework to expose them to DBR to understand what DBR is all about. Then, they become morecapable in conducting their own study, on their own practices.
When our master’s student goes up to doctoral program, they can sum up their study for master’s
degree, as the first iteration of DBR study to be submitted to a journal. They then can continue their
DBR for the second phase and more, in a remaining period in their doctoral program. This may be oneof the plausive ways to see more DBR studies conducted in our field by graduate students.
DBR has been thought of a powerful way to bridge the gap between academics and professionals.
Wilson’s (2014) criteria, such as relevance (does it relate to compelling problems currently faced byeducational practitioners?), usability (does it lead to actions?), and impact (does it make a realdifference in improving valued educational outcomes and practices?) to be placed in addition to
traditional expectations of rigor and internal validity of educational research, we shall lead our graduate
students to challenge to improve the quality of educational practices in their field, by engaging in a
DBR study.

REFERENCES
Anderson, T., & Shattuck, J. (2012). Design-based research: A decade of progress in educational
research? Educational Researcher, 41(1) , 16-25.
Easterday, M. W., Lewis, D. R., & Gerber, E. M. (2014). Design-Based Research process: Problems,
phases, and applications. International Conference of the Learning Science (ISLS) 2014
Proceedings (Vol. 1), 317-324.
Feng, X., Lu, G., & Yao, Z. (2015). Professional task-based curriculum development for distanceeducation practitioners at Master’s level: A design-based research. International Review of
Research in Open and Distributed Learning, 16 (2), 288-310.
Herrington, J., McKenney, S., Reeves, T. C., & Oliver, R. (2007). Design-based research and doctoral
students: Guidelines for preparing a dissertation proposal. In C. Montgomerie & J. Seale (Eds.),
Proceedings of World Conference on Educational Multimedia, Hypermedia andTelecommunications 2007, 4089-4097.

Suzuki, K. & Nemoto, J.(2015.8.18).Is Design-based Research desirable and feasible methodology for educational
media and technology? (Refreed concurrent session paper) presented at ICoME2015 (International Conference
for Media in Education), Northeast Normal University, China, August 17-20, 2015

Kelly, A. E., Lesh, R. A., & Baek, J. Y (Eds.) (2008). Handbook of design research methods ineducation: Innovations in science, technology, engineering, and mathematics learning andteaching. Routledge.
Kogo, C. (2013). Research methodologies of educational technology (Chapter 11). In Japan Society for
Educational Technology (Supervision), Methodologies of research in educational technology(Educational Technology Series, Vol. III). Minervashobo, 200-206 [In Japanese].
Lay, T., Cook, J., Dennerlein, S., et al. (2014). Scaling informal learning at the workplace: A modeland four designs from a large-scale design-based research effort. British Journal of Educational
Technology, 45 (6), 1036–1048, DOI: 10.1111/bjet.12197.
Ma, Y., & Harmon, S. W. (2009). A case study of design-based research for creating a vision prototype
of a technology based innovative learning environment. Journal of Interactive Learning Research,
20(1) , 75-93.
Masukawa, H. (2011). Learning sciences. In The Institute of Electronics, Information andCommunication Engineers (IEICE) (Eds.), Knowledge Base, S3-11, 2-4, 20-32. [In Japanese]
Available online: http://www.ieice-hbkb.org/files/S3/S3gun_11hen_02.pdf.
Nemoto, J., Shibata, Y., & Suzuki, K. (2011). Design-based research to improve learning design andlearning contents. Japan Journal of Educational Technology, 35(3), 259-268 [In Japanese].
Reeves, T. C. (2006). Design research from a technology perspective. In J. van den Akker, K.
Gravemeijer, S. McKenney & N. Nieveen (Eds.), Educational design research (pp. 52-66).
London: Routledge.
Suzuki, K., & Nemoto, J. (2013). Toward managing both educational improvement and researchaccomplishment: Let’s write a design-based research (Review). Competency Development of
Healthcare Professionals, 2(1), 45-53 [In Japanese].
Wang, F., & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning
environments. Educational Technology Research and Development, 53(4), 5-23.
Wilson, B. G. (April, 2014). Knowledge creation in design-based research projects: Complementary
efforts of academics and practitioners. A paper resented at the meeting of the American
Educational Research Association, Philadelphia, Available online: http://www.academia.edu/
6311234/
Zheng, L. (2015). A systematic literature review of design-based research from 2004 to 2013. Journalof Computers in Education, First online: 24 June 2015, DOI 10.1007/s40692-015-0036-z.