International Journal for Educational Media and Technology
2016, Vol.10, No. 1, pp. 45-52

Is Design-based Research desirable and feasible methodology foreducational media and technology?
Katsuaki Suzuki
Kumamoto University, Japanksuzuki@kumamoto-u.ac.jp
Junko Nemoto
Ehime University, Japan
nemoto.junko.nu@ehime-u.ac.jp
Design-based research (DBR) has been accepted as preferable research methodology fromits birth in early 2000’s. DBR has such advantages as avoiding novelty effect, ensuring
practical effectiveness to be created in a long-term educational practices, and contributing
to the advancement of theory base (as design principles), when compared to short-term
experimental designs that produce only statistical significance. On the other hand, DBR
creates challenges to be adopted as the research methodology, especially toward anadvanced degree, due to its long duration, a need for a research team with committed
practitioners, among others. This paper explored, by reviewing trends of current literature,
the way in which DBR can be adopted in more cases as preferred methodology in
educational media and technology studies. As the result, a set of guidelines for directing
graduate studies are proposed.
Keywords: Design-based Research, Research Methodology, Desirability, Feasibility, Educational
Media and Technology

Introduction
Design-based research (DBR) has been accepted as preferable research methodology from its birth inearly 1990’s. Special issues in major journals, including Educational Researcher，Journal of the
Learning Sciences，Educational Psychologist, and Educational Technology, were issued in 2003-2005.
Handbook of Design Research Methods in Education was published in 2008 (Kelly, Lesh, & Baek,
2008), introducing the latest thinking and current examples of DBR in education, which brought“innovations into real-world practices (as opposed to constrained laboratory contexts) and examining
the impact of those designs on the learning process. Designed prototype applications (e.g., instructional
methods, technology or materials) and the research findings are then cycled back into the next iterationof the design innovation in order to build evidence of the particular theories being researched, andpositively impact practice and the diffusion of the innovation (Kelly, Lesh, & Baek, 2008, p. i).
Some recent works using DBR include Feng, Lu and Yao (2015), where two cycles of DBR wereconducted that resulted in a new professional-task based master-level curriculum for distance educationin China, as well as a revised curriculum development model, which is capable of dealing multiple
roles in the industry with some degrees of prior knowledge and experiences of the target learners. Lay,et al. (2014) designed four solutions to scale up informal learning and tested them out in two different
workplaces (healthcare and building-construction sectors), refining the solutions as well as creating an
integrative model of scaling informal learning. In November 2013, Netherlands Institute forCurriculum Development has compiled an extensive digital book on “educational design research,”
freely available on their Website (http://international.slo.nl/publications/edr/). After an introductory
section, the book has a total of 51 DBR studies, ranging from (pre-)preliminary education, secondary
education, teacher education, and higher education and workplace learning.
Anderson and Shattuck (2012) reported that increasing numbers of studies have been conducted during
the last decade, as shown in Figure 1. According to them, over 2000 papers had been published in 10
years, with increasing numbers in recent years, shifting more about educational practices from theories.
Majority of papers (75%) were published in US, but papers from other countries were increasing, withmultiple iterations of practice and revisions reported in many papers (65%). In a systematic literature
IJEMT, Vol.10, No. 1, 2016, pp. 45-52 ISSN 1882–2290

45

International Journal for Educational Media and Technology
2016, Vol.10, No. 1, pp. 45-52

review on design-based research from 2004 to 2013, Zheng (2013) reviewed 162 selected studies in the
database of 219 social sciences citation index (SSCI). She found more than half of the reviewed studies
were about technological intervention (86%), lasted one year or less (69%), reported some revision ofintervention (74%), but reported only one iteration (50%). The last point, the number of iteration, wasconcluded differently in the two reviews, perhaps due to different selection procedure of target papers.

Figure 1. Number of articles using or discussing design-based
research (From: Figure 2 of Anderson & Shattuck, 2012)
DBR has such advantages as avoiding novelty effect, ensuring practical effectiveness to be created in along-term educational practices, and contributing to the advancement of theory base (as designprinciples), when compared with short-term experimental studies that produce statistical significance,
but with little impact in educational practice. On the other hand, DBR creates challenges to be adopted
as the research methodology, especially toward an advanced degree, due to its long duration, a need fora research team with committed practitioners, among others.
This paper explored the way in which DBR can be adopted in more cases as preferred methodology
in educational media and technology studies. Therefore, our research question was to examine thetrends and opinions on DBR in the literature so that a set of guidelines would be proposed to make
DBR a feasible alternative in directing graduate studies. We first examined the trends of DBR bysearching in ERIC (http://eric.ed.gov), CiNii (http://ci.nii.ac.jp/), and Google scholar(https://scholar.google.co.jp/), using "Design Based Research" as the keyword. We then explored
related articles from the references of the identified studies. To summarize the results of our
exploration, DBR and related other methods are compared, which will be followed by tips that were
found from the literature in conducting DBR. Finally, our ideas of directing graduate studies will be
proposed as a set of guidelines.
.

What is Design-Based Research?
Table 1 summarizes a comparison between traditional experimental design with DBR, as known as“design experiment,” made by Allen Collins in 1992, one of the pioneers in the movement of DBR.
Whereas a traditional experiment is conducted in a laboratory situation, with a control group, to test
pre-determined hypotheses, DBR is taking place in a naturalistic environment where multiple factors
interact each other. A simultaneous comparison group is not used in DBR; the comparison usually
made with practices in previous years, to check if any improvement has been accomplished bychanging the way the classroom was taught before.

Table 1.
Collins’ Comparison between traditional experimental design with design-based research
Traditional experimental design
Design-based research(aka. Design experiment)
Place
Laboratory
Complex situation (e.g., classrooms)
FactorChange one factor
Deal with multiple factors
Experimental Researcher control with purpose
A specific situation with no control
situation
IJEMT, Vol.10, No. 1, 2016, pp. 45-52 ISSN 1882–2290

46

International Journal for Educational Media and Technology
2016, Vol.10, No. 1, pp. 45-52

Procedure
Fixed
Flexible with alterations
Relation to
DisconnectedInteractive
society
Research
Validate hypotheses
Develop framework
style
StanceAs an experimenter
As a co-participant in design and analysis
Note: Based on Masukawa, 2011, translated by the first author
Table 2 summarizes characteristics of DBR to be pragmatic, grounded, interactive/iterative,
integrative, and contextual. It is important to notice that DBR is pragmatic in the sense of refining
both theory and practice. DBR not only addresses complex problems in real contexts incollaboration with practitioners, but it also integrates known and hypothetical design-principles
with technological affordances to render plausible solutions to these complex problems, andconducting rigorous and reflective inquiry to test and refine innovative learning environments aswell as to define new design-principles. That is to say, DBR tries to produce not only better
practices using theory, but also to improve theory through practice.
Table 2.
Characteristics of design-based research
Characteristics
Explanations
Pragmatic* Design-based research refines both theory and practice.
* The value of theory is appraised by the extent to which principles
inform and improve practice.
Grounded* Design is theory-driven and grounded in relevant research, theory
and practice.
* Design is conducted in real-world settings and the design process isembedded in, and studied through, design-based research.
Interactive, iterative * Designers are involved in the design processes and work together
with and flexible participants.
* Processes are iterative cycle of analysis, design, implementation, andredesign.
* Initial plan is usually insufficiently detailed so that designers canmake deliberate changes when necessary.
Integrative* Mixed research methods are used to maximize the credibility ofongoing research.
* Methods vary during different phases as new needs and issues
emerge and the focus of the research evolves.
* Rigor is purposefully maintained and discipline applied appropriate
to the development phase.
Contextual* The research process, research findings, and changes from the initialplan are documented.
* Research results are connected with the design process and thesetting.
* The content and depth of generated design principles varies.
* Guidance for applying generated principles is needed
Note: From Wang & Hannafin (2005), Table 2, p. 8
Figure 2 depicts the process of DBR, as compared with predictive (traditional experimental) research.
While traditional predictive research produces refined theory, as a result of hypothesis-driven
experiments with limited impact on educational practices, DBR starts with challenges in educational
practices. Solutions are then sought in collaboration with practitioners using existing theory andhypothesized design-principles. Multiple iterations of cycles of testing are required, as it has no control
group to compare, before a refined set of design-principles can be proposed as a part of
expanded/refined theory. However, as described in Introduction, mixed findings were observed in two
reviews as to how many iterations have been actually reported in previous studies.

IJEMT, Vol.10, No. 1, 2016, pp. 45-52 ISSN 1882–2290

47

International Journal for Educational Media and Technology
2016, Vol.10, No. 1, pp. 45-52

Figure 2: Predictive and Design-based Approaches in Educational Technology Research(Reeves, 2006)

What is Not Design-Based Research?
DBR is different from traditional experimental approach as we discussed in previous section. Table 3
summarizes how DBR can be compared with other similar research approaches that have been adopted
in the field of educational media and technology, namely Grounded Theory Approach (GTA), Action
Research, and Formative Evaluation. GTA has been widely employed as a substitute for traditional
experimental approach to produce qualitative explanations of educational practices, but it is notdesign-oriented. It focuses on understanding what is happening; not on how to improve the current
practice. Action research, on the other hand, focuses on how to improve own practice, thus considered
to be design-oriented, but not putting much emphasis on theory building. Formative evaluation is also
design-oriented, but not interested in theory building, whereas DBR aims for both improving practiceand refining theory. Although there are some differences in emphasis or aims in each approach, they
can represent variety of alternatives to the traditional experimental studies.
Figure 3 compares DBR with other research approaches by placing it in a spectrum ofreform/individuality description vs. causal/rule establishment, using the researcher’s stance as first,
second and third person. DBR is place in the middle, representing the second person stance, because
DBR is usually carried out in conjunction with practitioners (thus, you and I relationship), whereas
traditional experiment requires objectivity of the researchers (thus, third party view). Action research is
conducted to improve own practice (thus, always subjective using the first person view). The spectrum
also consider whether or not any development occur in the research (vertical comparison). DBR isalways involves creating a new methodology in educational practice, whereas interview is a second
person methodology with no intervention (i.e., just collecting second person data).
Table 3.
Comparison of Research Approaches
Grounded Theory
Design-based
Approaches
Research
Approach (GTA）
Purpose
Improvement of
Explanation of
Practice andPhenomena andConstruction ofConstruction ofTheoryTheory(Not General,＜design-oriented＞
but Domain specific=
Grounded Theory)＜descriptive＞
IJEMT, Vol.10, No. 1, 2016, pp. 45-52 ISSN 1882–2290

Action Research
Improvement ofSpecific Situation＜design-oriented＞

Formative
Evaluation
Improvement ofPractice(Class
Instruction orInstructional
Material)＜design-oriented＞

48

International Journal for Educational Media and Technology
2016, Vol.10, No. 1, pp. 45-52

TargetDomainEducational
Practice

Research
Process

Theory-driven
Practice→
Assessment ofeffect → Revisionof Practice→Assessment ofeffect →Proposal of
Design Principles→Improvement ofTheory･Making bothpractice
improvement andpaper submission
possible･More and more
recognizedCharacteristics

Humanities andSocial Science ingeneralCollection andAnalysis of Interview
data(Coding)→Concept
Extraction→Theoretical
Sampling andSaturation･Process alreadyestablished･Many reference
books available･Hypothesis finding

Humanities andSocial Science ingeneralStatus-quo
Analysis→
Revisionsuggested basedon theory→Implementation→Assessment(→Considerationof Generalizationand Limits）･Many studies
conducted innursing field, with
ample directions/suggestions, easy
to imitate･Not apt togeneralization

Educational
Practice
Development
based on theory ormodel→One-on -oneevaluation→Small group
evaluation→Tryout･Pass/fail on
posttest･Necessary step
beforeimplementation,
but ignored orskipped in manyoccasions

Note: From Suzuki & Nemoto (2013), Table 1, translated by the first author

Figure 3: Representative Research Methodologies placed in Methodology Spectrum(From: Kogo, 2013, Figure 11-1, p. 210, translated by the first author)
Note: *Original word is “Design Experiment”

How Should Design-Based Research Be Conducted?
Ma & Harmon (2009), when reporting a DBR case study, proposes a set of suggestions as to how to
conduct a DBR study. They added sub-steps in each of 4 DBR steps proposed by Reeves (2000) shownin Figure 2 (bottom) to make points of emphasis. In Step 1, they consider topic selection is important toavoid using DBR to a problem limited to a local situation, as DBR takes much effort in aiming attheory building. In Step 2 of developing solutions, they recommend selecting focused researchquestions that fit a particular iteration to avoid overdoing beyond getting answers to the questions.
They also suggest to identify a complex problem with no clear solution available, that calls forcollaboration with practitioners. For Step 3 of evaluation, they caution that deciding evaluation
methods itself require some time for try and error. For Step 4, they recommend to reflect and document
the methodology used in research, in addition to proposing design-principles, as methodology for DBR
is still not mature.

IJEMT, Vol.10, No. 1, 2016, pp. 45-52 ISSN 1882–2290

49

International Journal for Educational Media and Technology
2016, Vol.10, No. 1, pp. 45-52

Easterday, Lewis, & Gerber (2014) identified that one of the problems arising from the ill-definition of
DBR was the uncertainty of DBR process, and proposed a design process model, consisting of 6
iterative phases: focus, understand, define, conceive, build and test. They suggest that the design phases
are not carried out in a linear sequence but rather iteratively, and rapid iteration is a tenet of modern
human-centered design. To protect against the risks of designing interventions that are over-budget andbehind schedule by quickly testing the designer’s assumptions, they recommend for quickly building
low fidelity prototypes, testing them, and re-designing so that gradually evolving the intervention overtime, rather than design an entire intervention and discover only at the end that it does not work.
Herrington, et al. (2007) suggested guidelines for preparing a dissertation proposal for doctoral students.
Using the four phases of DBR by Reeves (2006) illustrated in Figure 2 (bottom), they mapped eachphase against the typical requirements of a research proposal as shown in Table 4. They recommendthat educational researchers who supervise doctoral work are engaged in design-based researchthemselves, so that their students begin to play active roles in that research agenda from the first day. If
the supervisor already be involved in long-term, meaningful coactive engagement with practitioners,
their students will be able to enter as apprentices to this ongoing collaboration, just as so-called“natural” or “pure” science (such as physics and chemistry) students are not expected to start theirresearch projects on their own.
Table 4.
Phases of design-based research mapped against typical elements of a research proposal
Phase of DBR (Reeves, 2006)
Element: The topics/elements that need to be described
PHASE 1: Analysis of practical Statement of problem
problems by researchers andConsultation with researchers and practitioners
practitioners in collaboration
Research questions
Literature review
PHASE 2: Development ofTheoretical framework
solutions informed by existing
Development of draft principles to guide the design of thedesign principles andintervention
technological innovations
Description of proposed intervention
PHASE 3: Iterative cycles of
Implementation of intervention (First iteration)
testing and refinement ofParticipants
solutions in practiceData collection
Data analysis
Implementation of intervention(Second and further iterations)
Participants, Data collection, Data analysis
PHASE 4: Reflection to produce
Design principles“design principles” and enhance
Designed artefact(s)solution implementation
Professional development Methodology
Note: From Table 1 of Herrington, et al. (2007), omitting “Position” column

Conclusions and Suggestions
This paper tried to collect useful guidelines of DBR so that DBR can be adopted in more cases aspreferred methodology in educational media and technology studies conducted by graduate students.
DBR is a strong methodology to have impacts not only educational practices, but also theory itself.
During a DBR study, a long involvement of researchers is required to gradually formulating
educational practice to reflect theory into practice, as well as practice informs the theory. Although
DBR has become a widely accepted research methodology, and although Reeves (2006) has
standardized procedures of DBR, it requires longer duration and deeper collaboration with practitioners,
which may be a barrier especially for graduate students. As Easterday, Lewis, & Gerber (2014) havepointed out, it may become necessary to pinpoint relevant aspects within an educational practice, tolower the risk of research in iterative activities required in DBR. Also needed in conducting a DBR
study is high-level skills as a practitioner, which may be expected for a graduate student with richexperiences as a teacher. However, those who has limited teaching experiences, just being a part of the

IJEMT, Vol.10, No. 1, 2016, pp. 45-52 ISSN 1882–2290

50

International Journal for Educational Media and Technology
2016, Vol.10, No. 1, pp. 45-52

DBR team may place him/her a burden of catching up with everyday challenges of the practice,without leaving enough mental capacity for research aspects of the DBR.
On the other hand, many reported studies deals with one iteration only, but with suggested revisions ofintervention, as shown by Zheng (2013). The fact that such DBR studies with only one iteration havebeen accepted as referred journal paper can be a good news for our students. As we experience
ourselves in conducting DBR studies (Nemoto, Shibata, & Suzuki, 2011), we realize that DBR requires
much time to examine the effects of revisions from previous iterations. It requires more time and effort
to propose design guidelines in a reliable and generalizable manner. We may need to speed up thestudy by adopting a notion of rapid prototyping to shorten the required duration for DBR studies, assuggested by Easterday, Lewis, & Gerber (2014).
Our suggestions that were drawn from this review may be summarized as the following guidelines.
Guideline #1: To advise a master’s student engage in a formative evaluation study byincorporating a model or theory to try to invent a solution to their own challenges in their setting.
It has been our recommendation in our graduate program to try to “make somebody happy” byconducting a formative evaluation study (see Table 3) in their own setting. It is to make the research
practical and useful to the educational practices of their field, rather than learning a “proper way” of
conducting a research. It can be regarded an action research, because they are engaging in from the first
person viewpoint (see Figure 3). By incorporating an existing model or theory to solve their own
problems, it may also be considered as the first step in DBR. It may be regarded, when they move up to
doctoral level, to be the first iteration of DBR.
Guideline #2: To provide our graduate students opportunities to participate in an on-going DBRof their supervisors, NOT as a part of their thesis, but as a part of their coursework.
It may be contrary to the advice that Herrington et al (2007) has suggested to let them engage inan on-going DBR of their supervisors as a part of their theses. Yet, we do not want take away an
opportunity to think deeply what study our students should be engaged in as their own research, bygiving them a topic from the on-going research. We may use our own DBR and ask our students toplay an active and increasingly important role, even if it were a part of their coursework. We think it isa good idea to have a course in which our students are to be exposed to DBR to understand what DBR
is all about. Then, they become more capable in conducting their own study, on their own practices.
Guideline #3: When our master’s student goes up to doctoral program, advise them to sum up
their study for master’s degree, as the first iteration of DBR study to be submitted to a journal.
This study identified the fact that many DBR studies have been published with only the first
iteration. Therefore, we can advise our students not to wait until completing multiple iterations in theirstudy, but just go ahead to submit their findings from the first iteration. They then can continue their
DBR for the second phase and more, in a remaining period in their doctoral program. This may be oneof the plausive ways to see more DBR studies conducted in our field by graduate students.
Guideline #4: To admit the values of research study by our graduate students from wider aspects
than traditional way, to make them of more practical value.
DBR has been thought of a powerful way to bridge the gap between academics and professionals.
Wilson’s (2014) criteria, such as relevance (does it relate to compelling problems currently faced byeducational practitioners?), usability (does it lead to actions?), and impact (does it make a realdifference in improving valued educational outcomes and practices?) should be considered in addition
to traditional expectations of rigor and internal validity of educational research.
The above guidelines have not yet empirically validated, but rather experientially drawn from our own
practice, as well as logically extracted from our review of literature. It is our hope that by following theabove mentioned guidelines, more professors would feel comfortable in directing DBR studies at
graduate school level. We then will become capable in leading our graduate students to challenge toimprove the quality of educational practices in their field, by engaging in a DBR study. The more DBR
study will be conducted in graduate schools, the more guidelines shall be proposed to better guide
future practices. This is exactly what DBR is striving for; gradual elaboration of design guidelines by
iterative and continuing practices.

References
Anderson, T., & Shattuck, J. (2012). Design-based research: A decade of progress in educational
research? Educational Researcher, 41(1), 16-25.
IJEMT, Vol.10, No. 1, 2016, pp. 45-52 ISSN 1882–2290

51

International Journal for Educational Media and Technology
2016, Vol.10, No. 1, pp. 45-52

Easterday, M. W., Lewis, D. R., & Gerber, E. M. (2014). Design-Based Research process: Problems,
phases, and applications. International Conference of the Learning Science (ISLS) 2014
Proceedings (Vol. 1), 317-324.
Feng, X., Lu, G., & Yao, Z. (2015). Professional task-based curriculum development for distanceeducation practitioners at Master’s level: A design-based research. International Review of
Research in Open and Distributed Learning, 16 (2), 288-310.
Herrington, J., McKenney, S., Reeves, T. C., & Oliver, R. (2007). Design-based research and doctoral
students: Guidelines for preparing a dissertation proposal. In C. Montgomerie & J. Seale (Eds.),
Proceedings of World Conference on Educational Multimedia, Hypermedia andTelecommunications 2007, 4089-4097.
Kelly, A. E., Lesh, R. A., & Baek, J. Y (Eds.) (2008). Handbook of design research methods ineducation: Innovations in science, technology, engineering, and mathematics learning andteaching. Routledge.
Kogo, C. (2013). Research methodologies of educational technology (Chapter 11). In Japan Society for
Educational Technology (Supervision), Methodologies of research in educational technology(Educational Technology Series, Vol. III). Minerva-shobo, 200-206 [In Japanese].
Lay, T., Cook, J., Dennerlein, S., et al. (2014). Scaling informal learning at the workplace: A modeland four designs from a large-scale design-based research effort. British Journal of Educational
Technology, 45 (6), 1036–1048, DOI: 10.1111/bjet.12197.
Ma, Y., & Harmon, S. W. (2009). A case study of design-based research for creating a vision prototype
of a technology based innovative learning environment. Journal of Interactive Learning Research,
20(1) , 75-93.
Masukawa, H. (2011). Learning sciences. In The Institute of Electronics, Information andCommunication Engineers (IEICE) (Eds.), Knowledge Base, S3-11, 2-4, 20-32. [In Japanese]
Available online: http://www.ieice-hbkb.org/files/S3/S3gun_11hen_02.pdf.
Nemoto, J., Shibata, Y., & Suzuki, K. (2011). Design-based research to improve learning design andlearning contents. Japan Journal of Educational Technology, 35(3), 259-268 [In Japanese].
Reeves, T. C. (2006). Design research from a technology perspective. In J. van den Akker, K.
Gravemeijer, S. McKenney & N. Nieveen (Eds.), Educational design research (pp. 52-66).
London: Routledge.
Suzuki, K., & Nemoto, J. (2013). Toward managing both educational improvement and researchaccomplishment: Let’s write a design-based research (Review). Competency Development of
Healthcare Professionals, 2(1), 45-53 [In Japanese].
Wang, F., & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning
environments. Educational Technology Research and Development, 53(4), 5-23.
Wilson, B. G. (April, 2014). Knowledge creation in design-based research projects: Complementary
efforts of academics and practitioners. A paper resented at the meeting of the American
Educational Research Association, Philadelphia, Available online: http://www.academia.edu/
6311234/IJEMT, Vol.10, No. 1, 2016, pp. 45-52 ISSN 1882–2290

52