DEVELOPMENT AND EVALUATION OF WEBSITE TO CHECKINSTRUCTIONAL DESIGN BASED ON THE ARCS MOTIVATION MODEL
Katsuaki Suzuki*, Akiko Nishibuchi*, Masayuki Yamamoto*, John M. Keller*** Iwate Prefectural University** Florida State University
JAPAN
U.S.A.
<ksuzuki@soft.iwate-pu.ac.jp><jkeller@mailer.fsu.edu>

ABSTRACT
Website that contained motivational design strategies forvarious instructional setting, including face-to-facelectures and self-paced learning materials, were designed
and developed to help lecturers/designers improve theirinstruction based on the result of user reaction
questionnaire. The Website has capabilities of collecting
questionnaire data, analyzing them, and suggesting weak
areas, based on Keller’s ARCS motivational design model.
Strategies to improve instruction are then suggested byretrieving from motivational strategy database that aresuitable for a given instructional setting (teaching mode,target audience, characteristics of learning objectives,etc.). Formative evaluation studies were conducted torevise the Website for usability and practical effects,
where the lecturers/designers worked through the Website
to come up with a set of their own motivational
enhancement strategies by referring to the weak areas thesystem analyzed and the motivational strategies thesystem proposed.

The former is called ID process, or Instructional System
Development (ISD) model, whereas the later is called ID
model. Systems science forms the basis of the former(i.e., formative feedback, plan-do-see, clear statement oflearning objectives, etc.), whereas psychology related tohuman learning forms the basis of the later (i.e.,
information processing model of human learning,
motivation theories, usability design, etc.).
Reigeluth[3] was among the first ID researchers who madeclear the distinction between systematic process models
for ID and design models. He prefers to call the former
Instructional Development Model, to distinguish from theID (Design) Model. Although the word ID is still used tomean both ID process and design models, this paper
follows the Reigeluth’s footstep.

1. INTRODUCTION

Reigeluth’s book in 1983, which is often called "TheGreen Book", was a major milestone in the history of ID
related research, by making available and easy to compare
the 8 dominant ID (design, not development) models atthat time. The most well known ID model, Gagne's
model of Nine Events of Instruction[4] [5], was included in
the Green Book, as the model that has the strongest
psychological underpinnings supporting the model.
Gagne, who turned himself from an establishedpsychologist to one of the founders of ID field, well
represents the close relationship between psychology and
ID. Among the 8 models in the Green Book, was John M.
Keller's ARCS Motivational Design Model[6].

1.1. What are ID models?

1.2. ARCS Motivational Design Model

Instructional Design (ID) models are practical summingof psychological as well as other researches, for helping
those who create educational and training materials.
Sometimes ID represents the processes of instructional
material development, such as Dick & Carey Model[1] and
ADDIE model[2] (i.e., Analysis, Design, Development,
Implementation, and Evaluation). ID, at other times,
represents a framework for the final instructional productso as to improve the effectiveness and appeal of thelearning resources.

ARCS Model, proposed by John M. Keller, was one ofthe pioneering works in the area of motivational design,
whose goal is to make instruction more appealing to thelearner. Motivation is often regarded as an input variableof ID that each of the learners brings into, and therefore,that should be taken care as a possible distracting factor of
instruction. Motivation is also regarded as a variable in
the instructional process, and therefore, should be taken

KEY WORDS
Instructional Design, Evaluation and Assessment in WBE,
ARCS Model, Formative Evaluation, Motivationcare to keep, or not to loose, during the instructional
activities.

perceived that applying ID models, such as the ARCS
model, may require a lot of effort.

ARCS model deals learner motivation as not only an
input and process variable, but also as an output variable.
The goal of motivational design is such that the learnerfeels to keep continuing the study as the result of learning
experience with well-designed instructional material. It is
the appeal of instruction that would make learner's
continuing motivation[7] high at the end. It is important toregard motivation as the goal of instruction, especially it
has always reported that Japanese students do well inmath and science in international surveys, but they are at
the same time score the lowest when asked if they likemath and science.

One of the burdens that seems to prevent from using suchmodels has been the implementation cost associated withthe application of rather abstract concepts to a particular
set of practical situations. There has been a gap between
the potential usefulness of ID models and the actual or
perceived difficulty for applying the models. There is aneed for a computer-based tool to bridge this gap.

The ARCS represents four factors affecting learner's
motivation, namely, Attention, Relevance, Confidence,and Satisfaction (i.e., ARCS model). The ARCS model
has been created by surveying related psychological andother research and theories, so as to depict all importantfactors for instructional designers, but in a simplistic(easy to apply) manner. It has been well received by andwidely used in the ID community, but not as wellrecognized by the professional community of educational
psychology with few exceptions[8] [9].

1.3. Purpose of the study
The purpose of this study was to bridge such a gap so as
to make it easier to apply ID models in practical settings.
It was to be done by providing a semi-automated strategy
selection mechanism based on one of ID models for
motivational enhancement of instruction, i.e., Keller’s
ARCS motivational design model. It was aimed toprovide a Web-based tool to collect evaluation data,
analyze them, suggest motivational enhancement
strategies that fit the given instructional setting, so that theusers could come up with a set of ideals to improve theirinstruction.

2. OVERVIEW OF THE WEBSITE
The research bases of the ARCS model are wide andthorough to cover all major motivation theories andconcepts. Curiosity arousal, inquiry arousal, variability
are included in Attention. Familiarity, goal orientation,achievement, affiliation, and power motives are in
Relevance. Goal setting, goal seeking behavior, successexperience, efficacy, learned helplessness, and attribution
theory are reflected in Confidence. Reinforcement, Peer
praise, Equity, Consistency, and Cognitive Feedback areincluded in Satisfaction, just to name a few. However,practitioners don't have time to refer to all of thepsychological concepts and theories, when designing
instruction.
It is therefore advisable to refer to the ARCS model thatcovers many findings of psychological research efforts.
One can save time by using the ARCS model, as
compared to study all of the basic psychological concepts
and theories, and to be able to judge when to apply which.
The ARCS model not only suggests four categories todivide motivational problems, but also suggests various
motivational strategies in each of the four categories fromdifferent application areas. The ARCS model has been
applied to such areas as courseware design[10], teacher
training[11],multimedia
production[12],science[13]
education , instructional message design[14], etc., so as
to provide a set of sample motivational strategies for thedesigners working in those areas. However, it is still

The Website “Check-and-Revise Your Motivational
Design” was designed to have the following features:
1)
2)
3)
4)
5)

Reaction Questionnaire Collection
Data Analysis
Motivational Strategy Database
Strategy Suggestion
Strategy Selection and Idea Plotting

2.1. Reaction Questionnaire Collection
Based on Keller’s ARCS categories and subcategories (3for each of the ARCS), Kogo and Suzuki[15] developed aJapanese version of reaction questionnaire to ask
motivational characteristics of instruction using 12 itemsof 9-point Likert scale. Items use Semantic Differential
scheme by asking how much the respondent felt about the
instruction, by placing a mark in between two adjectives
opposite to each other, such as (the instruction was)boring vs. interesting, useless vs. useful, etc. Web versionof this questionnaire was placed as a part of this system asshown in Figure 1. As the instructor or developer of the
instructional material sets up an account for each of thecourse titles, the system will create a file to store the datacollected on the 12 items, together with open-ended
comments and suggestions.

Figure 1. ARCS Reaction Questionnaire
2.2. Data Analysis
Means and Standard Deviations are calculated for each
item on the Reaction Questionnaire. The system then willshow the results in descending order of means, so as tohighlight the weak aspects of instruction. Three itemswith the lowest mean scores are determined as the areasthat need focus in revision, so that the effort in revising
instruction will be concentrated in the area that are
expected the largest improvement, as suggested in the
ARCS motivational design procedure.
2.3. Motivational Strategy Database
One hundred and fourteen motivational strategies, written
in Japanese, were included in the prototype of theWebsite’s database. They were taken from various
resources that explain the ARCS model and its strategies
by paraphrasing them to relatively easy to understandwording. Each strategy is identified as to which of the
ARCS subcategories that it should be used, together with
its applicability in terms of instructional settings (lecture,
self-regulated learning, etc.), target group’s ages, andinstructional objectives (knowledge, skills, attitudes, etc.).
All the strategies can be examined by the Website users,
by calling the “all strategy at glance” function.
2.4. Strategy Suggestion
By linking Data Analysis section of the Website with
Motivational Strategy Database, the system has a functionof displaying only the suitable strategies by the ARCS
subcategories. When the user select to see the best
strategies for a certain instruction, the lowest three items
are examined through the database to show only thosestrategies fit the ARCS categories as well as instructionalsetting. This function is to help the user concentrate on

Figure 2. Strategy Suggestion
the weak areas depicted by the questionnaire analysis,
although the user can select to further see the suggested
strategies in other relatively problem-less areas. Figure 2
shows a result screen showing a set of strategies selectedby the system based on the questionnaire results.
Figure 2 indicate, in Japanese, that the worst item in thequestionnaire result was "The instruction is not related tome," which was included as the first item of Relevance.
Ten motivational strategies are suggested from thestrategy database. The suggested strategies include, fromthe top of the box, "200200: Set problems in a realistic
situation, which students feel familiar," "200300: Prepare
various examples to try to fit various background of thestudents," and "202300: Deal with things related toknowledge and skills covered in previous lessons."
2.5. Strategy Selection and Idea Plotting
The system has a function of assisting the users to collectfrom the suggested strategies the ones they would like toincorporate in their idea generating session for the
improvement of instruction. The shopping cart model
was adopted to collect only the ones they liked.
In Figure 2, out of ten motivational strategies suggestedby the system for the first problem, it is shown that 5
strategies have been selected by the user. "202300: Dealwith things related to knowledge and skills covered inprevious lessons," which is displayed as the third in thesuggested list, is one of the strategies selected for the
inclusion of user's shopping cart. The right-most column
indicates if each of the strategies has been selected. The
remaining 5 strategies have not been selected, which canbe added to the user's shopping cart at any time.

The users are then prompted to come up with their idea ofrevision by referring the strategies they selected. The
final draft that contains selected strategies and their ideas
for revision can be saved in the text format, or printed out
for their references.

3. FORMATIVE EVALUATION STUDIESThe Website has undergone the following formativeevaluation studies[1].

Revisions were made and some functions were added
before going on to the small group formative evaluation:
Explanation of numbers 1-9 in reaction questionnaire
result was added. Minor differences in wording indifferent parts of the system were made uniformed.
Procedures that were not necessary were eliminated.
Colors were added to clarify the groupings of strategies
and of weakness of areas. The format for saving ideas for
improvement was shifted from CSV to text.
3.2. Small Group Formative Evaluation

3.1. Usability Testing (One-to-One Evaluation)
Two young college faculty members were participated in
an experiment to check and improve usability of theWebsite. Cognitive walkthrough method were adopted in
one-to-one evaluation sessions in which a user was asked
perform all the functions the Website provided: to see
general description of the system, the ARCS model, thereaction questionnaire sheet, to log in, to see thequestionnaire results, to check the enhancement strategies
the system proposed, to select from the proposed
strategies into “my list,” to download and print revisionideas that the user summed up. Previously collected datafrom students who took an undergraduate lecture, thatboth of the participants were familiar with, were used inthis experiment.
Both of the two participants could perform all the tasks
with no problem in 51 and 31 minutes, respectively.
Overall impressions of the system were favorably statedin the post experiment questionnaires. An interview
session was then conducted for about one hour each, bygoing through the tasks that they performed. Comments
and suggestions were noted in the interview sessions,which were categorized based on the urgency andexpected magnitude of improvement.

Ten undergraduate and graduate students wereparticipated in the small group formative evaluation tocheck and improve the effectiveness of the Website.
Participants examined six self-study print-based one-hour–long learning materials, which had been developed bysix of the participants. Content of the material varied,
including such topics as “How to assemble your own
PCs,” “Introduction to Knitting,” “Let’s study Piano
Codes,” and “How to calculate points in Mah-jongg.”
Expect for the one that each participant him/herself
developed, they evaluated the attractiveness of eachmaterial by filling out Reaction Questionnaire Section of
the Website, resulting in 9 sets of reaction data for each of
the material evaluated.
Six of the participants who authored the material thenreviewed the reaction data, and tried to come up with a setof revision ideas using the Website’s Analysis,
Suggestions, and Idea Plotting functions. As the results,
all of the six participants were able to reach at least one
revision idea, with an average of 3.67 ideas or a total of
22 ideas. Of 22 ideas, two were based on the comments
and suggestions in the open-ended section of reaction data,
whereas 20 were based on enhancement strategies
suggested by the system. Of 20 ideas based on suggested
strategies, 11 ideas were based only on strategies in the

Table 1: Number of strategies selected for "My Lists" by Ranks of Weakness

Rank
1
2
3
4
5
6
7
8
9
10
11
12
total

A
C3
R3
A3
A1

B
1 *
1 *
1 *
2 *

C

R1
A3
C3

1
2 *
0

S2

1

R3
C3
A3
C2

D
1 *
1 *
2 *
2 *

E

F

C3
A1
S2
A3

1 *
3 *
1 *
2 *

S1
R1
R2
S2

1 *
2 *
2 *
1 *

C2

2 *

A3

1 *

C2
A2
C1
R3
A1

R1
A3
C1
R2
C3
S1
C2

5 *
3
6 *
4
5
3
3

1 *
1 *
2 *
A2
1
1 *
3 *
A2 1
5
5
6
9
15
30
Note: * represent strategy that became basis for enhancement ideas
A3, C2, etc. refer area in the Reaction Questionnaire, cf. Table 2.

total
10
12
12
11
6
6
3
1
1
2
2
4
70

three weakest areas that the system called attention to theusers, 4 used strategies in the three weakest areas inconjunction with the ones from other areas, whereas only
5 ideas were based solely on strategies in less weakerareas. It was seen in this experiment that the Website
successfully facilitated to generate revision ideas,
especially in the area that needed improvement.
In terms of strategies selected to form the “My Lists,”
which were kept and used in idea plotting for revisions,the six participants selected a total of seventy strategies(Table 1). The average number of strategies selected into
“My Lists” was 11.67. Of 70 strategies selected, 34 werein the three weakest areas of each of the participants,
whereas other 36 were in other 9 areas. That is to say, anaverage of 1.89 strategies were taken from the three
weakest areas, whereas an average of 0.67 strategies weretaken from other 9 areas. This shows that the Website
facilitated to concentrate the revision effort to the weak
areas depicted by the questionnaire data.
Except one participant who selected from two of the three
weakest areas (i.e., Participant B in Table 1 selected no
strategies for category C3), all other participants selected
at least one strategy from all of their three weakest areas.
Each participant selected strategies to “My Lists” fromother 9 areas varied in number: from one (two participants,
A and C in Table 1) to 7 areas (one participant, E in Table1). Most (88%) of the strategies participants selected in
the three weak areas were in fact used as bases of revisionideas, whereas only about a half (47%) of the selectedstrategies in other 9 areas was utilized in formulating
revision ideas.
Ideas for further revisions were collected in thequestionnaires and interviews from the 10 participants ofthis experiment. Several buttons and links were alteredfor better usability and a clearer structure of the Website.
3.3. Case Study of a University Professor
A professor who taught undergraduate level course on
strategic information system used the Website to come up
with a set of ideas for course improvement for the next
year. This professor had no prior knowledge about the
ARCS model, or the ID models.
An announcement was given in the last class asking tovoluntarily submit reactions for the course by using theWebsite’s questionnaire. All of the 87 junior and senior
students who enrolled in the course received an e-mailwith the URL of the Website, with which they could
access to the reaction questionnaire with a click of amouse. Forty-one students sent their reactions, which theprofessor used in this experiment.
After the professor went through the tutorial session of
the Website, he saw the results of his students’ reactions
on the Website. Table 2 shows average point for each of
the ARCS questionnaire items in ascending order. First
three areas were identified as weakest: (1) A1: Whether or
not they felt sleepy, (2) C3: Whether or not they were able
to be creative in learning, and (3) S1: Whether or not thecontent was readily applicable.
He then used strategy suggestion and selected his own setof motivation enhancement strategies. A total of 13
strategies were suggested for the three weakest areas, ofwhich he selected four into his “My List.” Seventy-fivestrategies were suggested for other area, of which heselected 25. Finally he came up with a set of seven ideasof enhancement for the next year. It took him about one
hour to finish this task, including the tutorial session in
the beginning.

Table 2. Results of a Case Study of a University Professor
ARCS Reaction Questionnaire: Scale (1-9)
Items
A1
C3
S1

Felt sleepy ←→ Didn't feel sleepy
Not creative in learning ←→ Was creative in learning
Not readily applicable ←→ Readily applicable

C2
A3
C1
R3
S3
A2
S2
R1
R2

Steady progress impossible ←→ Steady progress was possible
Not stimulating ←→ Variable and stimulating
Objectives were vague ←→ Objectives were clear
Learning process wasn't fun ←→ Learning process was fun
Evaluation not consistent ←→ Evaluation was consistent
Curiosity wasn't aroused ←→ Curiosity was aroused
Effort wasn't recognized ←→ Effort was recognized
No relevance to me ←→ Relevant to me
Didn't want to acquire ←→ Wanted to acquire content
Note: number of responses=41, items are originally in Japanese.

Ave

SD

Strategies
suggested selected

4.93 1.33
4.98 1.80
5.45 1.73

3
4
6

1
1
2

5.56
5.66
5.68
5.71
6.12
6.24
6.24
6.63
7.00

16
10
11
3
4
6
7
9
9

4
6
4
2
0
2
2
2
3

1.73
1.34
1.51
1.43
1.33
1.40
2.10
1.78
1.81

During the interview after the task was completed, he
expressed that the ARCS model helped him
conceptualized the weak areas that need enhancement.
Also expressed was a convenience of thinking
improvement directly based on students’ reactions to thecourse. For him, the Website was a handy tool to thinksystematically of motivational enhancement to his course.
On the other hand, he was concerned with considerable
variation of available number of strategies in each area. It
was mainly because fewer strategies were proposed in thethree weakest areas than the other nine areas in his case.
He said he might have selected more strategies in thoseareas that had more favorable students reactions, becauseof the availability of more strategies in those areas. Morestrategies are needed especially in the areas with fewer
strategies in the database, for this system to be grown outof the stage of a prototype.

4. CONCLUSION
In this study, a prototype of the Website to check andrevise motivational design of instructional materials was
proposed. Positive results were obtained from formativeevaluation. When the strategies included in the database
will be enriched, this Website should become a handy toolfor designers/lecturers with limited ID expertise.
It is our hope that Web-based ID tools, such as the one
proposed by this study, will be made available more in the
future, so as to make the potential of ID models to be
fully utilized in educational practices. By combining thedevelopment of these ID tools with advancement of ID
models based on empirical studies, psychological researchfindings would become more readily available, throughthe window of ID, to support practitioners in making
better quality instruction.

ACKNOWLEDGEMENTPart of this research was supported by Grants-in-Aid for
Scientific Research (C)(2), 12680217, from Ministry of
Education, Culture, Sports, Science and Technology(MEXT), JAPAN.

REFERENCES[1] Dick, W., Carey, L., & Carey, J.O. (2000). The
Systematic Design of Instruction (5th Ed.). Boston, MA:
Addison-Wesley.
[2] Piskurich, G.M., Beckschi, P., & Hall, B. (Eds.)(2000). ASTD handbook of training design and delivery:

Instructor-led,
McGrow-Hill.

computer-based,andself-directed.

[3] Reigeluth, C.M. (1983). Instructional design: What is
it and why is it? In C.M. Reigeluth(Ed.), Instructionaldesign theories and models: An overview of their current
status. Lawrence Erlbaum Associates, U.S.A.
[4] Aronson, D.T., & Briggs, L.J. (1983). Contributions ofGagne and Briggs to a Prescriptive Model of Instruction.
In C.M. Reigeluth(Ed.), Instructional-design theories andmodels: An overview of their current status. Lawrence
Erlbaum Associates, U.S.A.
[5] Gagne R.M., Briggs, L.J., & Wager, W.W. (1992).
Principles of instructional design (4th Ed.). Harcourt
Brace Jovanovich, U.S.A.
[6] Keller, J.M. (1983). Motivational design of
instruction. In C.M. Reigeluth(Ed.), Instructional-design
theories and models: An overview of their current status.
Lawrence Erlbaum Associates, U.S.A.
[7] Maehr, M.L. (1976). Continuing motivation: An
analysis of a seldom considered educational outcome.
Review of Educational Research, 46(3), 443 - 462.
[8] Newby, T.J. (1991). Classroom motivation: Strategies
of first-year teachers. Journal of Educational Psychology,
83 (2), 195-200.
[9] Rieber, L.P. (1991). Animation, incidental learning,and continuing motivation. Journal of Educational
Psychology, 83 (3), 318-328.
[10] Keller, J.M., & Suzuki, K. (1988). Application of
the ARCS model to courseware design. In D.H. Jonassen(Ed.), Instructional Designs for Microcomputer
Courseware. New York: Lawrence Erlbaum, 401 –434
[11] Keller, J.M. (1984). The user of the ARCS model ofmotivation in teacher training. In K.E. Shaw(Ed.), Aspectsof educational technology volume XVII: Staff development
and career updating. Kogan Page, England.
[12] Keller, J.M., & Keller, B.H. (1991). Motivating
learners with multimedia instruction. Proceedings ofInternational Conference on Multimedia in Education andTraining '91, Tokyo, 313-316.
[13] Keller, J.M., & Kopp, T. (1987). Application of the
ARCS model of motivational design. In C.M.
Reigeluth(Ed.), Instructional theories in action: Lessons
illustrating selected theories and models. Lawrence
Erlbaum Associates, U.S.A.
[14] Keller, J.M., & Burkman, E. (1993). Motivation
Principles (Chapter 1). In M. Fleming, & W.H.Levie

(Eds.), Instructional message design (2nd
Educational Technology Publications, U.S.A.

Ed.).

[15] Kogo, C., & Suzuki, K. An analysis of the structureof course evaluation items based on ARCS motivation
model, II. Proc of ICCE/ICCAI 2000, 8th International
Conference on Computers in Education/International
Conference on Computer-Assisted Instruction 2000,
Taipei, Taiwan, 2000, 1577-8

Authors

Katsuaki Suzuki, Ph. D.
Professor, Faculty of Software and Information Science,
Iwate Prefectural University, majoring in Educational
Technology. Prof. Suzuki graduated from International
Christian University, Tokyo, and earned his Ph. D. in
Instructional Systems at Florida State University. Since
then, he has been working to promote the utilization ofvarious instructional design models in practical settings.
His research interests include ICT education, teacher
training, educational media utilization, and instructional
design in the e-learning settings. Member of JSiSE, Japan
Society for Educational Technology, Japan Association
for Educational Media Study, AECT, AERA, and AACE.

Akiko Nishibuchi
Graduate student, Faculty of Software and Information
Science, Iwate Prefectural University. Student Memberof Japan Society for Educational Technology.

Masayuki Yamamoto
Undergraduate student, Faculty of Software andInformation Science, Iwate Prefectural University.
Student Member of Japan Society for Educational
Technology.

John M. Keller
John M. Keller, Ph.D. in Instructional Systems
Technology and Organizational Behavior from Indiana
University, is a Professor of Instructional Systems andEducational Psychology at Florida State University. Since
earning his Ph.D., he has focused on the development ofapproaches to designing motivational systems He is best
known for his motivational design process that is called
the “ARCS model.” Also, he has contributed to the designof performance improvement and systematic training
design processes for several school districts, major
corporations, and government agencies. He has extensive
international experience and has been to Japan numerous
times where he has made presentations at conferences anduniversities, and to organizations such as NIER at the
Ministry of Education.