The Design of a Web-based Support System
for Material Design/Evaluation Based on Goal Based Scenarios
Junko Nemoto
Graduate School of Software and Information Sciences
Iwate Prefectural University
andGraduate School of Social and Cultural Sciences
Kumamoto University
Japannemoto@kumamoto-u.ac.jp
Makoto Miyazaki
Graduate School of Social and Cultural Sciences
Kumamoto University
Japan
maco@kumamoto-u.ac.jp
Katsuaki Suzuki
Graduate School of Social and Cultural Sciences
Kumamoto University
Japanksuzuki@kumamoto-u.ac.jp
Akihiro Abe
Faculty of Software and Information Sciences
Iwate Prefectural UniversityJapan
abe@soft.iwate-pu.ac.jp
Abstract: This paper describes a design process of a Web-based support system for material
design/evaluation based on the Goal-Based Scenarios theory (GBS). The system is based on apaper-based checklist developed by the first and third authors; it is intended to support
designers/educators to develop or revise learning contents using GBS, and includes four
elements: a self-assessment tool of the existing learning material, a GBS glossary, a GBS
sample gallery, and a development support tool. This paper focuses on a grand design of thesystem and the first element, which is a self-assessment tool design.

Introduction
Educators and designers are always facing challenges to create effective learning designs. From traditional
teaching styles, which are called teacher-centered designs, to learner-driven or learner centered-designs, newmethods have been appearing, and the integration of these methods and technologies have provided newlearning styles and support systems. Various practices, by practitioners and researchers, have produced new
instructional theories that will guide other practitioners to successful educational outcomes.
The authors have studied how to promote the effective use of a learning theory by taking the Goal-Based
Scenarios theory (GBS) as a case study. The authors started developing a tool for evaluating an existingmaterial, which is simple and handy for expected users who are involved in designing and developing GBSoriented or GBS-like learning materials (Nemoto & Suzuki 2004; Nemoto & Suzuki 2005). GBS is one ofproblem based learning (PBL) approaches that uses cases, examples, and problems (Jacobson 2008). The

developed GBS checklist is based on seven factors of the theory: Goal, Cover Story, Mission, Role, Scenario
Operation, Resources, and Feedback (Schank 1998; 1994; 1996; 1999). The users are able to know how mucha selected learning material follows the theory by filling in the checklist.
The advantage of the use of the checklist is to provide a place to reflect on the applied instructional strategies byreviewing a developed material with the checklist. This reviewing process would help users to revise andimprove their existing materials. In addition, the interview conducted for formative evaluation in the authors’previous studies showed extra advantages to use the checklist. An interviewee answered that the checklist
helps to reconfirm the material objectives, evaluation method, and instruction as well as understand thedifferences of the examined material design from GBS’s theoretical perspectives (Nemoto & Suzuki 2005).
Using the checklist is not only for revealing an application level of GBS, but also for confirming the strengths
and weaknesses of the strategies used in the material’s design. The feedback from using the checklist canbecome a useful resource, if the user has an access to it. But the formative evaluation showed that there aresome weaknesses in using the list.
First, the checklist is paper-based so that it is not easily accessible for potential users. Handling of thechecklist is not easy, because the checklist consists of some supplemental documents. Second, filling in the
checklist is not simple for the users; the checklist includes GBS jargon and requires the user to spend time tounderstand the questions on the checklist. Although the checklist has glossaries, for the keywords, anddescriptions to learn what GBS is, users who encounter GBS for the first time will have difficulty to use thechecklist appropriately. Therefore, the authors find that it is advantageous to create a web-based system that
expands the strengths of the checklist and mops up the weaknesses of the paper-based checklist.
The purpose of this study was to design a Web-based support system for GBS-based material design andevaluation, by expanding the existing paper-based GBS checklist. In addition to making the checklist easier touse, by adding features that cannot be done on paper, The study was intended to expand the application scenesof the checklist from evaluating existing learning materials to designing new materials. The aim was also forthe new support system to be designed to serve as an opportunity for novice designers to learn GBS, by showing
GBS examples and critical points when incorporating the notions of GBS-based designing principles. This
paper focuses on a grand design of the system and the first element, namely, a self-assessment tool design.

The Grand Design of the Web-based Support System
The web-based support system for material design and evaluation is intended to offer learning designers andeducators to reconsider the learning strategies that they have been using. Therefore, we have developed achecklist for the learners to use for their own benefit. The concept of the web-based support system is to raise
the awareness of learning designers and educators by providing a resource for them to self-assess their abilities
and discover how to make improvements on the developed material. The users, as result, deepen theirknowledge and skills of learning design by experiencing the system.
The web-based support system consists of four functions, as shown in Figure 1: a self-assessment tool of theexisting learning material, a GBS glossary, a GBS sample gallery, and a development support tool. The
system enables the user to access the four functions freely and independently, according to the user’s needs.
The first function is a self-assessment tool of the existing learning material that was developed based on theexisting paper-based checklist. This part has a diagnostic function to indicate the application level of GBS for
the target learning material. The user receives feedback after filling out the questionnaire provided.
Answering the questions gives the user a chance to reflect on the instructional strategies of the target material,
and the results shown after answering questions provides a diagram feedback about the material. Because theself-assessment tool includes a feedback sample that users will obtain when they use this function, they caneasily grasp the idea of the assessment.

The second function is a GBS glossary that provides a place for users to learn about GBS. The users are notrequired to use this function; they can use it when they feel the necessity. This is a supplemental resource toimprove the user’s knowledge.
The third function is a GBS sample gallery, in which the user can see examples of materials developed on thebasis of GBS. This part meets the users’ “SHOW ME” request so that they are able to see what GBS-oriented
materials would look like. Also, this part has exercises to practice using the assessment tool (checklist) thatcomes with a guideline. The purpose for providing the GBS material example and the guideline to use thechecklist, the user finds the relationship between the checklists and the examined material without reading any
manual or guide. This would encourage and attract more users to use the checklist.
The last (fourth) function is a development support tool that aids the user to create GBS-oriented contents.
The user can develop simple GBS contents by using this tool. Like the other functions, this one includes
examples. The user has actual experiences to develop the GBS-orientated contents. As mentioned above,
each function has examples so that a text-based manual is not necessary. Throughout the function trials, theusers are expected to get used to the idea of GBS and deepen their understanding of it.

Assessment tool
Development support tool

Assessment
tool

Tools

- Create a material--Analyze an existinglearning material-

Assessment
Sample

The guide to use example
ExamplesYOU
Designer

GBS
Samples

What
GBS
is？

GBSSample Gallery
Exercises- Confirm GBSof usingoriented materials the checklist

GBSGlossary

GBSGlossary- Learn GBS -Figure 1: The Grand Design of the System

The Self-assessment Tool Development with the Checklist
Figure 2 is an example of an assessment result that a user receives. Among the three functions, the top part isa description of the contents filled in by the user. This result is merely what the user answered in the selfassessment tool; this is a summary of the material analyzed.
The quadrilateral chart in the second part shows the result of the GBS analysis of the learning material; thischart is created automatically from the answers to the questions on the checklist. The four items of the chartare derived from GBS’s seven factors that are embedded in the developed material. Because designers usually
do not need to know the name of GBS’s seven design factors, the chart does not show them explicitly. Instead,the chart uses four perspectives to express what designers and educators should check to see if they are included
in designed materials: (1) sufficiency of objectives, (2) sufficiency of learning support, (3) sufficiency oflearning activities, and (4) sufficiency of scenarios. The user receives feedback from four perspectives alongwith the chart and documents. This design allows the user to look at the overview easily see the result; thedescription is in plain words that novice designers can usually understand. When the user wants to know moredetailed feedback on each perspective, more comments, that correspond to each question, will be provided byclicking “read details” link.
How sufficient the objectives are will show in the result of the questions about Goals of the GBS’s seven factors.
The sufficiency of objectives is general information about the objectives, but includes questions to confirm therelation to the other GBS factors. The sufficiency of learning support includes the feedback and resources ofthe seven factors; this result implies how the user considers learner support mechanisms and how much they arebuilt into the mechanism of the material. The sufficiency of learning activities is composed of scenario
operations, which indicates the degree of the learning activities, in the content, and helps the user recognize howmuch the learning activities relate to the goals. Last is the sufficiency of scenarios from the cover story, therole, and the mission of GBS’s seven factors. The context expands throughout the stories so that setting acover story and providing a role and mission is necessary.
The last part of the feedback window is the input from the users. The questionnaire has two types ofquestions: descriptive and multiple choice. The descriptive questions are listed at the end of the feedback as
the users confirm their self-analysis with other results.

■ General Information of the M aterial

Title

Case Studies – Management Ability

Target Users

For Managers

Objectives

The participants of the course will:- able to describe a position as a manager- able to respond and coach to the subordinates- Participate in discussion, and- Submit a final report

Evaluation
Methods

■ Quadrilateral Chart of the result
Sufficiency of scenarios

Sufficiency oflearning activities

Sufficiency oflearning support

Factors

Achievement

Sufficiency of scenarios％

Sufficiency of learning support％

Sufficiency of objectives％

Sufficiency of objectives％

Sufficiency of objectives●Sufficiency of scenarios(Item：Goals) See details

You have prepared a cover story and used scenariosthat the learnerspurse their learning. You have also provided role and mission, but the scope ofthe role and mission is not clear enough. Review again the scenario and role that are appropriate to the target learners.
●Sufficiency of learning support(Items：Feedback and Resources) See details

Thereisa scenario in the material, but decision making scenesare lucking. If the learnersdon’t have opportunitiesto make actionsthrough theactivities, the meaning of providing a scenario doesn’t work. The advantage of usingGBSis that the learnersand make actions with own thoughts. Add
learning activitiesand consider the strength to use a scenario.
●Sufficiency of objectives (Items：Scenario Operations) See details

In GBSenvironment, learners can confirm the result of decision making with feedback and deepen their understanding with resources.
Resources can be provided as references, web-links, paper-based documents and so forth. Discuss various types of feedback that are suitfor implementing method.
●Sufficiency of objectives(Items：Cover story, Role, and Mission) See details

The learning objectives are set. However, this assessment tool does not show how much your set objectives correspond to the scenario
provided. If there are several objectives, you needs to create various learning activities to cover the all objectives.
■Material Analysis on your own●The number of the scenario ： 6●The cover story used in introductory phase of learning●Relationship between given role and scenario operation●Types of feedback●Methods of feedback◆as a result of the action◆from a coach◆Story from professionals◆Others ( Details ：)●Learning objectives●Situation in which the learner learn

Figure 2: An example of the result of the self-assessment tool

The Algorithm of the Feedback
The feedback system is critical to attract users, and the amount of feedback must be appropriate for users. If
the user gets too little feedback, they will not be satisfied with the result. But a heavy volume of feedback candeter the user from reading the details. Therefore, the system employs a two-step feedback about the results inthe self-assessment. As shown in Figure 2, the results of the assessment include feedback from four
perspectives. The first feedback for improving the target materials is the visualized chart presents theassessment score, and the four comments in a few lines, and follows each perspective.
The type of feedback that the user receives varies depending on the user’s answer to the assessment. Eachgroup of questions in each perspective has two or three categories, and the feedback changes according to thenumber of categories in which the user has selected the right answer. That is to say, each perspective has fourto twenty-eight feedback messages, and the user receives feedback from forty-six, in total. This is an
automated response system with a variety of feedback. When the user wants to get more detailed feedback,h/she can obtain a full report with responses to each question.

The Development Process of the Self-assessment Tool (to be completed before
presentation)
The first author depicted the grand design of the web-based support system as well as the design of eachfunction. The second author regularly provided advice and comments to the design from a system developer’s
point view, and will develop the system. The third and forth authors reviewed the idea and examined thepossibility of implementation. In the presentation of this paper, the presenter will show a prototype of thesystem.

GBS Design Support System
for Material Design/Evaluation- What’s this? Evaluate■

GBS Self-assessment tool

Sample Result of GBS
Self-assessment Tool■

This website is for people who are involved in leaning material design.
Please select an existing materials and analyze it with the system. This system
will show the material strengthen and weakness as well as this systems support
your ability of analysis and development.
If you are interested in to create scenario type or simulation type of materials or
you already have it, you are the target users of the system.

Know about GBS■

What’s GBS■

GBS Glossary

Evaluate[GBS Self-assessment tool]

You can conf irm how much your choice of material is designed based on GBS.
We will recommend to visit the page to the person who already have a material toanalyze or who have a developed material by yourself .

See Sam ples■■

GBS material Samples
GBS Self-assessment
Exercise

Know about GBS[GBS Glossary ]If you are new to GBS, or want to know more, although you have some basic ideas
about it.

Develop■GBS Development
Support Tool

Examples of the Result of
GBS Development Support Tool■

SYSTEM CONCEPT■

Who’s system?

■

For what ?

See Samples[GBS Sample Gallery]

For people who would like to know some examples that are based on GBS.

Develop[GBS Design Support Tool]

This support our material development of GBS based materials.

Figure 3: An example of the Web Page Design (Top)GBS Design Support System
for Material Design/Evaluation
TOP Page > Evaluate
Evaluate■

GBS Self-assessment tool

Sample Result of GBS
Self-assessment Tool■

Evaluate
You are able to conf irm the degree of which the selected material’s application level of
GBS. If you have a material that you want to analyze or you have your own developedmaterials, please try the tool.

Know about GBS■

What’s GBS■

GBS Glossary

■

■

■

GBS Self-assessment Tool
Try the GBS self -assessment tool and see the analysis result of the materials.
Please answer the all of the questions and get the result.
You also f ind the other f actors that are essential f or learning material
development.

See Sam ples
GBS material Samples
GBS Self-assessment
ExerciseDevelop■GBS Development
Support Tool

Examples of the Result of
GBS Development Support Tool■■

Sample Result of GBS Self-assessment Tool
What’s kinds of result will you obtain by using the result?
If you want to see a sample of the result, please visit here.

SYSTEM CONCEPT■

Who’s system?

■

For what ?

Figure 4: An example of the Web Page Design

The Future Plan of this Study
The next step of this study is to complete the assessment function in the four functions of the web-based system.
To enhance the quality, a formative and usability evaluation of the self-assessment tool need to be conducted.
Several users, who have different levels of knowledge about GBS, will participate in the evaluation to use the
web-based support system. In addition, two other functions of the system, a GBS glossary and a GBS sample
gallery, will be used to enable the system to be released. The last function to be completed is the development
support tool, which the authors will need to create it from scratch; a literature review will be provided.
Planning the development phase and conducting the evaluation are critical steps for the completion of thesystem.
Evaluation Phases
Through three evaluation phases, the investigators plan to evaluate the developing self-assessment tools for six
types of users.
The first phase is to determine if, indeed, the system works as an independent tool that scaffolds the users’ use;
the system was designed for users to use without the assistance of a another person. The observation method
will be used in this phase. The learners’ level of metacognition and prior knowledge have been proven to be
critical factors in hypermedia-assisted learning (HAL) (Shapiro 2007); we need to establish if the support
system adequately works for all types of users.
In the second phase, the investigators evaluate whether all types of users are able to receive the same analyticalresults when they use the same learning materials as an assessment target; this is to verify the reliability.

Having several types of users use the system, the analytical results, provided by the system, reveals if thevarious types of users get the same results.
The third phase is to evaluate the degree to which the system contributes to the users’ motivation for learning
the design based on GBS. The investigators plan to use the ARCS model that categorizes into four
motivations, namely, Attention, Relevance, Confidence, and Satisfaction (Keller & Suzuki, 1988). We expect
users to obtain the knowledge and skills about GBS in the learning design and have confidence in it throughout
their use of the system.
Observation is the method used in the first phase; the method in the second phase is comparing the results thatthe users obtain; and, the questionnaire survey is the method in the third phase.
Participants of the Evaluations
For the evaluation, the authors set six types of users, anticipating them to have knowledge about GBSs as well
as experience in learning designs. The investigators categorized each classification into three ranks, and chose
to use six types of users (Tab. 1). Although the maximum number of types of users is nine, six of the nine types
are practical (e.g., none knows the GBS well unless the person has advanced experiences in learning design).
Therefore, six types of users will participate in one-on-one evaluation.

Knowledge about GBSs
Knowledgeable in GBS
Knowing Basic
information of GBS
Novice in GBS

Experience in learning designs.
Elementary
Advanced Learning
Intermediate
Learning Design
DesignLearning Design

○
○
○

×
○
○

×
×
○

Note: ○ is the target of the evaluation

Table 1: A matrix of users for one-on-one evaluation
Regarding the first classification, the knowledge level of the GBS, the authors will create a scale with several
journals about GBS (Tab. 1). The second classification, the experience in learning design (Tab. 2) will beassessed by referring to the ID competency advocated by ibstpi (2006).
Knowledge Level
Knowledgeable in GBS

Description
The person will be able to explain the GBS’s seven components
with an example.
Knowing Basic information of GBS
The person can explain the basic idea of GBS, but cannot
describe the seven components precisely.
Novice in GBS
The person does not know what GBS is.
Table 2: Knowledge Level of GBS for the Participants of the Evaluation
Experience
Description
Advanced Learning Design
Over 3 years experience
Intermediate Learning Design
1 to 3 years experience
Elementary Learning Design
Less than 1 year experience
Table 3: Experience in Learning DesignReferences
ibstpi (2006). Competencies: Instructional Design. [Available online] http://www.ibstpi.org/Competencies/
instruct_design_competencies.htm. Extracted on May 1, 2008.
Jacobson, J. M. (2008). A Design Framework for Educational Hypermedia Systems: Theory, Research, andLearning Emerging Scientific Conceptual Perspectives. Educational technology research and development,
56, 5-28.
Keller, J. M., & Suzuki, K. (1988). Use of the ARCS motivation modeling courseware design. In D. H. Jonassen(Ed.), Instructional designs for microcomputer courseware. Lawrence Erlbaum Associates, USA, Chapter
16.
Nemoto, J., & Suzuki, K. (2004). GBS checklist for training application. A paper presented at theInternational Symposium and Conference on Educational Media in Schools, Kansai University, Osaka,
August 3-4, 2004 (Proceedings, 75 - 82)
Nemoto, J., & Suzuki, K. (2005). A checklist development for Goal-Based Scenario fitness. [Goal-based
Scenario (GBS) Riron no tekioudo checklist no kaihatu] Journal of the Japan Educational Society, 29(3),309-318, 2005
Shapiro, M. A. (2008). Hypermedia Design as Learner Scaffolding. Educational technology research anddevelopment, 56, 29-44.
Schank, R. C. (Ed.) (1998). Inside multi-media case based instruction. Mahwah, NJ: Lawrence
Erlbaum Associates.
Schank, R. C. (1996). Goal-Based Scenarios: Case-Based Reasoning Meet Leaning by Learningby doing. In D. Leake (Ed.), Case-Based Reasoning: Experience, Lessons & Future Directions.
AAAI Press/The MIT Press.
Schank, R. C. (1994). What We Learn When We Learn by Doing. Technical Report #60, Evanston, IL: The
Institute for the Learning Sciences, Northwestern University.
Schank, R. C., Berman, T. R., & Macpherson, K. A. (1999). Learning by Doing. In Reigeluth, C. M. (ed),Instructional-Design Theories and Models: A New Paradigm of Instructional Theory VolumeⅡ.
Mahwah, NJ: Lawrence Erlbaum Associates.
Schank, R. C. & Cleary, C. (1995). Engines for education. Mahwah, NJ: Lawrence Erlbaum Assoceitates.
[Available online: http://engines4ed.org/hyperbook/index.html, Retrieved on April 2, 2008]