Paper for concurrent session

ICoME 2017

Three-Stage Design of a Question-Making Activity for Preparation in a
Blended Training Program

Kei AMANO
keiamano@kumamoto-u.ac.jp
Doctoral student / Assistant Professor
Graduate School of Instructional Systems /Center for Policy Studies,
Kumamoto University
Kumamoto, JAPANKatsuaki SUZUKI
ksuzuki@kumamoto-u.ac.jp
Professor
Research Center for Instructional Systems,
Kumamoto University
Kumamoto, JAPAN

Shigeki TSUZUKU
tsuzuku@kumamoto-u.ac.jp
Professor
Research Center for Instructional Systems
Kumamoto University
Kumamoto, JAPANNaoshi HIRAOKA
naoshi@kumamoto-u.ac.jp
Associate Professor
Research Center for Instructional Systems,
Kumamoto University
Kumamoto, JAPAN

Abstract​ ​In this paper, we discuss improvements made and their effect on the design ofpre-training preparation in a blended instructional design (ID) workshop. Through practiceresearch, the design was refined to comprise three stages of a question-making activity for
pre-training preparation: 1) submitting questions during the application process for the ID
workshop, 2) completing an assignment on how to use the ID model in participants’educational situations, 3) reflecting on the questions submitted during the first phase andresubmitting questions at the end of the pre-training preparation. As a result of the effectverification, this design is shown to encourage motivation for learning in such areas asparticipants’ confidence, satisfaction, and familiarity with learning content.
Keywords: Question-making activity, Pre-training preparation, Blended learning

BackgroundKumamoto University has held a blended training program of instructional design called“Introductory Class of Instructional Design (ID workshop)” since 2015 as a part of lifelong learning
activities. The ID workshop consisted of a preparation phase of one month (online), a face-to-face,
one-day program, and one month of a post-learning (online) phase.
In order to meet the needs each participant had, we introduced a task to create questions forlecturers as part of the application process for the ID workshop as a pre-training preparation. Then, in theface-to-face workshop, the lecturer answered the questions that had been submitted by the participants.
This Q&A activity received positive evaluation from participants as they were motivated to learn (Amano1

Paper for concurrent session

ICoME 2017

et al., 2016). However, there were some challenges in designing this aspect of the ID workshop. First,
although the ID workshop consisted mainly of this Q&A activity, the percentage of those who wrote
questions at the time of the application for the ID workshop dropped from 68% in 2015 to 49% in 2016.
In addition, only 56% of the participants submitted the entire assignment, which was required aspreparation for the face-to-face workshop in 2015. Thus, we needed to improve the design and support forpreparation before the face-to-face workshop so as to encourage participants’ motivation to learn in theworkshop.
In this paper, we describe improvements made to the pre-training preparation using design-based
research (based on Reeves, 2006). To solve the problems in the ID workshop, we developed three stagesof the question-making activity for preparation in the blended workshop and improved the support forparticipants when it comes to the assignment in the pre-training preparation. We further discuss whether
the problem we faced in the ID workshop was solved and what improvements were made with regards tolearning motivation.

Methods
Context
Blended ID workshops – workshops that combine face-to-face seminars with online andat-home preparations - were held in several districts both in 2015 (Tokyo, Osaka, Nagoya, Fukuoka) and2016 (Tokyo, Osaka, Nagoya, Fukuoka, Kumamoto). Although the places of workshops varied, eachworkshop followed the same program and all participants took part in and discussed in the samee-learning course. The goal of this workshop was to enable participants to analyze education cases andpresent proposals for their improvement using an ID model.

Design Based ResearchThis study employed a design-based research (DBR) approach. The purpose of design-based
research is to develop an optimal, research-based solution for the problem in a given context (Reeves,1999). The process of design-based research includes multiple cycles of development, testing, andrefinement (Reeves, 2006). Based on this process, we saw improvement in the ID workshop from 2015 to2016. As described above, the problem we faced was that there was incomplete participation in thesubmission of the assignment during the pre-training preparation for the 2015 ID workshop and thequestioning period in 2016, although the ID workshop consisted mainly of this Q&A activity. Our2Paper for concurrent session

ICoME 2017

purpose of using DBR was to solve these problems and to refine the design of the pre-training preparationin the blended workshop.

Development of Solutions
Improvement of Design in the Pre-training Preparation
Although the face-to-face workshop was composed based on answers from the lecturer to thequestions submitted during the application for the ID workshop, participants did not submit manyquestions in 2016. As Miyake and Norman (1979) pointed out, one cannot form a question unless hesufficiently understands the learning content; this problem seemed to be caused by a lack ofunderstanding about the content. With this in mind, we decided to add one more question-making activity
after testing and reporting the assignment, and thus redesigned the activity to include three stages whereapplication practices of knowledge are interposed by question making like hamburgers (Figure 1). This
activity was as follows: 1) Participants were required to review the syllabus of the training program andsubmit questions. 2) They were then required to read papers about ID and, to test their understanding of
the concept of the ID model, post a short report about how to use the ID model for the improvement of
their particular educational situations, which also let participants clarify their needs. 3) Participants wererequired to review the questions they submitted and determine whether or not they received an answer for
it during phase 1) and 2). Based on this reflection, participants were required to resubmit questions theywould like to ask to lecturer. By adapting this process, our goal was to let participants feel the relevanceof learning using the blended training program, and to use their training in their own educationalsituations. This further increased their confidence in their own learning by allowing them to recognize
how their questions had changed since the time of application, and thus realizing growth.

3Paper for concurrent session

ICoME 2017

Figure 1. Improvements to the three stages of question-making activities

Improvement of Participants’ Support in the Pre-training Preparation
The problem we faced was that only 56% of the participants submitted the entire assignment
required as preparation for the face-to-face workshop in 2015. According to the evaluative questionnaire,
many participants did not submit the entire assignment because some of them were embarrassed toregister an e-learning account and did not realize that this was required preparation for the face-to-faceworkshop. For learners unfamiliar with e-learning, it is important to promote dialogue between theteaching side and the learning side by using feedback and messages of support, and structure the design ofthe training program by showing the participants the learning objectives and learning strategies (Moore,1993). Based on this idea, we improved the design and support of the pre-training preparation andverified its effect. Before the workshop, the support we provided to participants as they prepared for theworkshop using e-learning was as follows: 1) preparing the participant for self-registration on Moodle
with whichever social networking account each participant had, 2) sending a reminder letter to informparticipants about the task deadline, and 3) providing some readings that explained the ID model. To
make this process smoother and to clarify preparation requests for participants, we modified these steps
and added some supporting materials (Table 1).
Table 1. Improvement of increased support for participants
conducting pre-training preparation using e-learning
Type ofImprovement

ModifyItems

AimImprovement content

TeachingMaterials

To provide examples ofID application and makeit easy to understand how
to utilize the ID model

Registrationofe-learning
account

To prevent confusion ofparticipants who are notaccustomed to e-learning
To increase motivation
for preparation

RemindermailAdd

To let participantsconfirm their learning
progress4

We changed the teaching materials from a readingsthat explained the ID model that was aimed atresearchers to ​excerpts of books that described
ID models along with cases that were written for
beginners​.
We changed the method of account registrationfrom the self-registration by the participant throughan existing social networking account to the way
the operator issues ID and password.
・In the message of the e-mail, we described ​theobjectives of learning, the method of evaluation,and the structure of the program ​and explained
why the pre-training preparation is necessary​.
・We stated that ​questions submitted by thedeadline would be addressed during theface-to-face workshop​.
We sent individual e-mails to students who werebehind on submitting assignments to urge them toparticipate.

Paper for concurrent session

ICoME 2017

To prevent confusion ofparticipants who are notaccustomed to e-learning

Supportingmaterial

We distributed a .pdf file in advance that explained
how to prepare, step-by-step, ​with e-learning
screenshots.

Testing Solutions in PracticeCompletion Rates and Evaluations of Impressions of the Entire ID Workshop
We examined the completion rates and evaluations of impressions using the ARCS model(Keller, 2010) in the 2015-2016 ID workshop. The completion rates were 33% (63 out of 179 participantssubmitted questions) in 2015 and 71% (144 out of 200 participants submitted questions) in 2016.
Evaluations of impressions for the blended workshop were conducted after the face-to-face workshop.
There were seven levels for the evaluation, with 1 being negative and 7 being positive for each item of theARCS model. The results are shown in Table 2. There was no significant difference in the mean of all
items (Attention; ​t ​(333.35) = 0.5645, ​n.s.​, Relevance; ​t​ (321.92) = 0.8572,​ n.s.​, Confidence; ​t ​(353.46) =0.7285, ​n.s​., Satisfaction; ​t​ (367.67) = 0.2503, ​n.s.​). Although the evaluations of impressions for theblended training program did not change from 2015 to 2016, the completion rates improved. Consideringthat the pre-training preparation load was increased in 2016, it can be inferred that the blended training
was improved as a whole. Of course, this effect cannot be considered to be solely due to the improvementof the pre-training preparations; other factors would contribute to it.
Table 2. Evaluations showing the average impressions for the blended workshop
Year
2015

Attention
6.30

Relevance
6.05

Confidence
5.01

Satisfaction
6.36(N=169)(0.93)(1.00)(1.10)(0.93)

2016

6.35

6.13

5.10

6.33

(N=201)(0.80)(0.81)(1.07)(1.13)※​​Numerical values ​are an average and the numbers within the parentheses in the lower row indicate standard deviation.

Comparison of Changes after Improvement of the Design and Support
To examine the influence of the improvement of the design of the question-making activity in thepre-training preparation, we compared the number of questions submitted at the time of application in2015 to those submitted in 2016, as well as to questions submitted at the end of the pre-training
preparation in 2016. Figure 2 shows the difference. Although the percentage of those who submitted
questions during the application process for participation in the ID workshop dropped from 68% (119 outof 174 participants) in 2015 to 49% (98 out of 201 participants) in 2016, it then increased to 71% (143 outof 201 participants) by adding one more question-making activity.

5Paper for concurrent session

ICoME 2017Moreover, the submission rates of the pre-training assignment were increased from 56% (95 out of169 participants) in 2015 to 90% (181 out of 201 participants) in 2016. The e-learning records illustrated
that almost all of the participants registered an account (unfortunately, one participant was unable toreceive the e-mail about the pre-training preparation, so she was not able to register her account). It wasalso confirmed that participants understood the necessity of the pre-training preparation, as one
participant commented, “In the beginning, I was worried about a workshop that has so many assignments,
but I was able to fully understand that it is a necessary step to learn,” in the questionnaire after theblended workshop. The ongoing problem of submission rates of the pre-training assignment might besolved by further improvement of the support in the pre-training preparation.

※​
1: 20 out of 57 participants did not submit the assignment.
※​
2: The participants who did not submit questions included comments that did not clearly ask questions, such as “Please take care of us.”Figure 2. Comparison of submitted questions

Final Report: Advantages of Using Three Stages for the Question-Making Activity
In the final report, which showed how participants critically analyzed the ID workshop using theARCS model, some participants pointed out that the added question-making activity motivated them tolearn. One participant described that this activity made him feel confident and satisfied as follows: “Asfor me, reviewing the question at the time of the application was where I learned the most. At the time ofthe application, I did not understand the content enough to ask questions, so I could only write a comment
about my passion for participating in the workshop. However, with regard to the contents learned throughprior-learning, it became clear that my understanding had advanced enough to ask questions to thelecturer.” Another participant described how thinking about the questions made him feel more familiar
with the learning content and its relevance as follows: “By reconsidering my own question, my awarenessof participation in the ID workshop increased. At the same time, I gained familiarity with the content.”

6

Paper for concurrent session

ICoME 2017Thus, it appears that the design improvement of adding questions encouraged motivation for learning byway of such means as participants’ confidence, satisfaction, and familiarity with the learning content.

Concluding Remarks
In this paper, we discussed design improvement to the pre-training preparation of the workshop
and its effect on participants. We were able to improve the pre-training preparation and refine the design
to include three stages of the question-making activity.
In educational practice research, researchers cannot control the research fields or the participants;
all we can do is make changes over time to improve upon our work. The problem we faced with our 2016
workshop was mitigated with some pre-workshop preparation support. Our future work is to continue
educational improvement research to refine the design of the ID workshop to solve any problems that wewill face in the future.

References
Amano, K., Morita, A., Tsuzuku, S., Suzuki, K., & Hiraoka, N. (2016). Design and evaluation of theflipped classroom using learner's past experience and preexisting knowledge, ​JSiSE ResearchReport​, 30(7), 21-28. [In Japanese]
Keller, J. M. (2010). Motivational design for learning and performance: The ARCS model approach. New
York: Springer.
Miyake, N. & Norman, D. A. (1979). To ask a question, one must know enough to know what is notknown. ​Journal of Verbal Learning and Verbal Behavior, 18​(3), 357-364.
Moore, M. G. (1993). Theory of transactional distance. In D. Keegan (Ed.), ​Theoretical principles ofdistance education​ (pp. 22–38). New York: Routledge.
Reeves, T. C. (1999). A research agenda for interactive learning in the new millennium. ​Proceedings ofEdMedia 1999​, 15-20.
Reeves, T. C. (2006). Design research from a technology perspective. In J. van den Akker, K.
Gravemeijer, S. McKenney & N. Nieveen (Eds.), ​Educational Design Research (pp. 52-66).
London: Routledge.

7