How Did Learners Use a Digital Badge as a Reflection Tool after
Completion of an Educational Program?

Kei AMANO
Kumamoto University
keiamano@kumamoto-u.ac.jp

Toshihiro KITA
Kumamoto University
kita@rcis.kumamoto-u.ac.jp

Katsuaki SUZUKI
Kumamoto University
ksuzuki@kumamoto-u.ac.jp

Naoshi HIRAOKA
Kumamoto University
naoshi@kumamoto-u.ac.jp

Shigeki TSUZUKUKumamoto University
tsuzuku@kumamoto-u.ac.jp

Abstract: A digital badge is a symbol not only of completion rewards but also of mastery associated with alearning process and its outcomes. Based on this idea, we designed a digital badge as a learning portfolio
linked to an instructional design workshop. This workshop followed a blended format and was offered as anopen seminar by Kumamoto University. The current investigation reports the results of a follow-up survey
conducted six months after the completion of the workshop, to verify the application and perceived usefulnessof the digital badge and the timing of browsing it. Results indicate usages of the digital badge to accesslearning outcomes and feedback, in addition to its utility as a learning reflection tool for participants. It isalso suggested that the digital badge as a learning portfolio might facilitate participants to apply theirlearning outcomes in the workplace. Other findings indicate that in order to allow novice learners to reflect
on their learning outcomes, it might be practical to combine the use of the digital badge that portably gathers
meaningful information about learning with opportunities that increase the timing of reflection such as
follow-up surveys. Implications and future directions of the study are discussed.
Keywords: Adult learning, Blended learning, Digital badge, Instructional design, Reflection

INTRODUCTION
Supporting learners’ reflection on their learning in
an educational program is an important component of
instruction. This reflection activity is an element of“First Principles of Instruction” identified by Merrill(2009) and is expected to integrate the learning
experience in an educational program with real-lifesituations where the learning outcomes might be used.
For working people, reflecting on learning outcomes
after training is considered crucial because they are
required to apply their learning to their job to recover
their investment in education. However, self-reflection
is difficult for novice learners or those who are
unfamiliar with reflection activities (Willis et al., 2016,p. 27). The problem with introducing a refection
activity for novices is that it is difficult to create ameaningful learning portfolio considering their lack ofself-reflection skills. Although systems supporting
learners’ self-reflection, such as e-portfolio, have been
proposed, using these systems effectively requires
teaching self-reflection skills. Because this skill is akind of metacognitive strategy (Gagne et al., 2005) that
focuses on the adoption of effective ways based onlearners’ extensive learning experiences, it takes a lotof time to master it. Therefore, the challenge is indesigning mechanisms to facilitate learners’ reflection
on their learning outcomes in a short-term program.
In the previous study, the researchers designed adigital badge to help learners create meaningfullearning portfolio (Amano et al., 2017). This digitalbadge was designed to display the accomplishment oflearning objectives with evidence such as online reportassignments and an asynchronous discussion record offorum posts made during a blended educational
program. By introducing the digital badge, we wereable to improve the certification of program
completion from seat-time based to mastery based.
This enabled us to check that learners reached theirlearning objectives at the end of the program and toensure the quality of the program. Further, the digitalbadge that could accumulate the program’s learning
outcomes was expected to be a useful tool for novices
who were unfamiliar with self-reflection and for
creating a learning portfolio.
A challenge that remained was that we did notverify the application and usefulness of the digitalbadge as a reflection tool. Therefore, the purpose ofthis study was to investigate through a follow-up
survey how and when participants use the digital badge
after completion of an educational program.

DESIGN OF FOLLOW-UP SURVEY USINGTHE DIGITAL BADGE
Overview of instructional design courses
The research field was instructional design (ID)courses that were a part of the university extension
courses held by Kumamoto University. The purpose of
these courses are as follows: 1) to allow participants togain basic knowledge of ID and to enhance theireducation from the viewpoint of effectiveness,
efficiency, and attraction; and 2) to enable participants
to apply ID in educational tasks related to their jobs andpresent proposals for improving their tasks. Table 1
shows the learning objectives and evaluation methods.
To allow participants to learn practical skills that might
be useful in improving their educational tasks, thelearning objectives were set as intellectual skills
focused not on memorization of knowledge but rather
on its application. In the post-learning assignment,
concrete ideas to improve educational cases and actionplans were included to facilitate participants’application of their learning outcomes to their jobs.
Thus far, a broad variety of professionals from several
fields, such as university lecturers, medical doctors,
nurses, and Japanese language teachers, have attended
these ID courses. These courses are composed of two
courses: an introductory course and a practical course.
In the introductory course, participants acquire basic
ID skills and consider improvement proposals foreducational cases presented by lecturers. In thepractical course, the goal is to present improvement
ideas for the educational cases that participants
engaged in based on the skills they acquired in the
introductory course. Both courses consist of priorlearning activities (online), a face-to-face program(one day), and post-learning activities (online). The
online phases served to allow participants toeffectively utilize the one day of the face-to-faceprogram. These online activities were required forsubmitting coursework items, and they were thus aprerequisite for evaluation to allow participants toselect the best learning path. Finally, participants
would acquire a digital badge if they met all theseevaluation criteria and prerequisites for evaluation oneach course of introductory and practical(Figure 1).
The digital badges had different purposes in eachcourse. In the introductory course, the digital badge
represented participants’ mastery of basic ID skills.
This badge was a ticket allowing attendance of thepractical course. After the practical course, the digitalbadge meant completion of both ID courses. In thisway, the courses were sequenced to let participants toachieve each learning objective in a step-by-stepprocess. The digital badge for the practical course
included the learning outcomes, which contained ideas
for improving the educational cases in which eachparticipant engaged. It was expected that participants

would use the digital badge from the practical coursein their occupations after completing both courses.
The learning management system (LMS) used for
e-learning courses was Moodle. We developed theonline assignment and digital badge using the default
features of Moodle.
Table 1. Post-learning assignment of the ID course
Introductory course
Learning objectives
Demonstrate how to use basic ID skills toimprove educational cases

Identify the problems of educational cases andselect appropriate solutions for it based on the
ARCS modelEvaluation methods and criteria
To score 80% or more in check testSubmit the final report and score 80% or more
based on the grading criteria. Based on thelearning in the course, summarize thefollowing items:
1) Analysis and improvement proposal of ID
course participants2) Action plans to use learning outcomes3) Three things that participants learned mostin the courseComments on other participants’ final reportson the discussion forum in Moodle
Practical course
Learning objectives
Propose ideas for improving educational casesOne or more of the ID tools can be applied topresent suggestions to improve participants’educational cases

Evaluation methods and criteria
Submit the final report and score 80% or more
based on the grading criteria. Based on thelearning in the course, summarize thefollowing items:
1) What participants want to continue in theireducational case2) What participants want to improve in theireducational case3) Three things that participants learned mostin the course
4) Action plans to use learning outcomes5) Appendix that includes an analysis of theireducational casesComments on other participants’ final reportson the discussion forum in Moodle

Design of follow-up survey using the digital
badge
Figure 2 shows the blueprint of the design of followup survey using the digital badge which was acquired

in the practical course. The follow-up survey tofacilitate participants’ reflection consisted of twophases: the issuance of the digital badge two or three
weeks after participants submitted the post-learningassignment, and the follow-up survey to checkparticipants’ progress six months after submission of
the post-learning assignment.
First, participants who met the passing criteria were
issued a digital badge that was linked to their learning
outcomes and displayed feedback from lectures. This
first trigger aimed to show participants that they had
achieved the learning outcomes. This also aimed to
inform participants about how they could improve theirjobs in the new fiscal year.
The second trigger to allow participants to reflect
upon their learning outcomes was conducted as afollow-up survey six months after completion of the ID

courses. Previous findings suggest that a follow-up not
only helps evaluate a training program but also
produces a “wake-up” effect (Nemoto et al., 2005). In
other words, a follow-up survey could let participants
recall their impressions about how the program wasconducted, what they learned, and how they applied
their learning in their current jobs. This suggests that afollow-up survey might trigger participants’ reflection
on their learning in an educational program and its
application to their jobs. Based on this idea, we
designed a follow-up survey using the digital badge.
We expected that the digital badge played the role of a
support tool for participants’ accumulation of learning
outcomes. The follow-up survey was also expected tofacilitate learners in browsing the digital badge andreflecting on how their learning could be connected totheir jobs.
CourseCompletion

To meet the criteria
to pass.
Feedback on learning
outcomes fromlecturers

Links to prerequisites forevaluation and artifacts

Practice using
basic skills

Links to prerequisites
for evaluation
Insisting onlearning outcomes
Links to prerequisites forevaluation and artifacts

Links to prerequisites
for evaluation

・The digital badges were issued only to participants
who met the passing criteria in each course.
・The digital badges were issued to participants twoor three weeks after submission of the post-learningassignment.

Figure 1. Mastery-based design of the ID courses

Participants wererequired to submit
their post-learningassignment forcompletion of the
program.

Participants perceive what
they have learned andrecall what they should doin the new fiscal year bybrowsing the learning
outcomes and feedback
from lectures linked to thedigital badge.

Participants recall theirlearning and perceive
their progress after
completion of the ID
courses by browsing thedigital badge once again.

Figure 2. Follow-up design to facilitate participant reflection upon their learning outcomes
after completion of the ID courses

RESULTS AND DISCUSSION
The follow-up survey was conducted in August
2017, six months after completing the ID courses. We
used the questionnaire module of Moodle for thisevaluation. Of the 72 participants who completed the
ID courses, 57 (79%) responded to the questionnaire.
In the follow-up survey, we confirmed how
participants conducted activities using the digitalbadge which was acquired in the practical course after
completing the ID courses by answering the following
questions: How do participants use the digital badge
and perceive the usefulness of it for learning and for
improving their jobs? and When do participants
browse their digital badge?

Application and perceived usefulness of thedigital badge
The frequency of browsing the digital badge after
the completion of the ID courses is presented in Figure
3. Six months after the completion of the ID courses,
more than half of all respondents browsed the digitalbadge only once. The second most were those who
browsed the digital badge once in two months. These
results show that not many participants browsed thedigital badge frequently.
Further, we investigated the purpose of browsing
the digital badge (Table 2). Of the 57 respondents, 36(63%) had some purpose in using the digital badge,
whereas 21 respondents (37%) did not have any
purpose. Regarding the respondents who had some
purpose, there was a tendency to use the digital badge
for their own reflection on learning and to increase
their motivation: for example, “I looked back at mydigital badge not to forget what I learned in the courses,”and “I looked back at my digital badge to be more
motivated to improve my educational cases.”Compared to this, there were less number ofparticipants who showed their digital badge to others:“I showed my digital badge to my friends to ask themto learn ID,” and “I showed my digital badge to otherparticipants to deepen my knowledge.” These results
suggest that participants tend to use their badges fortheir own advantage rather than to share learning
outcomes with others.
Supporting participants in facilitating application ofthem to their jobs and to reflect on their learning
outcomes is the primary purpose of letting participants
use the digital badge after the completion of their ID
courses. In both aspects, more than 60% of respondents
showed a positive reaction (Figure 4). Respondents
were also required to write the reason for their answer.
The reasons why they responded that the digital badge
is helpful to improve their educational cases are as
follows: “The comments and feedback given in thedigital badge pushed me to improve my work,” “Thedigital badge stimulated my confidence and motivation
for learning and helped me to maintain mycommitment to improving my work,” “Based on theinformation attached to the digital badge, such as thelearning outcomes and feedback from lecturers, I
thought about ideas to improve my educational cases.
It also sometimes helped to increase my motivation forthese actions.” These comments suggest that thelearning portfolio which is attached to the digital badge
played a motivating role for some participants andhelped them improve their educational cases.
From the perspective of the usefulness of the digitalbadge for learning after the completion of the ID
courses, those who responded positively commented asfollows: “If there was no digital badge, I would havesurely been in trouble. Although I usually make myown notes, the digital badge is more convenient to use
because the information is stored online, which is
better than using a notebook. By browsing the digitalbadge, I can remember what I was thinking, which I
sometimes forget,” “The digital badge helped me toconsistently review my learning from pre-assignment,
face-to-face workshop, to post-assignment of the ID
courses,” “I was able to reflect on my learning in thecourses from the comments and links attached to thedigital badge.” These comments suggest that the digitalbadge can help to conveniently access learning
outcomes and feedback that were perceived useful andto reflect on their learning.

Figure 3. Frequency of browsing the digital badge after the completion of the ID courses

Table 2. Purpose of using the digital badge (N=36, Multiple answers allowed)
Purpose of using the digital badge*1

N%*2

I looked back at my digital badge so not to forget what I learned in the courses

17

47

I looked back at my digital badge to be more motivated to improve my educational cases

1131

I looked back at my digital badge to review the progress of educational improvement

6

17

I showed my digital badge to my colleagues or business partner to show them my skills

6

17

I browsed other participants’ digital badges to get ideas for educational improvement

4

11

I showed my digital badge to my colleagues to encourage them to attend the ID courses

4

11

I introduced a digital badge into my educational program3

8

I showed my digital badge to my friends to ask them to learn ID

1

3

I showed my digital badge to other participants to deepen my knowledge

1

3

Others514%※Note: *1 The reasons were ordered based on the number of respondents.
*2 The percentage of respondents who selected each purpose was calculated out of 36 respondents who responded thatthey used the digital badge for some purpose.

Figure 4. Perceived usefulness of the digital badge after the completion of the ID courses

Timing of digital badge browsing
In this follow-up survey, we asked respondents toanswer the questionnaire by browsing the digital badge.
Because the digital badge accumulated learning
outcomes, it might be a helpful tool to makeparticipants reflect on their learning. The digital badge
was linked to an “activity report” page that is
implemented by default in Moodle and accumulates alllearning outcomes of each participant.
Access logs of participants’ activity report page areshown in Figure 5. In 2015, 57 participants acquired
the digital badge, and 72 acquired it in 2016. Comparedto the results of the ID courses in 2015, which did not
implement a follow-up survey, participants’ access totheir own activity report page increased six months
after the completion of the ID courses in 2016 when thefollow-up survey was held, although the number ofparticipants who acquire the digital badge also
increased. This result indicates that the follow-up

survey itself could be a key trigger that makes
participants browse the digital badge and reflect on
their learning outcomes.
Furthermore, some respondents mentioned theeffectiveness of the follow-up survey in the optional
free description field of the questionnaire: “I reviewed
my action plan after a long interval. I recalled myfeelings when I submitted the final report. I realizedthat the follow-up survey is necessary. Thank you forreminding me of that moment,” and “Although open
seminars and corporate training tend to be over forgood once they are completed, my motivation has
increased thanks to these ID courses, because they
include a follow-up survey. It was a significant resultthat I was able to perceive my outcomes of educational
practices.” From these comments of some participants,
it seems that the follow-up survey produced a “wakeup” effect as suggested by previous findings (Nemoto
et al., 2005). Further, it was confirmed that participants
browse their learning outcomes and reflect on theirprogress. This might have happened because the online
and portable form of the digital badge allowed more
space for storing participants’ learning outcomes. Ifthey did not have the digital badge, they would have
had to depend on their memory or some
portfolio/notebook they might have created. This
seems to make it difficult for novice learners to reflect
on their learning outcomes. As 53 out of 57
respondents (93%) browsed the digital badge, manyN※

participants use the digital badge for checking theirlearning outcomes at the follow-up survey. Thesefindings suggest that lecturers can use follow-up
surveys to facilitate reflection by participants.
Moreover, the digital badge helps participants toconveniently reflect on their learning outcomes. In
short, follow-up activities using the digital badge act
like “training wheels” for participants’ reflection after
the ID courses.

Implementation of the follow-up
surveyIssuance of the digital badge

Figure 5. Number of accesses to the activity report page attached to the digital badge※Note: * N shows the number of people who browsed the learning portfolio attached to the digital badge.

CONCLUDING REMARKSACKNOWLEDGEMENTS

To verify the application and usefulness of thedigital badge as a reflection tool, the current follow-up
survey was administered six months after thecompletion of the ID courses. The results presented
here suggest that accessing learning outcomes andfeedback through the digital badge might be useful forparticipants’ reflection on their learning outcomes.
Further, participants’ comments indicate that thedigital badge can facilitate the application of theirlearning outcomes in the workplace. Moreover, accesslogs in the LMS showed that the design of the followup survey with the digital badge might trigger
participants’ reflection on their learning outcomes.
These results also suggest that in order to allow novice
learners to reflect on their learning outcomes, it might
be useful to combine the use of the digital badge thatportably gathers meaningful information about
learning with opportunities that increase the timing ofreflection, such as follow-up surveys. Regarding
specific challenges in terms of the use of the digitalbadge, this study did not analyze how participants wereable to improve their jobs qualitatively based on theinformation attached to the digital badge. The current
investigation only examined how they use and perceive
the digital badge. Further investigation should be
conducted to determine the impact of the digital badge
on qualitative changes in participants’ jobs.

This work was supported by JSPS KAKENHI Grant
Number JP17K12948.

REFERENCES
Amano, K., Suzuki, K., Tsuzuku, S., & Hiraoka, N. (2017).
Designing a digital badge as a reflection tool inblended workshops. The Journal of Information and
Systems in Education, 16 (1): 12-17.
Gagne, R. M., Wager, W. W., Golas, K. C., & Keller, J. M.
(2005). Principles of instructional design (5th Ed.).
Wadsworth.
Merrill, M. D. (2009). First principles of instruction. In C.
M. Reigeluth & A. Carr (Eds.). Instructional design
theories and models: Building a common knowledge
base (Vol. III). New York: Routledge Publishers.
Nemoto, J., Hazelman, V. & Suzuki, K. (2005).
Instructional design workshop based on needs analysis
at University of South Pacific. In P. Kommers & G.
Richards (Eds.). Proceedings of ED-MEDIA 2005-World Conference on Educational Multimedia,
Hypermedia & Telecommunications: 268-275
Willis, J. E., Flintoff K., & McGraw. B. (2016). A
philosophy of open badges (Part 2). In Ifenthaler, D.,
Bellin-Mularski, N., & Mah, D.-K. (Eds.). Foundationsof digital badges and micro-credentials.