International Journal for Educational Media and Technology
2019, Vol.13, No. 1, pp.95-103

Reflection Support for Novice Learners:
Combining Digital Badges with Follow-Up Surveys
Kei Amano
Kumamoto University, Japan
keiamano@kumamoto-u.ac.jp
Shigeki Tsuzuku
Kumamoto University, Japantsuzuku@kumamoto-u.ac.jp
Katsuaki Suzuki
Kumamoto University, Japanksuzuki@kumamoto-u.ac.jp
Naoshi Hiraoka
Kumamoto University, Japannaoshi@kumamoto-u.ac.jp

A digital badge symbolizes not only the completion but also the mastery of a learning process and its outcomes.
Previously, we had designed a digital badge to serve as a learning portfolio linked to an instructional design workshop.
In this study, we created a follow-up activity in which we combined the digital badge with a survey aimed to support
and trigger learners’ reflection on learning outcomes and verified its effects. Results highlighted the use of digital badges
to access course knowledge and feedback and as a learning reflection tool. Furthermore, we observed that digital badges
as a learning portfolio might encourage learners to apply their learned skills in the workplace. To facilitate novice
learners’ reflection on their learning outcomes, combining digital badges that provide easily accessible, meaningfullearning information with follow-up strategies that stimulate reflection could be practical. Implications and future
directions are also discussed.

Keywords: Blended Workshops, Digital Badge, Follow-up Activity, Reflection, University Extension
CourseIntroduction
Supporting learners’ reflection on their learning in an educational program is an important component of
instruction. The reflection activity is an element of the “First Principles of Instruction,” identified by Merrill (2009),
and is expected to integrate the learning experience in an educational program with real-life situations wherein thelearning outcomes might be used. The act of reflecting on learned knowledge after training is considered crucial for
employees because they need to apply their learning to their job to recover their educational investment. However,self-reflection is difficult for novice learners or those who are unfamiliar with reflection activities (Willis, Flintoff, &McGraw, 2016). The problem with introducing a reflection activity to novices is that it is difficult for them to createa meaningful learning portfolio considering their lack of self-reflection skills. Although systems supporting learners’
self-reflection, such as e-portfolios, have been proposed (Barrett, 2010), their effective usage requires self-reflection
skills. However, because this skill is a type of metacognitive strategy (Gagne, Wager, Golas, & Keller, 2005) that
focuses on the adoption of effective methods based on learners’ extensive learning experiences, its mastery is timeconsuming. Therefore, the challenge is to design mechanisms to facilitate learners’ reflection on their learning
outcomes that can be used in short-term programs.
In a previous study, Amano, Suzuki, Tsuzuku, and Hiraoka (2017) designed a digital badge to help learners create ameaningful learning portfolio. The badge was designed to display the accomplishment of learning objectives withevidence such as online report assignments and an asynchronous record of discussion forum posts created during a
blended educational program. By introducing the digital badge, the researchers modified the program completion
certification from seat-time-based to mastery-based, allowing them to verify whether the learners reached theirlearning objectives at the end of the program and ensure the program’s quality. Further, the digital badge, whichcollected the program’s content into one virtual space, was a useful tool for creating a learning portfolio, especially
IJEMT, Vol. 13, No. 1, 2019, pp.95-103, ISSN 1882–2290

95

International Journal for Educational Media and Technology
2019, Vol.13, No. 1, pp.95-103

for novices unfamiliar with self-reflection.
However, the researchers did not clarify how an educational provider could use a digital badge as part of a learning
portfolio to support participants’ activity after completing an educational program. In other words, they did not
discuss how a digital badge could be used to ensure the education was effective, efficient, and engaging, although thebadge was designed to fulfill such functions. Therefore, the purpose of this study was to design a follow-up activity
that used a digital badge linked to a learning portfolio and verify the activity’s effects regarding participants’,especially novice learners’, reflection on their learning outcomes.

Literature Review
Digital Badges as Learning Portfolios
Digital badges are becoming popular as a mechanism for authenticating comprehension of online learning. Peck(2015) defined digital badges as “‘clickable’ graphics that contain metadata that can reveal information about theindividual or organization that issued the badge, the criteria met to earn the badge, the tool(s) used to assess theevidence, and the evidence of learning itself ” (p. 26). This type of digital badge is called a “micro-credential”
because it demonstrates that students mastered certain skills through an educational program and serves as analternative to a report card or diploma. Cheng, Watson, and Newby (2018) proposed not only the use of digitalbadges as alternatives for credentials but also as pedagogical tools to achieve more effective, efficient, and engaging
learning. One effective pedagogical use of the digital badge might be as a link to a learning portfolio (Amano et al.,2017; Gibson, Coleman, & Irving, 2016). The digital badge could be used for recalling and reflecting on learning
content because it allows the student to access learning materials, such as teacher and classmate feedback on
assignments and products like reports and presentation slides, after the course is completed, which can then be usedby the student to review and reinforce the skills they learned through the course (Amano et al., 2017).
The e-portfolio is a popular tool for supporting the collection of and reflection on learning outcomes, and there aremany similarities between the digital badge and e-portfolio. Regarding functional similarity, both record learning
pathways and include learning documentation, facilitating learning reflection and discussion (Buchem, 2016). The
digital badge differs in terms of autonomy (Buchem, 2016). E-portfolios require learners to create the portfolio bycollecting, organizing, and presenting digital evidence in a variety of media (Hartnell-Young et al., 2007). Therefore,
e-portfolios require advanced self-regulated learning skills. Conversely, digital badges are, by default, created byeducational institutions. In other words, digital badges represent skill acquisition without needing to be organized orpresented in a certain way by the student. This characteristic might be helpful for novice learners who have not yet
created a learning portfolio.

Follow-Up Activities after Training Programs
Brinkerhoff and Apking (2001) proposed the 40/20/40 Model, illustrating that training outcomes are not only
impacted by the training itself but also by post-training support. They also suggested that completion of the training
event itself should not be considered the training purpose but rather that participants learn the process needed tosolve problems in relevant contexts and achieve the training objectives set in advance.
Based on the importance of follow-up, previous studies have investigated how training participants’ activities can beeffectively supported after participating in educational programs, mainly in corporate settings. For example, Saks andBurke-Smalley (2012) reported that asking participants about behavior changes after training completion reminded
them about the training’s content, which subsequently triggered its implementation in the workplace. Moreover,follow-up not only helps evaluate a training program but also produces a “wake-up” effect (Nemoto, Hazelman, &Suzuki, 2005). In other words, a follow-up survey could help participants recall their impressions about how the
program was conducted, what they learned, and how they could apply their learning in their current jobs.
Interpersonal performance training in addition to classroom training, provision of workbooks for reference, andavailability of feedback opportunities on outcomes after training completion are also considered effective (Tews &Tracey, 2008). Thus, a wide variety of follow-up methods and their effects have been verified.
Kirkpatrick and Kirkpatrick (2016) classified various methods that were useful in corporate settings as follow-up
activities after training program completion (Table 1). In this classification, support, such as job aids and checklists,
represent useful tools to ensure that learners utilize what they learned during the training in their job settings.
Accountability refers to providing opportunities for taking action to those who are inactive when unsupervised.
Kirkpatrick and Kirkpatrick (2016) stated that “if something is measured, it means it is important,” and, therefore,asking learners about their outcomes might itself be a message to encourage them to reflect on what they learned in
IJEMT, Vol. 13, No. 1, 2019, pp.95-103, ISSN 1882–2290

96International Journal for Educational Media and Technology
2019, Vol.13, No. 1, pp.95-103

a course.
Based on the abovementioned findings, we designed our follow-up activity by considering both support andaccountability. First, regarding the support aspect, the digital badge linked to a learning portfolio was adopted to easilyreflect learners’ learning outcomes. Regarding the accountability aspect, we used a follow-up survey not only toprovide learners with the opportunity to reflect but also to guide them on what they should reflect on.
Table 1Required Drivers (from Table 7-2 of Kirkpatrick & Kirkpatrick, 2016)
Reinforce
Follow-up modules
Work review checklist
On-the-job training
Self-directed learning
Refreshers
Job aids
Executive modeling
Communities of practices
MonitorAction learning
Interviews
Observations
Self-monitoring
Key performance indicators

Support

Encourage
Coaching
Mentoring
RewardRecognition
Pay for performance

Accountability
Action plan monitoring
Dashboard
Work review
SurveyTouch bases/meetings

Methods
Overview of Instructional Design Courses
Our research focused on instructional design (ID) courses constituting part of the extension courses provided at
Kumamoto University. Their objectives are as follows: 1) to allow learners to gain basic knowledge of ID andenhance their job performance in terms of effectiveness, efficiency, and attractiveness of education for students;
and 2) to enable participants to apply ID in educational activities related to their jobs and present proposals for
improving their activities. Table 2 presents the learning objectives of and evaluation methods for the ID courses.
These courses comprise two sub-courses: an introductory course and a practical course. In the introductory course,
learners acquire basic ID skills and consider improvement proposals for specific educational cases presented bylecturers. In the practical course, the goal is to present ideas for improving the real-life cases of education programs
that learners are currently working on based on skills acquired in the introductory course. Both courses consist of
pre-learning activities (online), a face-to-face program (one day), and post-learning activities (online). Thus far, abroad variety of professionals from several fields, such as university lecturers, medical doctors, nurses, and Japanese
language teachers, have attended these ID courses. In this study, to allow participants to acquire practical skills that
could support their educational activities, the learning objectives were set as intellectual skills, focusing not onknowledge memorization but on its application. In the post-learning assignment, concrete ideas to improve specificeducational cases and action plans were included to facilitate participants’ application of their knowledge to theirjobs. The online phases allowed participants to effectively utilize the one day of the face-to-face program. These
online activities were required for submitting coursework items; therefore, they were a prerequisite for evaluation,
enabling participants to select the best learning path. Finally, participants would acquire a digital badge if they
fulfilled the evaluation criteria and prerequisites for evaluation in each course (Figure 1).

IJEMT, Vol. 13, No. 1, 2019, pp.95-103, ISSN 1882–2290

97

International Journal for Educational Media and Technology
2019, Vol.13, No. 1, pp.95-103

Figure 1. Mastery-based design of the instructional design courses
Table 2
Instructional Design Course Objectives and Evaluation Methods
Introductory course
Learning objectives•
Demonstrate how to use basic instructional design (ID) skills to improve education programs•
Identify the problems of specific cases of educational programs and select appropriate solutions based on the Attention,
Relevance, Confidence, Satisfaction modelEvaluation methods and criteria•
A score of 80% or more on the comprehension test•
Submit the final report and score 80% or more based on the grading criteria. Based on the course content, summarize
the following items in the final report:
1) Analysis and improvement proposal from ID course participants2) Action plans to use learning outcomes3) Three things that participants learned most in the course•
Comments on other participants’ final reports in the discussion forum in Moodle
Practical course
Learning objectives•
Propose ideas for improving educational programs related to the participant•
Present suggestions applying one or more of the ID tools to improve participants’ educational programs
Evaluation methods and criteria•
Submit the final report and score 80% or more based on the grading criteria. Based on the learning in the course,
summarize the following items in the final report:
1) What participants want to maintain in their education program2) What participants want to improve in their education program3) Three things that participants learned most in the course
4) Action plans to use learning outcomes5) Appendix that includes an analysis of their education program•
Comments on other participants’ final reports in the discussion forum in Moodle

The digital badges had different purposes in each course; in the introductory course, a digital badge represented
participants’ mastery of basic ID skills. This badge was an entry ticket to the practical course. After the practicalcourse, the digital badge indicated completion of both ID courses. Thus, the courses were sequenced in a way thatallowed participants to achieve each learning objective in a step-by-step process. The digital badge for the practicalcourse allowed participants to access the learning outcomes comprising ideas for improving the educational cases inwhich each participant was engaged. It was expected that participants would use the digital badge from the practicalcourse in their occupations after completing both courses.
Moodle was used as the learning management system (LMS) for the e-learning courses. We developed the online
assignment and digital badge using Moodle’s default features. Figure 2 provides a visual representation of uses ofthe digital badge. The digital badge was designed to help improve participants’ abilities in their workplace after
completing the program. By accessing the digital badge, participants could browse a portfolio of the skills they had
mastered, review their learning processes and outcomes, and apply their knowledge to activities, such as job
applications and advertising their educational skills to others, including their bosses and colleagues. By designing a
digital badge to constitute a skill mastery-based portfolio, we attempted to assist participants in applying theirlearning to their jobs.
IJEMT, Vol. 13, No. 1, 2019, pp.95-103, ISSN 1882–2290

98International Journal for Educational Media and Technology
2019, Vol.13, No. 1, pp.95-103

Figure 2. Visual representation of how a digital badge can be used (From Figure 4 of Amano et al., 2017)

Design of Follow-Up Activities Using a Digital Badge
A follow-up survey was conducted in August 2017, six months after the completion of the ID courses. We used
Moodle’s questionnaire module for this evaluation. Of the 72 participants who completed the ID courses, 57 (79%)
responded to the questionnaire. The follow-up survey was announced via e-mail on August 21; the email contained alink to the survey. The deadline for completing the survey was August 31.
The survey content focused on two aspects: it asked about the usefulness of the digital badge for supporting thereal-life application of knowledge (support), and it facilitated reflections on knowledge (accountability). Participants
could choose from five options for their answers: strongly disagree, disagree, neutral, agree, and strongly agree. The
participants were also asked to describe the reasons for their responses.
This survey was optional. The survey procedure was explained to participants before taking the survey.
Furthermore, there were no ethical issues related to the selection and treatment of subjects associated with thispaper. The authors had no conflicts of interest in conducting this study. Anonymous survey data were summarized
in a report, and we informed the participants that those who cooperated with the survey would have access to thereport. Through these actions, we aimed not only to achieve the study’s objective but also provide beneficial
information to participants.
Figure 3 shows the blueprint of the design of the follow-up survey using the digital badge acquired in the practicalcourse. The follow-up activity to facilitate participants’ reflection consisted of two phases: the issuance of the digitalbadge two or three weeks after participants submitted the post-learning assignment and the follow-up survey tocheck participants’ progress six months after submission of the post-learning assignment.
For the first phase, participants who met the passing criteria were issued a digital badge that was linked to theircourse achievements and displayed feedback from lecturers. This first reflection trigger aimed to show participantsthat they had achieved the learning outcomes and inform participants about how they could improve their jobs inthe new fiscal year.
In the second phase, participants’ reflection on their learned knowledge was triggered through a follow-up survey
six months after completion of the ID courses. Previous findings have suggested that follow-up surveys might be auseful tool to remind participants of what they learned and encourage them to utilize the learned skills in job-related
situations (Kirkpatrick & Kirkpatrick, 2016; Nemoto et al., 2005; Saks & Burke-Smalley, 2012). Moreover, a followup survey might be a useful tool to deliver instructional messages about what actions they can take after completingthe program. Based on these ideas, we designed a follow-up survey that asked participants to click on their digitalbadge to review course-related content. We expected the digital badge to not only act as a support tool forparticipants’ accumulation of course content but also help them browse the content connected to the digital badge
and reflect on how their learning could be incorporated into their jobs.

IJEMT, Vol. 13, No. 1, 2019, pp.95-103, ISSN 1882–2290

99

International Journal for Educational Media and Technology
2019, Vol.13, No. 1, pp.95-103

Figure 3. Follow-up design to facilitate participant reflection on their learning outcomes
after completion of the instructional design courses

Results and Discussion
In the follow-up survey, we confirmed how participants conducted activities using the digital badge that was
acquired in the ID courses’ practical sub-course by presenting the following questions: How do participants use thedigital badge and perceive its usefulness for learning and job improvement? When do participants browse thecontent that can be accessed through their digital badge?

Perceived Usefulness of a Digital Badge
Supporting participants in their application of and reflection on learned knowledge is the primary objective ofallowing participants access to the digital badge’s content after the completion of their ID courses. Regarding bothaspects of support and accountability, more than 60% of the participants showed a positive reaction (Figure 4).
Participants were also required to provide reasons for why they believed the digital badge is helpful in improving
their real-life educational programs; some participants provided the following positive responses: “The comments
and feedback given in the digital badge pushed me to improve my work”; “The digital badge stimulated myconfidence and motivation for learning and helped me maintain my commitment to improving my work”; and“Based on the information attached to the digital badge, such as the learning outcomes and feedback from lecturers,
I thought about ideas to improve my educational courses. It also sometimes helped to increase my motivation forthese actions.” These responses suggest that the learning portfolio attached to the digital badge played a motivatingrole for some participants and helped them improve their educational programs.
However, some participants had the following negative responses, as they viewed the digital badge not as a portfolio
but only as a reward: “I thought that I wanted to acquire a badge. But I have not used it at all once I acquired it,”and “I received a digital badge and was happy to finish the course. I didn’t use it at all.”From the perspective of the digital badge’s usefulness for learning after the completion of the ID courses, those
who responded positively commented as follows: “If there was no digital badge, I would have surely been in trouble.
Although I usually take my own notes, the digital badge is more convenient to use because the information is stored
online, which is better than using a notebook. By browsing the digital badge content, I can remember what I was
thinking, which I sometimes forget”; “The digital badge helped me to consistently review my learning from preassignment, to face-to-face workshop, to post-assignment of the ID courses”; and “I was able to reflect on mylearning in the courses from the comments and links attached to the digital badge.” These comments suggest thatthe digital badge can help participants conveniently access course content and feedback perceived as useful andreflect on their learning.
Some participants gave negative comments, possibly owing to job complications that made it so they could notbrowse the digital badge: “I didn’t browse the digital badge content after issuance because I was transferred to adifferent post due to personnel changes,” and “Everyday work was hectic, and I couldn’t review the digital badge.”
IJEMT, Vol. 13, No. 1, 2019, pp.95-103, ISSN 1882–2290

100

International Journal for Educational Media and Technology
2019, Vol.13, No. 1, pp.95-103

Figure 4. Perceived usefulness of the digital badge after the completion of the instructional design courses

Prompting Digital Badge Browsing
For the follow-up survey, we asked participants to answer questions by browsing the digital badge content. Because
the digital badge gathered course content in one place, it could be a helpful tool for learners to reflect on theirlearning. The digital badge was linked to an “activity report” page implemented by default in Moodle that
accumulates all the learning achievements of each learner.
Access logs of the participants’ activity report page are compared in Figure 5. The digital badge was acquired by 57
participants in 2016 and 72 participants in 2017. Compared to the results of the ID courses in 2016, which did not
implement a follow-up survey, participants’ access to their own activity report page increased six months after theissuance of the digital badge in 2017, when the follow-up survey was conducted, although the number ofparticipants who acquired the digital badge also increased. This result indicates that the follow-up survey itself could
be a key trigger that induces learners to browse the digital badge and reflect on their learning outcomes.
Furthermore, some participants mentioned the effectiveness of the follow-up survey in the optional free description
field of the questionnaire: “I reviewed my action plan after a long interval. I recalled my feelings when I submitted
the final report. I realized that the follow-up survey is necessary. Thank you for reminding me of that moment,” and“Although open seminars and corporate training tend to be over for good once they are completed, my motivation
has increased thanks to these ID courses because they include a follow-up survey. It was a significant result that I
was able to perceive my outcomes of educational practices.” From these comments, it seems that the follow-up
survey reminded participants of what they had learned and motivated them to use their new skills, as suggested byprevious findings (Kirkpatrick & Kirkpatrick, 2016; Nemoto et al., 2005; Saks & Burke-Smalley, 2012). Further, it
was confirmed that participants browsed the course content and reflected on their progress. This might have
happened because the online and easily accessible form of the digital badge allowed more space for storing
participants’ coursework. If they did not have the digital badge, they would have depended on their memory orsome portfolio/notebook that they might have created. As 57 of the 72 2017 participants (79%) browsed the digitalbadge, many participants used it to review their course achievements after completing the follow-up survey. Thesefindings suggest that lecturers can use follow-up surveys to facilitate reflection by participants at their convenience.

Figure 5. Number of times the activity report page attached to the digital badge was accessed
Note: N※ is the number of people who browsed the learning portfolio attached to the digital badge.
IJEMT, Vol. 13, No. 1, 2019, pp.95-103, ISSN 1882–2290

101

International Journal for Educational Media and Technology
2019, Vol.13, No. 1, pp.95-103

Implications: Practical Support for Novices’ Reflection in the Context of Lifelong Learning
Perceived usefulness of the digital badge was positive. The results indicated that the reasons for positive actions
were convenient access to the learning portfolio and feedback in the form of some participants’ comments,
confirming that a follow-up survey itself can provide learners opportunities to reflect on their learning outcomes.
These findings suggested that combining the use of the digital badge with an appropriately timed reflection, such as
follow-up surveys, could provide practical support for novice learners unfamiliar with reflection activities. A
recommendation derived from this study would be to use digital badges and follow-up surveys as novice learners’“training wheels” for reflection activities. The linking of the digital badge to the user’s learning portfolio allowed
participants to conveniently accumulate meaningful information about their training in one place. Moreover,providing an opportunity for reflection six months later prompted participants to reflect on their learning outcomes
and how to integrate their training into the workplace.
Figure 6 depicts these strategies using the classification by Kirkpatrick and Kirkpatrick (2016). Thus, this study
presents a novel method for follow-up activities implemented after the completion of a training program, using a
digital badge. It also suggests an effective follow-up activity for lifelong learning, which includes university extension
courses. Although some studies have demonstrated useful methods of follow-up activities in corporate settings(Saks & Burke-Smalley, 2012; Tews & Tracey, 2008), little has been reported focusing on lifelong learning settings.

Figure 6. Method for follow-up activities combining the use of digital badges and surveys
Some of the participants’ negative comments on the use of the digital badge after course completion suggest that
there are learners who regard the digital badge not as a learning portfolio but merely as a reward. Other comments
demonstrated that some learners could not browse the digital badge because of job-related circumstances that are
beyond their control. Although we cannot control learners’ job-related conditions, we will need to examine how tochange their perception of digital badges in the future by providing examples of the digital badge’s utility after
course completion.
It was also not clear which digital badge functions and characteristics facilitated job improvement and reflection.
Based on the hypotheses revealed in this study, richer qualitative and quantitative data need to be collected andanalyzed.

Conclusion
In this study, we designed and verified the effects of a follow-up activity after the completion of ID courses bycombining the use of a digital badge linked to a learning portfolio with a survey. The results suggest that accessing
course content and feedback through the digital badge could be useful for learners’ reflection on their knowledge.
Thus, this study presents a new application of the digital badge constituting support for reflection after an
educational activity. Further, participants’ comments indicate that the digital badge can facilitate the application ofknowledge in the workplace. Moreover, access logs in the LMS showed that the design of the follow-up survey withthe digital badge might trigger learners’ reflection on their learning outcomes. These findings also suggest that toallow novice learners to reflect on their learned skills, combining the use of the digital badge that gathers meaningful
information about learning with opportunities that trigger instances of reflection, such as follow-up surveys, could
be useful. In this way, this research suggests that combining the digital badge with follow-up surveys facilitatedreflective actions related to the participants’ workplace. However, these actions’ impact on the quality improvementof participants’ job performance was not verified.
Further studies are needed to change learners’ perception of digital badges as not only rewards but also learning
portfolios. This study only examined how learners used and perceived the digital badge; however, further
investigation should be conducted to determine how and to what extent the reflection support design that combined
digital badges with follow-up surveys impacted qualitative changes in learners’ jobs. In addition, further investigation
IJEMT, Vol. 13, No. 1, 2019, pp.95-103, ISSN 1882–2290

102

International Journal for Educational Media and Technology
2019, Vol.13, No. 1, pp.95-103

should be conducted using rich qualitative and quantitative data to clarify how digital badges affect reflection andjob performance improvement.

Acknowledgments

This work was supported by JSPS KAKENHI Grant Number JP17K12948.

References
Amano, K., Suzuki, K., Tsuzuku, S., & Hiraoka, N. (2017). Designing a digital badge as a reflection tool in blended
workshops. The Journal of Information and Systems in Education, 16(1), 12–17.
Barrett,
H.
(2010).
Balancing
the
two
faces
of
ePortfolios.
Retrievedfromhttps://electronicportfolios.org/balance/Balancing2.htm
Brinkerhoff, R. O., & Apking, A. M. (2001). High-impact learning: Strategies for leveraging business results from training.
Cambridge, MA: Perseus Publishing.
Buchem, I. (2016). Digital badge as (parts of) digital portfolios: Design patterns for educational and personal learning
practice (Part 19) In D. Ifenthaler, N. Bellin-Mularski, & D.-K. Mah (Eds.), Foundations of digital badges and microcredentials (pp. 343–367). Basel, BS: Springer International Publishing.
Cheng, Z., Watson, S. L., & Newby, T. J. (2018). Goal setting and open digital badges in higher education. TechTrends,
62(2), 190–196.
Gagne, R. M., Wager, W. W., Golas, K. C., & Keller, J. M. (2005). Principles of instructional design (5th ed.). Belmont, CA:
Thomson/Wadsworth.
Gibson, D., Coleman, K., & Irving, L. (2016). Learning journeys in higher education: Designing digital pathways
badges for learning, motivation and assessment (Part 7). In D. Ifenthaler, N. Bellin-Mularski, & D.-K. Mah (Eds.),
Foundations of digital badges and micro-credentials (pp. 115–138). Basel, BS: Springer International Publishing.
Hartnell-Young, E., Harrison, C., Crook, C., Pemberton, R., Joyes, G., Fisher, T., … & Underwood, J. (2007). Impact
study of e-portfolios on learning. Coventry, UK: British Educational Communications and Technology Agency.
Kirkpatrick, J. D., & Kirkpatrick, W. K. (2016). Four levels of training evaluation. Alexandria, VA: ATD press.
Merrill, M. D. (2009). First principles of instruction. In C. M. Reigeluth & A. Carr (Eds.), Instructional design theories andmodels: Building a common knowledge base (Vol. III) (pp. 41-56). New York, NY: Routledge Publishers.
Nemoto, J., Hazelman, V., & Suzuki, K. (2005). Instructional design workshop based on needs analysis at Universityof South Pacific. In P. Kommers & G. Richards (Eds.), Proceedings of ED-MEDIA 2005--World Conferenceon Educational Multimedia, Hypermedia & Telecommunications (pp. 268–275).
Peck, K. (2015). The future’s so bright I gotta wear shades. TechTrends, 59(1), 24–30.
Saks, A. M., & Burke-Smalley, L. A. (2012). An investigation into the relationship between training evaluation and thetransfer of training. International Journal of Training and Development, 16(2), 118–127.
Tews, M. J., & Tracey, J. B. (2008). An empirical examination of posttraining on-the-job supplements for enhancing
the effectiveness of inter personal skills training. Personnel Psychology, 61(2), 227–466.
Willis, J. E., Flintoff K., & McGraw, B. (2016). A philosophy of open badges (Part 2). In D. Ifenthaler, N. BellinMularski, & D.-K. Mah (Eds.), Foundations of digital badges and micro-credentials (pp. 23–40). Basel, BS: Springer
International Publishing.

IJEMT, Vol. 13, No. 1, 2019, pp.95-103, ISSN 1882–2290

103