1-G3
ePortfolios for Study Abroad Assessment and Improvement:
Monitoring Students’ Use of a Target Skill While Abroad
Miki Cutting
cutting@apu.ac.jp
Associate professor
Ritsumeikan Asia Pacific University
Beppu, Japan
Doctoral student
Kumamoto University
Kumamoto, Japan
Yoshiko Goda
ygoda@kumamoto-u.ac.jp
Associate professor
Kumamoto University
Kumamoto, Japan
Katsuaki Suzuki
ksuzuki@kumamoto-u.ac.jp
Professor
Kumamoto University
Kumamoto, Japan
Abstract:
ePortfolios, which are becoming a prominent tool for collecting experiential anddevelopmental evidence during study abroad, need more research as a means ofassessment. Our study made a systematic analysis of students’ ePortfolios while abroad
to understand how they used a target skill taught in pre-departure training, applying
Kirkpatrick’s training evaluation framework. The findings were then used to discuss how
to improve the training. Contextualized analysis of ePortfolio revealed various cultural
engagements in which the target skill was used. We found that ePortfolio also captured
students’ perspective changes and self-realizations as well as affective and behavioral
aspects in their learning. Furthermore, a progressive trend in difficulty level of students’questions regarding culture was observed. Quantitative data of the survey results werethen added to bolster the results. The evidence suggests that contexts simulated during
pre-departure training matched real-life situations abroad but did not sufficiently cover
the diversity of settings in which students can exercise the skill. We conclude that by
employing a systematic assessment approach using ePortfolios, rich learning evidence toimprove pre-departure training can be obtained.
Key words: ePortfolios, Assessment, Training evaluation, Education abroad

STUDY ABROAD ASSESSMENT
In this global era where people increasingly work across national boundaries, often interconnectedvirtually, higher education has been tasked with preparing students to better live and work together withpeople from different backgrounds by providing quality international education, such as study abroad
programs. However, outcomes in international education, such as intercultural competence, defined as“behaving and communicating appropriately and effectively in intercultural situations” (Deardorff 2006, p.
255), are complex and therefore challenging to assess (Deardorff 2015, Fantini 2012). Despite the natureof intercultural competence being experiential, developmental, and interactional (Deardorff 2015),

346

institutions tend to over-rely on pre-post surveys, standard instruments, and self-report Likert items, which
do not adequately capture students’ learning and development, behavioral aspects, and evidence to explain
the outcomes (Griffith et al. 2016, Deardorff 2015). Furthermore, evidence of student learning should be
used to understand and improve their learning (Suskie 2018), yet in the area of study abroad, research toindicate how assessment can lead to program improvement is lacking (Salisbury 2015). To capture evidenceof complex learning outcomes during study abroad, it is essential to determine which specific aspect toassess and how to monitor its development (Deardorff 2015, Fantini 2012). However, accessing direct
evidence of students’ learning, and monitoring development while the students are overseas, is notstraightforward, especially if faculty or staff are not accompanying them.
In the midst of these challenges, ePortfolios, which are web-based and can gather artifacts of learningand reflection that focus on learning and development (Eynon & Gambino 2017), are an emerging meansof assessment in international education allowing for collection and tracing of direct evidence of students’learning over time (Deardorff 2015). However, as Griffith et al. (2016) point out, a weakness of using
portfolios is that no standard assessment exists which enables meaningful comparisons among groups or
students’ submitted work, since content, platforms, and scoring methods are different across institutions,research, and contexts. In higher education in Japan, many institutions are still struggling with theimplementation of ePortfolios (Morimoto 2015), and in study abroad research, discussion of how to use
ePortfolios for measurement is still scarce (Shoraku & Yoshida 2018). With the expectation that ePortfolios
can assess complex learning outcomes during study abroad, research on the use of ePortfolios tosystematically assess students’ learning development overseas and to improve programming is needed.
In one of the few studies to examine study abroad ePortfolios in Japanese higher education, Cutting
et al. (2020) analyzed students’ daily reflections in ePortfolios documented abroad to see the impact of predeparture training. The researchers employed Kirkpatrick’s (2006) training evaluation framework, part ofwhich evaluates participants’ behavioral change “after” the training, so called Level 3 (Behavior) evaluation,and examined how frequently students wrote about using the target skill taught in pre-departure training.

RESEARCH DESIGN & METHODSThis research moves beyond the study of Cutting et al. (2020) to examine how the same target skill
is used and how it develops over time. It also adds self-ratings from students about their use of the skill tobolster results. Then, the findings are used to discuss how to improve pre-departure training.
Research questions:
1) What does the evidence in students’ ePortfolios tell us about how they use the target skill and how
it develops?

3472) How can we improve our pre-departure training from the evidence?

Portfolio
training

Level 2: Learning outcomes

Showcase

Level 3: Behaviors

Figure 1: Design of ePortfolio Assessment for Study Abroad

PROGRAM BACKGROUND
The study is based on the same short-term study abroad program as in Cutting et al. (2020), operated
in a local private university in Japan, but in the following year with different students. The program
dispatches a group of Japanese students to a US institution for 2 months for credit-bearing courses.
Enrollment requires completion of an intermediate English class in the home institution and students’
TOEFL (itp) scores range from 440 to 480. The program aims to equip students with communication skills
and intercultural competencies to work with people of diverse backgrounds in global society according to
the institutional mission. The participants are 15 students (4 male and 11 female; 4 second-year and 11
third-year).
FRAMEWORK FOR MONITORING
To optimize the study abroad experience, pre-departure training (7 weeks for 14 class hours) and reentry sessions (3 times) are provided by the first author in the home institution (Figure 1). The pre-departure
training course aims to promote intercultural communication and learning while abroad based on the
program goals. This includes “perspective taking,” in which students learn how to step out of theirethnocentric perceptions to see others’ viewpoints, and it is divided into sub-categories for practice andevaluation (Table1). The study uses the framework of Kirkpatrick’s (2006) training evaluation andexamines how students use what they learned from the pre-departure training while abroad. In pre-departure
training, students practice each skill in Table 1 in a simulation with a TA who acts as an American buddy
and their performance and attainment levels are evaluated as learning outcomes (Kirkpatrick’s Level 2:
Learning Outcomes). After the training, we examine if students are exercising the new skill (Kirkpatrick’s
Level 3: Behaviors). In this study, we focus monitoring on Skill b), due to its criticality in intercultural
understanding as well as its observability (Cutting et at 2020). Skill b) “Students can ask questions inEnglish about cultures, such as cultural differences or surprises.”348

Table 1 “Perspective Taking” Goals divided into Attitude/Knowledge/Skills for practice and evaluation“Perspective Taking”

Goal

Students possess the knowledge, attitude, skills, and English ability to attempt to broaden their perspectives.
AttitudeʀSuspend own judgement of others’ attitudes and behaviors.
ʀBeing curious and inquisitive to seek understanding through observation, questioning, and researching .

Knowledge

ʀStudents can differentiate facts, interpretation, and evaluation. (the D.I.E. method: Describe, Interpret, Evaluate).

Skillsa) Students can suspend their own judgement of others’ attitudes and behaviors.
b) Students can ask questions in English about cultures, such as cultural differences and surprises.
(appropriately, not based on their own judgement).
c) Students can explain their own views to others (appropriately) to help mutual understanding.
d) Students can critically reflect and express their own views, assumption, and/or their perspective change.

Target skill
to monitor(Adapted from Cutting et al. 2020)
ePORTFOLIOS
Throughout the program, ePortfolios are employed to deepen students’ learning through selfreflection and to provide evidence for learning assessment. In the pre-departure course, students learn how
to reflect on their experiences and why it is important. While abroad, students write daily reflections andset daily goals as a non-compulsory yet highly recommended activity (Figure 2). Upon their return, students
review their ePortfolios and create showcase portfolios. Figure 2 shows an example of students’ dailyreflections during study abroad. The top column contains instructor’s guided reflection questions withmotivational messages posed daily and the second column has students’ writing. The bottom column isfeedback from a trained peer advisor, who was a program participant of the previous year.
ʴePortfolio during study abroadʵʀWhat did you experience today? What did you notice or learn from the experience? Why is it important and how will
you utilize it in your life?

࣭How was your achievement of the day? What is your goal tomorrow?

(Instructor)
Instruction for reflection & messages(Student)

1) Daily Reflection:
ʀSomething I did, noticed, felt, or learned.
ʀWhy it is important.
ʀHow I will use it in my life.

2)Goal setting & evaluation:
ʀMy achievement of the day.
ʀMy goal for tomorrow.

(Peer Advisor)

Comment to the reflection

Figure 2: ePortfolios during study abroadePORTFOLIO ANALYSISThe first phase of data analysis was to thoroughly read students’ daily ePortfolio reflections andextract descriptions which referred to their use or intended use of the target skill following Cutting et al.
(2020). This qualitative coding was conducted based on the deductive approach of Sato (2008), first by pre-349

setting a code label for the target skill, and then allocating the code to segments which specified students’use or intention to use the skill (Cutting et al. 2020). A researcher in the field of international education
supported the first author in extracting appropriate segments. When agreement was reached on all selected
segments, we concluded that theoretical saturation had been achieved. The segments were then set inchronological order and use of the target skill over time was analyzed. 15 students’ reflection entries, fromdays 1 to 19, were analyzed. Then from day 19 to 29, a self-rating survey which asked if they were using
the skill or not was collected. The total number of entries for 15 students over 19 days were 226 entries,
average of 79% submission rate per day. (The total participants was 16. However, one student who did not
submit a self-rating survey was excluded from this study to align with reflection data.) Prior to this research,
all participants provided written and verbal consent to make use of their ePortfolio submissions.

RESULTSThe extracted segments showing students’ use or intended use of the target skill are listed, student
by student, in Table 2. 8 students out of 15 (53%) wrote about using the skill, and there are 12 segments intotal. While these do not represent every experience they had, their authentic contexts where the skill wasused are captured in detail. First, students asked their cultural questions to their “buddies” the most, who
are student volunteers at the hosting institution to support the international students. The findings signified
that buddies acted as important cultural guides for international students. In addition, ePortfolios discovered
other interlocutors, such as “RAs,” “volunteers,” and a “dorm resident.” Variety of cultural questions was
also documented, for example, personal observations, local recommendations, and politics.
ePortfolios importantly captured students’ perspective changes and self-realizations. For example,
Student C (day 19) who asked about World War II to an American dorm resident reflected on the realizationof his/her narrow views. Furthermore, this student felt the need for academic vocabulary. Student D (day11) also asked about politics and reflected on the necessity of listening skills. Although here we see students’recognition of the need for language skills in complex cultural dialogues, Byram & Golubeva (2020) claimthat many models of intercultural competence “do not take account of linguistic competence” (p.74). Moremodels of intercultural competence which integrate language aspects need to be developed. ePortfolio
entries also capture students’ feelings, as in “thankful,” “exciting,” and “frustrating” in their culturalinteraction, providing additional aspects related to their learning. Griffith et al (2016) refute that existingmeasures in intercultural communication often do not sufficiently collect affective and behavioral evidence
which are essential in viewing interactions with local hosts. These ePortfolios provide affective andbehavioral evidence which are hard to capture by standard instruments.
Table 2 Extracted descriptions from ePortfolios on using the skill to “ask questions about culture”
Students’ descriptions
context
Student A About cultures, I felt there are so many people who are half naked. Texas is surely hot, but Japan is …I ‫ڹ‬Buddy
Day 1
don’t know what makes such difference, so I will try to ask a buddy about it.
Observation
Student A I asked a buddy about a kick skater. Lots of people are riding them and I learned people can rent one ‫ݱ‬Buddy
Day 7
after installing an application.
Observation

350

Student B After class, I sent a message to a buddy and asked which coffee shops he recommends around school. ‫ݱ‬Buddy
Since I am in the US, I felt I should explore different coffee shops to study. …I truly respect our Local
Day 4
buddies’ spirit to help. So thankful.
recommendation
Student B I asked my buddy about politics in the US, because I couldn't understand the pros of the Republicans in ‫ݱ‬Buddy
Day 14
my Multicultural Society class. He told me that …I wanna know about politics of the U.S. more and more. Politics‫ݱ‬RA

Student C (The RA) asked questions and asked about Japanese language to me. Using this time, I asked the same
What was asked
Day 6questions back to her and explained using drawings and photos. With these, she talked to me a lot.
(language)

Most shockingly, when I asked him (a dorm resident) about bombing in Hiroshima, he said American ‫ݱ‬Dorm resident
Student C aircrafts … I wasn’t taught such a thing, so later I looked into it. It seems Japan was banning people to.. War
Day 19World is big. My view is narrow. And topics are difficult. I really felt I need more vocabulary for academic
conversation, but it was a lot of fun!
I asked (volunteers) about the election in the US which I learned in class because I wanted to see if I can ‫ݱ‬Volunteer
Student D talk about something academic. However, their vocabulary was difficult, and speed was fast, so it was Politics
Day 11
hard to catch all. But they explained some words to me and gave various examples, so I was thankful. I
still lack listening skills. I will work on it.
Because of a comment from my teacher, I learned my gestures could give a negative image to some people. ‫ݱ‬Buddy
Student E I also asked questions about an expression I know to a buddy and found out people may consider it as Expression
Day 4
racist. Asking questions is surely important and I want to ask questions more.
I talked to our buddies on the street. Starting from “it’s hot again,” I kept asking questions about whatever ‫ݱ‬Buddy
Student F I saw in town. This is my first time to stay this long overseas, so this new culture is very exciting …. My Town
Day 7
goal for the day was proactively communicating with buddies and I feel I achieved it well.
observation
I will write about the questions I asked to RAs today and what I thought about their answers. Question 1: ‫ݱ‬RA
Student G In the parking lot, it seems…Question 2: In Texas, do Republicans….Then I heard Texas is Republican, Observation,
Day 10but.... Their answers were interesting. I want to keep asking more questions from now on.
Politics, etc.
After the tour, some students were asking questions to the guide and understanding by nodding. However, ×Guide atStudent H I couldn’t understand and felt terrible. I had lots I wanted to ask about this organization but not being Volunteer siteable to understand what they say was frustrating. I have a few things I want to ask, so I will ask the staff Volunteer site
Day 6next time I go there.
Student H (At a volunteer site, in a freezer room) There was a foreign volunteer next to me and I wanted to ask ×Volunteer
Day 18various questions about volunteering to the person. But I was so cold that no word came from my mouth. Volunteering(Interlocutors and question content are highlighted. Students’ reflections of their action are underlined. Japanese entries were translated
into English by the first author)

Next, to explore the development of skill usage over time, we chronologically arranged the segmentsfrom day 1 to 19 (Table 3). From day 1 to day 9, students tended to ask questions about observations in
town and local recommendations as in student A, B, and F. From day 10, questions about politics arose as
in student B, C (war), D, and G. This progression is not rigid, but thinking of cultural learning, questions
about personal observations are relatively easy to ask and “visible” part of cultures. Politics are more
complicated to discuss and comprehend, thus requiring elaborate interaction. Furthermore, intricate
discussions involve certain English levels. Therefore, it is not difficult to guess why political questions startto emerge later than town observations and this may show developmental stages in this skill.
Table 3 Context for skill use in chronological order (students’ cultural questions and self-ratings)
Student A
Day1~3Student C

Day16~19
Day19~26

Student F

Student G

Student H

×Guide atvolunteer site
Volunteer site‫ݱ‬Buddy
Expressions‫ݱ‬Buddy
Observation‫ݱ‬Buddy
Observation‫ݱ‬Volunteer
Politics

Day10~12
Day13~15

Student D Student E‫ݱ‬Buddy
‫ݱ‬RA
Local
What was asked
recommendation (language)

Day4~6
Day7~9Student B‫ڹ‬Buddy
Observation‫ݱ‬RA
Observation,
Politics, etc.

‫ݱ‬Buddy
Politics‫ݱ‬Dorm resident
War×Volunteer
Volunteering

Self-rating: “I ask questions about cultures (cultural differences and/or surprises)”
Strongly agree

Somewhat agree

˜: Intention to use the skill, 㾐: Used the skill, ×: Couldn’t use the skill

351

Don’t agree much

SELF-RATING
From days 19 to 26, a self-rating survey with the question “I ask questions about cultures” wasconducted to collect quantitative data. In Table 3, Students A, B, and C, who wrote about using the skill
most frequently had the highest self-rating. Whereas, Student H, whose ePortfolio revealed no success inasking questions rated the lowest, which matches the ePortfolio evidence. The ePortfolios of Student H
showed that his/her difficulty in comprehending English hindered asking cultural questions. Since Jackson(2020) states that researching “host-sojourner interaction” to identify what facilitates or hinders“intercultural communication and relationship building” is still new (pp.448-449),” this area needs to be
examined further. It must also be noted that there were 7 students who did not refer to using the skill in
their ePortfolios; 4 of these answered, “Strongly agree,” 2 “Somewhat agree,” and 1 “Don’t agree much.”
One of the main reasons there are no such references is likely that some of these students missed submittingreflections several times, and some entries were brief and lacking detail. Therefore, ePortfolio assessment
requires adequate submission as well as content, and thus motivating and encouraging students to document
their learning in ePortfolios is imperative.
USE OF DATE FOR COURSE IMPROVEMENT
Lastly, we discuss how the findings help improve the program. According to Suzuki (2015), it is
crucial to make the training context similar to real-life settings in order to promote learning transfer (Cutting
et al 2020). To examine how much our training context resembles real situations, we compared both. In
pre-departure training, students practice the target skill with a TA acting as a “Buddy” by asking questions
about “things they saw” in a simulated environment. The ePortfolios revealed that interaction with “Buddies”
and asking questions about “observations” are the highest frequency (Table 2), which indicate our training
context matches students’ real-life situations. However, ePortfolios also showed that use of the skill
extended to more diverse situations and involved other interlocuters. To optimize affordances in the hostenvironment, we will embed such elements in simulation practice in pre-departure training. As discussed
earlier, cultural questioning tends to progress from simple observations to deeper topics such as politics,
and English comprehension is integral in cultural discussions. This suggests that we can add levels in our
simulations following this pattern and integrating advances in English.

CONCLUSION
Our study attempted to use ePortfolios to perform a systematic assessment of study abroad participants’learning, by honing in on a target skill taught during pre-departure training and monitoring and analyzing
students’ use of that skill during study abroad, using Kirkpatrick’s framework. We then discussed how to
use the findings to improve the pre-departure training. Contextualized qualitative analysis of ePortfolios
revealed diverse situations where cultural questions emerged and local people engaged with, all of which
expanded the scope of students’ sojourn experiences. ePortfolio importantly captured students’ perspective352

changes and self-awareness, as well as affective and behavioral aspects in learning. Furthermore, a trend
toward transition in difficulty levels in students’ cultural questions was observed. The evidence suggests
the pre-departure training context matches students’ sojourn experience but does not sufficiently cover
diverse situations and complexity levels of the context of their skill use. According to Jackson (2020), thefuture research should investigate intercultural discourse in study abroad in a systematic and in depth way
to inform “types of interactions” students routinely engage in, “learner speech act behavior in various
situations,” intercultural misunderstandings and negotiation strategies (p.452), and to use the evidence toprovide pre-departure training with examples of real-life scenarios. Although this study is limited by its
scale and focus, we hope our examination took a step in that direction using ePortfolios.
ePortfolios enables study abroad researchers and practitioners to open the door of students’ life abroad,filled with excitement, struggles, and surprises in their intercultural encounters. By having a clear focus inassessment and monitoring students’ learning in a systematic way, complex evidence of learning abroadcan become a rich and solid resource for improving pre-departure education and beyond.

REFERENCES
Byram, M. & Golubeva, I. (2020). Conceptualizing Intercultural (Communicative) Competence andIntercultural Citizenship. The Routledge Handbook of Language and Intercultural
Communication (Jackson, J. ed.). Oxon: Routledge. 70-85
Cutting, M., Goda, Y., & Suzuki, K. (2020). Instructional Method and Impact of Pre-Departure Training
for Study Abroad: From the Viewpoint of Transfer of Learning [In Japanese]. Journal of Japan
Association for College and University Education, 42(1): 95-104
Deardorff, D. K. (2015). Demystifying Outcomes Assessment for International Educators. Sterling, VA:
Stylus.
Deardorff, D.K. (2006). Identification and Assessment of Intercultural Competence as a Student Outcomeof Internationalization. Journal of Studies in International Education, 10(3): 241-266
Eynon, B., and Gambino, L. M. (2017). High Impact ePortfolio Practice A Catalyst For Student, Faculty,
and Institutional Learning. Sterling: Stylus Publishing, LLC.
Fantini, A.E. (2012) Multiple Strategies for Assessing Intercultural Communicative Competence. The
Routledge Handbook of Language and Intercultural Communication (Jackson, J. ed). Oxon:
Routledge. 390-406
Griffith, R. L. Wolfeld, L. Armon, B.K., Rios, J. & Liu, O.L. (2016). Assessing Intercultural Competencein Higher Education: Existing Research and Future Directions. Research Report, ETS RR16-25:
1-44
Jackson, J. (2020). The Language and Intercultural Dimension of Education Abroad. The Routledge
Handbook of Language and Intercultural Communication, Oxon: Routledge.
Kirkpatrick, D. L. & Kirkpatrick, J.D. (2006). Evaluating Training Programs. Oakland: Berrett-Koehler
Publishers, Inc.
Morimoto, Y. (2015). Educational Big Data as e-Portfolios and Learning Analytics [In Japanese]. Computer& Education, 38:18-27.
Salisbury, M. (2015) How We Got to Where We Are (and Aren’t) in Assessing Study Abroad Learning.
Assessing Study Abroad (Savicki, V. & Brewer, E. eds). Sterling: Stylus.
Shoraku, A. & Yoshida, M. (2018). Learning Outcomes of Overseas Studies in an Undergraduate Program[In Japanese]. Kobe Journal of Higher Education, 26: 59-78.
Suskie, L. A. (2018). Assessing Student Learning: A Common Sense Guide. San Francisco: Jossey-Bass.
Suzuki, K. (2015). Kenshu Sekkei Manual [Training design manual-translation]. Kyoto: Kitaoji-shobo.
Vande Berg, M. P., Paige, R. M., & Lou, K. H. (2012). Student Learning Abroad. Sterling: Stylus
Publishing LLC.

353