International Journal of Educational Media and Technology
2021, Vol. 15, No. 1 pp.117-128 .

Analysis of Behaviors Documented in Reflective Journals
in Study Abroad ePortfolios to Improve Pre-Departure Training
Miki Cutting
Ritsumeikan Asia Pacific University
Graduate school of Kumamoto University
cutting@apu.ac.jp
Yoshiko Goda
Kumamoto University
ygoda@kumamoto-u.ac.jp
Katsuaki Suzuki
Kumamoto University
ksuzuki@kumamoto-u.ac.jp
Abstract:
In study abroad research, capturing learning evidence and using it to improve pre-departure training is indemand. Our study applied Kirkpatrick’s training evaluation for study abroad and analyzed students’reflective journals documented over time for evidence of how students used a target intercultural skill taught
during pre-departure training. The findings were then utilized to discuss how to improve the training. In-depth
analysis of reflective journals indicated that most common contexts in which sojourners used the skill resembled
our training context and signaled the importance of buddy schemes. However, it also revealed sojourners’
various cultural engagements and their stages of advancement of cultural inquiry which were not covered in the
training. Students’ reflective journals, importantly, documented encounters with language difficulties in complex
cultural inquiry with affective evidence. This raised issues that training needs to incorporate linguistic
components to handle advanced cultural dialogues.
Key words: Assessment, Education abroad, Intercultural training, Instructional design, Kirkpatrick’s training
evaluation

Introduction
As students’ mobility across the globe is restricted due to the pandemic, and higher education institutions have
had to move to online alternatives to provide international education, the value of opportunities to physicallystudying in foreign countries has become recognized more than ever. However, merely sending students
overseas does not necessarily lead to expected outcomes, therefore it is critical to provide research-based
intercultural mediation to optimize the sojourn experience, such as by pre-departure training (Jackson & Oguro,
2017; Jackson, 2020a; Vande Berg et al., 2012). A recent government-let large-scale study in Japan (Kawai-juku,2018) has also revealed the importance of pre-departure training, yet studies in pre-departure training andtraining effectiveness are still scarce (Cutting et al., 2020) and development of pre-departure training needs tobe discussed nation-wide (Kawai-juku, 2018).
In the field of training evaluation, Kirkpatrick’s model of training evaluation, which divides evaluation into 4levels: reaction (Level 1), learning (Level 2), behavior (Level 3), results (Level 4), and formalizes the significance ofeach level in training, has been used as a de facto standard (Suzuki, 2015a). Behavior (Level 3) is “the degree towhich participants apply what they learned during training when they are back on the job” (Kirkpatrick &Kirkpatrick, 2016, p.13). Behavior (Level 3) is “the missing link in moving from learning to results” and therefore
the most critical level (p.49).
According to Suzuki (2015b), to promote behavior (Level 3), it is crucial to make the training context resemble
real-life settings. For intercultural intervention to maximize study abroad experiences, it therefore raises thequestion as to how we can make our training context similar to students’ affordances overseas. Jackson (2020b)

IJEMT, Vol. 15, No. 1, 2021, ISSN 1882–2290

117

International Journal of Educational Media and Technology
2021, Vol. 15, No. 1 pp.117-128 .

urges that in future research we first need to obtain evidence from study abroad by investigating sojourners’
intercultural discourse in a systematic and in-depth way and then use such evidence for real-life scenarios in
pre-departure training. Suskie (2018) also argues that evidence of student learning should be used to understandand improve their learning. Yet in the field of study abroad, institutions too often rely on pre/post surveys,
standard instruments, and self-reported Likert items, that do not adequately capture students’ learning anddevelopment, behavioral aspects, and learning evidence to explain outcomes (Griffith, et al. 2016; Deardorff,
2015; Cutting, 2015). Moreover, research to indicate how study abroad assessment can lead to program
improvement is lacking (Salisbury 2015).
Cutting et al. (2020) used sojourners’ learning evidence documented in daily reflections to evaluate behavior(Level 3) in study abroad. The study first identified the possibility that sojourners were not utilizing a criticalintercultural skill taught during training. In the following year, the researchers redesigned the training andensured students’ attainment of that skill prior to study abroad to promote behavior (Level 3). During training,skill practice and evaluation through roleplays matching authentic sojourn settings were conducted several times
to assure students’ skill acquisition. After training, the study collected the students’ daily reflections while abroadand extracted documentation of use of the skill. The study found that students’ documentation of use of theskill significantly increased after this change of training and suggested the impact of training design on students’behavior (Level 3) in study abroad (Cutting et al. 2020).
However, the evidence in the study was not analyzed in sufficient depth to understand how students
implemented the skill, so these findings were not available to use to improve the training. As stated above, reallife evidence in study abroad should be used for pre-departure training content and learning improvement.
Analysis of sojourners’ descriptions of use of the target skill in a designated country may provide criticalinformation for pre-departure training.
The current study focuses on sojourners’ ePortfolio reflections during study abroad following Cutting et al.
(2020), and takes further steps to conduct in-depth analysis of sojourners’ writing about how they are using thetarget skill, which was taught in pre-departure training, to understand their real-life situations in study abroad.
Then this real-life evidence is used to discuss how pre-departure training can be improved. The present focus
is therefore to create an evaluation cycle to investigate behavior (Level 3) of Kirkpatrick’s model in-depth, andaims to use the findings to further develop the pre-departure training programs.

Research Design & Methods
Research questions
Cutting et al. (2020) evaluated behavior (Level 3) by collecting students’ reflection records and analyzing how
much students documented their use of the target skill while abroad. This study moves beyond Cutting et al.
(2020) and conducts in-depth analysis of the evidence from reflection records to understand how students areusing the skill with host nationals. Students’ self-ratings about their use of the skill, conducted in the same
ePortfolios, are also added to bolster results. The findings are used to discuss how to improve pre-departure
training (Figure 1). The current study follows the same study abroad program, pre-departure training content,
and target skill as in Cutting et al. (2020), but with different students in the following year.
Research questions are as follows:
1) What do students’ reflections tell us about how they use the target skill during study abroad?
2) How can evidence from their reflections improve training?

IJEMT, Vol. 15, No. 1, 2021, ISSN 1882–2290

118

International Journal of Educational Media and Technology
2021, Vol. 15, No. 1 pp.117-128 .

Figure 1. Research framework

Short-term study abroad and pre-departure training
This study abroad program is operated by a private university in Japan. Aiming to prepare students to work ina global society according to the university’s mission, the program dispatches a group of Japanese students,(approximately 15 to 20) to a US partner institution for 2 months for credit-bearing courses created for thisprogram (Figure 1). For enrollment, participants need to have completed the intermediate English course
offered in the institution and their English levels range from TOEFL ITP 460 to 500. The participants in thisstudy are 15 students (4 male and 11 female, 2nd and 3rd year).

Training design: Learning goals and target skill
To optimize students’ learning during study abroad and to achieve program goals, a pre-departure training (7weeks, 14 class hours) course was provided, with the learning goals of Intercultural communication, English
communication, and Perspective taking refined through research in Blair & Cutting (2015) and Cutting (2015; 2016).
The Perspective taking goal requires suspending one’s own judgement of others and seeking others’ viewpoints(Table 1) following theories and models in intercultural learning and the D.I.E. (Describe, Interpret, Evaluate)
method. Perspective taking has several sub-goals as in Table 1 and our study focuses on skill (b): Students can ask
questions in English about cultures, such as cultural differences and surprises by following Cutting et al. (2020). We decided
to monitor this skill, since this skill is relatively observable and identifiable compared to other skill goals, suchas Suspending own judgement and Critical reflection of own views.
More importantly, this skill of cultural inquiry requires other sub-goals in Perspective taking (Table 1). To ask
cultural questions, attitude goals of suspending own judgement along with curiosity and inquisitiveness are exercised.
The skill also practices the knowledge goal of the D.I.E. (Describe, Interpret, Evaluate) method to understand others’
perspectives and interpretations. This skill gives opportunities for students to use English and interact withothers for cultural learning. According to Byram (1997, pp.98-99), asking questions to “acquire new knowledgeof a culture and cultural practices” is “pivotal” in intercultural communicative competence. Since we focus ona single intercultural skill, our study is limited in scope and cannot represent the intricate phenomena of
sojourners’ learning. However, outcomes in international education are complex and broad, thus it is critical toidentify specific indicators and evidence in assessment (Deardorff, 2015). Our study is an attempt to focus ona specific indicator to identify and trace the evidence in complex intercultural learning.

IJEMT, Vol. 15, No. 1, 2021, ISSN 1882–2290

119

International Journal of Educational Media and Technology
2021, Vol. 15, No. 1 pp.117-128 .

Table 1
Target skill (b) in the goal of “Perspective Taking” (Adapted from Cutting et al., 2020)
Goal“Perspective Taking”
Students possess the knowledge, attitude, skills, and English ability to attempt to broaden theirperspectives.
・Suspend own judgement of others’ attitudes and behaviors.
・Being curious and inquisitive to seek understanding through observation, questioning, and research.

Attitude
Knowledge
Skills・ Students can differentiate facts, interpretation, and evaluation. (The D.I.E. method: Describe,Interpret, Evaluate).
(a) Students can suspend their own judgement of others’ attitudes and behaviors.
(b) Students can ask questions in English about cultures, such as cultural differences andsurprises. (Appropriately, not based on their own judgement).
(c) Students can explain their own views to others (appropriately) to help mutual understanding.
(d) Students can critically reflect and express their own views, assumption, and/or their perspective
change.

Training implementation: Skill practice in roleplays
In pre-departure training, skill practice is conducted through roleplays. To make the roleplay settings authentic,
we preset the situational description in the roleplay that students ask cultural questions to their ‘buddies’ about something
they ‘observed’ in the US. This is because our study abroad program employs a buddy scheme where students in
the host institution voluntarily support international students as buddies, often by showing them around thearea. Thus, students have opportunities make observations when they go out with buddies. Trained TAs act asbuddies and conduct roleplays (Figure 2). Students’ performance is evaluated and practice is conducted several
times until their skill attainment is assured by using the TOTE model in instructional design (Cutting et al.,2020).

Figure 2. Roleplay practice in training: Asking cultural questions to a TA who is acting as a buddy

Training evaluation: Reflective journals and self-rating survey in ePortfolios
Throughout the program, ePortfolios are implemented to deepen students’ learning through self-reflection andto provide evidence for learning assessment (Cutting et al., 2020). We used ePortfolios to evaluate training
effectiveness because this system can record learning reflection over time and self-ratings can be conducted,
which can indicate their use of the target skill taught during pre-departure training. The employed ePortfolio
is Manaba, a learning management system (LMS) widely used in Japanese institutions featuring the functions ofreports, tests, surveys, polls, and projects. In pre-departure training, students practice writing reflections inreports. When they are abroad, they start writing daily reflections on a voluntary basis. The portfolio reflections
contain the instructor’s guided reflection questions based on Kolb’s (1984) experiential learning model, such as
to write something I did, noticed, felt, or learned. Why it is important. How I will use it in my life accompanied by goal
setting. Trained peer advisors, who were program participants of the previous year, provide prompt feedback
to the sojourners’ daily reflections (Cutting, et al., 2020). Self-rating about use of the target skill is done insurveys in the same ePortfolios. After study abroad, students review their reflections and create a showcase
portfolio to represent their learning outcomes of study abroad.

IJEMT, Vol. 15, No. 1, 2021, ISSN 1882–2290

120

International Journal of Educational Media and Technology
2021, Vol. 15, No. 1 pp.117-128 .

(Instructor)
Instruction for reflection & messages(Student)

1) Daily Reflection:・Something I did, noticed, felt, or learned.
・Why it is important.
・How I will use it in my life.

2)Goal setting & evaluation:
・My achievement of the day.
・My goal for tomorrow.

(Peer Advisor)

Comment to the reflection

Figure 3. Reflective journals in ePortfolios used during study abroad

Research methods
As was done in Cutting et al. (2020) students’ descriptions about usage of the target skill while abroad werecollected from their daily reflection records. The researchers read students’ reflection records thoroughly andextracted the descriptions which indicated students use or intended use of the target skill, following thedeductive coding approach of Sato (2008). A researcher in the field of international education supported thefirst author in the selection of appropriate segments. The researchers discussed each selected segment and whenagreement was reached on all the segments, it was concluded that theoretical saturation had been achieved.
Prior to this research, all participants provided written and verbal consent to make use of their ePortfolio
submissions.
An in-depth analysis of the selected descriptions was conducted, which identified how students used their skillsin the following areas:• Interlocutors: To whom did students ask cultural questions?
• Content of cultural questions: What are their cultural questions?
• Usage: Were they able to ask questions?
To identify the context, the researchers read the extracted descriptions along with the original data andconducted deductive coding to identify the interlocutors and content of cultural questions. Then code labels wereassigned and modified throughout the coding process iteratively to represent the interlocutors and content of cultural
questions. The other researcher supported this process and the coding was completed when agreement wasreached across all the codes.
Next, we arranged the selected segments and their code labels in chronological order and examined if there wasany tendency or developmental process in students’ skill usage over time. In addition to this qualitative analysis,a self-rating survey with the question “I ask questions about cultures” was given to the students to bolster
results as quantitative data. We limited our analysis of students’ daily reflection records to days 1 to 19 tocorrelate with a self-rating survey that was completed immediately after this period. Finally, the evidence andanalysis of students’ portfolios was compared with pre-departure training and ways to improve pre-departure
training was discussed.

IJEMT, Vol. 15, No. 1, 2021, ISSN 1882–2290

121

International Journal of Educational Media and Technology
2021, Vol. 15, No. 1 pp.117-128 .

Results
This study used the daily reflection data of 15 students out of 16 participants of the program who submitted
both reflections and self-ratings. The total number of reflection entries of 15 students up to day 19 was 226.
The average of students’ portfolio submissions per day was 79%. In their reflections, 8 students out of 15 (53%)
wrote about using the skill, 12 segments in total. Table 2 shows the list of 12 segments written by 8 students(Students A to H) from days 1 to 19 in chronological order. Through deductive coding, we identified ‘Interlocutor’
Table 2Students’ description of using the target skill and identified real-life context
Identified context

Students& Dates

Students’ descriptions in reflective journals about using the skill

Student A
Day 1

Interlocutor

Content ofUsage
cultural questions

About cultures, I felt there are so many people who are half naked. Texas issurely hot, but Japan is …I don’t know what makes such difference, so I
will try to ask a buddy about it.

Buddy

Observation△

Student B
Day 4

After class, I sent a message to a buddy and asked which coffee shops herecommends around school. Since I am in the US, I felt I should explore
different coffee shops to study. …I truly respect our buddies’ spirit to help.
So thankful.

Buddy

Local
recommendation✓

Student E
Day 4

Because of a comment from my teacher, I learned my gestures could give anegative image to some people. I also asked questions about an expression I
know to a buddy and found out people may consider it as racist. Asking
questions is surely important and I want to ask questions more.

Buddy

Expression✓

Student C
Day 6[The RA] asked questions and asked about Japanese language to me. Using
this time, I asked the same questions back to her and explained usingdrawings and photos. With these, she talked to me a lot.

RA

Language✓

Student H
Day 6After the tour, some students were asking questions to the guide (at thevolunteer site) and understanding by nodding. However, I couldn’t
understand and felt terrible. I had lots I wanted to ask about thisorganization but not being able to understand what they say was frustrating.

Guide(Volunteer)at Volunteer
site

Volunteer site×

Student A
Day 7

I asked a buddy about a kick skater. Lots of people are riding them and I
learned people can rent one after installing an application.

Buddy

Observation✓

Student F
Day 7

I talked to our buddies on the street. Starting from “it’s hot again,” I kept
asking questions about whatever I saw in town. This is my first time to staythis long overseas, so this new culture is very exciting ….

Buddy

Observation✓

Student G
Day 10I will write about the questions I asked to RAs today and what I thought about
their answers. Question 1: In the parking lot, it seems…Question 2: In
Texas, do Republicans….Then I heard Texas is Republican, but.... Theiranswers were interesting. I want to keep asking more questions from now on.

RA

Observation,
Politics, etc.

✓

Student D
Day 11

I asked (volunteers) about the election in the US which I learned about inclass because I wanted to see if I can talk about something academic.
However, their vocabulary was difficult, and the speed was fast, so it was hardto catch all. But they explained some words to me and gave various examples,so I was thankful. I still lack listening skills. I will work on it.

Volunteer

Politics✓

Student B
Day 14

I asked my buddy about politics in the US, because I couldn't understandthe pros of the Republicans in my Multicultural Society class. He told me that
…I wanna know about politics of the U.S. more and more.

Buddy

Politics✓

Student H
Day 18[At a volunteer site, in a freezer room] There was a foreign volunteer next tome and I wanted to ask various questions about volunteering to the person.
But I was so cold that no word came from my mouth.

Volunteer

Volunteering×

Student C
Day 19

Most shockingly, when I asked him [a dorm resident] about the bombing
in Hiroshima, he said American aircrafts … I wasn’t taught such a thing, solater I looked into it. It seems Japan was... World is big. My view is narrow.
And topics are difficult. I really felt I need more vocabulary for academic
conversation, but it was a lot of fun!

Dorm
resident

War✓(Japanese descriptions were translated by the first author)✓: Used the skill, △: Intention to use the skill, ×: Couldn’t use the skill

IJEMT, Vol. 15, No. 1, 2021, ISSN 1882–2290

122

International Journal of Educational Media and Technology
2021, Vol. 15, No. 1 pp.117-128 .

and ‘Content of cultural questions’ from each segment and allocated codes, which are displayed in the right columns
in Table 2. We highlighted the words or phrases which showed the ‘interlocutors’ and ‘content of cultural questions.’
The identified ‘Interlocutors’ are as follows: Buddies, Volunteers, Residential assistants, Dorm residents. The identified‘Content of cultural questions’ from are: Observation, Politics/War, Volunteer site/Volunteering, Local Recommendation,
Expressions, Language. The category of ‘Observation’ signals students’ questions about what they saw and observed.
Questions about politics and wars are labelled as Politics/War. Volunteer site/Volunteering is questions about avolunteer site students visited weekly or about volunteering there. Local Recommendation is questions to local
people about their recommended places to visit. Expressions includes questions about certain English
expressions the students encountered and the cultural acceptability of these expressions. Language refers toquestions posed to learn about the language. We also checked if the skill was used or not from the descriptions
and signified by ✓when the skill was used, × when the skill was not used, and △ for the intention to use theskill.
From Table 2, ‘interlocutors’ and ‘content of cultural questions’ were extracted and listed in Tables 3 & 4 to see
frequency and tendency. Students asked cultural questions to their buddies (students in the host institution who
support international students) most frequently (6 descriptions), followed by volunteers students worked withwhile volunteering (3), RAs (Residential Assistants in the dorm) (2), and a dorm resident (1). The most frequently
asked cultural question is Observation (4), followed by Politics (3)/War (1), Volunteer site/Volunteering (2), Local
recommendation (1), Expressions (1), and Language (1).
Table 3
Types of interlocutors and frequency of skill use
Interlocutors
Frequency

Buddy

Volunteer

RA (Residential assistant)

Dorm resident

6

3

2

1

Table 4
Types of content and frequency of skill use
Content of cultural
questions
Frequency

Observation

Politics/War

Volunteersite/Volunteering

Local recommendation
Expressions
Language

4

Politics 3, War 1

2

1

C

Transition of use of the skill over time
Table 5 displays students’ cultural questions from Table 2 in chronological order to see any transition ordevelopment in using the skill. In the earlier stage from days 1 to 10, students asked questions about Observations
the most (days 1, 7, 7, 10) along with questions about a Local recommendation (day 4), Expression (day 4), Language(day 6), and Volunteer site (day 6). After these, questions about Politics emerge from days 10 to 14, followed byanother political question about War in day 19. In cultural learning, questions about Observations are relatively
easy and simple to ask as visible parts of cultures. Politics are more complicated to discuss and comprehend, thus
requiring elaborate interaction. Although the data is limited and the progression is not rigid, examining students’use of the skill over time, rather simple cultural questions appeared at the beginning and more complex cultural questions
emerged later.

IJEMT, Vol. 15, No. 1, 2021, ISSN 1882–2290

123

International Journal of Educational Media and Technology
2021, Vol. 15, No. 1 pp.117-128 .

Table 5
Cultural questions in chronological order
Dates

Interlocutor

Content of cultural questions

Student A Day 1

Buddy

Observation

Student B Day 4

Buddy

Local recommendation

Student E Day 4

Buddy

Expression

Student C Day 6

RA

Language

Student H Day 6

Volunteer (Guide at
Volunteer site)

Volunteer site

Student A Day 7

Buddy

Observation

Student F Day 7

Buddy

Observation

Student G Day 10

RA

Observation, Politics, etc.

Student D Day 11

Volunteer

Politics

Student B Day 14

Buddy

Politics

Student H Day 18

Volunteer

Volunteering

Student C Day 19

Dorm resident

War

Simple cultural
questions

More complex
cultural questions

Self-rating
From days 19 to 26, a self-rating survey with the question “I ask questions about cultures” was conducted tocollect quantitative data. Table 6 contains the self-rating results of Students A to H as well as their description
content and its frequency. Students A, B, C, D had the highest self-rating Agree about using the skill. Amongthem, Students A, B, C wrote about using the skill 2 times in reflective journals, which is more frequent than
the rest, and their ratings of using the skill seem to be supported by their reflection record. Students E and F
rated Somewhat agree and they documented their use one time in their reflections. Students G and H rated
themselves as Somewhat disagree, the lowest among the participants. Student H, especially, wrote s/he could not
ask cultural questions two times in the reflections and his/her negative self-rating also seems to be supportedby the documented evidence.
Table 6Students’ descriptions in reflective journals and their self-ratings
Dates

Interlocutor

Content of cultural questions

Usage

Frequency ofdescription

Self-rating

2

Agree

2

Agree

2

Agree

Student A Day 1

Buddy

Observation

△

Day 7

Buddy

Observation✓

Student B Day 4

Buddy

Local recommendation✓

Day 14

Buddy

Politics✓

RA

Language✓

Dorm resident

War✓Student D Day 11

Volunteer

Politics

✓

1

Agree

Student E Day 4

Buddy

Expression✓

1

Somewhat agree

Student F Day 7

Buddy

Observation✓

1

Somewhat agree

Student G Day 10

RA

Observation, Politics, etc.

✓

1

Somewhat disagree

Student H Day 6

Guide atVolunteer site

Volunteer site×

Somewhat disagree

Volunteering

×2

VolunteerStudent C Day 6
Day 19Day 18It must also be noted that there were 7 students who did not document their use of the skill in reflective journals.
The right column of Table 7 contains a list of those students. Among these, 4 answered Agree, 2 answered
Somewhat agree, and 1 Somewhat disagree. Overall, no student answered Disagree, which indicates all participants

IJEMT, Vol. 15, No. 1, 2021, ISSN 1882–2290

124

International Journal of Educational Media and Technology
2021, Vol. 15, No. 1 pp.117-128 .

asked questions to some extent.
Table 7
Total self-ratings of use of the skill
Self-rating

Agree
Somewhat agree
Somewhat disagree
Disagree

Total number of
respondents

Ratio

8
4
3
0

53%
27%
20%
0%

The respondents who documented The respondents who did NOT
their use in reflective journals
document their use

4
2
2
0

4
2
1
0

Challenges students faced in using the target skill while abroad
Although we provided practice to use the target skill prior to study abroad, some students documented thatthey struggled in using the skill due to difficulty in English. As in Table 8, Student H could not ask questions
because s/he did not understand English speech at a volunteer site. Student D reported English vocabulary
was difficult and the speaking speed was fast when s/he asked questions about politics. Likewise, Student C felt
difficulty in conversing about political topics and wrote about the need for vocabulary for academic
conversations. Students also documented their emotions while using the skill, as Student H wrote “frustrating”
and Student C wrote “it was a lot of fun!” Students’ reflective journals capture affective evidence in students’learning.
Table 8Difficulties students faced in using the skill
Student H (Day 6)

Student D (Day 11)

Student C (Day 19)

After the tour, some students were asking questions to the guide and understanding by nodding.
However, I couldn’t understand and felt terrible. I had lots I wanted to ask about thisorganization but not being able to understand what they say was frustrating.
I asked (volunteers) about the election in the US which I learned in class because I wanted to see if I
can talk about something academic. However, their vocabulary was difficult, and speed was fast,
so it was hard to catch all.
Most shockingly, when I asked him (a dorm resident) about the bombing in Hiroshima, he said
American aircrafts … I wasn’t taught such a thing, so later I looked into it. It seems Japan was...
World is big. My view is narrow. And topics are difficult. I really felt I need more vocabulary foracademic conversation, but it was a lot of fun!

Discussion
How to improve pre-departure training
This in-depth analysis revealed detailed contexts of how students used the skill over time and what challenges
they faced during study abroad. Using these findings, we now discuss how our pre-departure training can be
improved. The current pre-departure training provides roleplay practice in simulated settings for asking cultural
questions to promote behavioral changes during study abroad. As noted earlier, it is critical to make the training
context similar to real-life settings for training effectiveness (Suzuki, 2015b). Thus, we compare this training
context with real-life evidence captured in students’ reflective journals. The differences between training andstudy abroad evidence are listed in Figure 4.
1) Buddies and observations - matched training simulation
First, in our training, students practice asking cultural questions to their buddy, and their questions are about
what they saw or noticed (observations). In their daily reflections, students most frequently asked cultural
questions to their buddies and the most common questions were based on observations, which indicate our training
context matches students’ real-life situations.
2)

Buddies as important cultural guides

IJEMT, Vol. 15, No. 1, 2021, ISSN 1882–2290

125International Journal of Educational Media and Technology
2021, Vol. 15, No. 1 pp.117-128 .

The importance of host receptivity and the buddy scheme in students’ learning abroad is claimed by Jackson(2020a) and Cutting & Megumi (2017). Notably, these findings signified that buddies served as cultural guides
for international students providing opportunities for cultural inquiry. A further suggestion is that interculturaltraining between buddies and sojourners may promote reciprocal deeper learning.
3) Additional contexts – not addressed during training
Students’ journal evidence, however, revealed that the interlocutors were not only buddies. These included dorm
residents, RAs (resident assistants) and volunteers at a volunteering site. The content of their questions was also notconfined to observation and ranged from local recommendations, and expressions, to politics. Since training focused on
only asking cultural questions to buddies about observation, these findings suggest the need for contextual
expansion of the training as seen in Figure 4.

Figure 4. Comparison between training context and real-life context

4) Issues of complex cultural questions and language
In our study, students’ cultural questions, when arranged in chronological order, indicate a tendency that
relatively simple cultural questions (i.e., observations, local recommendations, expressions) are asked at the beginning andmore complex questions (politics, war) emerge later. Some descriptions revealed students felt it was difficult to ask
complicated cultural questions because of language difficulties. The more complex the questions are, the moreadvanced English is required. Thus, imagining why political questions emerge later is not difficult. This finding
raises critical issues. Although we observed students’ struggles in English in deeper intercultural dialogues, manymodels of intercultural competence do not address linguistic competence (Byram & Golubeva, 2020; Deardorff& Arasaratnam-Smith, 2017) and research on integrating intercultural development and language competence
is much needed. Moreover, according to Jackson (2020b), researching “host-sojourner interaction” to identify
what facilitates or hinders “intercultural communication and relationship building” is still new” (pp.448-449).
Our findings indicate language issues as hindering factors in complex intercultural inquiry. Thus, incorporating
linguistic components to handle advanced cultural dialogues may be necessary in training.

Affective evidence in reflective journals
Griffith et al. (2016) observe that existing measures of intercultural communication often do not sufficiently
collect affective and behavioral evidence which are essential in viewing interactions with local hosts. The
reflective journals in ePortfolios captured students’ affective aspects in their learning, such as their feelings offrustration and excitement in addition to behavioral evidence.

Kirkpatrick’s model for pre-departure training
Lastly, Kirkpatrick’s model suggests the importance of evaluating behavior (Level 3) after training in workplaces.
However, how behavior can be analyzed in study abroad settings and how such findings can improve training
have not been discussed. This study therefore may provide suggestions for using Kirkpatrick’s model tounderstand students’ behavior during study abroad and use real-life evidence for pre-departure training. We
also found cases where students self-rated that they were using the skill, but no evidence was seen in their

IJEMT, Vol. 15, No. 1, 2021, ISSN 1882–2290

126

International Journal of Educational Media and Technology
2021, Vol. 15, No. 1 pp.117-128 .

reflections. A possible reason is lack of submissions and/or insufficient content within, and this limited ourdata collection and analysis. Ample submissions and content are indispensable for analysis of reflective journals.

Conclusion
Capturing students’ learning evidence abroad has long been a challenge and using such evidence to improve
pre-departure training is still lacking. Our study applied Kirkpatrick’s evaluation model and conducted in-depth
analysis of behavior (Level 3) documented in sojourners’ reflective journals to see how they were using the target
skill over time. Then we employed the results in a discussion on how to improve pre-departure training.
Sojourners’ journal evidence revealed students most frequently asked cultural questions to buddies and thecontent was about observations, which matches our pre-departure training context. The results signaled theimportance of buddy schemes for intercultural interaction. However, sojourners had cultural engagement withmore diverse people in various content, which suggested expansion of the training context. Moreover, students
seem to start asking more complex cultural questions later, such as about politics. The evidence uncovered sojourners’
language difficulty with such complex cultural questions and suggested incorporating linguistic components tohandle advanced cultural dialogues. Our reflection data also captured affective evidence of study abroad and enabled
us to witness students’ lives abroad, filled with excitement, struggles, and surprises. To optimize students’learning abroad, more research is needed to better understand sojourners’ behaviors in intercultural engagement
to improve training. Reflective journals in ePortfolios may be a key to connect students’ learning abroad tofuture training.

Acknowledgment
The authors would like to thank Dr. Tatsuya Hirai for his continuous support with our research and Dr. Peter
Roux for critical suggestions.

References
Byram, M., & Golubeva, I. (2020). Conceptualizing intercultural (communicative) competence and intercultural
citizenship. In J. Jackson (Ed.), The Routledge handbook of language and intercultural communication (pp. 7085). Oxon, OX: Routledge.
Byram, M. (1997). Teaching and assessing intercultural communicative competence. Clevedon: Multilingual matters Ltd.
Blair, D., & Cutting, M. (2015). Assessing global learning at home and abroad: Assessing global Learning:measurement, implications, and applications, Webinar presentation in NAFSA Global Learning Faculty
Conversation Series, October 28, 2015.
Cutting, M. (2015). Application of "significant learning" theory in a short-term study abroad program:Categorization of learning outcomes in study abroad and a new approach to learner-centerededucational design. [In Japanese]. Intercultural Education, 41, 111-126.
Cutting, M. (2016). Assurance of learning in international education: Assessment and outcomes in our studyabroad programs. [In Japanese]. Journal of Japan Association for College and University Education, 38(2),67-76.
Cutting, M., Goda, Y., & Suzuki, K. (2020). Instructional method and impact of pre-departure training for study
abroad: From the viewpoint of transfer of learning [In Japanese]. Journal of Japan Association for Collegeand University Education, 42(1), 95-104.
Cutting, M., & Megumi, M. (2017). A practice of Japan-US global collaborative education: Intercultural
collaboration of both students and teachers and use of the buddy system [In Japanese]. Intercultural
Education, 46, 93-109.
Deardorff, D. K. (2015). Demystifying outcomes assessment for international educators. Sterling, VA: Stylus.
Deardorff, D. K. & Arasaratnam-Smith, L.A. (2017). Intercultural competence in higher education. Oxon, OX:
Routledge.
Eynon, B., & Gambino, L. M. (2017). High impact ePortfolio practice: a catalyst for student, faculty, and institutional
learning. Sterling: Stylus Publishing, LLC.

IJEMT, Vol. 15, No. 1, 2021, ISSN 1882–2290

127

International Journal of Educational Media and Technology
2021, Vol. 15, No. 1 pp.117-128 .

Griffith, R. L. Wolfeld, L. Armon, B.K., Rios, J., & Liu, O.L. (2016). Assessing intercultural competence inhigher education: existing research and future directions. Research Report, ETS RR16-25, 1-44.
Jackson, J., & Oguro, S. (2017). Intercultural interventions in study abroad. Oxon, OX: Routledge.
Jackson, J. (2020a). Online intercultural education and study abroad. Oxon, OX: Routledge.
Jackson, J. (2020b). The language and intercultural dimension of education abroad. In J. Jackson (Ed.), The
Routledge handbook of language and intercultural communication (pp. 442-456). Oxon, OX: Routledge.
Kolb, D.A. (1984). Experiential learning: experience as the source of learning and development. Englewood Cliffs, NJ:
Prentice Hall, Inc.
Kawai-juku.(2018). Nihonjin no kaigairyugaku ni kansuru kokasokutei [Survey of the impact of study abroadon Japanese people in 2018 -translation]. [In Japanese]. Retrieved January 5th, 2021, fromhttps://www.mext.go.jp/a_menu/koutou/ryugaku/1411310.htm
Kirkpatrick, D. L., & Kirkpatrick, J.D. (2006). Evaluating training programs. Oakland, CA: Berrett-Koehler
Publishers, Inc.
Morimoto, Y. (2015). Educational big data as e-portfolios and learning analytics. [In Japanese]. Computer &Education, 38:18-27.
Salisbury, M. (2015). How we got to where we are (and aren’t) in assessing study abroad learning. In V. Savicki.,& E. Brewer (Eds), Assessing study abroad (pp. 15-32). Sterling, VA: Stylus.
Shoraku, A., & Yoshida, M. (2018). Learning outcomes of overseas studies in an undergraduate program [InJapanese]. Kobe Journal of Higher Education, 26, 59-78.
Suskie, L. A. (2018). Assessing student learning: A common sense guide. San Francisco, CA: Jossey-Bass.
Suzuki, K. (2015a). Proposal of inverted triangle model for training design based on 4 levels of evaluation. [InJapanese]. Proceedings in the 45th Annual Conference of Japanese Society for information and Systems in Education,
September.
Suzuki, K. (2015b). Kenshu Sekkei Manual [Training design manual-translation]. Kyoto: Kitaoji-shobo.
Vande Berg, M. P., Paige, R. M., & Lou, K. H. (2012). Student learning abroad. Sterling, VA: Stylus Publishing
LLC.

IJEMT, Vol. 15, No. 1, 2021, ISSN 1882–2290

128