ICoME 2021

Evaluating Impact of Pre-Departure Training on Study Abroad
Outcomes by Adapting Kirkpatrick’s Evaluation Framework
Miki Cutting
cutting@apu.ac.jp
Associate professor
Ritsumeikan Asia Pacific University
Beppu, Japan
Doctoral student
Kumamoto University
Kumamoto, Japan
Yoshiko Goda
ygoda@kumamoto-u.ac.jp
Associate professor
Kumamoto University
Kumamoto, Japan
Katsuaki Suzuki
ksuzuki@kumamoto-u.ac.jp
Professor
Kumamoto University
Kumamoto, Japan
Abstract:
Study abroad outcomes are complex and not easy to obtain. In the midst of these challenges,
evaluation of pre-departure training for study abroad is needed, yet little is researched. How
can we evaluate the effectiveness of pre-departure training when study abroad outcomes are
intricate? In this study, by employing Kirkpatrick’s training evaluation model and designing
pre-departure training to promote training transfer, we evaluated training impact on study
abroad outcomes. For evaluation, we employed both qualitative and quantitative measures,such as the Intercultural Development Inventory (IDI), showcase portfolios, and surveys. The
findings suggested that training designed for training transfer enhanced attainment of our study
abroad outcomes. Furthermore, our attempt indicates that Kirkpatrick’s model is an applicable
framework to evaluate pre-departure training for study abroad outcomes.
Key words: Intercultural competence, Instructional design, Outcome assessment, Showcase
portfolios, Training transfer

INTRODUCTION
In the field of study abroad, merely sending students overseas is not enough for intercultural learning,
and intentional intervention is considered important (Jackson, 2020; Vande Berg et at. 2012). It has beenreported that students who are studying abroad have limited meaningful interaction with local students(Jackson & Oguro, 2017) and even when intercultural contact increases, one’s bias may enlarge due tounpleasant experiences (Yashiro et al. 2009). Intentional interventions, such as pre-departure training,support sojourners’ learning abroad, yet Halenko & Jones (2017) claim that its research is still undeveloped.
To evaluate pre-departure training, we need to know its impact on study abroad (SA) outcomes. However,outcomes in international education are, as Deardorff (2015) defines, developmental, experiential, andcomplex in nature, and not straightforward to assess. How can we know if training is effective when SA

ICoME 2021

outcomes are complex? Is there any model or framework to be used? Blair (2017, p.122) reports, “there
simply is no single instrument, method, or assessment moment that allows us to capture the complexity ofthe data.” Our study focuses on this critical issue from the viewpoints of training evaluation and discusses
evaluation framework of pre-departure training.
Training evaluation model and training transfer
In the field of training evaluation, Kirkpatrick’s evaluation model has been the de facto standard andthe model consists of four phases in evaluating: Level 1: reaction, Level 2: learning, Level 3: behavior,
Level 4: results (Suzuki, 2015). Kirkpatrick & Kirkpatrick (2009) explains importance in an evaluation
design to identify a few critical behaviors, which are “specific, observable, measurable behaviors thetraining graduates should perform” that will likely lead to expected results, emphasizing the criticality ofbridging training to behavioral change after training. In the research of training transfer, positive training
transfer is “the extent to which the learning that results from training experience transfers to the job andleads to meaningful changes in work performance” (Baldwin et al., 2009, pp. 41-42).
Training evaluation model for pre-departure training
A study by Cutting et al. (2020a) employed Kirkpatrick’s model to evaluate the effectiveness of predeparture training design for study abroad focusing on training transfer in Level 3 (behavior). They firstfound training transfer was not occurring after their initial pre-departure training, so they redesigned
training to focus on training transfer utilizing Kirkpatrick’s model. Through monitoring participants’ dailyreflections while abroad, the study suggested that transfer-focused training impacted training transfer(Cutting et al., 2020a). However, the study did not examine if training led to SA outcomes.

RESEARCH DESIGN & METHODS
Although pre-departure training is considered integral, it lacks evaluation research. Evaluating its
impact is especially difficult when SA outcomes are experiential and complex. This research targets predeparture training designed to enhance training transfer by Cutting et al. (2020a) and evaluates training
impact on SA outcomes using Kirkpatrick’s model.
Our research question is as follows:
RQ: Does pre-departure training designed to enhance training transfer bring out expected SA outcomes?
Pre-departure training for study abroad
The study abroad program in this research is operated by a Japanese private university which sends
a group of Japanese students to a US institution for 2 months to take credit bearing classes. The programaims to prepare students to work in a global society according to the mission of the institution. To maximize
study abroad experiences, pre-departure training (7 weeks, 14 class hours) as well as re-entry sessions (3times-monthly, 6 class hours) are offered. The participants are required to have completed an intermediate
English course offered at the institution and their English levels range from TOEFL ITP 460 to 500.
Adapting Kirkpatrick’s model to design training evaluation

ICoME 2021

We designed pre-departure training evaluation by following the flow of Kirkpatrick & Kirkpatrick(2019). According to Kirkpatrick’s model, analysis starts from refining Level 4 (results), which is often an
organizational impact such as ROI, after identifying organizational needs. This study focuses on evaluating
training effectiveness on SA outcomes after training, thus adapting Level 4 for our educational context.
We first set SA outcomes considering the needs of the program. Then we identified critical behaviors
in study abroad (Level 3) which are expected to lead to these outcomes. As learning goals in training (Level2), we identified required knowledge, skills, and attitudes. This evaluation scheme is exhibited by adding
our own study abroad context and using the ADDIE model in ID in Figure 1.

Figure 1. Application of Kirkpatrick & Kirkpatrick (2009) in designing pre-departure training evaluation

Study abroad (SA) outcomes
To optimize students’ learning in SA, our pre-departure training created goals for interculturalcommunication, English communication, and perspective taking. As mentioned earlier, preparing students
to broaden perspectives instead of enlarging biases and enhance meaningful intercultural interactions are
critical for maximizing the SA environment. Thus, perspective taking goal requires suspending one’s own
judgement of others and seeking others’ viewpoints (Cutting et al., 2020b) to broaden perspectives as wellas having meaningful interactions with local people. This goal was refined through previous research(Cutting, et al., 2020b) and employs the theory of DMIC (Developmental Model of InterculturalCompetence) by Bennet (1986), which explains people’s perspective shifts from ethnocentric views to more
ethnorelative views in intercultural competence (Hammer, 2011).
Based on these, SA outcomes of perspective taking are set as follows:
SA outcomes: Perspective development by suspending one’s judgement and seeking others’ perspectives.
Critical behavior (CB) for SA outcomesTo enhance attainment of SA outcomes, pre-departure training aims that student possess theknowledge, attitude, skills, and English ability to attempt to broaden their perspectives and has sub-goals
in knowledge, attitudes, and skills (Cutting et al., 2020b). Among these, we selected a sub-goal as a critical

ICoME 2021

behavior in Level 3. Following Cutting et al. (2020a; 2020b), we selected the following sub-goal to monitor
for training transfer and in this research, we call it critical behavior using Kirkpatrick’s term:
CB: Students can ask questions in English about culture, such as cultural differences and surprises.
Pre-departure training was designed to enhance training transfer and assured the attainment of
the goal by all participants by providing performance tests repeatedly through roleplays in simulated
settings (Cutting et al., 2020a; 2020b). ePortfolio systems are used throughout the program to enhance,
document, and evaluate students’ learning (Cutting et al., 2020b).
Methodologies: Evaluation of SA outcomesTo evaluate the impact of pre-departure training on study abroad outcomes, we use Intercultural
Development Inventory (IDI), surveys, and showcase portfolios. The methodologies are listed in Table 1.
Table 1. SA (study abroad) outcomes and evaluation methods
Explanation

Evaluation tools

Showcase portfolios
SurveysAnalyzed perceived impact of training CB

Interviews

Analyzed personal changes after SA

IDI (Inventory)
Perspective development bySAoutcomes suspending one’s judgement of others
and seeking others’ viewpoints.

Methodologies
Measured pre/post increase-compared with a controlled group
Analyzed if CB led to SA outcomes-compared with a controlled group

IDI (Intercultural Development Inventory)

To evaluate SA outcomes, we employed The Intercultural Development Inventory (IDI), since
IDI has been created based on the DMIS theory which provided foundation for our goal. IDI is a 50-item
questionnaire to assess developmental stages in people’s worldviews towards differences. To see ourtraining impact, we compared the results with a control group who took the previous training. The previous
training had the same goal, but training did not focus on training transfer without assuring the attainmentof the critical behavior and lack of training transfer was found (Cutting et al., 2020a). Here, we call transferfocus training T-focus training and non-transfer-focus training Non-T-focus training.
Showcase portfolios

Since inventory does not indicate the causes of the results, we analyzed students’ showcase
portfolios in which they wrote about their attainment of SA outcomes and investigated if training of thecritical behavior led to SA outcomes. Barret (2010) explains that in most showcase portfolios, students
reflect on their achievement of learning goals by thinking over their selected evidence and write about theirlearning, its significance, and next learning goals. Our students created showcase portfolios in our re-entry
sessions by reflecting on, selecting, and exhibiting their outcomes as well as their next goals. In their
showcase portfolios, we examined how they developed their perspectives, especially the effect of thecritical behavior. We analyzed their descriptions of outcomes in showcase portfolios following thedeductive coding approach by Sato (2008) and examined whether outcomes were attained by the criticalbehavior. Then, we compared the results between T-focus and Non-T-focus training.

ICoME 2021

Surveys

To bolster our data, we conducted surveys to investigate if training of the critical behavior led to SA
outcomes. We conducted closed questions as well as open-ended questions. Questions were a 4-point Likert
scale from 4 (Strongly agree) to 1 (Strongly disagree). The survey, which was anonymous and consented
upon, was given to all the participants in the T-focus training 6 months after their return. A limitation ofthis study is that the survey was created after T-focus training was designed and therefore we only presented
the results of T-focus training.

RESULTSIDI results
To understand training impact on SA outcomes, we conducted IDI before and after study abroad
for students in T-focus training and Non-T-focus training. In Non-T focus training, 21 out of 22 students(95%) and in T-focus training, 15 out of 16 students (94%) completed pre-posttests. T-tests indicated IDI
increase in both training. To compare effect of training onto post-test by eliminating effect of pre-test, we
conducted ANCOVA to perform pre-posttest analysis where the pre-test is used as a covariate, the training
types as an independent variable, and the post-test as a dependent variable. There is a significant effect oftraining type on the post-test after controlling for the pre-test, (F [1, 33] =4.29, p<0.05) and students in Tfocus training had statistically higher increase in post-test. Also 87% of students had an increase in IDI in
T-focus training, whereas, 57% of students had an increase in the Non-T-focus training.
Table 2. Results of IDIPre

IDI
Non-T-focus (n=21)
T-focus (n=15)

M
85.3
84.3

IDI

Post
SD
(13.39)(11.51)

M
90.5
96.7

Non-T-focus

SD(13.68)(12.82)
T-focusPre (SD)

Post (SD)Pre (SD)

Post (SD)85.3(13.39)

90.5(13.68)

84.3(11.51)

96.7(12.82)

Non-T focus (n=21)
T-focus (n=15)

Increase
1213

Decrease
9
2

t

p

-2.32
-4.72

<.031
<.0003

ANCOVAF (1,33)=4.29, p=0.046

Increase
57%
87%

Showcase results
Showcase portfolios were submitted by 21 out of 22 students (95%) in Non-T-focus training and
16 out of 16 students (100%) in T-focus training. In Non-T-focus trainees’ showcase, there was no evidenceof CB, which is asking questions to seek others’ perspectives. For example, this student explained his/her
perspective development from observation: Something I was able to notice as visible differences as, for example,
some labor workers seemed to be black people, but I did not see such difference among students. On the other hand,

8 T-focus trainees out of 16 (50 %) indicated their perspective change by seeking others’ viewpoints byasking question as CB.

ICoME 2021

Table 3. Results of showcase portfolio analysis: Evidence of SA outcomes obtained by the critical behavior
Extracted descriptions of goal attainment in showcase portfolios

Non-T-focus

None

T-focus

When I encountered something different from myself or Japanese people, instead of feeling strange, I could accept it as a difference. And I
gained an attitude to ask and search why there’s such a difference.
I tried to ask questions when I didn’t understand or was curious. Well, mostly about music. At the beginning, I was not asking questions
much, so I was often listening to others’ questions. But what I am curious about is not necessarily what others want to know, so I thought it ismyself who needs to ask. I think my questions to teachers increased recently. Self-solution and solution by friends decreased, I think.
I was feeling these volunteers (at the volunteer center) were volunteering as part of a school requirement. But when I asked for reasons why
they were coming, I heard lots of voices which surprised me such as, ‘because it is fun,’ ‘because I can meet and talk with lots of people.’
I told M that we learned about slavery systems in Japan like this and asked him ‘how is it really?’
Since I was naturally interested in other cultures, I think I was able to ask questions. But I was not as aware of my own culture as I thought, sowhen I was asked about my own culture, I looked into it.
When I visited my buddy’s house, his/her parents were saying they like sushi. When I asked what kinds they like, I was surprised that themother said she likes sea urchins, since I saw on TV that foreign people do not eat sea urchins much. So, I was tempted to say, ‘foreign people do
not eat sea urchins, so you don’t like them, do you?’ But then this question is from my assumption. So instead, I asked, do people here eat
sea urchins? Then she said, ‘not so much.’
I was curious about American food, since I did not want to gain weight. I often hear people say that American food makes people fat. But when
I actually asked people about it, healthy food tends to cost more than unhealthy junk food, and because of that reason, many people buy junkfood. When I went to the supermarket, salads were indeed expensive.
By going to America, I was able to change my stereotypes greatly (finding various views of policies and regulations, etc.) … By recognizing this,I was able to ask questions with care and to share my culture and values.
I was talking (with a dorm resident) about how history is taught in each country. After listening to his explanation, I asked about the case of theHiroshima atomic bomb. I was shocked at what he said and felt “What? Really?” After that, I researched to see if it was true. Then I found…

Student 1
Student 2
Student 3
Student 4
Student 5
Student 6
Student 7
Student 8-Bold type shows identified segments that indicate when students sought others’ perspectives by asking questions.

Figure 2. An example of students’ showcase portfolios of study abroad outcomes
Survey results
As survey results, 11 students (68%) completed survey and the mean of perceived effectivenessof training critical behavior ranged from 3.5 to 3.9 out of 4.0, highly positive in all questions.
Table 4. Results of surveys: effectiveness of training critical behavior
Practice of critical behavior led me to… (n=11)
Level 3 behavior in study abroad(3) Become more conscious to ask cultural questions(4) Ask cultural questions during study abroad(5) Use English more
Study abroad outcomes(7) Broaden my perspectives
SA outcomes (1)(6) Learn new cultures with curiosity
SA outcomes (2)(8) Deeply communicate with local people SA outcomes (2)(9) Mutually understand in a host country
SA outcomes (2)

M

SD

3.8
3.5
3.9

.39
.50
.29

3.9
3.8
3.5
3.6

.29
.39
.66
.48

Next, data from open-ended questions in the same surveys were open coded to identify the impactof training. Table 5 shows the responses and coded impact. We found that skill training had impact onperspective development (n=4), suspending judgement (n=5), promote critical behavior (n=4), consideringothers (n=2), and arouse interest (n=2).

ICoME 2021

Table 5. Students’ comments in surveys: effectiveness of training critical behavior
T-focus
Student (a)Student (b)Student (c)Student (d)Student (e)Student (f)Student (g)Student (h)Student (i)Student (j)Students’ comments about impact of critical behavior training

Coded impact

By asking and learning about cultures, my views broadened.
Through question practice, I noticed I was often making my own judgement unconsciously.
Because of that, now I can deal with things doubting my views in a positive way and thinking“what if…”
After study abroad, I came to think of ways to talk to each other in a considerate manner forothers. I really stopped feeling “no way!” or “that’s weird,” but instead, I started
questioning “why” and deepening my cultural understanding. Not only cultural things, I cansee my persona change.
I was able to ask questions smoothly during study abroad.
By learning how to ask questions, I became not hesitant to ask questions to others.
I was able to become interested in cultures, especially America and American nationality as amulti-ethnic country, and was able to ask questions proactively.
Instead of making judgement only through my bias, I think first before making judgement toview things.
I stopped my self-conclusion. I stopped my self-centered interpretation such as “maybe that’sthat, and started asking questions even about a small thing.
I became more curious about other cultures.
Casual things can be totally different, so I decided to be careful of talking about it and started
thinking that I want to learn how to talk without leading to bias or prejudice.

Perspective development
Suspend judgement,
Perspective development(attempt)
Consideration for others,
Suspend judgement,
Perspective development
Promote critical behavior
Promote critical behavior
Arouse interest
Promote critical behavior
Suspend judgement
Suspend judgement,
Promote critical behavior
Arouse interest
Consideration for others,
Suspend judgement

Lastly, to understand the impact of the skill training, we conducted interviews 6 months after study
abroad. The following interview of a student explains his/her personal transformation after training.
‘After coming back, I often had a chance to talk to international students in English during my club activity and often
acted as a translator. I tried to explain about cultural things and I tried to take initiative to ask them about it. I asked
about their preference of timing of taking a bath and ways to take one and tried to explain how ‘time’ is viewed here.
I became like this because I learned it is better to think before making judgements during the pre-departure training.
Knowledge of intercultural understanding also helped. Previously, when some members said they could not do a task
because they were busy, I often assumed that it was an excuse because everyone is busy. But now, without assuming,
I came to work with the person by asking and discussing other ways. My thoughts became flexible. I had a fixed
thinking before, but now I stop a second and think.’

DISCUSSION
Our study employed Kirkpatrick’s framework to evaluate if T (transfer)-focus pre-departure training
led to expected SA outcomes, which is perspective development by suspending judgement and seeking
others’ views by asking questions. Asking cultural questions in English was set as the critical behavior (CB).
IDI results showed T-focus training had significantly higher impact on attainment of SA outcomes than
Non-T-focus training. Moreover, in showcase portfolios, 50 % of T-focus trainees indicated their attainmentof outcomes by CB, whereas Non-T-focus trainees showed no evidence of practicing CB for the outcomes.
The survey after T-focus training indicated a positive perception of training impact on SA outcomes. Lastly,
open-ended responses revealed that T-focus training produced various perceived outcomes such as
considering others and arousing interest, in addition to the expected outcomes.
The limitation of this study was its narrow focus, but this was to manage the complexity of SA
outcomes and trace learning evidence. Moreover, Level 4 in Kirkpatrick model was not examined, which
involves organizational impact. Thus, future research is needed to understand training effectiveness in othereducational contexts. Despite the fact that our study is limited in its scope, our adaptation of Kirkpatrick’s
model in pre-departure training gave us a pathway to evaluate training impact of study abroad outcomes.

CONCLUSION

ICoME 2021

In the field of study abroad, adequate preparation is necessary especially prior to interculturalexperiences (Savicki, 2008; Deardorff, 2009), nonetheless, its effectiveness is rarely examined undercomplexity of assessing study abroad outcomes. Our study approached this issue by employing
Kirkpatrick’s framework and evaluated the impact of pre-departure training on study abroad outcomes. The
findings from qualitative and quantitative analysis using different evaluation tools suggest our transferfocused training impacted attainment of expected outcomes. Furthermore, we conclude Kirkpatrick’s model
is applicable in evaluating pre-departure training for study abroad outcomes. We believe our adaptation of
Kirkpatrick’s framework using ePortfolio systems can be utilized beyond study abroad settings, such as
field trips, teaching internships, and many other programs which send students to a real-world experience
after training. Future research is needed to examine organizational impact as Level 4.

REFERENCES
Baldwin, T. T., Ford, J. K., & Blume, B. D. (2009). Transfer of training 1988-2008: An updated review andnew agenda for future research. In G. P. Hodgkinson & J. K. Ford (Eds.), International review
of industrial and organizational psychology (24), 41-70. Chichester, UK: Wiley.
Barret, H. C. (2010). Balancing the two faces of ePortfolios. Educação, Formação & Tecnologias, 3(1), 614.
Bennet, M.J. (1986). A developmental approach to training for intercultural sensitivity. International
Journal of Intercultural Relations, 10(2), 179-196.
Blair, S. (2017) Mapping intercultural competence: aligning goals, outcomes, evidence, rubrics, andassessment. In D. K. Deardorff & L. A. Arasaratnam-Smith (Eds), Intercultural competence inhigher education (110-123). Oxon, OX: Routledge.
Cutting, M., Goda, Y., & Suzuki, K. (2020a). Instructional method and impact of pre-departure training forstudy abroad: From the viewpoint of transfer of learning [In Japanese]. Journal of Japan
Association for College and University Education, 42(1), 95-104.
Cutting, M., Goda, Y., & Suzuki, K. (2020b). ePortfolios for study abroad assessment and improvement:
Monitoring students’ use of a target skill while abroad. Proceedings in the International
Conference for Media in Education, 346-353.
Deardorff, D. K. (2015). Demystifying outcomes assessment for international educators. Sterling, VA:
Stylus.
Hakenko, N., & Jones, C. (2017). Explicit instruction of spoken requests: an examination of pre-departure
instruction and the study abroad environment. System, 68, 26-37.
Hammer, M.R. (2011). Additional cross-cultural validity testing of the Intercultural Development Inventory.
International Journal of Intercultural Relations, 35(4), 474-487.
Jackson, J. (2020). Online intercultural education and study abroad. Oxon: Routledge.
Jackson, J., & Oguro, S. (2017). Intercultural interventions in study abroad. Oxon, OX: Routledge.
Kirkpatrick, J. & Kirkpatrick, W. (2009). The Kirkpatrick four levels: a fresh look after 50 years, 19592009.http://www.managesmarter.com/managesmarter/images/pdfs/trg_20090417_kirkpatrickw
hitepaper.pdf
Sato, I. (2008) Qualitative data analysis-Its principles, methods, and practice [In Japanese]. Tokyo:
Shinyosya.
Suzuki, K. (2015). Proposal of inverted triangle model for training design based on 4 levels of evaluation.
Proceedings in the 45th Annual Conference of Japanese Society for information and Systems in
Education, September.
Yashiro, K., Machi, E., Koike, H., & Yoshida, T. (2009). Intercultural training [In Japanese]. Tokyo:
Sanshusha.
Vande Berg, M. P., Paige, R. M., & Lou, K. H. (2012). Student learning abroad. Sterling, VA: Stylus
Publishing LLC.