Poster Presentation

Analysis of Study Abroad SummerProgram

Tanya Erdelyi, Tetsuya Fukuda, and Keita Yagi
International Christian University
The purpose of this research is to investigate the extent of improvement in English test scores
after Japanese university students attended an English program overseas. Study-abroad
programs are generally associated with gains in English abilities. These gains are due to thelarge number of English classes and amount of exposure to English in authentic situations.
In a previous study, participants in a study-abroad program at a bilingual Japanese university
showed more gains with statistical significance in listening in the lower and intermediatelevels, and in structure in the higher level, than those who did not study overseas. In thecurrent study, similar results were gained with statistical analyses. This time, a comparison
was added between students who had graduated from a Japanese high school and those who
had graduated from an international school or school overseas. Our hypothesis was thatstudents who attended a Japanese high school would benefit from studying overseas during
university more than the other students because of the extensive exposure to English andpossible gains in motivation. This was rejected in terms of probability; however, its effectsize was found to be small, thus necessitating further investigation into this issue.

The purpose of this study was to examine the improvement in English test scoresof students who participated in a short-term study-abroad program at a bilingual
university in Japan. A few features should be noted about this university and its
programs in order to gain a better understanding of the context. First, before
entering the university, many of the students lived overseas, often in English
speaking countries. This results in a general trend of higher English proficiency
amongst the new students. Additionally, during the first year of their studies,most new students participate in a semi-intensive English program emphasizing
English for academic purposes. Finally, the university offers the Study English
Abroad Program, or SEA Program. Students accepted for this program have116

Analysis of Study Abroad, OnCUE Journal, 11(2), pages 116-122

the opportunity to study abroad for six weeks during the summer. Some of theparticipating ESL programs are located in Australia, Canada, England, and theUnited States. Students who do not participate in the SEA Program must attend
additional academic English courses. Considering this information, the difference
between those who participated in the SEA Program and those who did not were
measured in this study. Additionally, the impact of the SEA Program on those
who graduated from a Japanese high school and those who graduated from aninternational school or school overseas was investigated.

Background

Short-term study abroad programs have been popular at universities in Japan as
methods of increasing student intercultural communication competence. Thisis done through their direct overseas experience and increasing their foreign
language skills and motivation to learn their target language. According toKinginger (2009), study abroad is defined as “a temporary sojourn of pre-definedduration, undertaken for educational purposes” (p. 11). She holistically analyzed
the effectiveness of study-abroad programs and claimed that studying abroad
can improve learners’ language abilities in all aspects. In particular, the abilities
related to social interaction are improved, though learners’ individual differences
are huge and cannot be ignored (Kinginger, 2013).
At the present research site, Watanabe (2000) studied the impact of the SEA
Program by comparing the TOEFL ITP scores of the students who participatedin this overseas program and those who did not. He compared three groups;
advanced, intermediate, and intermediate low language proficiency students.
He found that the SEA Program participants who were at intermediate andintermediate low levels gained higher TOEFL scores in the listening section.
They also gained higher total TOEFL scores than those who did not take partin the SEA Program. He further found that the SEA Program participants whowere at an advanced level gained higher TOEFL scores in the structure section
than those who did not participate in the SEA Program. He did not, however,find a statistically significant difference between their total scores.
Considering these previous studies, it is possible to hypothesize that the117

Erdelyi, Fukuda, & Yagi

students who participate in the SEA Program can improve their English morethan those who do not. A further hypothesis can be made that the students whowent to a Japanese high school benefit more from the SEA Program than those
who went to an international school or school overseas. This is because the lattergroup of students’ English proficiency tends to be very high. Based on thesehypotheses, the following research questions were addressed:
RQ1: Who made more improvements in English, those who participated in
the SEA Program or those who did not participate?
RQ2: Who benefited more from the SEA Program, those who attended aJapanese high school, or those who attended an international high school or ahigh school overseas?

Placement of Students

The current study was conducted at a bilingual university in Japan. Along withtest scores from a mandatory TOEFL ITP taken at the beginning of the school
year, background information about the students was collected for placement
purposes. This information included the type of high school they attended( Japanese, international, or overseas), and their language experience while living
abroad. After placement, the students study English together in small sections of
20-22 students multiple times per week. The English program consists mainlyof academic reading, writing, and skill building classes. The amount of requiredclasses is dependent on their placement into the four different proficiency levels,and their acceptance into the SEA Program. Each academic year consists ofthree terms. Students placed in the highest proficiency level participate in theprogram for two terms. The next group studies for three terms. The third group
participates in the program for four terms, and the lowest proficiency level for fiveterms. Students can apply for and attend the SEA Program during the summerof their first or second year. Those who do not participate in the SEA Program
are required to take additional academic skills classes in English, such as Lectures
and Critical Discussion, Grammar for Written Communication, Speaking andListening, and Academic Debate. An optional IELTS is taken at the end of theEnglish program.
118

Analysis of Study Abroad, OnCUE Journal, 11(2), pages 116-122

Data Collection and Analysis

Data from the 333 first-year students who took both the TOEFL ITP and theIELTS were collected. To answer RQ1, the data were divided into two groups,students who participated in the SEA Program, n = 142, and those who did not,n = 191. A second division of the data from SEA Program participants was madefor answering RQ2, students who attended a Japanese high school, n = 132, andstudents who attended either an international high school or one overseas, n =10. Due to the small number of attendees in these two types of non-Japanesehigh schools and to facilitate a t-test during the data analysis, these students werecombined into a single group.

Results and Discussions

RQ1 asks whether students in the SEA Program (and studied overseas) or nonparticipants showed improvement in English. Table 1 shows the descriptive
statistics of the two groups. Of the 142 SEA students, their mean score of thepre-program test (TOEFL) was 500.66, and that of the post-program test(IELTS) was 5.73. All the scores were converted to CEFR levels (EducationalTesting Service, 2017; IELTS, 2017), and the mean improvement was 1.18. The
mean score of the pre-program test for the students who did not attend the SEA
Program was 528.55 for TOEFL, and that of the post-program test was 6.12 for
IELTS. Their scores were also converted to CEFR levels, and showed that thenon-SEA students improved by 1.07.
Additionally, to compare the two groups in terms of gains in English test scores,
an independent t-test was conducted. There was a significant difference between
Table 1
Descriptive Statistics of SEA and Non-SEA
PrePostCEFRnM

SDM

SD

M

SD

SEA

142

500.66

41.92

5.73

.52

1.18

.50

Non-SEA

191

528.2550.00

6.12.75

1.07.52

Group

119

Erdelyi, Fukuda, & Yagi

the SEA and non-SEA participants; t(331) = 2.02, p = .04. The results showthat those who participated in the SEA Program improved their English morethan those who did not. However, the effect size, Cohen’s d, was unremarkable.
Effect size is “the degree to which the phenomenon is present in the population”(Cohen, 1988, p. 9), and Plonsky (2015) strongly suggests reporting effect sizes
whenever applicable. In this study, the effect size was 0.22, which is considered lessthan small. The reason for statistically significant results but small effect size could
be the large sample size, a factor that often affects statistical significance (Plonsky,2015). Therefore, it could also be concluded that the extra academic skills classes
taken by the non-SEA students are an adequate alternative to the SEA Program.
This is beneficial information to the university for the promotion of the extra
academic skills classes to those who could not attend the SEA Program.
To answer RQ2 (Who benefited more from the SEA Program, those whoattended a Japanese high school, or those who attended an international highschool or a high school overseas?), the SEA program students were further
divided into two groups. These groups were those who attended a Japanesehigh school and those who attended an international school or a high schooloverseas. Table 2 shows the descriptive statistics of those two groups. Of the 142
participants in the SEA program, 132 students went to a Japanese high school,
and the mean score of the pre-program test (TOEFL) was 498.82, and that ofthe post-program test (IELTS) was 5.69. The scores were converted to CEFR
levels and the mean improvement was 1.19. The other 10 students went to a local
school overseas (n = 2) or to an international school (n = 8), and the mean scoreof the pre-program test was 525.00, and that of the post-program test was 6.20.
Table 2
Descriptive Statistics of JPN and INT
PrePostCEFRGroupnMSDMSDMSDJPN

132

498.82

41.33

5.69

.51

1.19.51

INT

10525.00

44.15

6.20.421.10.32120

Analysis of Study Abroad, OnCUE Journal, 11(2), pages 116-122

Their scores were also converted to CEFR levels, and examination showed thatthese students improved by 1.10.
In order to compare the two groups in terms of gains in English test scores,
another independent t-test was conducted. This time, no significant difference
between the two groups was found; t(140) = .82, p = .43, with a less than small
effect size, Cohen’s d = 0.22. It is uncertain why the results were insignificant.
Perhaps more significant results could be found if other unknown factors such as
willingness to communicate or time spent on studying English were investigatedin a more complex statistical model.

Conclusion

Two hypotheses were investigated in the current study. The first one was that
SEA Program students would make more gains in English ability than those whodid not participate. The second was there would be higher gains in English abilityfor students who had studied in a Japanese high school instead of an international
high school or one overseas. After a statistical analysis of the participants’ pre- andpost-program test scores, the difference between SEA and non-SEA students wasfound to be statistically significant, but the effect size was small. As for the second
hypothesis, the difference was insignificant, and the effect size was also small.
In order to explore further gains in English ability from short-term studyabroad programs, additional variables could be examined. These include, but are
not limited to, types of accommodation while studying abroad, ESL class hours,and length of stay in an English-speaking country prior to university. Applying
these analyses to other cohorts or contexts is also suggested.

Acknowledgments

This research paper is based on the hard work done by the instructors and staff ofthe ELA program at International Christian University.

References

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.).
New York, NY: Psychology Press.
121

Erdelyi, Fukuda, & Yagi

Educational Testing Service. (2017). TOEFL ITP® overall performancedescriptors. Retrieved from https://www.ets.org/toefl_itp/research/performance-descriptors/
IELTS. (2017). Common European Framework. Retrieved from https://www.
ielts.org/ielts-for-organisations/common-european-framework
Kinginger, C. (2009). Language learning and study abroad: A critical reading ofresearch. Basingstoke, UK: Palgrave Macmillan.
Kinginger, C. (Ed.). (2013). Social and cultural aspects of language learning instudy abroad. Amsterdam, The Netherlands: John Benjamins.
Plonsky, L. (2015). Advancing quantitative methods in second language research.
New York, NY: Routledge.
Watanabe, Y. (2000). TOEFL gain scores: What do they mean to the English
language program? ICU Language Research Bulletin, 15, 117-132.

Author bios

Tanya Erdelyi is an English Instructor in the International Christian University’s
English for Liberal Arts Program. Her research interests include peer feedback,collaborative learning, and classroom communities. tanyae@icu.ac.jp
Tetsuya Fukuda is an Instructor in the English for Liberal Arts Program atthe International Christian University. His research interests include program
evaluation, test evaluation, and psychological aspects of language learners. Recently,
he published papers on L2 motivation and school belonging. tfukuda@icu.ac.jp
Keita Yagi is currently an English instructor in the English for Liberal Arts Programat the International Christian University. His research interests include SLA,
material development, and teacher education. ykeita@icu.ac.jp
Received: October 1, 2017
Accepted: November 11, 2018122