Feature Presentation
Exploring IELTS Scores in an Englishfor Academic Purposes Program
Using Cluster Analysis
Tetsuya Fukuda
International Christian University
Kimberly Klassen
Tokyo Woman’s Christian University
We describe an investigation into individual differences in an English for Academic
Purposes program. The aim is to explore characteristics of successful students in order tobetter support those less likely to succeed. Students’ (N = 349) IELTS scores and survey
responses were analyzed using cluster analysis. The survey concerned test preparation, priortest experience, and extrinsic motivation. Six subgroups with distinct characteristics wereidentified. Recommendations for program evaluation are made based on the findings.

English for Academic Purposes (EAP) is defined as “language research andinstruction that focuses on the communicative needs and practices of individuals
working in academic contexts” (Hyland & Shaw, 2016, p. 1). According toJordan (2002), EAP is considered to be a key area within English for SpecificPurposes (ESP). An important aspect of EAP programs for curriculum writers
and teachers is formative evaluation of the program’s objectives: are students
achieving the learning goals, and why are some students more successful thanothers? Proficiency tests such as IELTS are often used to gauge students’ academic
English abilities after completing an EAP program. Many factors are involved in
students’ success on such proficiency tests, not the least of which are individualdifferences. The challenge for program heads is to tease apart these influences in14

Exploring IELTS Scores Using Cluster Analysis, OCJSI 5, pages 14-20

order to better support those learners who are less successful. Here, we describe
an approach to identify patterns among students of a large cohort in order todetermine whether a certain combination of factors can account for students’performance on IELTS, the program exit exam. The approach employs cluster
analysis as a complementary analytical tool, which is easily adaptable in terms of
the factors of interest to the program evaluation.
Cluster analysis is a multivariate procedure used to group people, as opposed
to grouping variables as in factor analysis (Skehan, 1986). Measures are takenof a population sample on different variables, and based on patterns of scoreson these variables, the sample is grouped into clusters of similar people (Skehan,1986). Cluster analysis is often used to complement other traditional statisticalprocedures, as it can shed light on the role of individual differences that may
not be readily apparent otherwise, as in Gu and Johnson’s (1996) study on
vocabulary learning strategies.
To investigate students’ performance on the exit exam from our EAP
program, we used cluster analysis as a complementary procedure to prioranalyses (Erdelyi et al., 2018; Yagi & Fukuda, 2020), which had indicated muchindividual variation. This example analysis from our EAP program may be ofinterest to researchers seeking alternative approaches to program evaluation.
Our research question is: what patterns among EAP program participants can befound in terms of IELTS test scores, test preparation, prior test experience, andextrinsic motivation?

MethodThe data were collected from 349 first-year students at a Japanese university(n = 183 from the 2020 cohort and n = 166 from the 2021 cohort). Students
join the EAP program in April, and after completing the program in February
the following year, they take IELTS in March as an exit test to determine theiracademic English proficiency.
The data consist of two components: IELTS scores and responses to a survey.
Table 1 shows the four 5-point Likert-scale items in the survey. They weredrawn from our previous exploratory study (Yagi & Fukuda, 2020). The survey
15

Fukuda & Klassen

aimed to ascertain how prepared, experienced, and motivated the participants
were regarding the IELTS test. The responses to Item 1 and Item 2 wereconceptualized as preparation. The responses to Item 3 represent experience, andthose to Item 4 were future use (i.e., extrinsic motivation represented by the needfor the test score in the future).
The IELTS scores ranged from 4.5 to 8.5 (M = 6.37, SD = 0.81). Since
Table 1The Question Items and the Options in the Survey
Item 1

How many hours did you study to prepare for the IELTS exam?
More than 10 hours
Between 5 and 9 hours
Between 2 and 5 hours
Between 1 and 2 hours
None

Item 2

Did you participate in the information sessions for IELTS held atGuidance Seminar in December and Preparation Seminar in January?
Both days
January only
December onlyPart of one day

Item 3

Had you taken IELTS before you took it in March?
4 or more times
3 times
2 times
1 timeNever

Item 4How likely are you to use the IELTS results to study abroad in the near
future, such as on an exchange program or graduate school?
Very likely
Quite likely
Somewhat
Not very likely
Not at all16

Exploring IELTS Scores Using Cluster Analysis, OCJSI 5, pages 14-20

test scores and responses to survey items were used in the analysis, all data
were standardized to z-scores. To answer the question of what patterns can beidentified among participants, the data were analyzed employing cluster analysis
with four factors explained above. The cluster analysis was conducted following
the procedure in Staples and Biber (2015).

Results

As a result of the cluster analysis, the participants were categorized into six
groups, which are shown in Figure 1. ANOVAs were conducted to confirm thevalidity of this six-group solution: the groups were significantly different in all
four factors (p < .001), and the effect sizes were medium to high (.49 to .69 in etasquared).
The results from the analysis found that the six clusters of participants havethe following characteristics. Cluster 1 (n = 16) were most experienced with thetest. They did not prepare much, and their IELTS scores were above average.
Cluster 2 (n = 50) reported the lowest need for the test scores in the future. They
prepared less than average, and their scores were low. Cluster 3 (n = 65) had anaverage need for the test score in the future. However, they did not prepare forFigure 1. Six-cluster comparison 2020+2021
17Fukuda & Klassenthe test, and their scores were low. Cluster 4 (n = 123) had the most interest in
using the score for the future. They prepared, but their scores were below average.
Cluster 5 (n = 33) prepared the most, but the scores were below average. Theirinterest in using the score for the future was also below average. Cluster 6 (n =62) had the highest scores but prepared less than average. Their test experience
was just above average.

Discussion

As noted above, cluster analysis is an exploratory tool, and the findings can raise
unexpected insights into the participant group (Staples & Biber, 2015). Here,
we focus on three such observations among the clusters, which we will explore in
future research.
First, we are interested in exploring further the effect of test experience on
test scores. For example, Clusters 1 and 6 had the most experience, scored thehighest, and prepared the least. This finding suggests that test experience may be
more valuable than attending test information sessions before test day. Currently,
there is no specific IELTS test preparation course available to the majority ofthese students before the exam, and yet having practical test experience seems tobe helpful. Opening test prep courses is, therefore, an important consideration
for the EAP program heads.
Secondly, we need to look further at the interaction between motivationand test preparation. For example, Clusters 2 and 5 both reported the lowest
future need for the score (i.e., extrinsic motivation), yet Cluster 5 prepared themost for the test of all groups. It was unexpected that students with low extrinsic
motivation would prepare a lot for the test. Thus, we need to further refine thefactor of motivation for the purposes of the program evaluation analysis.
Lastly, we found that the factors of test experience, test preparation, andextrinsic motivation were not sufficient to explain the differences in test scores
between all the participants. When we look at Clusters 3 and 6, these two groups
seem to show similarities in terms of average test experience, below average test
preparation, and an average need for the test in the future. However, in termsof test scores, Cluster 3 scored below average while Cluster 6 scored well above
18

Exploring IELTS Scores Using Cluster Analysis, OCJSI 5, pages 14-20

average. What this suggests is that there are factors beyond those examined inthis analysis that need to be explored to account for differing test performance.

Conclusion

Cluster analysis can be a useful complementary statistical tool for EAP program
evaluation, as it attempts to group participants according to patterns of factors
chosen by the researcher. This can bring to light individual differences that are notobvious through traditional analyses. Our findings have indicated several waysthat the program heads can better support students in the exit exam process. For
example, we found that some students identified a need for the test score, yet did
not prepare much for the exam. Thus, we recommend making test prep courses
available to these students before they sit the exam. We also found that having
test experience was more valuable than attending the test information sessions.
Again, this highlights the need for focused test preparation courses in which
students can get hands-on experience with the IELTS format. Finally, we were
not able to collect program entrance test scores due to Coronavirus restrictions.
Our future program evaluations will include a comparison of entrance and exit
test scores, as well as fine tuning complex factors such as motivation.

References

Erdelyi, T., Fukuda, T., & Yagi, K. (2018). Analysis of study abroad summer
program. OnCUE Journal, 11(2), 116–122. https://jaltcue.org/files/
OnCUE/OCJ11.2/OCJ11.2_pp116-122_Erdelyi_Fukuda_Yagi.pdf
Gu, Y., & Johnson, R. K. (1996). Vocabulary learning strategies and language
learning outcomes. Language Learning, 46(4), 643–679. https://doi.
org/10.1111/j.1467-1770.1996.tb01355.x
Hyland, K., & Shaw, P. (2016). Introduction. In K. Hyland, & P. Shaw (Eds.),
The Routledge handbook of English for academic purposes (pp. 1–13).
Routledge.
Jordan, R. R. (2002). The growth of EAP in Britain. Journal of English
for Academic Purposes, 1(1), 69–78. https://doi.org/10.1016/s14751585(02)00004-8
19

Fukuda & Klassen

Skehan, P. (1986). Cluster analysis and the identification of learner types. InV. Cook (Ed.), Experimental approaches to second language learning (pp.
81–94). Pergamon Press.
Staples, S., & Biber, D. (2015). Cluster analysis. In L. Plonsky (Ed.), Advancing
quantitative methods in second language research (pp. 243–274). Routledge.
https://doi.org/10.4324/9781315870908-14
Yagi, K., & Fukuda, T. (2020). Exploratory investigation into successful and lesssuccessful learners of English with interviews. Language Research Bulletin,
35, 84–94. https://doi.org/10.34577/00004769

Author bios

Tetsuya Fukuda (PhD in Education, Temple University Japan) is a senior associate
professor at Juntendo University. He teaches academic English courses and isinterested in psychological factors in language learning. He has published papers
in national and international journals, including more than 10 book chapters.
t.fukuda.wv@juntendo.ac.jp
Kimberly Klassen (PhD, Applied Linguistics, Cardiff University, Wales) is anassociate professor at Tokyo Woman’s Christian University. Recent publications haveappeared in Language Teaching, Reading Psychology, The Reading Matrix and The
Curriculum Journal. She has taught in Korea, Taiwan, the United Arab Emirates
and Japan. kimik@lab.twcu.ac.jp
Received: January 20, 2023
Accepted: May 18, 2023

20