Proficiency Scores and Classroom Performance
Richard BLIGHT, Meredith STEPHENS

gained by incorporating forms of standardized testing in

Introduction

university curricula. Standardized tests（which include

In recent years, studentsʼ performance on various highthe TOEIC test） provide a clear focus and specific goals

stakes external exams（such as the TOEIC, TOEFL,. Such tests
for students to strive towards （p.１３）

Eiken）has been strongly emphasized at Japaneseprovide a common direction for teachers who may

universities. There appear to be various reasons forpreviously have been pursuing individual preferences forthis trend. Firstly, TOEIC（et al）test scores provide ateaching content and testing systems. The clear focus

clear differentiation between studentsʼ language levels,that standard tests provide also tends to foster enhanced

which fulfills an essential purpose for employers needing

teamwork among teaching staff, which is another factor

to quickly identify the most able recruits. They alsothat has a demonstrated correlation with achievement

assist to provide a form of institutional accountability（asgains. In

well as individual merit）in terms of simple numericalnumerical achievement scores allow for the adoption ofscores. The adoption of high-stakes exams at colleges

relevant targets. Hence Edwards（２００６）argues that thealso tends to conform to the cultural direction by overtly

adoption of forms of standardized testing in university

valuing the formal accreditations that underlie such test

education importantly provides towards the critical goaladministrations.

of increased accountability, both on the part of the

Yet this direction causes many concerns for teachers,

addition, thetransparency

providedbyteachers and students.

who are frequently aware of what can be significant

Another major advantage of using the TOEIC overdiscrepancies between studentsʼ TOEIC scores and theirtraditional university in-house exam systems is that theobserved performance in the classroom. In order to

TOEIC gives equal weighting to listening and reading

investigate these concerns, we conducted a classroom-.
skills（each section comprises ５０％ of the final grade）

based research study that compared a studentʼs TOEIC

Gaining a good TOEIC score hence requires students toscore against results on three standard classroom tests.

improve their listening skills, a critical area of language

We wanted to investigate to what degree the results on

acquisition that has traditionally been either undervalued

the tests provided a similar（or different）measure of aorstudentʼs English language ability.

university testing back to the high school system has

neglected. A

significant

washback

effectfrombeen previously reported, so listening ability would most
likely also tend to become more seriously fostered in the

Review of Current Issues

preceding years of education.

Using external exams such as the TOEIC to evaluate
studentsʼ

is

testing provides for the continuation of a focus on test

controversial from a number of points of view. Perhaps

score performance. Such a direction has been heavily

central to this debate is that the TOEIC is a ʻproficiency

stressed in the preceding high school years, but tends totestʼ（i. e., it measures a studentʼs general language

become less relevant and meaningful in the non-abilities）, while performance is traditionally measured instandardized testing environments currently found atterms

many universities. Indeed, the students have beenof

performanceʻ achievementon

university

testsʼ, whichcourses

Also important is that the incorporation of TOEIC

focus （ morenarrowly）on the content studied during classes.

educated in a system that strongly emphasizes individual
point scores to such an extent that they can easily

Advantages of a TOEIC Focus
Edwards（２００６）argues that many general benefits are
大学教育実践ジャーナル 第１０号 ２０１２

underestimate the value in pursuing alternative goals（such as a focus on language fluency）, particularly if
９Richard BLIGHT, Meredith STEPHENS

their performance cannot be simply represented inand listening sections）. The hypothesis that there

terms of numerical point scores.

would be equal gains by students in all three groups on

Finally, TOEIC tests provide all students with anaccount of the additional English study（regardless of theopportunity to demonstrate their individual skill andcourse content）was confirmed for eleven of the twelve

study application. The TOEIC puts all students on a

study groups, with the single exception being the non-ʻlevel playing fieldʼ, regardless of the name, ranking, orEnglish majors in the TOEIC Preparation class, who

status of the tertiary institution they attend. Students

showed more substantial improvement on the TOEIC

can identify and develop various areas of their skills in a

Reading section. Hence the benefits of the Test

range of language contexts and structures. In this way,

Preparation course were limited to assisting non-English

the TOEIC provides an important chance for students tomajors to improve their reading score, and provided noachieve and demonstrate individual performance, rather

additional benefits for English major students or on thethan claiming upon the status of their university.

listening section of the test, beyond the general benefits
provided by other forms of English study.

Disadvantages of a TOEIC Focus
Foreign language teachers working in Japan have

ʻPutting the Horse before the Cartʼ

cautioned against problems associated with placing too

Zemach（２００５）expresses frustration at test results（on.
much emphasis on studentsʼ test scores（Guest, ２００５）

the TOEFL）being considered more important than

They argue that this direction ignores other vital

mastery of the language. Tests provide insight into alanguage goals, and in particular fails to address thestudentʼs language abilities, but are a form of temporary

studentsʼ need to develop basic communication skills

measure of the studentʼs learning progress, and never（Lyster, ２００５）. They

withcomprehensively represent the state of the studentʼs

TOEIC scores as perpetuating fundamental mistakes

learning. Indeed, we would describe this situation ofmade in secondary curriculums, where the memorization

prioritizingof grammatical patterns and difficult vocabulary islearning over the process of learning as failing to put the

typically prioritized over oral and written communication

horse

Zemachconsequently

tasks. The teachers describe students who achieved

recommends offering TOEFL classes tostudents who

good TOEIC scores, but who demonstrated difficulties

have already taken classes or demonstrated proficiency

holding. She argues that a solidin other key skills （２００５, p.１７）

the

most

see

basic

the

of

preoccupation

conversations. Brownan

beforeinaccurate, temporary
the

cart.

measureoffoundation of communicative skills is necessary to

discusses this paradoxical situation further :achieve good test scores, and certainly before students
If we are serious about teaching students to actuallyattempt to memorize less common points of grammaticalcommunicate

distinction.

inasecondlanguage, we

mustabsolutely stop limiting ourselves to the relatively

Similarly, Garant（２００３）and Kobayashi（２００１）indicate

easy testing of their second language knowledge

that in Japanese high schools the studentsʼ test scores（as with grammar and vocabulary tests）, or their

are typically considered more important than their

receptive language skills（as in listening and reading

communicative

cites

the

case

of

tests）. We will have to face the many challenges ofcommunicative English classes being cancelled at thetesting their performance in written and oral

end of Junior High School because of an approachingcommunication. （Brown in Sunga ２００３, p.１４）

High School entrance examination. Furthermore,,
One interesting study was conducted by Robb（１９９９）
whoskills. GarantinvestigatedtheeffectivenessofJapanese teachers expressed that the goal of English

TOEIC

education was to see that the students passed their

preparatory courses at a Japanese university. He

English tests. The ability to communicate was seen

followed two student groups（English majors vs. non-asEnglish majors）who took three different classes : TOEIC

accomplishedPreparation, Business English, and Four Skills English.

examination process. （Garant, ２００３, p.１２３）

very

important, but
without

only

ifinterfering

it

could

be

with

the

At the end of the year, he checked for improvement in
the studentsʼ TOEIC scores（in both the TOEIC reading
１０

A further concern then becomes that the TOEIC
大学教育実践ジャーナル 第１０号 ２０１２

Proficiency Scores and Classroom Performanceexam（taken at the end of university）may end up being

Combining TOEIC Preparation with Modern EFL

assigned the same artificial significance as the university

Pedagogy

entrance exam（taken at the end of high school ; see

There are alternative ways to boost TOEIC scores,. So the problem of mistaking the
Matsuyama, １９７８）

rather than the methods used in test preparation

fundamental goal of language acquisition with instead,courses. Krashen, for example, argues that developing

the goal of achieving a good score on a test, looks set toreading skills promotes broad language improvement :be extended into the university years. Further, acommon flaw of the test preparation courses is that

When children read for pleasure, when they get

students are repeatedly asked to tackle questions that

hooked on books, they acquire, involuntarily andare way beyond their current language level, and as awithout conscious effort, nearly all the so-called

result they become discouraged when they consistently

language skills many people are so concerned about :

get

of

They will become adequate readers, acquire a large

vocabulary that they are presented with in the TOEIC

the

wrong

answers. Similarly, therange

vocabulary, develop the ability to understand andfar exceeds their vocabulary knowledge. The ʻtest

use complex grammatical constructions, develop a

preparationʼ approach of persistently exposing students

good writing style, and become good（but not

to materials at too high a difficulty level has a strong

necessarily perfect）spellers. （２００４a, p.１４９）

negative impact on motivation levels, and yet（as areward for effort）provides only limited benefits. This

Many of the skills that Krashen relates to ʻgood readingʼapproach

are also required to perform well on the TOEIC test,

also

intuitively

encroachesoncommonprinciples of learning, which generally assume that

includinglearning

competence. Henceoccursbest

withinprotected

stagesofvocabularyknowledge

and

one

to

way

grammaticalfacilitateandevelopment. Frequently, ʻover-exposureʼ to too much

improvement in TOEIC scores is to get students actively

information tends to overload students（leading to

involved in an extensive reading program. Fluent

processes of mental shutdown）rather than leading to thereading without recourse to a dictionary is necessary todesired massive absorption of new information. Rather,enjoyeffective

reading, andtheenjoymentofreading

isinnecessary in order to persevere with the demanding

difficulty level, combined with repeated recycling andtask of learning. Harold Palmer, who was invited topractice with the material.

Japan by the government in １９２２ to promote TEFL,

learning

requiresmeasuredincreases

involveconsidered that reading should be as fluent or natural as

students in practicing masses of previous test questions.

speaking or hearing, not the word for word puzzling out

But TOEIC questions are always ʻnarrowʼ in focus andoflacking a general context ; the common mistake thenthis warning（originally made in pre-war Japan）is still

becomes for teachers to practise volumes of test

equally as relevant today. A perusal of the questions inquestions ʻin isolation,ʼ rather than to master broad

any TOEIC exam reveals the kind of written language

reading skills, which would ultimately lead to higherthat is likely to be found in newspapers. This language

TOEIC

to

is considerably more sophisticated than the language

intensively studying ʻformʼ rather than ʻmeaning.ʼ By

used in graded readers. In order to succeed in thecontrast, Krashen

arguesthatʻmeaningʼ

is

more

TOEIC exam, students must be familiar with thefundamental

tolearningalanguage :Language

language that is typically found in newspapers andacquisition

only

happens

when

Test

preparation

coursesscores. Class

typically

time

is

hence

also

devoted. Unfortunately,
meaning （Masukawa, １９７８, p.２４６）

understandtelevision broadcasts. However, in order to attain this. Krashen also suggests thatmessages （２００４b, p.１）

level, students must first progress through the levels ofteaching grammatical rules should be delayed until

difficulty such as are found in graded readers series.

students have reached an advanced level : I would firstNovice readers are likely to waste their time launching

give acquisition a chance, and then use conscious

straight into difficult newspaper English, without first.
knowledge to fill in some of the gaps （２００４b, p.１）

developing

Unfortunately in Japanese education, the pressure ofdevelopment.

we

a

solid

foundationforreadingskill

pending exams means it is very difficult for teachers tofollow Krashenʼs advice to give acquisition a chance .
大学教育実践ジャーナル 第１０号 ２０１２

１１Richard BLIGHT, Meredith STEPHENS

Study Methodology

Summary of Results

This study compares the results of ２１ students（withThe tests for the conversation textbook（ʻLetʼs Talk !）two classes per week）, who were being taught according

and two OUP readers（ʻReader ＃１ , ʻReader ＃２ ）were

to three distinct themes : A TOEIC preparation course,

each marked out of １００ points, and so could be easily

the popular Let’s Talk ! textbook（Leo Jones, CUP,

plotted on the same graphical scale. However, the２００２）, and readers in the Oxford Graded Reader series

TOEIC test is recorded on a different scale（１０ pts−９９０. The alternative tests used were
（OUP, Level １）

pts）, so it was necessary to consider how to adjust the

publishedtextbook

TOEIC test scores into a similar １００ point scaling,materials. These tests covered vocabulary and listening

thereby enabling the TOEIC results to be plotted onto

dictations from the textbook materials.

the same graph. We examined the studentsʼ TOEIC

resources

associated

with

the

This study examines the studentsʼ relative performance

scores, and found that the test scores ranged from a lowon the TOEIC course, which includes a wide range ofof ２０５ pts to a high of ５３５ pts. This was compared topossible test content, and on the OUP / CUP courses

the studentsʼ scores on the other three tests, which

which feature more narrowly restricted test content

ranged from a low of ４２ pts to a high of ９６ pts. In orderbecause the associated tests aim to measure learning

to adjust the TOEIC scores to the same １００ pt scale, thejust on the textbook materials. In particular, we werehighest scores on the two scales were denoted asinterested to investigate how the studentsʼ performanceequivalent. Hence TOEIC ５３５ pts was set to ９６ pts onon the various tests compared. Would students withthe １００ pts scale, and the other TOEIC scores werehigh TOEIC scores also get high scores on the otheradjusted

tests ? There seemed to be at least a good chance thatcomputation. This provided the desired TOEIC scaling

this might NOT be the case, given the different teachingfrom a low of ３７ pts to a high of ９６ pts, which could thenmethodologies underlying the three sets of materials.

be plotted on the same graph as the other three test

We hoped that by comparing the results for the different.
scores（see : Figure １）

tests, a pattern might be observable which could be
used

to

predict（tosomeextent ） the

studentsʼ

according

to

the

same

mathematical

When we study these results, which are presented in
the order of studentsʼ TOEIC scores（from low to high）,
we observe that first ; the TOEIC scores have a wider

performance on future tests.
Given the nature of this project as a small-scale

range（３７−９６）than the other tests. The ʻLetʼs Talkʼ andtests seem to be the most closelyclassroom investigation, it is appropriate to identify

ʻReader ＃２

limitations associated with the research method. First,

correlated, and grouped in a band around the ７０−９０ pts

the results can only be verified to a limited degree, since

range. The ʻReader ＃１ results follow a similar generalthey refer to single test administrations for a small grouppattern, but are much more widely dispersed. Byof students（n＝２１）on the tests. The capacity tocontrast, the TOEIC scores appear to be the least

generalize from the results is also limited to the degree

related to any of the other three tests.

that this sample of one university class is representative

There are also patterns observable for individual

thestudents. First, Student̲２１ achieved the highest TOEIC

significant nature of the limitations just described, it wasscore（９６ pts）and scored the second highest on thedetermined not to provide a statistical analysis of, but was average in the otherʻReader ＃２ test（９５ pts）

correlation coefficients between the test scores. Rather,

tests（８４, ８０ pts）. Similarly, Student̲１５ with the highest

of

other

university

classes. Further, duetothe benefits of the present study would be realised byscore on ʻReader ＃１ （９６ pts）was more average on the

representing the test scores on a common graphical. Student̲８ did very
other three tests（８８, ８５, ７１ pts）

scale, which would then allow for any potential patterns

well on all three non-TOEIC tests（９２, ８６, ８５ pts）but

to be observed. Teachers could then consider the. Finally,much worse on the TOEIC test（６０ pts）

results and how they might impact upon their teachingStudent̲２０ achieved the second highest TOEIC score（８２methods, and use the results to inform their teaching.

pts）, but scored much lower on the three other tests（６４,６３, ４２ pts）
. At the low end of the TOEIC scale,Student̲１ with ３７ pts was low on the ʻReader ＃１ test
, but average on the other two tests（７５, ８０ pts）.
（６３ pts）

１２

大学教育実践ジャーナル 第１０号 ２０１２

Proficiency Scores and Classroom Performance

Figure１ : TOEIC vs. Classroom Tests
at proficiency gains would appear more likely to have

Discussion of Results

the added effect of increasing studentsʼ test scores,

While this study has not determined the causes of ourwhich is a good example of the situation previously

findings, we would like to provide our interpretations

identified（re : Putting the horse before the cart ）.

where we believe these are relevant. The benefits ofInstead of repeatedly studying test questions that mighttypical test preparation courses appear to us to be quite

be used, students could be actively engaged in an

limitedfor

test

extensive reading and/or listening program（to giveandsome examples）. Extensive reading can provide many

strategies , but the TOEIC（and similarly for the othersubstantial learning gains, including boosting knowledge

major tests）is a General Proficiency Test that aims toof common vocabulary terms, which would also improve

preparationanumbercoursesofpractice

reasons. First, thetest-taking

skillsmeasure a studentʼs General Language Proficiency

TOEIC scores, as well as increasing reading performance（GLP）. In Applied Linguistics, GLP is（another critical factor in test scores）.

acomplexscientific construct, which aims to provide a measure of

There is also a problem with the methods of study

broad language performance so as to provide insight into

incorporated

the nature of a studentʼs language abilities. Hence, in

workbooks consist of large numbers of test questions

inorder to improve TOEIC scores, language exercisesthat are presented out of context. The students

should be aimed at increasing GLP, a goal that contrasts

consequently are not provided with the opportunity tomarkedly from the objectives of test preparation courses.

absorb important contextual cues, which in real-life

Further, the inherent value to a studentʼs language

activities tend to facilitate the learning process. Instead,acquisition in memorizing the answers to commonly

natural learning processes are replaced by repetitious

preparation

occurring test questions, or in practicing test strategies

andworkbooks

questionable.

information, frequently in the guise of grammaticalsubstantial gain in GLP within the confines of a １５-weeksemester. However, it also seems somewhat incongruent

memorization

courses. TOEIC（while ignoring the testʼs primary focus on GLP）is
We are aware that it is difficult to achieve any form oftiring

test

tend

to

exercises. Hence

present

massesoftheseunrelatedpuzzles to solve, while at the same time blocking natural
language acquisition processes.
Finally, thematerials

being

presented

in

test

to use such a program to simply aim for improvements

preparation courses are generally not pitched specificallyon the margins, rather than for any form of serious

enoughlearning. Teaching programs and methods which aim

example, be achieved with a graded readers program.

大学教育実践ジャーナル 第１０号 ２０１２

atstudentsʼ

individuallevels, ascan, for

１３

Richard BLIGHT, Meredith STEPHENS

The other three tests used in this study were, byReferences

contrast, ʻachievement testsʼ which examined knowledge

Edwards, N.（２００６） The historical and social foundations ofof the materials studied during the preceding classes.

standardized testing : In search of a balance betweenHence all students had an equal chance of learning thematerials and achieving good results on the tests, and it
is likely that this situation has contributed significantly
to the studentsʼ performance on the three non-TOEIC
tests being more closely related.

learning and evaluation.

Shiken, １０（１）, ８−１６.

Guest, M.（２００５） The problem of the talking sandwich.

The

Daily Yomiuri, October ２１, p.１４.
Chapman, M.（２００５） TOEIC : Claim and counterclaim. The
Language Teacher, ２９（１２）, １１−１５.
Garant, M.（２０００） EFL testing and university admission in
Finland and Japan.

Asian Journal of English Language

Teaching, １０, １１５−１３５.

Conclusions

Krashen, S.（２００４a）The

It appears that the current trend of increasing

Power

of

Reading. Portsmouth :Heinemann.

educational accountability means that TOEIC scores andKrashen, S（２００４b） Applying the comprehension hypothesis :other high-stakes English language tests will continue toSome suggestions. Research presentation at The １３ thgrow in importance for university students. Clearly,

International

teachers are obliged to do their best to help students

Teachers Association of the Republic of China）, Taipei,

attainthebestpossiblescores, but

need

to

simultaneously maintain a commitment to alternative
pedagogical goals, such as communicative competence.
Teachers should not, for example, revert to traditional
grammar translation techniques in order to boost test
courses, since the studentsʼ class time is seriously

SymposiumonLanguage

Teaching （English

Taiwan, November １３.
Kobayashi, Y.（２００１） The learning of English at academic
high schools in Japan : students caught between exams andinternationalisation.

Language Learning Journal , ２３, ６７−７２.

Ichiyama, M., Critchley, M., Fiyouzat, R., Iijima, Y., & Suzuki,
T.（２００１）Encounters. Tokyo : Nanundo.
Lyster, R.（２００５） Effective pedagogy for continued language

limited. Communicative methodologies should instead

growth in immersion. Research presentation at Ritsumeikan

be adapted to foster general language competence gains

University, Nov ５.

simultaneously with the promotion of test scores.

Masukawa, K.（１９７８） The oral method : A critical survey in

We have discussed in this paper the types of
discrepanciesclassroomsthatteachers

between

TOEIC

frequently
scoresandobserveinclassroom

performance. It is intrinsic to the process of educationthat learning achievement needs to be measured, but it
is simplistic to interpret a studentʼs language ability as
their most recent TOEIC score. Further, the resourcesthat should be apportioned to pursuing the goal of testscores need to be carefully balanced against more

Koike K., Matsuyama, M., Igarashi, Y. & Suzuki, K.（Eds.）,
The Teaching of English in Japan, pp２４２−２５５. Tokyo :Eichosha Publishing.
Matsuyama,

The

Teaching

communication

abilities may get low scores because they have notmastered basic grammar points. Proficiency scores andcommunicativecompetence

are

not

in

Japan, pp３５−４６. Tokyo :effect of

direct test

preparation on the TOEIC scores of Japanese university
students.

good

EnglishRobb, T.（１９９９） A study of the

７６.

with

of

Eichosha Publishing.

inherently more difficult to measure. High scores do not
Furthermore, students

examinations : College

Koike, K., Matsuyama, M., Igarashi, Y., & Suzuki, K.（Eds.）,

fundamental language learning processes, which arenecessarily correlate with English communication skills.

M.（１９７８） Entrance

entrance examinations and English education in Japan in

Teaching English as a Second Language, ３（４）, ６７−

Sunga, N.（２００３） An interview with James Dean Brown on
language testing and assessment.

Guidelines, ２５（１）, １３−１６.

Zemach, D.（２００５） From A to Z : TOEFL Drive. Essential
Teacher, ２（３）, １６−１７.

necessarily

compatible classroom goals, and program administratorsneed to perceive the inherent value in pursuing both
directions in university courses.

１４

大学教育実践ジャーナル 第１０号 ２０１２