Seeking the Benefits of E-testingin Assessing Studentsʼ English Proficiency
Mutsuko NAGASAKI, Akira NAKAYAMA, Sunao ORIMOTO, Neil HEFFERNAN
Ehime University

Although both tests served well enough to attain．
Introduction

the initial aims of the EECʼs programs, they have caused

The English Education Center（EEC）at Ehime

some problems in terms of operation. The biggest

University started using an official standard English

problem is that they both require manpower. In orderproficiency test infor the approximately ，. We used the Global Test ofEnglish Communication（GTEC）for Students fromtofirst year students to takethe test at the same time, about, and since then have been using the Test of

office staff,

members of faculty,

student assistances（SAs）andpart-

English for International Communication（TOEIC）Bridge.

time assistants are asked to work as proctors orBoth tests are paper and pencil tests（PPT）, but theassistant proctors and receptionists on the day of the

EEC began to explore the possibilities of using e-testing

test. It especially requires great manpower to collect

as a means of measuring university studentsʼ English

question and answer sheets and carefully count them in

proficiency. This paper aims to explore the possibleorder to avoid the leakage of exam questions. Careful

benefits

and adequate preparation is necessary to secure theseofComputerized

e-testing, focusing

on

two

Assessment

Systeme-tests ; theforEnglishtesting staff and enough classrooms, which demands

Communication（CASEC）and the GTEC College Testconsiderable expense in terms of time and effort for the

Edition. A trial of these tests was conducted to

office staff.

examine the possibilities of implementing them in the

Yet another difficulty is in scheduling the test itself.

EEC curriculum. The results of the trail are reported

It takes GTEC for Students about

and some suggestions for their use are made at the end

Bridge about

days until the studentsʼ score reportsof the paper.

are

to

returned

mentioned, since

theweeks and TOEIC

university. As

previously％ of studentsʼ final grade is decided

by one of these official standard English proficiency tests,．
Why e-testing ?

the test has to be conducted at least a few weeks before

The EEC provides about，first year students

the end of the semester in order for the scores to bewith an official standard English proficiency test twice

available to teachers. Consequently, it has been held in

every year at the beginning of June and December.

the middle of each semester since

The aims of the usage of the standard test are first to

regular classes held at the same time as the test need toestablish a higher validity of studentsʼ grade evaluation,

be canceled. The scheduling is always quite a difficultand secondly to obtain the appropriate objective dataand controversial issue at the university.

. As a result,according to which the EEC streams its English classes

Furthermore, since the test is conducted in thefor the second semester. Twenty percent of student

middle of the semester, the studentsʼ proficiency has tograde evaluation is based upon the results of this test,

be evaluated much earlier than would be ultimately

and two tests（GTEC for Students fromdesirable. This is problematic in terms of giving a fair

TOEIC Bridge from

to

and

until now）have been used.

and accurate evaluation to students.

These tests were chosen from the perspective of both

One way to solve these problems is the use of an e-their appropriate levels for our first year students andtesting system with which students take a test online oralso for their cost.

on a CD-ROM using a computer. Under this system,

大学教育実践ジャーナル 第号

Mutsuko NAGASAKI, Akira NAKAYAMA, Sunao ORIMOTO, Neil HEFFERNAN

they can take the test on their own, following directions

order of test items presented on the screen is decided byshown on the computer screen. It also omits the wholetest takerʼs responses. If they answer incorrectly, a

processes of distributing and collecting test materials

slightly easier question will be given next. If they giveand counting them. This requires fewer proctors,

a correct answer, the next item will be slightly morewhich will result in less effort on the human resourcesdifficult. Thatside of the equation.

items are chosen from an item bank in order toAnother advantage of e-testing is that it provides

is, according

tolearnersʼ

responses,

accurately reflect their level of proficiency.

allowsIn sum, the use of an e-test will undoubtedly suitflexibility in making the test schedule and helps solve thethe EEC for its testing purposes in that we will noscheduling problem. In addition, this flexibility will

longer have to set up one specific test time and date on

contribute to being able to more fairly evaluate studentsʼ

which all the students have to sit for a test ; and theEnglish proficiency, since it is possible to have the test

leakage of exam questions will become a moot point. If

towards the end of the semester.

several test dates are set, students can take it during atest scores immediately

after

the

test. It

specific period in a computer room following the．
E-tests selected for trail

directions

on

the

screenbythemselves. It

willcontribute to flexibility in test scheduling.

． Selection
In order to improve the above situation, the EEC
has begun to consider ways to promote more efficiency
in the process of carrying out our English proficiency
test, and thus, in, conducted some research in． Two Candidates : CASEC and GTEC College Test
EditionTable

andsummarize the content of the CASEC

and GTEC College Test Edition. As Table

shows,

order to find an alternative official standard English test.

CASEC is composed of four sections, with the first two

Among several e-tests, we nominated CASEC, GTEC

parts assessing knowledge of vocabulary or set phrases

College Test Edition, Versant Speaking test, and thewhich are frequently used in daily life. Sections

TOEIC Speaking and Writing（SW）test, as prospective

assess learnersʼ listening ability to understand main ideas

tests, after reviewing the content, level and cost of each

as well as specific information given in daily situations.

test. Both the Versant Speaking test and the TOEIC

Though Sections

SW test were refined and well-organized tests, but they

Sectionare very challenging and may not be appropriate for low

answers. Hence, accuracy of spelling is necessary tolevel students ; the approximate English proficiency level

give a correct answer.

to

and

are multiple choice questions,requires test takers to physically type theirof most Ehime University students is considered to be

Regarding the GTEC College Test Edition（Table ）,low-to-high intermediate. Another problem is that theits components and contents are quite similar to those ofabove-mentioned tests are designed to measure speaking

the TOEIC Bridge test. It has

sections, listening andproficiency only, while the current TOEIC Bridge isreading, and each is composed ofparts. It evaluates

designed to measure listening and reading proficiencies.

learnersʼ receptive skills.

In addition, they cost more than the TOEIC Bridge,

The other characteristics of CASEC and GTEC

which is another issue that needs to be considered.

College

CASEC and GTEC College Test Edition ） seem to satisfy

requirements, score reports, and the price are summa-

all the conditions that we require at the EEC, and also

rized in Table ．

Test

Edition, suchascomputerhardwareeliminate the problems mentioned above. Above all,they are computer adaptive tests（CAT）based on item
response theory（IRT））, which can measure test takersʼ
English proficiency efficiently within a limited time.
The use of CAT is expected to solve some problems
inherent in such tests as the TOEIC Bridge. That is,
the problem of all students having to take the test at thesame time is avoided, therefore allowing more flexibility
in terms of human resources and time. In CAT, the
大学教育実践ジャーナル 第号

Seeking the Benefits of E-testing in Assessing Studentsʼ English Proficiency
Table

Table

大学教育実践ジャーナル 第号: Summary of the CASEC test

: Summary of the GTEC College Test EditionMutsuko NAGASAKI, Akira NAKAYAMA, Sunao ORIMOTO, Neil HEFFERNAN: Other Characteristics of CASEC and GTEC College Test Edition

Table

computers in total were used, and the students were．
The Trial
The trial tests of the CASEC and GTEC College
Test Edition were conducted on July
Ehime University. In total,
CASEC andth,atstudents took theasked to take the both tests anytime between ：

am to

：

SAs,

pm. Three faculty members from the EEC,

and an office staff member from each testing company
were present to operate the trial.

took the GTEC. Out of these students,

were from the department of Engineering andwere from Law and Letters. The former group of
students was asked to volunteer to take the test by a．
Results． Test Results
Table

supervisor in their department, while the latter students

shows the mean of both tests : the CASEC

were asked to do so by their English teacher. The rest

and GTEC College Test Edition. The total averageof the students were volunteers from all other faculties,score of the CASEC waswith the exception of the Medical faculty. The students

）

was

．
Table

underwent the two tests in a row, and a half of them．

）, and that of the GTEC.

shows the results of the correlation analysis

took CASEC first and GTEC next, while the other halfamong the three different tests : the TOEIC Bridge,

did the opposite. There were

CASEC, and GTEC for College. Since the CASEC has

students who took

both tests. Therefore, only the results ofstudents

four different sections, which are not compatible to thewere used for correlation analysis with their TOEIC

other two tests, we summed up the scores of Part

Bridge test scores.

Part

On test day, three computer rooms which hold

Table

Table

and

of CASEC in order to compare the scores in theReading section. In the same manner, the scores of: Mean average for each section: Correlation Matrix among the three tests （ N ＝）大学教育実践ジャーナル 第号

Seeking the Benefits of E-testing in Assessing Studentsʼ English Proficiency

Partand Part

were summed up to compare thescores in the Listening section.
As

Table

The numbers in the tables are the percentage of people
who chose the corresponding answers. Graphsdemonstrates, thecorrelation

be-

and

demonstrate the same results using a line graph.

tween the average of the total scores of the TOEIC

The results demonstrate two important elements.

Bridge and the CASEC are stronger（r＝. ）than theFirst, the students had similar opinions towards bothcorrelation between those of the TOEIC Bridge and thetests. As can be seen in the tables and graphs, theGTEC（r＝. ）. On the other hand, in each sub section

results of the CASEC and GTEC College Test Edition（e.g., the Listening and Reading section）, the GTEC has

were quite similar. From Statements

stronger correlations with the TOEIC Bridge.

majority of the students marked either agree or

to

, the

strongly agree , which means they were in favor of the． Questionnaire
Table

and

e-tests, althoughillustrate the results of a questionnairethepercentages

of

CASEC

were

slightly higher than those of the GTEC. With regard to

distributed to students regarding the CASEC and GTECStatementsCollege Test Edition. The questionnaire was given ineither disagree or strongly disagree in both tests,

Japanese. As can be seen from these tables, in the leftwhich

row of the tables, the questionnaire had nine statements

percentages of the two tests were very similar ;and the students were told to choose their opinion fromtherefore, it is reasonable to conclude that there wasfiveitems,

agree,

orstrongly

disagree,strongly

agree.

answered the questionnaire :

In

means

they

e-testing. The

almost no difference between the two types of e-tests

total,

students

from the studentsʼ points of view.

on theth,th, fourSecond, theresults

on theCASEC andth answered it immediately following the tests.

showedthat

thestudents

seemed to feel more comfortable in taking PPT than etests. Approximately

号

againstneither,days after the tests, and one student who took the tests

大学教育実践ジャーナル 第

weredisagree,answered it on thethe day after the trial tests, andand , the majority of the students marked％ of the students who took the％ who took the GTEC felt that they

Table

: Results of Questionnaire for the CASEC

Table: Results of Questionnaire for the GTEC

Mutsuko NAGASAKI, Akira NAKAYAMA, Sunao ORIMOTO, Neil HEFFERNAN

Graph

Graph

: Results of questionnaire for the CASEC: Results of questionnaire for the GTEC College Test Edition

could concentrate on PPT more than e-tests. Moreover,％ of the students answered that they actually
preferred PPT to e-tests, while about
took the GTEC and％ of those who％ of those who took the CASEC

of about

words, the TOEIC Bridge and CASEC do

not include any long passages. The existence of a longreading passage of GTEC might affect the results seen inthis research.

claimed they preferred e-testing. In spite of the factthat the students were favorable to the contents of etests, as the answers of Statements

toshowed, itturned out that they liked PPT more than e-tests.

． Questionnaire
The results of the questionnaire indicate that theparticipants felt more comfortable in taking PPT than etests, although they showed positive attitudes towards e-．
Discussion

tests. The reason for this might be that they are notaccustomed to taking e-tests. In general, almost all theIn this section, first the results of the trial tests, andtests, including the National Center Test for university

then those of the questionnaire are discussed. Third,

admissions which Japanese students must take before

the possibility of operating e-tests is considered from anentering university, are PPT. Thus, it is no wonder

administrative point of view.

that they are accustomed to paper-based tests rather
than e-tests, which probably resulted in their preference． Test Resultsfor PPT.

The results indicate that there was a strong

One of the solutions for this problem is to encouragecorrelation between the average of the total scores of thestudents to work on e-learning. Fortunately, the EEC

TOEIC Bridge and the CASEC（r＝. ）. With regard tohas been using the e-learning program ALC Net

the correlation between the TOEIC Bridge and theAcademy as a part of our curriculum. The students are

GTEC（r＝. ）, there was a medium correlation. The

required to complete five short sample tests of TOEIC-

reason why the CASEC had a stronger correlation withlike questions in this program each semester as classthe TOEIC Bridge than the GTEC might be attributed

assignments. The

to whether each test has a long reading passage in the

program is to encourage students to become familiar

reading section or not. While the GTEC has a passage

with the content of the TOEIC test and practice taking

present

purposeofusing

大学教育実践ジャーナル 第号

theSeeking the Benefits of E-testing in Assessing Studentsʼ English Proficiency

it, because we use the TOEIC Bridge as a proficiency

Table: Estimated numbers of computers available
for e-tests

test. However, now we can also use the program tohave students get used to e-tests themselves.
Since we have already implemented an e-learning
system in which students can practice taking TOEIC-liketests online, we should consider using it more effectively
to promote their adaptability to e-tests. From an
educational point of view, it is also important to facilitate
it for their future because common aptitude tests forwork such as SPI, CAB, and GAB）

often require

Regarding the total time required to take one of thee-tests outlined in this paper, about

minutes tohour

students to take them online. In an era of the internetandand computer technology, more tests will be computer-depending on which e-test we use and how we conduct

ized ; hence, it is imperative to motivate students tothe procedures related to the administration of the test.

become accustomed to e-tests.

It seems feasible to have seven to ten test rounds a day.

． Possibility for Operating E-tests，

minutes should be necessary for one test round,

Considering the number of test-takers, at mostFinally, thepossibilitiesfor

implementing

operating e-tests will be discussed. Table

computers

andbeing available. If we carry out seven to at most ten

shows therounds of testing a day, all students will be able to sit for

estimated number of students who will take an e-test
each semester, which is made based on the enrollmentof

students will take the test, withthe test within one or two days.
This estimation shows the possibility of operating an, and the number of the students who repeatede-test, but some administrative problems still remain.

st

We are not sure yet whether it is possible to arrange theone of the required English courses in the EEC in thesemester,
Table

and

nd

semester,.

academic schedule to conduct the test on two days of theshows the number of the computers in fiveacademic year. Even if we could, whether we canrooms in the Center for Information Technology building

actually use these computer rooms simultaneously allat Ehime University（CITE）, which we might be able to

day during the required periods is uncertain. In

use for e-testing purposes. They are generally available

addition, we need to figure out how to certify eachto faculties for classes if they are reserved in advance.

student. In the e-testing system, students take the test

Although there are more computer rooms in several

at different times, meaning it is crucial to ensure theother buildings on campus, using only the computers inaccuracy of student identifications before each round of

one building has a strong advantage because of thetesting.

administrative efficiency of operating the test. CITE
has the most computers on campus.
Table: Estimated numbers of first year students
taking e-tests．
Conclusion
To summarize, the benefits of the use of e-testing inplace of PPT are :（ ）it can be operated with much lessmanpower ;（ ）it allows flexibility in creating the test
schedule ; and（ ）this flexibility will make it possible toevaluate studentsʼ English proficiency more accurately
because the test can be conducted towards the end of
the semester. They will help solve problems that the
EEC has faced in recent years.
From the results of the trial of the CASEC and the
GTEC

College

Test

Edition, andconsideringthesimulation of carrying out these classes at Ehime
University, it can be surmised that both tests arereasonable and feasible candidates as a proficiency test
for the EEC. On the other hand, the results of the
大学教育実践ジャーナル 第号

Mutsuko NAGASAKI, Akira NAKAYAMA, Sunao ORIMOTO, Neil HEFFERNAN

questionnaire showed that the students prefer PPT to etesting. As discussed, students in todayʼs increasingly
globalized world should be encouraged to become
accustomed to taking e-tests because they will likely see
more of these types of computerized tests as they movethrough both their academic and working careers.
They will benefit from having good adaptability to etesting in the future. Further research is necessary to
determine how we can effectively promote studentsʼ
adaptability to taking computer-based tests.

References
Bachman, L.F. （）. Fundamental considerations inlan-guage testing. Oxford, UK : Oxford University Press.
Benesse Corporation. What is GTEC College Test Edition ?
Retrieved from http : //www.benesse.co.jp/gtec/cte̲college/index.html
Educational Testing Service（ETS）. The format of TOEIC
Bridge. Retrieved from http : //www.toeic.or.jp/bridge/
about/tests/
The Japan Institute for Educational Measurement（JIEM）, Inc.
The summary of CASEC. Retrieved from http : //casec.
evidus.com/test-summary/index.html）GTEC is a test which assesses the four skills of English :listening, reading, writing, an speaking. However, GTEC
College Test Edition which is composed of two skills,
listening and reading, was chosen for this trial considering
the budget of the university.
）IRT is a measurement theory to estimate the ability levelsof test takers and the characteristics of test items such asdifficulty and discrimination. IRT estimates are not basedon specific samples ; hence, they are stable across different
groups of individuals and across different administrations.
As a result, tests can be tailored to individual test takersʼlevels of ability and designed in the way they measurethese abilities（Bachman,）According to the）.

CASEC Can-Do , the test takers can

exchange ideas on general topics（e.g., daily conversation,
not in the business situations and contexts）in English.
）The

GTEC Can-do indicates that the test takers canunderstand about

percent of the contents that the native

speakers say in the general situation（e.g., sightseeing）andthe test takers also can read and write in English only on
their personal E-mails.
）The SPI（Synthetic Personality Inventory）is produced by
Recruit Career Co., Ltd., and the CAB and GAB are provided by SHL Japan Ltd. The SPI is the most widely usedaptitude test in Japan, and GAB is the second most commonly used test. While these are general aptitude tests,
CAB is for those who want to do computer-related work.
大学教育実践ジャーナル 第号