Research Digest
Practicality and Validity in Three
Achievement Tests of Vocabulary

George M. Harrison
College of International Relations
Nihon University, Mishima
Teachers wanting to bring more cohesion into a course, or who
want to be informed of how well their students have learned vocabularytaught in class, often write their own short-term achievement tests.
Because these tests are often used only one time, and because theyshould be fair in terms of the language construct being tested, a teacher
is often faced with the task of creating test items which are not only
easy to write and score, but which approximate as best as possible
vocabulary knowledge studied in class (Nation, 2001). Many teachersdeveloping classroom tests, myself included, often go through thisprocess intuitively, without much systematic attention to the effort that
goes into developing, administering, and scoring a test, or to how well
the test does its job. The purpose of this study, therefore, is to find out,
given a choice of three vocabulary test formats, which is most usable
in terms of practicality and validity.

Three Test Designs
Student-centered test. Nation (2001, p. 375) presents a test which
deals with a variety of word knowledge aspects. With this weekly test,
39

Harrison

the teacher chooses 10 words for the class which will be on the test;in addition, each student chooses their own 10 words (ktotal = 20), andgives them to the teacher who then directs the student to either write
a sentence, write the word’s collocates, its definition, or derivations of
the word (same word family).
Word association test. Read (2000, 2004) presents a selective
word association test he developed to measure students’ depth ofknowledge of items in Xue and Nation’s (1984) University Word
List (UWL). As with traditional productive word association tests, it
included paradigmatic and syntagmatic relationships. Read hadoriginally set out to measure depth of knowledge of items in the UWL
and had established concurrent validity (correlating it with a definition
matching test) and reliability (using Rasch modeling). He found it wasdifficult, however, to make the items consistent in how the associations
related to the prompt and therefore limited his items to one word form,
adjectives. He also found guessing to be a big factor with individual
items because learners could identify connections among the item
choices. In response, he developed a format where some items had
two correct choices in each box, while others had one correct choiceon one side and three on the other. Here are example test items withtwo correct answers in each box:
Suddenbeautifulquicksurprisingthirstychangedoctornoise

schoolCommoncomplete light ordinary sharedboundary circle name party

Read, 2000, p. 184

Read recognized that good guessing may still be responsible formuch variance, and in looking at the items, we can see that if a learner
did not know the meaning of sudden, she could look at the choices
40

OnCUE Journaland see that paradigmatic choices such as beautiful and thirsty are
unlikely matches with some syntagmatic choices such as doctor or
school. Developing distractors that avoid this may be difficult, andeven more so if the choices are to be limited to the first 2,000 mostfrequent words.
Sentence completion tasks. Read (2000) also describes a blankfilling task as one type of classroom progress test that requires learners
to draw from the immediate context to produce the correct word. In
the blank, the initial letter is provided to reduce cognitive load. Hereare two of Read’s examples (p. 174):a) Many diamond mines are l__________ in South Africa.
b) Modern jetliners fly at an a__________ of 35,000 feet.
These items appear to test students’ ability to recall and produce aword (lemma) given a context (i.e., the conceptual and grammaticalhints in the sentence). Since learners are expected to use vocabulary
studied in class, Read justifies not using multiple initial letters whenother responses are possible.
Comparing these three tests in just one area may not be a fairassessment of their use. For example, the student-centered progresstest appears easiest to develop, but may be problematic for scoring
and validity. Analyzing these three tests in terms of the practicality ofdeveloping, sitting, and scoring them and in terms of their perceived
validity within the context of their use would be more revealing. I posethe following research questions:
How do the three tests rank against each other in terms ofa) the resources required (for making, taking, and marking thetest)? (RQ1)b) how well they appear to test the knowledge studied in the
textbook? (RQ2)
41

HarrisonThis is an effort, in other words, to determine which test is mostusable for this context in terms of (a) practicality, and (b) validity.

Methodology
To answer these questions, a sample of 12 of the original 34
vocabulary items taught were selected by taking every third one fromthe order in which they occurred in the course textbook (Harrison,2007). Each of the three tests (see Appendix) contained these 12 items.
The “student-centered” test was compromised to some extent because
the choice of items was not the students’. Still, this was justifiable forpractical reasons: (a) to be able to compare the three tests, and (b) theclass in which I used the textbook ended 10 weeks prior to the study.
To help in choosing distractors for the word association test andthe words in the sentence completion test, I consulted an electronic
dictionary containing the Oxford Dictionary, Thesaurus, and Collocation
databases. For the vocabulary not studied in the textbook and whichwere not loanwords in Japanese, I checked the General Service List
(West, 1953) to see if the distractors I came up with were in the first
2,000 most frequent words. If not, I searched for a new distractor.
The word association test underwent the most adaptation from theoriginal test design it was based on. In items with multi-word units(MWUs), three modifications were made: (a) nodes (sometimes morethan one word) were identified and used as prompts, (b) choices
included multi-word parts that made up the unit (sometimes withellipses cueing which side of the choice the syntagmatic relationshiplay), and (c) the number of correct choices per item varied (in an effort
to hasten the test-writing and reflect the various meanings addressed in
the textbook). Efforts were made to keep the choices to the same part ofspeech or grammatical unit, which worked fine with the paradigmatic
associations, but was impractical for the syntagmatic choices.
The three tests were given to each participant in the following order:
42

OnCUE Journal(a) the sentence completion test, (b) the student-centered test, and (c)the word association test. This order was chosen to minimize test effect.
After each test, I conducted a group interview, asking students how
they arrived at their answers. At the end, I asked general impression
questions about the three tests.
Participants. The tests were given to two male third-year Japanese
university students from an advanced English course which ended 10
weeks prior to the study. They were selected because I had used my
textbook in that class and they would more closely resemble students
who had just studied the material than would students who had neverencountered the text. Both students are motivated to learn English andare members of the campus English club. One student, Y (short for
Yuki; pseudonyms used), had reviewed the textbook during the 10
week break.

Results and Analysis
Practicality. Table 1 shows the results for the time and effort put
into the test. In terms of time, the student-centered test looked to besuperior, with the sentence completion test in second, and the word
association test last. Because the tests were given to only two students,
the time in marking each is extrapolated to 20 students (actual time
times 10), a conservative estimate compared to many university
English classes in similar contexts. Time to mark, therefore, is a very
rough estimate of the time it would take a teacher to score the tests
and should be considered carefully.
In terms of effort, the sentence completion test required the least
effort, with the student-centered test next, and the word association
test once again in last place.

43

Harrison

Table 1. Time and Effort Resources
Time (minutes)

Test

Effort perceived(scale of 1 to 10, 1 = least
effort)make take2 mark Avg.

make

take

mark1

total

Sentence
Completion

65

15

15

95

4

4

2

3.33

Studentcentered

10

17

50

77

1

5.5

10

5.50

Word
Association

135

10

20

165

9

2

83

6.33

Other
resources(to make)electronic
dictionary

electronic
dictionary

Results for two test candidates, but extrapolated to 20 candidates
Based on students’ judgments
3This could have been made easier if there were a systematic number of correct
choices.
1
2

Since time and effort are different scales, I cannot give an overall
quantitative ranking of the tests, making it difficult to provide adefinitive answer to the first research question. Also, each of the threescales (making, taking, and marking) cannot be assumed to be equalin value, although that is how I treated them.
I can say, based on these data, that the word association test was
the least practical for writing; it took a great deal of time and mental
effort to develop distractors that were semantically or grammaticallylikely, but which were also within the 2,000 most frequent words.
The student-centered test took very little time and effort to create, but
was difficult and time-consuming to grade due to some ambiguous
responses. The sentence completion test seemed the most balanced
in terms of time and effort it took to create, sit, and score. In similarclasses with close to 30 students, the sentence completion task test
may be most practical; for classes less than 20 students, the studentcentered test would be a better choice.
Validity. The first test, the sentence-completion test, measured
productive recall of the word or MWU based on the context provided.
Some of the student comments in response to the question, “What were
you thinking when you chose your answer” are shown in Table 2.
44

OnCUE Journal

Two things can be said based on the data and test results: (a) test
takers engaged semantic and grammatical clues in context to help
them arrive at answers, and (b) sentences that were similar to thosein the textbook or which were true for students’ lives were easier thansentences that do not fit these situations.
The student-centered test measured receptive recall (of the items),
but some language production was needed to complete the test tasks.
Student comments answering the questions, “Which were easy?”“Which were difficult?” and “Why?” can be found in Table 3.
Table 2. Interviewee Comments After Sentence-Completion Test
Item

Interviewee comments

Interpretation
Semantic and grammatical

H: “It was easy. We can see ‘simple’ and ‘but’
complicatedand ‘relationship’.”
Y: “Yeah, we studied this in the class.”
H: “It’s a negative word because of ‘lonely’. At

tease

first I thought ‘bully’ or ‘ignore,’ but thefirst word is ‘t’ so I thought ‘tease’.”
Y: “The situation.”
T: “I thought why they do exercise is to relievestressclues
Textbook context

stress, and that’s what I do.”was same and Ss
remembered it
Semantic and grammaticalclues
Textbook context
was same and Ss
remembered it
Connected context topersonal experience
Semantic clue; possibly
collocation

H: “We studied this word, but I can’t
remember it.”
Y: “To and from work, so that means the timecommuteto come to a place or back to a place. I
want to live closer [to school], so fromthe sentence, it’s true for me.”
(Note: Y got partial credit for the synform

Textbook context was thesame, but Ss couldn’t
recall form
Connected context topersonal experience“commit”)
Both Ss realized their answer is not correct.
effortNote: The collocation in the test, “makemore of an effort,” is longer than the
textbook’s “make an effort.”

45

The generative use may
have made the item
more difficult

Harrison

Table 3. Interviewee Comments After Student-Centered Test
Item

Interviewee comments
H: “I know this word”

Interpretation
Ss know thein a day that things areword andcomplicated” [sentence]
Y: “I’m a little confused

the situationlack of concrete

how to use themeaningcomplicatedstuff

Answer produced“I have so much works

word ‘stuff.’”-

H: “difficult”

may havebeen
difficult to

Y: “I remember using thisI hear youat the pub for your“I understand you.”[definition]

party”

instance ofuse.

H: “I live close to this uni.

take longremember
S remembers an-

H didn’t use theIt doesn’t take long tosame sensecome to here.”that I was

Y: “It didn’t take long fortesting, but

me to notice that

Y did.

my friends break thepromise.” [sentence]

make aneffort

Y: “It was easy for me to

Y: “You should make an

Y is drawing

make the sentence

effort to be loved

on his life

because it’s aby your girlfriend.”

experience.

useful word and[sentence]

I use the word indaily life.”

Students said, after taking all three tests, that they liked this version
the best. When asked to elaborate, Y said, “It’s fun if I can show I know
the meaning of the word,” and H (Hiro) said, “Because we have toknow the exact meaning of the word and it’s useful for us in daily life.”
Several comments related to the fact that they felt they were given
more of a chance to show what they knew about the word comparedto the other two tests.
46

OnCUE Journal

Students viewed the student-centered test as most similar to theway they are expected to use the language in real life. In terms of target
language use domain, it was similar to several tasks in the textbook.
Furthermore, the sentences and definitions produced by one student, Y,
were similar to those in the book, but those of H, were mostly different.
Since student Y reviewed the textbook over the 10-week break, but H
did not, at least some degree of validity can be established. If thetest were used in the middle of a course, as it should be, perhaps the
responses of student H would have more closely resembled those of
the textbook, and he would have got a better score.
This test clearly separated the meaning and use aspects ofvocabulary (allowing me to control which to measure), but presented
difficulties in grading. It was difficult, for example, to decide whether
to accept answers that related to a different sense of the word thanwhat I was looking for. In the middle of a course, the teacher may
explicitly instruct students to use the sense that was studied, or test theunderlying meaning. Another difficulty I had lay in the ambiguity indrawing the line between the grammaticality of the use of the item andthat of the rest of the sentence. It seemed unfair to penalize grammarthat was not related to the item, but it was sometimes necessary when
it was in close proximity to the item.
The word association test measured receptive recognition in
the two scales of meaning (paradigmatic use) and use (syntagmaticuse). Student comments are listed in Table 4. This test appeared tosuccessfully measure students’ knowledge of the senses of the words
in the meaning scale. H, for example, explained that he knew one
sense of “get to” and “I hear,” but not the one I was testing. Y, who had
been reviewing the textbook, correctly answered these items. Since
the textbook directly taught these new senses, the test achieved its
goal.

47

Harrison

Table 4. Interviewee Comments After Word Association Test
Item

gossip-I hearget toInterviewee comments
H: “It’s natural to associate ‘gossip’with ‘talk about’ but I thinksometimes it also may mean
‘laugh’”Interpretation

Y: “I wonder if there are two or more
answers, even though I chose one
for each. Sometimes I chose two.”

H knows the meaningof the word, but isassociating what
people do when theygossip (laugh).
Test familiarity; also needto systematize numberof correct As

H: “Some of the words [‘I hear,’ and‘get to’] I couldn’t understand themeaning.”

H knows one sense of
the word, but not thatwhich is tested.

Word associations seemed to work relatively well even when someitems confounded grammatical functions and collocations among
their syntagmatic choices. However, for one item, gossip, one student
was convinced that laugh should be a paradigmatic association. This
relates to the non-sensitivity of the test; i.e., in an effort to find plausible
choices, many of my distractors were similar in meaning.

Discussion and Final Comments
Several limitations emerged in developing and using these threetests. First, this may not be a completely accurate estimation of thetests’ usability in their intended context because the participants
selected for the project were not, at the time of the study, in a classusing this textbook. More students would also be more desirable fora valid analysis. Second, in future studies, the word association test
should have a systematic number of correct choices per item. Scoring
was difficult and students experienced doubt with the unequal numberof choices per item. Third, students need some familiarity with the test
format. One student was confused on the student-centered test andwrote sentences instead of definitions for part of it. Finally some control
for test effect between the tests should be put in place. Items such as
48

OnCUE Journal

commute and make an effort became progressively easier apparently
because they were learned in the tests and subsequent interviews.
Given these limitations, the research questions are answeredwith caution. In response to RQ1, I have chosen to rank the sentencecompletion test as most practical in a class with close to 30 students
or more, and the student-centered test as the most practical for smallerclasses. The word association test is less practical than the other two.
Answering RQ2 is more complex. The word association test
measured meaning and use aspects equally even when these were nottaught equally in the text. Thus, it did not meet the criteria for a fairassessment, and would be more suitable for a theoretical construct.
The student-centered test was superior in the students’ eyes and it
gave them a chance to demonstrate their knowledge. And, in termsof aspects of word knowledge, it was easy to design the test to matchthe aspect I wanted. It was, however, less controllable for consistentgrading and it did not measure production. The sentence completion
test measured productive recall, but combined use and meaning in
the prompts, making it difficult to isolate which aspect students werethinking about to arrive at the answer. If the teacher and students
wanted to know which aspects of a word to focus on, this test may not
be as informative.
In terms of validity for the purpose of informing me and my studentsof their current knowledge of the vocabulary items, the studentcentered test appeared to be more valid than the sentence-completion
test, which was in turn more valid than the word association test. If ateacher in a similar context were to choose a test based on practicality
and validity, I would recommend the student-centered test especiallyin smaller classes. However, if the teacher specifically wanted tomeasure productive recall, or had a large class, I would recommend
the sentence completion test. This and future studies investigating
short-term achievement tests can help teachers decide which test
method is most informative and practical for their course.
49Harrison

References
Harrison, G. (2007). University issues. Mishima, Japan: Unpublishedtextbook manuscript.
Nation, I. S. P. (2001). Learning vocabulary in another language.
Cambridge: Cambridge University Press.
Read, J. (2000). Assessing vocabulary. Cambridge: Cambridge UP.
Read, J. (2004). Plumbing the depths: How should the construct of
vocabulary knowledge be defined? In P. Bogards & B. Laufer(Eds.), Vocabulary in a second language: Selection acquisition,and testing (pp. 209-227). Amsterdam, Philadelphia: John
Benjamins Publishing Company.
West, M. (1953). A general service list of English words. London:
Longmans, Green and Company.

Xue, G., & Nation, I. S. P. (1984). A university word list.
Language learning and communication, 3(2), 215-229.
Appendix
Table 5. Aspects of Word Knowledge (based on Nation, 2001) ofPrimary Focus in the Textbook (Excerpt)
Form
Item

Meaning
word
parts

form &
meaning

concept &referentsRP(relationships)
R P (misunderstanding)RPspokenwritten

R

R

R

R

R

R

put sb downRR

not take long
to…

R

R

RP

RP

make fun of

R

R

RP

RP

lead sb totease
get to sb

RR

take on…

R

complicatedsend the wrong
messagerelate to…
not that great a…

(LW)

associations

RP

make an effort
gossiping

RP
RP

RP
RP
RP

R

RP

50

RP

RP

OnCUE Journal

Item

Use
collocations[& MWU]R P thingsgrammatical
functionsR P has got
R P sb…by
doing stR P to st/sb
RPacomplicatedsend the wrong
messagerelate to…
not that great a…
make an effort
gossipingput sb down

constraints on use

RP
RP
RP

RP

R P put sb down

not take long to…
make fun oflead sb totease
get to sb

R P start / realize
RP
R P sb to do stRP
R P too muchworktake on…

R = receptive knowledge, P = productive knowledge, LW = loan word usedin Japanese (not necessarily with same collocations)

Table 6. Test Results
1

2

3

4

5

6

7

8

9

10

11

12

complicated

not take longget tostresscommuteI hear what
you’re saying

percent

tease

downtime

totalgossip

stuff

1

1

0

1

1

1

1

0

1

0.5

0

0

7.5

62.50

1

1

0

0

0

1

0

0

1

0

0

0

4

33.33

Y

1

1

1

1

1

1

1

1

0

1

1

1

11

91.67

H

1

1

0.5 0.5 0.5 0.5 .5

0 0.5 0.5

0

0

5.5

45.83

Y 0.671

1

10.75 0.67 10.26 85.50

StudentCentered
Word
Association

make aneffortsend thewrong
message

Student
Y
Sentence
Completion H

Test

H 0.67 0.25

1 0.67 1 0.5

1 0.67 0.5

1

0

1

1

0.5 1 0.67 0.5 0.33 7.09 59.08

Msentence-completion = 47.92; Mstudent-centered = 68.75; Mword association = 72.29

51

Harrison

Vocabulary Test [student-centered test ]Follow the directions for each part. You will be asked to either write asentence, a definition, or collocations.
I. For the following words or multi-words, write a sentence. Try toshow that you understand its meaning and how the word is used.
1. complicated_________________________________________________________
2. make an effort_________________________________________________________
3. not take long to_________________________________________________________
4. downtime_________________________________________________________
5. stuff_________________________________________________________
II. For each of the following, write the meaning.
6. send (somebody) the wrong message________________________________________________________
7. gossiping________________________________________________________
8. tease_________________________________________________________
9. get to (somebody)________________________________________________________
10. (a) commute________________________________________________________
11. I hear you________________________________________________________
52

OnCUE Journal

III. For the following word, write at least one collocation.
12. stress________________________________________________________

Sentence Completion Test (excerpt)
1. Relationships were simple at first, but lately things have gotten
c__(complicated)___.
2. The American high school teacher never touches students
because she doesn’t want to send them the wrong m_(message)_.
3. I’m not working hard. I need to make more of an e_(effort)_.
4. We know it’s not polite to talk about other people’s weak points,
but we still g_(gossip)_.
5. It didn’t t__(take)__ l__(long)___ for people to notice I’m fromanother part of Japan.

Word Associations Test (excerpt)
Directions
Below each vocabulary word are two boxes. The left box has atleast one choice that relates to the meaning of the vocabulary word.
The right box has at least one choice that matches the grammar or
collocation of the vocabulary word. Each box has at least one correct
answer. Circle all the correct associations with the vocabulary word..
Example: exhausted
smellyused upvery tired
dirty

I’m…
That train ride was…

…mood
movie was…

1. complicatedconfusingstrangesurroundedcuriousrelationshiphas gotten

53

freelyquestionHarrison

2. message
plananswermisunderstanding
opiniontalk…
send the wrong…

forget to have a…
teach a…

productnote

make an…
try an…

plan an…
take an…

lies
kindness

party
light

…for them to see…to apologize…to realize…to effort

3. effort
attempt
deal

4. gossip
sharecomplainlaugh attalk about

5. not take longquickly happen
suddenly do

count upconnect to54