JLTA Newsletter No. 30
日本言語テスト学会
The Japan Language Testing Association

JLTA Newsletter No. 30
発行代表者：浪田克之介
発行所：日本言語テスト学会 (JLTA) 事務局〒389-0813 長野県千曲市若宮 758 TEL 026-275-1964
e-mail: youichi@avis.ne.jp URL: http://jlta.ac2010 年 (平成 22 年) 12 月 15 日発行
FAX 026-275-1970＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊JLTA 2010 annual conference
Yo In’nami(Toyohashi University of Technology)
I would like to take this opportunity to thank you all for joining the JLTA
annual conference held at Toyohashi University of Technology. The keynote
speech by John Read started by critically reviewing Coxhead’s (2000) Academic
Word List. Although this well-established list provides guidelines for constructing
a vocabulary test, Read argued that adopting a wider approach consideringcollocations and formulaic expressions is needed to complement the Academic
Word List. The symposium by Yasuyo Sawaki, Hideki Sakai, Rie Koizumi,
Tomoko Ishii, and John Read reflected the conference theme, “diagnostic testing inlanguage teaching”; this is a flourishing area in language testing and learning, since
the need to link testing with learning has increased over the years. The conference
was overall very well received, as shown by the 19 presentations (12 of which werein English, while 8 were delivered by overseas scholars from as far as Finland) anda relatively large number of attendees (86).
Nevertheless, there is room for improvement. First, although five of thepresentations ran simultaneously owing to time constraints, the number should havebeen restricted to just three or four at most. Many participants complained that they
had difficulty selecting from the high quality of presentations delivered
simultaneously. Second, although we posted information about our annual
conference on several mailing lists, we were able to attract only two testing
companies for commercial exhibition. Given the important roles tests play insociety as well as in tertiary education, we need to consider how to better appeal totesting publishers and companies in order to obtain better knowledge about thelatest test products and to enable publishers and companies to learn from thecutting-edge research in the field.
-1-

across EIKEN levels. 4) The 4000 wordfamily level may be a reasonable guideline
to comprehend the pilot TEAP, based on
the 95% benchmark.
Some future implications, such as
investigation of the relationship between
vocabulary, comprehension and test itemdifficulty, are also discussed in the end.
Reported by Yukie Koyama(Nagoya Institute of Technology)

第 14 回 日本言語テスト学会
全国研究大会 研究発表
於：北海学園大学
平成 22 年 9 月 11 日（土）研究発表：
Using word frequency lists to investigate
the vocabulary used in a pilot version ofa new university entrance exam
Jamie Dunlea(Society for Testing English Proficiency)

Modality and context effects inestimating the lexical knowledge of thetertiary-level Japanese learners ofEnglish with the yes/no test format
Toshihiko Shiotsu(Kurume University)
John Read(University of Auckland)

The purpose of this study was toinvestigate the validity of a pilot version of
TEAP (Test of English for Academic
Purposes) in comparison with the reading
and vocabulary sections (from Grade Pre-2
to Grade 1) of EIKEN by analyzing the
vocabulary used in the texts of these tests.
TEAP, developed by STEP and Sophia
University, is meant to measure the English
ability of high school students who areplanning to study at a university level. The
analysis tools for this research were thefollowing two versions of Range by P.
Nation, the GSL (General Service List) /AWL (Academic Word List) version and
the BNC version. The BNC version wasdeveloped in order to analyze the words
beyond the first 3000 word level because
the GSL/AWL version does not classify
these words.
The materials analyzed were 1. Pilot
version of TEAP reading section and 2.
Vocabulary and reading sections of EIKEN
tests (from Grade Pre-2 to Grade 1), andboth 1. & 2. were administered in 2009.
The main findings of this research are as
follows; 1) Vocabulary profiles of thereading sections of the EIKEN tests
analyzed showed clear differences acrosslevels, with higher percentage of more
frequent words in the lower level tests. 2)
Vocabulary profiles for the TEAP weregenerally closest to the Pre-1 test of EIKEN,
but fell between Pre-1 and Grade 2 forsome levels. 3) The number of words
needed to cover 95% of running words inthe tests increased in a consistent way

In this presentation, Dr. Shiotsu and Dr.
Read reported on the extended version of
the Yes/No test. Their research was very
innovative in three ways. First, it presents
the target words both in spoken and written
form. Secondly, it explores how theaddition of two types of sentence-based
context influences performance on the
Yes/No task. Thirdly, it investigates
whether reaction time adds a significant
dimension to the measurement ofvocabulary knowledge with this test format.
They developed two forms of an English
Yes/No test, based on samples of items
from the BNC word frequency lists. Thus,
twelve Yes/No test versions varying inform (A vs. B), mode (oral vs. written) andcontext condition (none vs. syntactic vs.
semantic) were produced.
With the sample of 346 Japanese
university students, the Yes/No test proved
to be a reliable measure in both the oral andwritten modes, and the oral and writtenversions of the same test tended to have
similar difficulties. In terms of the context
effect, the results indicated that providing
semantic context might make the same
Yes/No test more difficult, especially in theoral mode. The reaction time data showedthat the participants were consistently-2-

slowest in responding to the syntactic
contexts, whereas the latencies for the
no-context and semantic conditions werequite similar, and did not account forreading and listening comprehension test
performances in any useful way.
It was pointed out that it is necessary toestablish the criteria on which a non-word
can be made effectively because it might
affect reaction time of learners.
Reported by Katsumasa Shimada(Momoyama Gakuin University)

construct) of lexico-grammatical ability.
After the presentation, a few importantquestions from the floor facilitated a deeper
and lively discussion.
Reported by Tomoyasu Akiyama(Bunkyo University)

Diagnosing reading and writing in asecond or foreign language
Ari Huhta(University of Jyväskylä)
J. Charles Alderson(Lancaster University)

Investigating
theconstructoflexico-grammatical knowledge in an
academic ESL writing test
Yasuhiro Imao
(University of California, Los Angeles)

The past decade has witnessed a growinginterest in diagnostic language testing in thefield of language test development andresearch. In this presentation, the authors
claim that despite a number of attempts todevelop such tests, they have rarely been
based on a theory of language learning or atheory of diagnosis. The present paper
reported on an international 4-year(2010-2013) research project into thediagnosis of reading and writing abilities in
L2. The project brings together scholars
from various relevant fields, including
applied linguistics, psychology andassessment. The purpose of the project(DIALUKI – diagnosing reading andwriting in a second or foreign language) is
to identify the cognitive and linguisticfeatures which predict a learner’s strengths
and weaknesses in those areas. The paper
described the three main studies presenting
a number of instruments that had beendeveloped for the learners leaning English
as a foreign language and Finnish as asecond language in Finland. Besides thetheories that underlie a range of diagnostic
testing methods, a number of useful andinteresting tools were presented anddemonstrated, including a range ofcognitive and psycholinguistic measures in
informants’ L1 as well as diagnostic tools
for detecting L1 dyslexia, in order toexamine their applicability of L2 diagnosis.
The presentation continued for 40 fullminutes and ran out time for Q & A session.
Though the presentation was quite

Assessing grammatical ability inwriting skills is a challenging task, in that a
number of components seem to beinterwoven in the construct of grammaticalability. Based on the definitions oflexico-grammatical knowledge by Pupura(2004) and Rimmer (2006), this paper
examines the relationship between theconstruct of lexico-grammatical knowledge
and human rating data patterns. 200 ESL
writing samples were assessed by
ESL-trainedteachers
using
three
assessment
criteria(accuracy,
appropriateness,andrange).
The
Confirmatory factor analysis of humanrating data patterns showed that theaccuracy and range factors were moderately
correlated, suggesting that these two factors
are statistically within the same construct.
However,
appropriatenesswasnotcorrelated with the two, suggesting thatappropriateness is not included in the same
construct of accuracy and range. Further
sophisticated statistical analyses show thatthe three assessment criteria seemed to bedifferentiated. This paper gives an insight
into investigation of the construct oflexico-grammatical ability and issues ofhuman raters in productive skills. This
study also clearly showed a possibility of
exploring a theoretical framework (i.e.

-3-informative and insightful, one of the things
which was lacking involved the diagnostic
feedback. That is, it seems remaining to be
explored what feedback should be given to
the student in what way, so it may help him
or her to promote his or her learning
processes.
Reported by Yoshinori Watanabe(Sophia University)

performance
and
found
that
the
multiple-choice form was the easiest while
the timed dictation form was the mostdifficult. Focusing on the written form,
Moore (1937) determined that themultiple-choice form was useful formeasuring learners’ spelling ability. Yet it
remains unclear which type of spelling testis the most appropriate for EFL learners.
Twenty words were used in both Northby’sand Moore’s study (i.e., ninety, succeeded,similar, nickel, admission, carnival,
appearance, excitement, planned, finally,
orchestra,occasionally,
convenience,
exhibition, schedule, arriving, various,
humor, Saturday, and island).
The current study compares the fourtypes of written spelling tests (i.e., story
form, timed dictation, list form, andmultiple-choice) among 80 mid-level highschool Japanese learners of English.
Participants completed a pre-test of
approximately 90 regular words (i.e., withsound-letter correspondences; Mori, 2007)
to determine their knowledge of English
spelling rules. Participants subsequently
completed the four types of spelling tests(randomly administered); answers were
carefully analyzed using for example
Cook’s (1997) categorization for L1 and L2
spelling errors (i.e., insertion, omission,
substitution,transposition,
grapheme
substitution, and other) to classify
participants’ misspellings. The materials
which were used in this study were basedon the previous findings (Northby, 1936;
Moore, 1937). However, to determine thelevels of the target words, Gakken’s corpus
data and JACET 8000’s vocabulary lists
were considered as a standard. The spelling
tests are effective tools to examine learners’
level of proficiency in writing mechanics.
Thus, the results of this research will bebeneficial and helpful for Japanese teachersof English to decide which form to use in
his or her English lessons.

Diagnosing the English speaking abilityofcollegestudents
inChina--Development and validation of
the College English Diagnostic Speaking
Test
Zhongbao Zhao
(Shanghai Jiao Tong University)
This presentation was canceled.

A comparison of four types of spelling
tests among Japanese EFL learners:
Focusingonsound-letter
correspondences
Sachiyo Takanami(Graduate School,
University of Tsukuba)
The purpose of this study is to compare
and to determine the difficulties of fourtypes of the written form spelling tests
among Japanese learners of English.
Relationships between English spellings
and sounds (i.e., pronunciations) are quitedifficult for learners of English to acquiredue to their complicated orthographic
system. Yet spelling and reading areimportant aspects of literacy that cannot bedisregarded in language competence.
Nortyby (1936) compared five forms ofspelling tests—namely, story form (i.e., fillin the appropriate words in a given passage),
timed dictation (i.e., write whole sentences
in a limited time), list form (i.e., write thewords as pronounced), multiple-choice(choose the correct spelling from among
several choices), and oral form (spell theword orally)—for diagnostic purposes andinvestigated the variability of student

Learner
adaptedtesting:An
individualistic approach to language
assessment-4-Dr. Kwon presented a detailed overview
of test development for Levels 2 and 3 of
the NEAT, with a target date of 2013 forgeneral use. Piloting sessions of prototypes
for the NEAT began in 2009, and by theend of 2011 a grand total of 180,000
students will have participated in piloting at
various stages of development.
The NEAT Levels 2 & 3 are to be used
for gate-keeping by universities, withscores on the Level 3 exam pertaining tothose departments where students will needa satisfactory amount of practical English
language skills to study. Departments that
require not only practical English language
skills but also more advanced ability in
Academic English will use scores on Level
2. The distinctions correspond roughly tothose of abilities in BICS and CALP
(Cummins, 1980). In 2012, the NEAT will
be administered at schools nation-wide andvalidation procedures on the outcome will
be conducted to decide on the efficacy of
eventually replacing the CSAT presently
being used by universities for this purpose.
The NEAT Level 2&3 team isaddressing the great number of practicaland theoretical issues that a project of thismagnitude entails. Because development of
the NEAT is still in the midst of on-goingvalidation procedure of hypothesis testing,the content of Dr. Kwon’s presentation
focused to those areas for which concretemeasures to remedy perceived challenges
could be reported.
Concerning practical challenges, thereis the matter of managing the material andhuman resources for writing and rating thevarious items/tasks across administrations,and administering the exam itself, whilesafeguarding both security and consistency.
An estimated 600,000 high-school
students are projected take the NEAT
Levels 2 & 3 via the Internet within acertain time period during the school year.
With a projected capacity to administer theexam to 50,000 candidates for a single
145-minute sitting, it would take 12
administrations within 3 days to accomplish.
If a decision were made to implement 2

Parviz Alavinia(Urmia University)
This presentation was canceled.

The National English Ability Test of
Korea
Oryang Kwon
(Seoul National University)
For several years, The Republic ofKorea has been in the vanguard among
Asian nations coordinating expertise in thefield of language testing and foreign
language education with government
initiatives in order to enhance the English
language abilities of its human resources inan age of globalization.
In 2007, a project was initiated todevelop a comprehensive testing programof all four English language skills –Speaking, Listening, Reading, and Writing.
Its objectives are two-fold: 1) to promote
communicative competence in English as
an objective for teaching and learning in its
high schools and universities, and 2) tobetter inform stakeholders in the contexts of
Korea’s educational, governmental andcivilian institutions than the variety of
instruments that are presently being used.
The National English Ability Test, or
NEAT, is to have three different versions,each with content corresponding to thetarget language use domains for three
distinct groups of stakeholders and contexts.
Level 1 will be designed to measure theEnglish language proficiency of its
university students and graduates. Level 2
is to evaluate both proficiency andachievement of high school students ingrades 11 and 12. Level 3 will evaluate
both proficiency and achievement of highschool students in grade 10. The test
content for levels 2 & 3 will be tied closely
to the high school curriculum for English
language ability in all four skills proscribedby the Ministry of Education; hence theinclusion of ‘achievement’ within theconstruct.

-5-

sittings rather than one, the result would bea two-fold increase in needed resources.
The need for equating scores across thevarious forms used for administrations overtime presents another challenge; the use of
IRT and anchor items to equate thedifferent forms over time would be ideal,
but the practical limitations are formidable.
In the area of quality control and security
for item writing and rating, the plan is tocreate a pool of qualified secondary school
teachers via a 60-hour regimen of training
and on-going assessment for certification.
Eighty teachers were trained and certified
in 2010. The goal of the program is to haveobtained a pool of trained, certified 960
teachers by the year 2012.
Several issues involving test content forLevels 2 & 3 and their respective
theoretical constructs remain areas fordeliberation. On one hand, the methods forassessment are to complement the highschool curricula. On the other hand, thismatch between testing and teaching on anation-wide scale is intended to result in an
increased
levelofcommunicativecompetence in English that can be verified
by actual performance at universities andeventually, in society at large.
To match testing and teaching at
high-schools, how should Levels 2 & 3 be
defined in terms of operational objectives tobe taught and assessed? Dr. Kwon
presented a theoretical model based on
BICS & CALP and showed how it could beincorporated in components both in thehigh-school curricula and in the test
blueprint of test items and tasks for the
NEAT.
Governments and educators throughout
Asia share many of the same kinds ofproblems in their endeavors to improve theability of its people in the use of English
Language and achieving a positive andmeaningful washback from reforms in theface of adverse social and economic
paradigms. However, the comprehensive
and careful approach as presented by Dr.
Kwon leads me to believe that the initiative
to develop the NEAT in Korea has a very
good chance of success, and I look forward

to reports on progress and results in the
future.
Reported by Jeff Hubbell(Hosei University)

Validationofthelistening
comprehension component of the Centre
Test in Japan: Listening in the realworld, in the Course of Study, and in theCentre Test
Kozo Yanagawa(Graduate School, University of
Bedfordshire)
The presentation aimed to identify
importantdifferences
betweenthepresenter’s definition of real life listening,listening in the Courses of Study
curriculum guidelines, and the listening
component of the Center Test. The
definition of real life listening was derived
from a comprehensive framework ofcontextual parameters developed by thepresenterworkingwithinWeir’ssocio-cognitive
frameworkof
test
validation.
Yanagawa described 10 importantdifferences between real life listening andlistening as taught in Japanese high schools.
Results from a survey of teachers andstudents concerning which of theseelements should be better reflected in theCenter Test showed strongest support
among the teachers for increasing thevarieties of English (American, British,etc.) and better reflecting adjustments topronunciation in connected speech (Sandhi
variation).
Yanagawa’s concern for improving
listening instruction in Japanese highschool EFL classes was clearly evident. He
stressed that one of the aims of introducing
a listening component to the Center Test
was positive washback but suggested thispotential had not been realized. The
presentation was followed by a lively andinformative discussion with the audience. It
is worth noting that the study concentrated
on two of the five kinds of validity
described in Weir’s framework: Cognitive-6-and Contextual Validity. The concern withpositive washback would suggest that any
appeal to change in the Center Test would
benefit from also explicitly addressing
Consequential Validity. Indeed, any attempt
at an integrated, evaluative validation ofsuch a high stakes test would benefit from abalanced consideration of all of theelements of validity in the socio-cognitive
framework proposed by Weir, including
Scoring Validity.
Yanagawa’s work to define aframework of contextual parameters
relevant to EFL listening comprehension
tests will be a welcome contribution tolanguage testing, and the presentersuggested the full framework would bemade available in the future.
Reported by Jamie Dunlea(Society for Testing English Proficiency)

improvementof
test
performance.
Furthermore, surveys and interviews after
the tests showed that some memories or
habits of note-taking training impaired theirtest scores. Even though many participants
admitted the benefits of note-taking
strategies, lack of practice time ofnote-taking skills or insufficient time fortakings notes while testing was the maincause for the absence of its positive effects.
The speaker and the floor shared the sameview that more systemic training and moretraining time should be needed.
Reported by Iimura Hideki(Tokiwa University)

Comparison of Japanese and native
English-speaking raters’ perspectives onoral English performance
Takanori Sato
(Graduate School, Sophia University)

Effects of note-taking strategy training inlistening comprehension tests
Junghyun Kim(Sookmyung Women’s University)

Althoughvariousstudieshavesuggested the similarities and thedifferences in language performance ratings
between native speakers (NSs) andnon-native speakers (NNSs) of a particularlanguage, there is only a limited number of
researchwhich
conducted
directcomparisons between NS and NNS raters.
In addition, there were variations among theresults in the previous studies. Thus, thepresent study investigated the differences
between NSs’ and NNSs’ evaluation ofJapanese EFL learners’ oral English
performance.
A total of 30 university students
performed monologues on three topics.
Then, 4 NS raters and 4 NNS raters
assessed the learners’ performance based onthefollowing
criteria:
OverallCommunicative
Effectiveness(OCE),
Grammatical
Accuracy,
Fluency,
Vocabulary Range, Pronunciation, andContent Elaboration/Development. The
raters first assessed OCE and then the otheranalytical measures. A 6-point scale wasused for each measurement. The result
demonstrated that although the significant
difference was not found between the mean

This study investigated how students’
test performance would be influenced after
receiving the training of note-taking skills
in TOEFL listening tests. Three different
groups of Korean university students werecompared: (a) a previously untrained groupthat was not allowed to take notes during
the listening tests, (b) a previously
untrained group that was allowed to takenotes during the listening tests, (c) apreviously trained group (the experimentalgroup) that was allowed to take notes
during the listening tests. Based on thecontents of test preparation book by Lee(2006), the experimental group was given
two lectures on effective note-taking
strategies of iBT TOEFL listening
comprehension tests.
The results showed that the groups thatwere allowed to take notes performed betterin listening comprehension tests than thegroup that was not allowed, and that evenafter receiving note-taking strategies
training, students did not show significant-7-

scores of OCE, the Japanese NNS raters
evaluated accuracy and pronunciation
higher than the NS raters. It was suggested
that whereas the raters shared the common
views of the communicative effectiveness,
Japanese NNS raters’ leniency toward
accuracy and pronunciation was induced bysharing the common native language withthe students. It was also revealed that while
all of analytical measures of the NS raters
significantly predicted OCE, only fluency
and content elaboration were found to be
the significant predictors in the NNS raters’case. This difference may be due to recent
pedagogical approaches in Japan, such as
meaning-focused, communicative language
teaching.
In the 10-minute discussion, thefollowing topics were presented: reasons orideas for Japanese raters’ leniency on
grammar, the difference between thelengths of teaching experiences of the NS
and NNS raters, and the necessity ofcomparison between the NS and NNS
raters.
Reported by Katsuyuki Konno(Graduate School, University of Tsukuba)

non-native instructors. The instructors werealso found to have different opinions about
the use of group discussion task for theplacement test. One of the questions thatwere raised from the audience was the issueof rater training. The researcher explainedthat the lack of rater training could be oneof the causes of the rater disagreement.
Another issue addressed the reason forusing pronunciation for placement decision.
The researcher indicated that the instructors
have currently started discussing this issue.
It seems odd to use this aspect of student
oral performance for placement decision,
although it may have a great advantage ofpracticality of administrating the test.
Hidetoshi Saito(Ibaraki University)

Oral reading fluency as an assessment
instrument
Masanori Suzuki(Pearson Knowledge Technologies)
This presentation showed that oral
reading can also be an effective assessment
instrument. Reading passages out loudfluently with proper expression involves thereader’s ability to recognize whole words
rapidly and effortlessly and to express themeaning of the passage using appropriate
pausing, intonation, and phrasing. The
presenter presented three measurable
aspects of oral reading ability ― reading
rate, reading accuracy, and expressiveness― which provide a barometer of a student’s
literacy.
This presentation described a validation
and usability study which involved the testdevelopment team together with a US local
school district and then demonstrates how
the assessment tool saved teacher time,
gave feedback and encouragement to
students, created a digital portfolio ofstudents’ reading performances, and could
potentially involve parents in the process ofreading together with their children. It
also reported on the relation between (ORF
Oral Reading Fluency) and language
learning progress.
The presentation

Native and non-native raters’ judgmentof English pronunciation at a placement
test
Ji Eun Lee(University of Illinois at Urbana
Champaign)This qualitative research examined an
aspect of validity of a placement test for the
ESL program at the University of Illinois at
Urbana-Champaign. The ESL instructors
judge international incoming students’intelligibility of pronunciation on a group
discussion task. The placement decision isbased partly on the results of this test. The
researcher interviewed eight instructors
with varied teaching experience anddifferent native languages. The results
showed disagreement among the instructors
on the definition of “intelligibility.”
Native speaking instructors rated students’intelligibility more severely compared to-8-新旧 TOEIC®テストの比較検証--４テス
ト・セット 800 問の受験データ分析
法月健（静岡産業大学）
伊藤彰浩（西南学院大学）
島谷浩（熊本大学）

concluded with suggestions for using suchan automated ORF test in the context ofJapan to create a standard for English
education in Japan.
Reported by SHIOKAWA Haruhiko(Teikyo Kagaku Daigaku)

法月・伊藤・島谷・木下（2009）(以下、分析 1)では新旧 TOEIC®の難易度を分析し、新旧テスト間の難易度に有意差があり、表面上は従来通りの問題形式でも項目やテキストの特性に顕著な差が存在する可能性を示唆した。しかし、分析 1 は新旧１テストずつの比較である
ため個々のテストの特性の差の可能性とテスト実施の順番が旧→新の順であったため新テストに練習効果の可能性が指摘されていた。
この指摘に基づき、分析 1 で使用したテストを含め、新旧それぞれ２テスト・セット（計４テスト・セットの 800 問）を使い、日本の３大学の学生 86 名を 2グループに分け、グループ間で新旧テス
トの受験順序を変えてテストを実施した。
素点に基づいて新たに分析に加えた
新旧２テスト・セットの平均値を 2 グループで比較したところ、グループ間に有意差はなかったが、読解部門のみ新テス
トが有意に高かった。一方、等化した 4
テスト・セットのラッシュ項目難易度の平均値の比較においては、分析１に使用した新テストの聴解項目のみ他の部門の項目よりも有意に低くくなる結果が示された。以上から、新旧テスト間には若干の難易度差があるが、これは新旧テ
ストの恒常的な違いというよりは個々のテストの特性の要因である可能性が高く、今回の分析において新旧テストのいずれかに有利な練習効果が生じた可能性は低いことが分かった。
さらに、等化した４テスト・セット８
００問を難易度別に内容分析を行った。
その結果、例えば、難易度の高い聴解項
目は①音声中の状況になじみがない、②
音声中の表現と正答選択肢の内容とのDo test practice and keyword list help
oral summary test performance?
Hidetoshi Saito(Ibaraki University)
Saito reported on a study which
compared participants’ performance on anoral summary task (story retelling based onreading a passage) in four conditions: (a)practiced with a keyword list, (b) practiced
without a keyword list, (c) only a keyword
list, and (d) no practice and no keyword list
(impromptu). He tested the viability ofpredictions derived from two hypotheses:
The Test Practice Effect Hypothesis―testpractice facilitates better performance on
performance test tasks than no test practice,andtheRobustTest
Context
Hypothesis―test takers’ performance isunaffected by immediate task planning andother manipulations of task aspects (e.g.,concreteness of information) in testing
contexts. The results show that somepredictions derived from each of the two
hypotheses and that neither of the
hypotheses is superior in explaining all the
results. One notable finding in his study isthat the use of keyword list positively
influences performance, only whenaccompanied by practice. Without practice,
keyword list does not seem to benefit theoral summary test performance; it merely
seems to provide psychological security.
The findings of this study are helpful indetermining how teachers can help learners
better prepare for an oral summary test,
which in turn would enhance thedevelopment of learners’ abilities for oralsummary.
Reported by Atsushi Mizumoto(Kansai University)-9-

関係が間接的、③誤答選択肢の表現が音声中に頻繁に使用される、などの特徴が挙げられた。
報告者 谷 誠司
（常葉学園大学）る作文の評価を広めていくことを第一と考えており、詳細な説明は別途付けたいという回答があった。ライティングの
採点表の信頼性と妥当性を統計的、質的両面から分析した野心的な取組みであり、今後の展開に期待できるが、９人の教員の感覚を代表的と考えていいものか、そしてネイティブ教員の総合評価（同じ採点表を使ったものではなく）を基に併存的妥当性を見るのが適切かどうかについて少々疑問が残った。
報告者 松本佳穂子（東海大学）

EFL ライティング・ルーブリックの信頼性と妥当性の検証
大年順子 （岡山大学）
金志佳代子（兵庫県立大学）
久留友紀子（愛知医科大学）
正木美知子（大阪国際大学）
山西博之（関西外国語大学）
発表者の先生方が開発されたライティング採点表(「ルーブリック ２００
９」=analytic scale)の信頼性と妥当性を詳細に検証した研究である。３０人の大学生の作文を日本人教員５名、ネイティ
ブ教員４名が採点し、上記採点表を使用した日本人教員の結果については、５つの項目（「内容・展開」、「構成」、「文法」、「語彙」、「綴り・句読点」）と全体スコアの相関によりかなり高い信頼性が「綴り・句読点」を除く４項目について得られた。ネイティブ教員は総合評価（holistic scoring）を行い、その結果と日本人教員の採点結果の相関を基に併存的妥当性の検証がなされたが、そこでは「綴り・句読点」に加えて「構成」
についての相関が有意ではなかった。更なる妥当性の検証のためにＫＪ法による教員のコメントの詳細な分析がなされ、日本人教員とネイティブ教員が重要視するポイントについて共通点や相違点が明らかになった。イントロダクショ
ンにおけるトピック・センテンスの導入の仕方などの「構成」面の違いが見られたので、今後は問題のある部分の再検討
や共通理解のための説明書きの作成を進めていく。
会場からは、「文法」や「語彙」という評価点の中に適切さの概念が含まれていないのは不十分ではないかという指摘があったが、まずルーブリックによ英語エッセイの評価に求められる教師の特性とは―指導経験，環境，及び言語能力を背景に―
長橋雅俊（筑波大学大学院生）
本発表は特性の異なる評価者が英語作文を評価し、その採点の一貫性と配点の厳しさを分析した研究である。参加者として，教育経験などが多岐にわたる現職教師 10 名、および教職課程（英語）を履修中の学部生 10 名と、英語教育学を専攻し教職免許を有する大学院生 4名より協力を得た。評価者の変数には、指導経験年数、勤務校での指導の実態、普段指導している学習者の習熟度レベ
ル、そして言語熟達度／言語判断力が用いられた。評価者 24 名は、作文 25 編ずつを 6 段階の分析的評価法（内容展開、構成、正確さ、全体）で 2 回採点し、これらの間にはウェブサイト上での訓練
タスクが課せられた。この訓練タスクは、
誤文訂正、パラグラフ整合性などを問う
課題からなり、その中で各評価者の言語判断力が測定された。以上の採取データ
より，教師が勤務している高校の生徒習得度、作文の採点の厳しさや判断の一貫性、評価者の言語能力をそれぞれに多相ラッシュ・モデルで計測、および分析し
た。

-10-

訓練前（1 回目）の採点では、一貫性に欠ける（ミスフィット値がつけられた）教師は 10 人中 2 人（20％）であったのに対し、学生は 14 人中 5 人（36％）に上り、指導経験が一貫した判断に貢献していたことが窺われる。評価者の言語判断力に関しても、教師は学生に比べ明らかに高かったが、この測定値が低かった教師は該当の学生同様、はずれ値が多くみられた。訓練後（2 回目）の採点では、一貫性に欠けると判断された評価者の人数は減少し、トレーニング効果が見られた。また、教師が勤務している学校のレベル差や評価者の言語判断力が採点の厳しさに影響している可能性も指摘された。
報告者 村上京子（名古屋大学）

英語 Can-Do 調査分析に基づく TUFS
言語フレームワーク構築の試み
長沼君主（東京外国語大学）
工藤洋路（東京学国語大学）
吉冨朝子（東京外国語大学大学院）
東京外国語大学（TUFS）における「英語自律学習支援プログラム」は，2008
年度に設立された東京外国語大学英語
学習支援センター（ELC）により提供されており，学生への総合的な学習サポートを目的としている。このプログラムは，①Speaking Corner（ネイティブスピー
カーの英語講師とのスピーキングセッション。テーマを決めて，気軽に英語を話すことを楽しむ）, ②Writing Corner（ウェブベースの自動採点プログラム(Criterion)を利用したテーマ別ライテ
ィング。アドバイザーとのセッションで対面による指導も受けられる）,③e-Learning
Programs （ TUFS
e-Learning System を利用した速読・多聴学習支援プログラムなどを通して，リーディングやリスニングのトレーニングを行う。主専攻及び副専攻英語科目においては学習課題が設定されており，単-11-位認定条件となる）, ④English Library（Graded Readers などを取り揃え，多読を通した英語学習を推進。スピーキン
グ・セッションのモデルダイアログなどの多聴用コンテンツもポッドキャスト
等で配信），の 4 つから構成されている。
このように，インプット学習だけでなく，
アウトプット学習機会も設けられており，4 技能を総合的に網羅するプログラ
ムとなっている。英語を主専攻語または副専攻語として履修している学習者には，一定量の授業外学習課題をレベルに応じて課され，これらの学習の成果として，ポートフォリオ評価により，CEFRに準拠した「TUFS 言語パスポート」が発行される。CEFR に準拠しつつも，東京外国語大学独自の言語フレームワー
ク（TUFS 言語フレームワーク）を開発するため，その基礎資料として，「英語アカデミック Can-Do 調査（長沼，宮嶋，
2006）」の結果を，IRT を用いて困難度の分析をし，示唆を得た。現在は Can-do
評価タスクを開発中である。フロアーからは多くの質問があり，大変活発な議論がなされた。
報告者 髙波幸代（筑波大学大学院）

インハウス CAT の設計手法－実施済み
テストデータを利用したシミュレーシ
ョン－
秋山實（東北大学大学院教育情報学教育部／
株式会社ｅラーニングサービス）
アダプティブテスト（Computerized
Adaptive Test；CAT）は、受検者の能力水準に応じた難易度の問題項目を提示
して、短時間で効率良く高い精度で能力を測定できる。すでに大規模テストのCAT 化は実現しているが、本研究は比較的小さい特定の組織、例えば大学、企業、団体等が構築した項目プールを念頭に置き、その項目プールを用いたときの最適アルゴリズムと実施条件を探るためのシミュレーションシステムを開発することが目的であった。当日の発表では、
CAT の特徴を詳細に説明した後、インハ
ウス CAT の定義とその必要性、さらにCAT の仕組みが報告された。また、その設計の難しさも丁寧に説明されたので、本研究の意義がフロアにいた参加者にも正確に伝わった。CAT の最適実施条件
は項目プールの大きさと受験者特性によって異なるので、本研究のシステムは、
主に初期条件（能力値の初期値、初期テストレット）、項目選択（Owen の方法とフィッシャー情報量）、能力推定モデル（1・2 母数ロジスティックモデル）、終了条件（能力推定値の標準誤差 SE、
SE の変化値、受験項目数）などを制御できるように工夫されている。このシステムを用いた実際のシミュレーション
実験の報告に対し、フロアから難易度別の項目提示率に関する質問があったが、質疑応答に十分な時間を取ることができず、個別に対応することとなった。なお、実際の CAT では受験者の心理的な側面も考慮することも必要であると思われるが、本研究はシミュレーションシス
テムの開発が目的であるから、そこまで射程に入れることができない。この点はCAT 化を実現した組織が検討すればよいことであり、システムの問題とは言えない。本研究のシミュレーションシステム
は教育機関に公開される予定であると報告されていたので、今後、CAT の設計へ本研究のシステムが活用されることを期待したい。
報告者 服部 環（筑波大学）

日本人 EFL 学習者におけるワーキング
メモリ容量と記憶表象との関係：動詞分類課題による検証高木修一(筑波大学大学院生)-12-

読解プロセスとは、テキストの記述に関する心的な表象、つまりイメージを作り上げることである。読解力はこの心的表象、つまりイメージ作りができるかど
うかで決まる。本研究の目的は、心的表象の構築プロセスに関する学習者要因
として、ワーキングメモリ（WM）容量について検証することである。心的表象は
テキスト記述のみから作られる表層構
造、テキストベース、そしてテキスト記
述と読み手のスキーマなどの相互作用から作られる状況モデルの３つに区分される（e.g., Van Dijk & Kintsch,1983）。本研究においては、EFL 学習者（24 名）において WM 容量が大きい学習者のほうが、より精緻な表象が構築されるのではないかという仮説を設定し、イベント索引化モデルの枠内で使用されている動詞分類課題を使用し、その関係性を検証した。結果として、読み手のWM が記憶表象に与える影響が明らかになった。
報告後、「L1 で認知レベルが低い学習者の場合読解力を伸ばすにはどの次元を強調すべきか」、や「内容理解能力との相関はあるか」、などの質問がなされた。ワーキングメモリの容量の大きい
学習者は読解の処理能力も高いということは、メモリ容量が大きく処理能力が早いコンピュータを彷彿し、PC にはな
い人間だけが可能にする「想像力」や「創造力」との関係性も調査すると興味深いかもしれない。ワーキングメモリは、学習者の読解や理解という複雑な作業を行う処理能力である。しかし、これを測定するのは容易でない。報告者が指摘したように、動詞分類課題という手法に限界が感じられたが、迅速な情報処理とはどう意味をもつのかを示唆する貴重な研究であると言えよう。
報告者 李洙任
（龍谷大学）

スピーチにおける自己評価の妥当性：質
的観点から
深澤 真
（茨城県立竹園高等学校）

ワークショップ：

生徒は自己評価を行うことについて
どのように感じているのか。自己評価は生徒にどのような利益をもたらすのか。
これらは自己評価に対して我々が持つ疑問であるが、本発表からこれらの答えとなるヒントを得ることができた。
本研究では高校生がスピーチを行った後に自己のパフォーマンスに関しての評価を行ったが、その評価に対して「客観的な評価が難しかった」や「スピーチで緊張して正確な評価ができなかった」等の意見が得られた。これらの意
見から自己評価に対しての生徒自身の不安をうかがうことができる。生徒は普段から自己評価を行っていた訳ではな
く、自分自身を評価するということに慣れていなかった。もし自己評価の練習を重ねればこれらの不安が徐々に取り除かれるのではないかという意見が出されたが、その時間を捻出するという課題が残されているようだ。しかし、このような自己評価の客観性に関する不安がある中でも、生徒は自身の発表を振り返る機会としてや自分の英語力を反省する機会として自己評価を見ていた。評価
項目（発音、文法、流暢さ、態度など）については普段から指導が行われていたが、それらに注意を向けさせるという点においても自己評価には良い波及効
果があると考えられる。
自己評価が成績に考慮され得るか、もしくはされるべきかという議論の余地は十分にあるが、動機付けという点において自己評価は有益な方法であることは認められた。課題はうまく自身を評価できない生徒への対処や不安を取り除
く指導であると思われる。
報告者 佐藤敬典（メルボルン大学院）

This year’s two-hour workshop
conducted by Professor Akiyo Hirai of
University of Tsukuba and her student
Naoko Ito concerned the use of a three-way
analysis of variance (ANOVA) foranalyzing language testing and learning
data. The example of the three-way design
data demonstrated at the workshop included
three variables (rater, strategy, andproficiency), wherein it was investigatedwhether raters (trained/untrained) were able
to assess learners’ speaking performance
without being influenced by learners’
test-taking strategy (more/less) andproficiency level (high/mid/low). The rater
was a between-design variable, whereas thestrategy and the proficiency level wererepeated-design variables. The data wereanalyzed using the SPSS advanced modeland found to show a three-way interaction.
The data were then collapsed into two-way
designs and analyzed for each variable.
Although this can be done via pull-downmenus, the lecturers stressed the advantage
of programming syntax, especially whenanalyzing complex interactions.
The audience posed several questions in
response to which Professor Hirai and Ms.
Ito stated the following: It is technicallypossible to enter as many variables as
possible into an ANOVA design, but
including less than three variables isrecommended because of difficulty ininterpreting interactions; it is suggested thatreporting an ANOVA table along with agraph showing an interactional effect will
be effective, because visual information
greatly helps us interpret such an effect.
Further, researchers are encouraged toreport not only p values but also effect sizes(e.g., [partial] eta squared), because effectsizes indicate the proportion of varianceaccounted for by a variable. Overall, thepresentation was very well received, and分散分析―3 元配置デザインを中心に講師
平井 明代（筑波大学）
伊藤尚子（筑波大学大学院生）-13-the question-and-answer session was truly
insightful.
Reported by Yo In’nami(Toyohashi University of Technology)

講演：Issues in the assessment of academic
vocabulary
John Read(University of Auckland)
学術語彙知識の測定に関しての講演であった。
最初に，Nation (2001)をもとに，学術語彙知識の構成概念（construct）をどのように定義するのかという話から始まった。学術語彙の特徴の一つとして，幅広い学術文において見られ，学術文以外ではあまりみられないことがあげられる。
次に，学術語彙リスト(The Academic
Word List [AWL])についての説明がなされた。AWL は，ビクトリア大学で学部生向けに使われた学術文 350 万語のコーパスに基づいて，頻度や使用される学問領域の幅に従って抽出されている。AWL
は，学術語彙分野においてコーパスに基づいて体系的に作られた最初のリスト（domain）を規定したこと。二つ目は，テストされるべき単語を抽出する際の基礎ができたこと。三つ目は，テスト結果から学習者が知っている AWL 単語の割合を解釈できること。
また，AWL は，学術語彙テストの作成
以外に教育でも役立つことが紹介された。たとえば，AWL で頻繁に使われる単語，コロケーションの提示である。
AWL は，必ずしもあらゆる学問分野のテキストを集めているわけではなく理系分野のテキストが少ない，また，wordfamily が単位の基礎となっており一つの word family でも複数の意味を持っている場合テスト作成時には気をつけなければならない，といった AWL の限界も同時に紹介された。
報告者 片桐一彦（専修大学）

シンポジウム:Diagnostic testing in language teaching
Coordinator & Panelist
Yasuyo Sawaki(Waseda University)
Panelist
Hideki Sakai(Shinshu University)
Rie Koizumi(Tokiwa University)
Tomoko Ishii(Rikkyo University)
DiscussantJohn Read(University of Auckland)
The symposium entitled “Diagnostic
testing in language teaching”, which also
was the conference theme, was reallyfulfilling. Prof. Sawaki overviewed current
research on diagnosing second language
ability. As an example of a diagnostic test
DIALANG was explained. As a 2nd
approach, extracting detailed information
about learner performance from existing L2
assessments was explained. She talked
about
group
level
feedbackandである。また，広く頒布されており，無料である。
そして，AWL は学術語彙分野の診断テストとしても役に立つことが，主に 3 つの点から説明された。一つ目は，AWL は体系的な方法でもって学術語彙の領域-14-individualized feedback with examples andtheir limitations.

Prof. Ishii gave a presentation on her
investigation into a systematic andintegrative way to look at vocabulary sizeand depth. She described vocabulary tests
addressing
four
different
aspects:
vocabulary size, multiple meanings,
derivatives and lexical choice. She
suggested that by accumulating and using
the data on these four aspects, diagnostic
judgments can be made.

Prof. Sakai and Koizumi introduced the
ELPA English Diagnostic Test of Grammar.
The test is targeted at Japanese secondary
school students, and has two characteristics:
each test item belongs to one of the five NP
groups classified according to their internal
complexity; distracters can indicate
test-takers’ error patterns, or tendencies for
misunderstanding.
Afteradetailed
description of the test, they concluded thatteachers can use students’ diagnostic
information from the test and start remedial
teaching based on the information.

After three presentations, Dr. Read
commented on and summarized theirstudies, emphasizing the need to use andintegrate high-stakes proficiency andachievement tests (introduced by Sawaki)
and simple tests of aspects of language
knowledge for classroom use (Sakai,
Koizumi and Ishi) for diagnostic
assessment.

Several questions were posed by theaudience. Some responses from the-15-

panelists were: Edit Grammar is one ofmany possible ways to receive diagnostic
information and that integrated tests canalso be used; tests of more complex
vocabulary construct can be designed for
diagnostic testing; we should reconsider thevalue of integrative measures such as cloze
tests, dictation and elicit imitation. The
symposium was really insightful for
researchers and educators.
Rintaro Sato
(Nara University of Education)

事務局よりお知らせ
学会賞発表：
典さんに学会賞（最優秀論文）が授与されました。

総会において，本年度から始まった
The JLTA Best Paper of the Year の受賞者発表がおこなわれました。本年度の受賞者は，佐藤敬典氏（メルボルン大学大学院生）。後日，渡部良典・最優秀論文
表彰委員会委員長より，受賞者の佐藤孝◆ The JLTA office would be grateful if you could update us on your recent achievements
relevant to the field of language testing and evaluation. Any information on yourpresentations, publications, awards, and so forth would be greatly appreciated. The relevanceof the information will be evaluated by the office and given in the newsletter in due course.
日本言語テスト学会事務局〒389-0813 長野県埴科郡戸倉町芝原 758
TEL 026-275-1964
FAX 026-275-1970
e-mail: youichi@avis.ne.jp
URL: http://www.avis.ne.jp/~youichi/JLTA
編集： 広報委員会
委員長 片桐一彦 (専修大学)，副委員長 齋藤英敏 (茨城大学)委員
秋山實 (東北大学大学院／株式会社ｅラーニングサービス)，
長沼君主 (東京外国語大学)，佐藤臨太郎 (奈良教育大学)-16-