日 本 言 語 テ ス ト 学 会 (JLTA)
第 14 回（2010 年度）全 国 研 究 大 会 発 表 要 綱

Handbookof
the 14th Annual Conferenceofthe Japan Language Testing Association

大会テーマ (Conference Theme): Diagnostic Testing in Language Teaching

日時： 2010 年 9 月 11 日（土）8:15－17:00
会場： 豊橋技術科学大学 A棟 1階・2階(〒441-8580 愛知県豊橋市天伯町雲雀ヶ丘1-1, TEL (代表) 0532-47-0111)
主催：日本言語テスト学会
事務局 〒389-0813 長野県千曲市若宮758

TEL: 026-275-1964 FAX: 026-275-1970

THE JAPAN LANGUAGE TESTINGASSOCIATON
758, Wakamiya, Chikuma, Nagano 389-0813, Japan
Telephone: +81-26-275-1964 Fax: +81-26-275-1970
https://e-learning.ac/jlta.ac/
E-mail: youichi@avis.ne.jp

Conference Venue:
TOYOHASHI UNIVERSITY OF TECHNOLOGY
1-1 Hibarigaoka, Tempaku, Toyohashi, Aichi 441-8580, Japan
Telephone and Fax: +81-532-47-0111
http://www.tut.ac.jp

全 国 研 究 大 会 本 部 委 員
浪田 克之介
中村 優治
中村 洋一
小泉 利恵
塩川 春彦

（北海道大学名誉教授）（慶應義塾大学）（清泉女学院短期大学）（常磐大学）（帝京科学大学）

全 国 研 究 大 会 運 営 委 員
塩川 春彦
島田 勝正
伊藤 彰浩
法月 健
島谷 浩
中村 優治
藤田 智子
Soo-im Lee
小泉 利恵
前田 啓朗
土平 泰子
小林 美代子
印南 洋（帝京科学大学）（桃山学院大学）（西南学院大学）（静岡産業大学）（熊本大学）（慶應義塾大学）（東海大学）（龍谷大学）（常磐大学）（広島大学）（文京女子大学）（熊本大学）（豊橋技術科学大学）

全 国 研 究 大 会 実 行 委 員
印南 洋
小泉 利恵
塩川 春彦

（豊橋技術科学大学）（常磐大学）（帝京科学大学)

研 究 発 表 審 査 委 員
塩川 春彦
島田 勝正
伊藤 彰浩

（帝京科学大学）（桃山学院大学）（西南学院大学）

2

Table of Contents（目次）
1. Overall Conference Schedule（全体案内）

4

2. Conference Committee Information（学会事務局からのお知らせ） 10
3. Abstracts（アブストラクト）

12

4. Workshop Information（ワークショップ情報）35

5. Access to the Conference Venue（会場へのアクセス）36

6. Commercial Exhibits（展示協賛企業）

42

3

Overall Conference Schedule
Timetable
September 10, 2010 (Friday)
15:00－17:00 Pre-Conference Workshop “Analysis of Variance: Focusing on three-way designs”
Akiyo Hirai (University of Tsukuba)
Naoko Ito (Graduate School, University of Tsukuba)(conducted in Japanese; Room A207, 2nd floor, Building A)
16:00－17: 30 Board Meeting(Toyohashi Green Hotel, Lounge)
September 11, 2010 (Saturday)
8:15－
Registration(In front of Room A105)
8:45－9:00
Opening Ceremony(Room A101)
9:05－9:45
Presentation Part I
9:50－10:30
Presentation Part II
10:35－11:15
Presentation Part III
11:20－12:00
Presentation Part IV
12:00－13:00 Lunch Break(Committee Meetings: Room A109)
13:00－13:40 Presentation Part V
13:50－14:50 Keynote Speech(Room A101)
15:00－16:35 Symposium(Room A101)
16:40－16:55 General Business Meeting(Room A101)
16:55－17:00 Closing Ceremony(Room A101)
18:00－20:00 Banquet(Toyohashi Green Hotel, Lounge)
Commercial Exhibits: In front of Room A105
Participants’ Lounge: Room A105 (Drinks are available for free.)
Headquarter:
Room A109

大会日程表
2010 年 9 月 10 日（金）15:00－17:00 ワークショップ 「分散分析―3 元配置デザインを中心に」
講師：平井 明代（筑波大学）
・伊藤 尚子（筑波大学大学院生）（A207 教室）
16:00－17:30 理事会（豊橋グリーンホテル ラウンジ)2010 年 9 月 11 日（土）8:15－
受付
（A105 教室前）
8:45－9:00
開会行事（A101 教室）
9:05－9:45
研究発表 Part I
9:50－10:30
研究発表 Part II
10:35－11:15
研究発表 Part III
11:20－12:00
研究発表 Part IV
12:00－13:00 昼食（役員会：A109 教室）
13:00－13:40 研究発表 Part V
13:50－14:50 基調講演
（A101 教室）
15:00－16:35 シンポジウム（A101 教室）
16:40－16:55 総会（A101 教室）
16:55－17:00 閉会行事
（A101 教室）
18:00－20:00 懇親会（豊橋グリーンホテル ラウンジ)
協賛企業展示： A105 教室前
参加者休憩室： A105 教室（飲み物がございます（無料））大会本部：
A109 教室4

Program of the 14th JLTA Annual Conference
September 11 (Saturday)
8:15－
Registration (In front of Room A105)(During this time period, presenters can check the connection between the PC and the projector.)
Conference Attendance Fee: Members: ¥1,000; Non-members: ¥3,000 (except Students: ¥1,000)

8:45－9:00

Opening Ceremony (Room A101)
Coordinator Yo In’nami(Conference Organizing Committee Chair, Toyohashi University of Technology)
Greetings
Kiyokatsu Jinno (Vice President, Toyohashi University of Technology)
Greetings
Katsunosuke Namita (Emeritus Professor, Hokkaido University)

9:05－12:00

Presentation 1 (Presentation: 30 mins; Discussion: 10 mins)

12:00－13:00

Lunch(Committee Meetings: Room A109;

Part I
Part II
Part IIIPart IV

9:05－9:45
9:50－10:30
10:35－11:15
11:20－12:00

Participants’ Lounge: Room A105)

13:00－13:40

Presentation 2 (Presentation 30 min; Discussion 10 min)

13:50－14:50

Keynote Speech (Room A101)
Coordinator
Kazuhiko Katagiri (Senshu University)
Introduction of the lecturer Katsunosuke Namita(Emeritus Professor, Hokkaido University)
Theme: Issues in the assessment of academic vocabulary
Lecturer
John Read (University of Auckland)

15:00－16:35

Symposium (Room A101)
Theme: Diagnostic testing in language teaching
Coordinator & Panelist
Yasuyo Sawaki (Waseda University)
Panelist
Hideki Sakai (Shinshu University)
Rie Koizumi (Tokiwa University)
Tomoko Ishii (Rikkyo University)
DiscussantJohn Read (University of Auckland)

16:40－16:55

General Business Meeting (Room A101)
Selection of the chair
Reporter
Youichi Nakamura(JLTA Secretary General, Seisen Jogakuin Junior College)

16:55－17:00

Closing Ceremony (Room A101)

18:00－20:00

Banquet (Toyohashi Green Hotel, Lounge)
Coordinator
Katsumasa Shimada (Momoyama Gakuin University)
5

Part V

13:00－13:40

日本言語テスト学会第 14 回全国大会プログラム
9 月 11 日（土）8:15－
受 付（A105 教室前）
（PC 利用発表者：発表教室で機器接続確認）
学会参加費：会員 1,000 円、非会員 3,000 円（ただし学生は 1,000 円）
8:45－9:00

開会行事（A101 教室）
総合司会 印南 洋（研究大会実行委員長・豊橋技術科学大学）
挨拶 神野 清勝（豊橋技術科学大学副学長）
挨拶 浪田 克之介（北海道大学名誉教授）

9:05－12:00

研究発表 1（発表 30 分, 質疑応答 10 分）

12:00－13:00

昼
食（役員会：A109 教室

Part I
Part II
Part IIIPart IV

9:05－9:45
9:50－10:30
10:35－11:15
11:20－12:00

休憩室：A105 教室）

13:00－13:40

研究発表 2（発表 30 分, 質疑応答 10 分）

13:50－14:50

基調講演（A101 教室）
司会
片桐 一彦（専修大学）
紹介
浪田 克之介（北海道大学名誉教授）
演題： Issues in the assessment of academic vocabulary
講師： John Read (University of Auckland)

15:00－16:35

シンポジウム（A101 教室）
テーマ： Diagnostic testing in language teaching
コーディネーター兼パネリスト： Yasuyo Sawaki (Waseda University)
パネリスト：
Hideki Sakai (Shinshu University)
Rie Koizumi (Tokiwa University)
Tomoko Ishii (Rikkyo University)
討論者：
John Read (University of Auckland)

16:40－16:55

総16:55－17:00

閉会行事（A101 教室）

18:00－20:00

懇親会（豊橋グリーンホテル ラウンジ）
司会 島田 勝正（桃山学院大学）

Part V

13:00－13:40

会（A101 教室）
議長選出
報告 中村 洋一（JLTA 事務局長・清泉女学院短期大学）

6

Presentation Overview
Part

Time

I

9:05
－
9:45
II 9:50
－
10:30

第1室
第2室
第3室
第4室
第5室
(A114 教室) (A106 教室) (A108 教室) (A110 教室) (A111 教室)
DunleaTakanami
Sato
法月・
秋山
伊藤・
島谷
Shiotsu & Alavinia
Lee
大年・
高木
Read
金志・
久留・
正木・
山西
Suzuki

長橋

深澤

Huhta
& Yanagawa
Alderson

Saito

長沼・
工藤・
吉冨

--

Zhao

--

--

--

III 10:35－
11:15
IV 11:20
－
12:00
Lunch

Imao

V 13:00
－
13:40

Kwon

Kim

7

Presentation Details
The 1st Room (Room A114)
Chair Part I
Yukie Koyama (Nagoya Institute of Technology)
Part II
Katsumasa Shimada (Momoyama Gakuin University)
Part III
Tomoyasu Akiyama (Bunkyo University)
Part IV
Yoshinori Watanabe (Sophia University)
Part V
Yosuke Yanase (Hiroshima University)Part
Presenter (Affiliation)
Title
I
Jamie Dunlea (Researcher, Society for Using word frequency lists to investigate the vocabulary used
in a pilot version of a new university entrance exam
Testing English Proficiency [STEP])II
Toshihiko
Shiotsu(Kurume Modality and context effects in estimating the lexical
knowledge of the tertiary-level Japanese learners of English
University)with the yes/no test format
John Read (University of Auckland)
III
Yasuhiro Imao (University of Investigating the construct of lexico-grammatical knowledge
California, Los Angeles)
in an academic ESL writing test
IV Ari Huhta (University of Jyväskylä)
Diagnosing reading and writing in a second or foreign
J. Charles Alderson (Lancaster language
University)
V
Zhongbao Zhao (Shanghai Jiao Tong Diagnosing the English speaking ability of college students
University)
in China--Development and validation of the College
English Diagnostic Speaking Test
The 2nd Room (Room A106)
Chair Part I
Randy Thrasher (Okinawa Christian University)
Part II
Kozo Yanagawa (Graduate School, University of Bedfordshire)Part III
Jeffery K. Hubbell (Hosei University)
Part IV
Jamie Dunlea (Researcher, Society for Testing English Proficiency [STEP])
Part V
Hideki Iimura (Tokiwa University)Part
Presenter (Affiliation)
Title
I
Sachiyo Takanami (Graduate School, A comparison of four types of spelling tests among Japanese
University of Tsukuba)
EFL learners: Focusing on sound-letter correspondences
II
Parviz Alavinia (Urmia University)
Learner adapted testing: An individualistic approach tolanguage assessment
III
Oryang Kwon (Seoul National
The National English Ability Test of Korea
University)
IV Kozo Yanagawa (Graduate School,
Validation of the listening comprehension component of the
University of Bedfordshire)
Centre Test in Japan: Listening in the real world, in the
Course of Study, and in the Centre Test
V
Junghyun Kim (Sookmyung
Effects of note-taking strategy Training in listening
Women’s University)
comprehension tests
The 3rd Room (Room A108)
Chair Part I
Katsuyuki Konno (Graduate School, University of Tsukuba)
Part II
Hidetoshi Saito (Ibaraki University)
Part III
Haruhiko Shiokawa (Teikyo University of Science)
Part IV
Atsushi Mizumoto (Kansai University)Part
Presenter (Affiliation)
Title
I
Takanori Sato (Graduate School,
Comparison of Japanese and native English-speaking raters’
Sophia University)
perspectives on oral English performanceII
Ji Eun Lee (University of Illinois at
Native and non-native raters’ judgment of English
Urbana Champaign)pronunciation at a placement test
III
Oral reading fluency as an assessment instrument
Masanori Suzuki (Pearson
Knowledge Technologies)
IV Hidetoshi Saito (Ibaraki University)
Do test practice and keyword list help oral summary test
performance?
V--8

第 4 室（A110 教室）
司会
Part I
谷誠司 (常葉学園大学)
Part II
松本佳穂子 (東海大学)
Part III
村上京子 (名古屋大学)
Part IV髙波幸代 (筑波大学大学院生)
Part
Presenter (Affiliation)
TitleI
法月健（静岡産業大学）
新旧 TOEIC®テストの比較検証--４テスト・セット 800
伊藤彰浩（西南学院大学）
問の受験データ分析
島谷浩（熊本大学）II
大年順子（岡山大学）
EFL ライティング・ルーブリックの信頼性と妥当性の金志佳代子（兵庫県立大学）
検証
久留友紀子（愛知医科大学）
正木美知子（大阪国際大学）
山西博之（関西外国語大学）
III
長橋雅俊（筑波大学大学院生）
英語エッセイの評価に求められる教師の特性とは―指導経験，環境，及び言語能力を背景に―IV 長沼君主（東京外国語大学）
英語Can-Do調査分析に基づくTUFS言語フレームワー
工藤洋路（東京学国語大学）
ク構築の試み吉冨朝子（東京外国語大学大学院）
V--第 5 室（A111 教室）
司会
Part I
服部環 (筑波大学)
Part II
Soo-im Lee (龍谷大学)
Part III
佐藤敬典 (上智大学大学院生)
Part
Presenter (Affiliation)
Title
I
秋山實（東北大学大学院教育情報 インハウス CAT の設計手法－実施済みテストデータ
学教育部／株式会社ｅラーニング を利用したシミュレーション－
サービス）II高木修一 (筑波大学大学院生)
日本人EFL 学習者におけるワーキングメモリ容量と記
憶表象との関係：動詞分類課題による検証
III
深澤真（茨城県立竹園高等学校） スピーチにおける自己評価の妥当性：質的観点から
IV --V---9

Conference Committee Information
To All Participants
Going to the Venue
University car parks are not available for this conference. Please use public transportation to come to the venue.
Registration
1. The conference registration site is located in front of Room A105 (1st floor, Building A).
2. The conference attendance fee is ¥1,000 for members (including institutional members) and ¥3,000 for
non-members (¥1,000 for non-member students). If non-members apply for membership at the registration desk,
the conference attendance fee will be ¥1,000. (The JLTA annual fee is ¥10,000 and the admission fee is ¥1,000.)
3. Please wear your conference name card throughout the conference.
4. The banquet fee is ¥3,000. The banquet registration is conducted at the the registration desk. The banquet will
be held in the lounge of the Toyohashi Green Hotel, which is at a two-minute walking distance from the east gateof JR (Japan Railway) Toyohashi Station. After the conference, the shuttle bus will take you to the hotel free ofcharge.
5. The conference handbook is available at the registration desk.
Lunch and Participants’ Lounge
1. Lunch can be purchased at a nearby convenience store. The store is 100 meters from the university main
entrance gate.
2. Complimentary drinks are available in Room A106.
3. No smoking is permitted on campus.
Accommodation
We are afraid that we provide no accommodation services through our association. Please make the arrangements
by yourself.
Emergency Contact E-Mail Address: koizumir@tokiwa.ac.jp (Rie Koizumi).

To Presenters
1. Presenters will have 30 minutes to present their paper, followed by 10 minutes for discussion.
2. Please register at the registration desk first. Please go to the designated room 10 minutes prior to the startingtime of the presentation.
3. If you are not a member and have not paid the ¥3,000 “Presentation fee” (different from “Attendance fee”)
before the conference, please pay it at the registration desk. This rule applies to every presenter on the program.
4. You are expected to connect your computer to the projector and operate it yourself. The projector and connector
cable can be found in the room. There is no sound system and as such you cannot play sounds from the computer.
LAN internet access is not available. During the registration period, the presenters can check the connection
between the PC and the projector.
5. Please bring your handouts in case the PC or the projector does not work.
6. If a letter of invitation is required, please send an e-mail message to Rie Koizumi at koizumir@tokiwa.ac.jp

To Chairs
1. One chair is assigned to each presentation.
2. Please make sure that the presentation does not exceed the allotted time.
3. Please start the presentation at the time designated in the program. Please do not change the starting time andthe order of the presentations.
10

学会事務局からのお知らせ
大会参加者へのご案内■駐車場
学会用に駐車場は準備しておりません。公共の交通手段をお使いください。
■受付
1. A 棟 1 階の A105 教室前で行います。
2. 学会参加費は、会員 1,000 円（個人・賛助会員を含む）、非会員 3,000 円（ただし学生は 1,000 円）です。非会員の方でも、受付で入会手続きを行えば学会参加費は 1,000 円となります。JLTA 年会費は10,000 円、入会費は 1,000 円です。
3. 学会中は、名札をお付けください。
4. 懇親会費は 3,000 円です。受付でお支払いください。懇親会は豊橋グリーンホテルのラウンジにて開かれます。ホテルは JR 豊橋駅東口より徒歩 2 分の場所にあります。大会終了後、懇親会場まで、無料の貸し切りバスを用意いたしましたのでご利用下さい。
5. 『JLTA 第 14 回（2010 年度）全国研究大会発表要綱』は受付で配布します。
■昼食・休憩室
1. 昼食は受付では販売いたしません。会場から徒歩 5 分（正門から 100 メートル先）の場所にミニストップがあります。
2. 飲み物は A106 教室にございます（無料）。
3. 大学内はすべて禁煙です。
■宿泊
宿泊の斡旋はいたしておりません。
■緊急連絡先の E メールアドレスkoizumir@tokiwa.ac.jp (小泉利恵)

発表者へのご案内
1. 30 分の発表と 10 分の質疑応答の時間があります。
2. 受付を済まされ、発表開始 10 分前には、発表会場にお越しください。
3. 非会員で、「研究発表費（参加費とは別）」の 3,000 円を事前に支払われていない方は、受付時に支払いをお願いいたします。これは、プログラムに掲載する共同研究者すべてに適用されます。
4. ご発表にプロジェクタを使われる方は、PC との接続作業は各自で行ってください。発表会場にはプロジェクタとともに接続ケーブルもございます。PC からの音声を会場のスピーカーから出すことはできません。インターネットへの直接接続はできません。朝の受付時に接続を確認できます。
5. 予測できない不具合に備え、ハンドアウトのご持参をお勧めします。
6. 出張依頼状などが必要な方は、事務局までご連絡ください。

司会の方へのご案内
1. 1 発表につき 1 人の司会をお願いしています。
2. 時間オーバーにならないようにお気をつけください。
3. 決められた時間に発表を始めてください。発表時間や順番を変えないでください。

11

Abstracts

Keynote Speech(Room A101)

ISSUES IN THE ASSESSMENT OF ACADEMIC VOCABULARYJohn Read (University of Auckland)

There are two basic issues that need to be addressed in the design of any vocabulary test. The first is how to selectthe lexical units that will be the target items for the test, and the other is the type of test format to use indetermining whether the learners have acquired the target items. I will discuss these two issues in relation to theassessment of vocabulary size, and academic vocabulary in particular. Coxhead’s (2000) Academic Word List
(AWL) is now well established as the standard reference for high frequency word families in academic texts
and the conventional means of testing those words is by the use of relatively decontextualized formats such as
multiple-choice, gap-filling, matching and translation. However, the AWL is not above criticism. Read andChapelle (2001) argue that the construct of academic vocabulary should be defined from an interactionalist
perspective, taking account of learner knowledge of vocabulary within particular disciplinary areas. In a morecomprehensive critique, Hyland and Tse (2007) present evidence that the AWL does not adequately cover therange of academic disciplines, but they also argue that no general academic vocabulary list can satisfactorily
represent the various uses of lexical items across academic fields. Thus, we need to consider whether a different
approach is required to the selection and testing of academic vocabulary items, one which takes more account of
the contexts in which words are used. Another dimension to the question is how to deal with the collocations andformulaic expressions that occur frequently in academic discourse. Vlach-Simpson and Ellis (2010) have recently
published an Academic Formulas List (AFL), which is designed to complement the AWL, but which raises freshquestions about whether the formulas can simply be treated as another kind of lexical item for assessment
purposes, using the conventional test items, or whether new kinds of test format should be developed. In keepingwith the conference theme, I will discuss these matters from the standpoint of diagnostic language testing.

12Symposium(Room A101)

Diagnostic testing in language teaching
Coordinator & Panelist
Yasuyo Sawaki (Waseda University)
ysawaki@waseda.jp
Panelist
Hideki Sakai (Shinshu University)
Rie Koizumi (Tokiwa University)
Tomoko Ishii (Rikkyo University)
DiscussantJohn Read (University of Auckland)

Introduction
Coordinator: Yasuyo Sawaki (Waseda University)
Diagnosing second language (L2) ability aims at identifying strengths and weaknesses of language learners, so thatthe results can inform instruction and learners’ self studies. Despite the well-recognized importance of this topic for
effective second language instruction, historically, diagnostic assessment was a largely ignored area in the field of
language assessment (Alderson, 2005). Recently, however, interests in this topic are growing rapidly in the field,
and this movement appears to have been propelled by the development of DIALANG, a web-based diagnostic
self-assessment instrument in 14 different European languages, during the last decade.
This symposium provides an introduction to the topic of diagnosing L2 ability in general and presents
illustrative examples of recent efforts toward extracting diagnostic information from learner test performance on
the assessment of four skills as well as grammar and vocabulary, both in international and Japanese contexts. The
first paper by Sawaki provides a general overview of recent developments in L2 learner diagnosis in the field anddescribes recent works on related topics in the assessment of reading, listening, speaking, and writing skills inlarge-scale international assessment contexts. The other two papers focus specifically on diagnosing language
ability of Japanese learners of English. The second paper by Sakai and Kiozumi focuses on the development effort
of a diagnostic grammar assessment for Japanese secondary school students, while the third paper by Ishii
proposes a new approach to assessing different aspects of Japanese English learners’ vocabulary knowledge. Afterthe presentation of these three papers, issues that emerge from those studies and future directions in diagnosing
second language ability in the Japanese context will be discussed.

Paper 1: Current research on diagnosing second language ability: An overview
Yasuyo Sawaki (Waseda University)
The primary purpose of this presentation is to provide a general overview on recent developments in research on
diagnostic assessment in the field of language assessment. An important aspect of diagnosing second language(L2) ability, or understanding L2 learners’ strengths and weaknesses to inform instruction, is to provide learners
with detailed performance feedback. This line of work has largely been proceeding in two directions in the field.
One is to develop a new assessment designed specifically for diagnosis from scratch, as can be seen in the recent
development of DIALANG (Alderson, 2005, Alderson & Huhta, 2005) in Europe. The other approach is to
extract detailed information about examinee performance characteristics from an existing assessment that is not
specifically designed for diagnosis. In this case, an important goal is to generate score reports containing detailed
feedback on test results. So far, albeit the history is fairly short, previous research relevant to L2 ability diagnosis
has focused primarily on the second approach, many of which have taken place in large-scale international
language assessment contexts.
13

The first area of work for enhancing performance feedback in existing assessments represents
techniques employed to provide group-level performance feedback, including scale anchoring (Beaton & Allen,1992) and the can-do statement method. These approaches offer information about performance characteristics of
examinees at different ability levels. However, a limitation of these approaches is that everyone in the same
performance level receives the same feedback. Thus, the feedback does not necessarily reflect performance
characteristics of a specific learner. In an attempt to address this issue, language testing researchers are exploringother approaches that allow extraction of performance feedback tailored to individual learners’ needs. One iscognitive diagnosis (Lohman & Ippel, 1993), which combines cognitive psychology and educational measurement
to extract learner’s performance characteristics based on a detailed analysis of test tasks and learners’ item response
patterns. Cognitive diagnosis has been applied to analyses of examinee performance on selected-response reading
and listening comprehension assessment items. Another relevant area of work is automated scoring of learner
responses to performance-based speaking and writing assessments, where linguistic characteristics of speechsamples and essays are analyzed from various perspectives to identify characteristics of learner responses that
distinguish different levels of task performance.
In this presentation, illustrative examples of previous applications of the different areas of work above
will be described. Challenges and issues emerging from the previous work and possible directions for future
research will be discussed as well.

Paper 2: Diagnostic information from the ELPA English Diagnostic Test of Grammar
Hideki Sakai (Shinshu University)
Rie Koizumi (Tokiwa University)
The English Diagnostic Test of Grammar (EDiT Grammar) is a diagnostic grammar test mainly targeted at
Japanese secondary school students. The purpose of the EDiT Grammar is to identify the grammaticalweaknesses of learners that cannot be easily identified by teachers during teaching activities and to present
information that will be useful for teaching in the future. It focuses on the students’ knowledge of basic English
noun phrases (NPs), especially their internal structures, because previous research has suggested the difficulty
faced by Japanese learners of English in acquiring these phrases. An example is provided below.
Instruction: Choose the most appropriate English phrase for the underlined part of the Japanese sentence from 1
to 4.
丘の上の白い建物は、私の学校です。(NP Group 3)(The white building on the hill is my school.)1. the hill on the white building (Japanese word order)
2. the white building on the hill*
3. the white on the hill building (Premodification)
4. on the hill the white building (Postmodification)
The EDiT Grammar has two main characteristics. First, each test item belongs to one of the five NP groups:(a) Group 2 ([determiner] + premodifier + head noun; e.g., my green sweater), (b) Group 3 (NP + prepositional
phrase; e.g., a surprise party for my brother), (c) Group 4 (NP + present/past participle phrase; e.g., books written
in English), (d) Group 5 (NP + relative clause; e.g., things I bought yesterday), and (e) Group 6 (NP + to-infinitivephrase; e.g., books to read). The test was administered to 107 ninth-year students, of which only 30% to 67%
could select correct answers in each NP group; the test items under Group 2 were the easiest, whereas those underGroup 6 were the most difficult. The low percentages of correct answers support the claim that greater emphasis
must be placed on the need for systematic teaching of NP internal structures in Japan.
The second characteristic of our diagnostic test lies in its distractors, each of which can indicate a test-taker’s
14

error patterns, or tendencies of misunderstanding NP structures. Distractors were created on the basis of five error
patterns: failure to correctly understand (a) premodification structures, (b) postmodification structures, (c) phrase
structures, and (d) clause structures, and (f) errors of using the Japanese word order. The results showed that atleast 10% of the learners selected all the distractor types, and that the highest percentage of errors made by them
involved (a) premodification structures (27%). Since (a) distractors appear to be the easiest errors to detect, we
argue that even a simple rule causes learning difficulties in Japanese learners of English.

Paper 3: Synthesizing vocabulary size and depth measures
Tomoko Ishii (Rikkyo University)
Over the last few decades, the importance of vocabulary in second language came to be well recognized andmuch research has been conducted in the area. One of the focused areas of research is how learners’ vocabularyknowledge can be assessed. The research interests have lied not only in how many words learners know
(vocabulary size), but also in how well they know those words (depth of knowledge), as knowing a single
meaning for a word is not enough to use and understand the word in the actual language use. There are manythings learners need to know about a word, such as grammatical behaviour, shade of meaning, collocation,
restriction on the usage, and many more. Reflecting the complex nature of vocabulary knowledge, many
researchers have devised various measures for depth of vocabulary knowledge in order to tap into different typesof knowledge.
However, most of the depth measures in the past have addressed only a single depth aspect, and not
many researchers have investigated how size and depth approaches can be combined in a test battery. As aconsequence, it is sometimes difficult to interpret the learners’ performance on one test of vocabulary depth,
especially when they do not perform well – is it because they do not know the words well enough, or is it simply
because the number of the words they know is limited? Looking at a single aspect never tells us an answer for this.
Also, it seems the filed has shown more interest in research-focused assessment, and not much has been discussed
about diagnosing vocabulary knowledge.
Addressing these issues, this paper describes how one size-depth vocabulary test battery wasdeveloped for a specific student group (Japanese university students), and proposes a scoring scheme thatcombines size and depth scores in a principled way. The test battery proposed in this paper taps into four aspectsof vocabulary knowledge: vocabulary size, multiple meanings, derivatives, and lexical choice. By accumulating
the data on these aspects from Japanese students, an attempt is made to capture their typical development ofvocabulary knowledge, and to make diagnostic judgment.

15

Paper Presentation
The 1st Room (Room A114) Part I
Using word frequency lists to investigate the vocabulary used in a pilot version of a new university
entrance exam
Jamie Dunlea (Researcher, Society for Testing English Proficiency [STEP])
j-dunlea@eiken.or.jp
Identifying relevant levels of vocabulary knowledge can improve the validity of a language test by allowing forcomparison of the vocabulary included in the test with texts relevant to the TLU domain. This paper uses theRange vocabulary analysis tool to investigate the usefulness of analyzing vocabulary levels for a pilot version of atest of academic English proficiency. The test is intended for use as a university entrance exam for a Japanese
university. The original version of the Range program defined three vocabulary levels based on the General
Service List and the Academic Word List. Together, these lists represent a total of almost 3000 word families, andhave been shown to cover more than 85% of running words in a variety of texts. However, Nation and others
have claimed that up to 98% coverage of running words in a text is necessary for learners to be able to read
unmodified texts without undue strain, and that the 3000 word families in the GSL/AWL are not sufficient toprovide this coverage. To address this problem, Nation has developed a further list of 14 levels, each containing
1000 word families derived from the BNC spoken corpus. The present research used both the GSL/AWL versionof Range and the BNC version.
In addition to the pilot test, the reading and vocabulary sections of the upper four grades of the EIKEN testing
framework were also analyzed for comparison. While recognizing that the use of word families and frequency
lists have many limitations, the paper will demonstrate that the analysis of vocabulary levels can help to improvethe content validity of a pilot test by identifying similarities and differences in the vocabulary used compared totext types relevant to the TLU domain identified for the test. In addition, at a practical level, the analysis has also
demonstrated that the Range software can be a useful and easy to manipulate source of information to help item
writers identify relevant vocabulary targets and for identifying potentially problematic items in reading texts inadvance.

16

The 1st Room (Room A114) Part II
Modality and Context Effects in Estimating the Lexical Knowledge of the Tertiary-Level Japanese
Learners of English with the Yes/No Test Format
Toshihiko Shiotsu (Kurume University)
toshihiko_shiotsu@kurume-u.ac.jp
John Read (University of Auckland)
Lexical measures have been shown to function well not only to produce estimates of vocabulary size but also as
general indicators of proficiency level, for purposes such as placement of learners in a language program and forvocabulary acquisition research. The simplest kind of vocabulary measure is the Yes/No format, in which
test-takers report whether they know each of a sample of target words.
The present study extends the Yes/No research in three ways. First, it presents the target words in spoken as wellas written form (cf. Milton and Hopkins, 2005). Secondly, it explores how the addition of two types ofsentence-based context influences performance on the Yes/No task. The third innovation is to investigate whether
reaction time adds a significant dimension to the measurement of vocabulary knowledge with this test format.
Two forms of an English Yes/No test were developed, based on samples of items from the British National
Corpus word frequency lists compiled by Nation (n.d.). Thus, 12 Yes/No test versions varying in form (A vs. B),
mode (oral vs. written) and context condition (none vs. syntactic vs. semantic) were produced.
A total of 270 Japanese university students contributed as participants, each taking both forms of the Yes/No test
under contrasting mode or context condition, along with a listening and a reading comprehension test.
Performance data from further 45 students taking the two Yes/No forms under the same mode and context
conditions provided links in the dataset for a Rasch-based comparisons of the modality and context effects on
item difficulty.
The presentation will outline the overall design of the study and present a selection of the results. With the given
sample of learners, the Yes/No test proved to be a reliable measure in both the oral and written modes, although
the same words appeared to become more difficult under the oral mode. In terms of the context effect, the results
indicated that a richer semantic context produced more consistent measures and better correlations with thelistening and reading tests than a bare syntactic context or the no-context condition. The reaction time data
showed that the participants were consistently slowest in responding to the syntactic contexts, whereas thelatencies for the no-context and semantic conditions were quite similar. Contrary to expectation, there was noclear association between the oral Yes/No test and listening comprehension on the one hand, or the written modeand reading comprehension on the other.

17

The 1st Room (Room A114) Part III
Investigating the construct of lexico-grammatical knowledge in an academic ESL writing test
Yasuhiro Imao (University of California, Los Angeles)
yasuimao@ucla.edu
Grammar has played an important role in language education and has been tested in various formats. However, atheoretical framework on which grammatical ability is based did not exist until recently. In addition,
lexico-grammar as a construct in performance assessments is often operationalized as impressionistic evaluationof language samples rather than comprehensive analyses of them. Based on the definitions of lexico-grammaticalknowledge by Purpura (2004) and Rimmer (2006), this study attempts to model the construct oflexico-grammatical knowledge in an academic ESL writing test. It employs a corpus-based analyses approach toanalyze written language samples to examine the relationships between the construct and human ratings of thesamples.
Two hundred writing samples taken from the University of California, Los Angeles (UCLA) English as a Second
Language Placement Examination (ESLPE) were analyzed for errors, inappropriate uses, and range of uses,
based on past studies of written academic discourse. The frequencies of identified elements were standardized
and bundled into groups (parcels) to be hypothesized as indicators of three factors (accuracy, appropriateness, andrange) in confirmatory factor analysis (CFA). The results showed that the accuracy and the range factors weremoderately correlated while the correlations between these two and the appropriateness factor were
non-significant. The findings suggest that accuracy and range may be two components of the same construct,whereas appropriateness is related but a somewhat different construct.
The analytic lexico-grammar human ratings (accuracy, appropriateness, and range) were judged by experienced
ESL instructors, who had been trained for the study. The relationships between the lexico-grammaticalknowledge based on language elements and analytic lexico-grammar human ratings were also examined by
CFA to investigate how the three factors would affect human ratings. The results revealed that threelexico-grammar factors not only had effects on associated analytic ratings, but also on other ratings. The findings
suggest that raters may not have been able to clearly differentiate the three constructs.

18

The 1st Room (Room A114) Part IV
Diagnosing reading and writing in a second or foreign language
Ari Huhta (University of Jyväskylä)
ari.huhta@jyu.fi
J. Charles Alderson (Lancaster University)
Diagnostic language testing is a growing area of language test development and research. However, what areclaimed to be diagnostic tests are often little more than placement tests, and are very rarely based on a theory oflanguage learning, or a theory of diagnosis. Moreoever, several recent studies have merely attempted to retrofit
diagnostic information to tests that have been developed as proficiency tests - see, for example, the special issueof Language Assessment Quarterly, 2009, Volume 6 (3). Whilst useful Q Matrices and procedures for statisticalanalysis have been developed, they remain uninformative unless the tests being analysed are solidly based on
relevant applied linguistic theory.
In this paper, we report on an international 4-year (2010-2013) research project into the diagnosis of reading andwriting abilities in L2. The project, entitled DIALUKI – Diagnosing reading and writing in a second or foreign
language - seeks to identify the cognitive and linguistic features which predict a learner’s strengths andweaknesses in those areas. The project brings together scholars from applied linguistics, psychology andassessment to engage in multidisciplinary work and to develop innovative ways of diagnosing the developmentof L2 abilities.
This paper will describe the three main studies which are planned for the duration of the project. Study One
explores a range of potential diagnostic measures, and Studies Two and Three will involve longitudinal studies of
L2 development and experimental learning trials. We present details of the instruments being used in Study One,
where several hundred learners of English as a foreign language and Finnish as a second language will be studiedcross-sectionally in Finland; the learners’ ages range from 10 to 18. The study explores the diagnostic potential ofa range of cognitive and psycholinguistic measures (e.g., working memory, phonological processing, ability toprocess non-words) in informants’ first language (Finnish or Russian) as well as diagnostic tools for detecting L1
dyslexia, in order to examine their applicability for L2 diagnosis. We also include measures of L2 vocabulary,
motivation and background information on the informants, and examine the relationship of all variables tomeasures of first and second / foreign language reading and writing abilities.
It is expected that this project, being firmly based on theories of first and second language reading and writing,will make a major contribution to understanding how to develop and validate diagnostic instruments andprocedures.

19

The 1st Room (Room A114) Part V
Diagnosing the English speaking ability of college students in China--Development and validation of theCollege English Diagnostic Speaking Test
Zhongbao Zhao (Shanghai Jiao Tong University)
michaelzhao998@hotmail.com
This paper reports the development and validation of the College English Diagnostic Speaking Test (CEDST)
designed in the context of EFL teaching and learning at the tertiary level in China. The process of developing andvalidating the CEDST is expected to demonstrate the usefulness of such kind of test as a tool for identifying thestrengths and weaknesses of students’ English speaking ability. And it is also hoped that the study would yield aprocedural framework to be used by college English teachers to design their own diagnostic oral English
language tests. The CEDST is designed as a 15-minute face-to-face interview test, which employs three task
types: reading aloud, presentation and role-play. A checklist is designed for the examiner to record eachtest-taker’s performance with respect to the following six categories of criteria: intelligibility, vocabulary range
and accuracy, grammatical accuracy and complexity, fluency, communicative effectiveness, use ofcommunicative strategies, and coherence and cohesion. The score report form is designed to include a composite
grade and individualized feedback detailing students’ strengths and weaknesses. As part of the a posteriori
validation of the CEDST, a case study was conducted with 14 sophomore students from a national key university
in Shanghai. The participants were engaged in five sets of CEDST tests within a time span of three months. For
each test, they got a detailed score report form. At the end of the tests, a questionnaire survey and a follow-up
face-to-face interview were conducted to collect test-takers’ opinions on the usefulness of the test. The datareported in the paper include 14 self-assessment checklists, 5 sets of CEDST scores, and 14 questionnaires andinterviews. Analyses of the data indicate that the self-assessment results correlate very well with the test scores.
The majority of the students consider the individualized feedback accurate, appropriate and useful. It is therefore
concluded tentatively that the CEDST is quite effective and useful in detecting the strengths and weaknesses ofstudents’ speaking English ability. Limitations of the current study and suggestions for further study are presented
at the end of the paper.

20

The 2nd Room (Room A106) Part I
A comparison of four types of spelling tests among Japanese EFL learners: Focusing on sound-letter
correspondences
Sachiyo Takanami (Graduate School, University of Tsukuba)
stw324_423tknm@yahoo.co.jp
The purpose of this study is to compare and to determine the difficulties of four types of the written form spelling
tests among Japanese learners of English. Relationships between English spellings and sounds (i.e.,
pronunciations) are quite difficult for learners of English to acquire due to their complicated orthographic system.
Yet spelling and reading are important aspects of literacy that cannot be disregarded in language competence.
Nortyby (1936) compared five forms of spelling tests—namely, story form (i.e., fill in the appropriate words in agiven passage), timed dictation (i.e., write whole sentences in a limited time), list form (i.e., write the words aspronounced), multiple-choice (choose the correct spelling from among several choices), and oral form (spell theword orally)—for diagnostic purposes and investigated the variability of student performance and found that themultiple-choice form was the easiest while the timed dictation form was the most difficult. Focusing on thewritten form, Moore (1937) determined that the multiple-choice form was useful for measuring learners’ spelling
ability. Yet it remains unclear which type of spelling test is the most appropriate for EFL learners. Twenty words
were used in both Northby’s and Moore’s study (i.e., ninety, succeeded, similar, nickel, admission, carnival,
appearance, excitement, planned, finally, orchestra, occasionally, convenience, exhibition, schedule, arriving,
various, humor, Saturday, and island).
The current study compares the four types of written spelling tests (i.e., story form, timed dictation, list form, andmultiple-choice) among 80 mid-level high school Japanese learners of English. Participants completed a pre-testof approximately 90 regular words (i.e., with sound-letter correspondences; Mori, 2007) to determine theirknowledge of English spelling rules. Participants subsequently completed the four types of spelling tests(randomly administered); answers were carefully analyzed using for example Cook’s (1997) categorization for
L1 and L2 spelling errors (i.e., insertion, omission, substitution, transposition, grapheme substitution, and other) toclassify participants’ misspellings. The materials which were used in this study were based on the previous
findings (Northby, 1936; Moore, 1937). However, to determine the levels of the target words, Gakken’s corpus
data and JACET 8000’s vocabulary lists were considered as a standard. The spelling tests are effective tools to
examine learners’ level of proficiency in writing mechanics. Thus, the results of this research will be beneficial
and helpful for Japanese teachers of English to decide which form to use in his or her English lessons.

21

The 2nd Room (Room A106) Part II
Learner Adapted Testing: An Individualistic Approach to Language Assessment
Parviz Alavinia (Urmia University)
pevinia2006@yahoo.com
Half a century of stringent theorizing and practice in the testing enterprise, principally triggered by the publicationof Lado’s seminal book entitled Language Testing, has engendered a wide-ranging assortment of myriad
breakthroughs in various facets of language assessment. Though testing now enjoys an unprecedented glorious
status, there still seem to exist some notions which are in urgent need of reconsideration, among which a reference
can be made to the focal (yet partly neglected) role of individuals, particularly when it comes to considering their
desires and preferences, in bringing about more beneficial backwash effects. As most of the time it is the case thattest designers tend to be unaware of minute individual differences (IDs) both in terms of possessing varied
personality-induced inclinations and holding disparate views of the whole testing context, the results gained
through the administration of most current tests are thought to underrepresent the testees’ true abilities in the areas
under scrutiny. Thus, in an attempt to bring these learner IDs and preferences into the foreground of attention, theresearcher in the present study has opted for a more individualistic approach to assessment, materialized throughhis devised method and coined terminology LAT (Learner Adapted Testing). In effect, LAT differs from othersimilar assessment theories in that it seeks to set about a thoroughgoing reappraisal of the paramount role ofindividuals in the tortuous process of test development and implementation. To put it in a nutshell, the outstanding
results of the study with several groups of learners unanimously point toward the fruitfulness and success of theresearcher’s individualistic approach (LAT) vis-à-vis the commonplace practice of testing.

The 2nd Room (Room A106) Part III
The National English Ability Test of Korea
Oryang Kwon (Seoul National University)oryang@snu.ac.kr
The Korean Ministry of Education, Science and Technology (MEST) has been developing a new National
English Ability Test (NEAT) for past several years. The new test consists of three levels: Level 1 for college
students and adults, Level 2 for high-school graduates, and Level 3 for high school students. Level 1 is being
developed by a consortium composed of some major universities in Seoul and the Chamber of Commerce of
Korea; Levels 2 and 3 are being developed by a project team sponsored and supervised by the Korea Institute forCurriculum and Evaluation (KICE). The Level 2 and Level 3 tests are of great importance because they may
replace, from 2016, the College Scholastic Ability Test (CSAT), which is the current national college entrance
examination. As the new test is going to include all four skills of listening, speaking, reading and writing, its
washback effect on secondary and even elementary schools will be significant, and this will eventually improve
the overall English competence of Korean students. Recently, the KICE team had a public hearing on the test
framework, item writing, rating, and administration of the Level 2 and Level 3 tests. This paper will introduce
these two level tests at the current state of development.

22

The 2nd Room (Room A106) Part IV
Validation of the Listening Comprehension Component of the Centre Test in Japan: Listening in the realworld, in the Course of Study, and in the Centre Test
Kozo Yanagawa (Graduate School, University of Bedfordshire)kozo@msh.biglobe.ne.jp
Validation is an attempt to investigate whether or not or to what extent a test measures what it is meant to measure.
The validity of the Listening Comprehension Test of the Centre Test in Japan (henceforth J-NCT [L]), developed
and administered by the Daigaku Nyushi Centre (National Centre of University Entrance Examination), is still
open to question. This is a key issue: If the test is not measuring listening ability appropriately, then how could
we expect the listening ability of Japanese learners of EFL at high schools to be enhanced? The purpose of thisstudy is, therefore, to investigate the validity of the J-NCT [L], specifically in terms of two critical criteria forvalidations, that is contextual validity and cognitive (theory-based) validity (Weir 2005), and aims to highlight
aspects of the J-NCT [L] that might be reformed to enhance its validity and so lead to further improvements in thelistening ability of Japanese learners of EFL at high school.

The 2nd Room (Room A106) Part V
Effects of Note-taking Strategy Training in Listening Comprehension Tests
Junghyun Kim (Sookmyung Women’s University)
selinainse@gmail.com
This paper investigates how students’ test performance would be influenced after receiving the training ofnote-taking skills in TOEFL listening tests. Three different groups of 107 Korean university students werecompared: (1) a previously untrained group that was not allowed to take notes during the listening test, (2) apreviously untrained group that was allowed to take notes during the listening test, and (3) a previously trainedgroup (i.e., the experimental group) that was allowed to take notes during the listening test. The researcher had
given the experimental group two lectures on effective note-taking strategies of iBT TOEFL listening
comprehension tests employing the contents of the test preparation book by Lee (2006): the common expressions
used in lectures, and the specific note-taking strategies such as summarizing, note-taking process, symbols, andabbreviations. The research question of the study was, “How would performance be affected given note-taking
allowed conditions after the students’ taking lectures on effective note-taking strategies?” Two hypotheses weremade; (1) the learners will perform better when they are simply allowed to take notes given the typical academic
listening situation (Zara, 2005 ; the pilot study by the researcher in 2007), and (2) the learners will perform betteror feel comfortable when they are allowed to use both L1 and L2 (Barbier et al., 2006; Dunkel, 1988). To analyze
the main effects of the study, a one-way ANOVA was conducted. Surveys and interviews were also done to
examine the research question. The results of this study were somewhat mixed. First, as hypothesized, the groups
that were allowed to take notes performed better in listening comprehension tests than the group that was notallowed. Second, even after receiving note-taking training, students did not show significant improvement of thetest performance. Furthermore, some memories or habits of note-taking training impaired their test scores. Even
though many participants admitted the benefits of note-taking strategies, lack of practice time of note-taking skills
or insufficient time for taking notes while testing was the main cause for the absence of its positive effect. Because
the current iBT TOEFL allows note-taking throughout the entire test session, the note-taking skill which was notimportant in the past now assumes a greater importance. Future research will need to investigate the effects ofmore systemic training and more training time. The present research can contribute not only to test preparation
but also to actual academic environments for English learners, especially in EFL contexts.
23

The 3rd Room (Room A108) Part I
Comparison of Japanese and native English-speaking raters’ perspectives on oral English performance
Takanori Sato (Graduate School, Sophia University)
sa_taka_119@live.jp
In oral English performance assessment, it is not uncommon for native English-speaking (NS) and non-native
English-speaking teachers to act as raters. In this context, it is important and necessary to understand thedifference in their perspectives on learners’ oral performance so that test administrators can choose appropriate
raters according to the intended purposes.
The present study examined how Japanese and NS English teachers assess the overall effectiveness of Japanese
students’ oral English performance. In particular, the study aimed to explore what performance aspects eachgroup of raters mainly focuses on when judging learners’ overall performance. Four Japanese teachers and four
NS teachers were asked to rate three monologs of 30 undergraduate college students; then, their scores andwritten comments were analyzed and compared. First, the raters were asked to assign a single score (1 to 6) for
each monolog on the basis of their intuitive judgment of the performance (labeled as overall communicative
effectiveness). Following this, the teachers were provided with five analytic rating criteria—grammaticalaccuracy, fluency, vocabulary range, pronunciation, and content elaboration/development—and asked to assignscores for all the monologs. Finally, they were requested to write what aspects they focused on most whenassessing overall communicative effectiveness. The scores and comments elicited from the two groups of raters
were analyzed to identify the differences in scoring and to examine what aspects contribute to the overall
judgments.
The results showed that the Japanese raters assigned significantly higher scores for all the analytic criteria with the
exception of content elaboration/development, although their overall judgment of the monologs was almost thesame and not significantly different. The scores assigned by the Japanese raters showed that only fluency andcontent elaboration/development significantly predicted overall communicative effectiveness. The scoresassigned by the NS raters, on the other hand, revealed that all the five criteria significantly predicted the overall
score, although fluency and content elaboration/development were relatively strong predictors. The raters’ writtencomments also showed differences: the Japanese raters paid more attention to aspects that were not directly
assessed by the given criteria.
These results suggest that Japanese raters may prioritize fluency and content when assessing Japanese learners’
overall oral performance and that NS raters may be more sensitive to linguistic aspects (i.e., grammar,
pronunciation, and vocabulary) and take these constructs into consideration in their overall judgment. Thus, theresults of this study contribute to an understanding of the difference in the Japanese and NS English teachers’
perspectives on learners’ oral performance.

24

The 3rd Room (Room A108) Part II
Native and non-native raters’ judgment of English pronunciation at a placement test
Ji Eun Lee (University of Illinois at Urbana Champaign)lee296@illinois.edu
This study investigated EPT (ESL placement test) raters’ judgments of test takers’ pronunciation, and how theirvarious teaching experiences and different native language affected on their judgments. The EPT is administered
to international incoming students whose TOEFL score does not meet the campus or departmental cutoff scoresin the University of Illinois at Urbana Champaign. The oral section of EPT is designed to measure students’pronunciation, and employs the group discussion format. 8 raters with different teaching experience and native
language were interviewed, and their responses were analyzed qualitatively. Results showed that native speakerof English were more severe than non-native speakers of English even though teaching experience had more
effect on their different judgment of test takers’ pronunciation. Among raters, there is no agreed definition of
intelligibility, and factors the raters feel contribute most to intelligibility varied. Therefore, raters’ comments
manifested different constructs of underlying intelligibility and resulted in different rating scales. In addition, it
was shown that raters had different perspective of group discussion format, and how to use group discussion
format efficiently in order to measure students’ pronunciation and intelligibility was suggested.

25

The 3rd Room (Room A108) Part III
Oral Reading Fluency as an Assessment Instrument
Masanori Suzuki (Pearson Knowledge Technologies)
masanori.suzuki@pearson.com
Oral reading, or reading aloud, has a long history in teaching and learning in Japan as a classroom teachingpractice or self-learning method. A number of recent studies (e.g. brain activation studies) have suggested thatreading aloud stimulates parts of the brain important for language development and helps improve generalmemory functions. These recent experimental findings have help gain a wider acceptance of oral reading as ateaching or self-learning method. However, oral reading is rarely used as an assessment tool in Japan.
This presentation argues that oral reading can also be an effective assessment instrument. In the US, oral reading
fluency (ORF) is defined as “the ability to read a text quickly, accurately, and with proper expression” (National
Reading Panel, 2000). Reading passages out loud fluently with proper expression involves the reader’s ability torecognize whole words rapidly and effortlessly and to express the meaning of the passage using appropriate
pausing, intonation, and phrasing. Three measurable aspects of oral reading ability that contribute to ORF arereading rate (words correct per minute), reading accuracy (words correct/words attempted), and expressiveness(appropriate pausing, intonation, and phrasing). Together, these provide a barometer of a student’s literacy, andcan provide an early warning if the student begins to fall behind.
Traditional ORF assessment requires individualized testing using “pencil and paper” in which a teacher follows
along with a student who reads a passage. However, this is very time-consuming. A computerized oral reading
fluency testing system has been developed to automatically and efficiently evaluate the reader’s oral reading
performance. The automated scoring system uses speech processing technologies and can report on the threemeasurable aspects of the performance: reading rate, reading accuracy, and expressiveness and pausing.
The presentation describes a validation and usability study which involved the test development team together
with a US local school district and then demonstrates how the assessment tool saved teacher time, gave feedback
and encouragement to students, created a digital portfolio of students’ reading performances, and could potentiallyinvolve parents in the process of reading together with their children. It will also report on the relation between
ORF and language learning progress, in the context of an assessment taken annually by up to 4 million children in
the US alone.
The presentation concludes with suggestions for using such an automated ORF test in the context of Japan tocreate a standard for English education in Japan.

26

The 3rd Room (Room A108) Part IV
Do Test Practice and Keyword List Help Oral Summary Test Performance?
Hidetoshi Saito (Ibaraki University)
cldwtr@mx.ibaraki.ac.jp
The purpose of this cross-sectional study is to compare participants’ performance on an oral summary task (story
retelling based on reading a passage) and to test the viability of predictions derived from two hypotheses, the test
practice effect hypothesis―test practice improves test performance (e.g., Bygate, 2001)―and the robust testing
context hypothesis―test performance resists influences from manipulations of test task features (e.g. Iwashita etal., 2001).
Eighty-two university students registered in three EFL classes participated in the study. All participants took aspeaking proficiency test (TSST) approximately two weeks before oral summary test task. Participants practiced
the oral summary task in class, and it was part of their final tests. The participants were divided into two groups.
Each group was instructed to perform three summaries in a quiet room. All participants orally summarized one
passage given on the spot (the impromptu condition) and another passage that they practiced at home using aself-selected keyword list (the fully-assisted condition). Moreover, when summarizing an additional passage thatparticipants practiced at home, group 1 participants was not allowed to use the keyword list (the practiced
condition). Group 2 participants summarized a new passage using keywords created on the spot (thekeyword-assisted condition). Three passages used for summary were counterbalanced across candidates.
Three trained raters rated audio-recorded performances for fluency, accuracy, and content using a three-item
rating scale. The four-facet measures assessing participant’s ability, item difficulty, rater severity, and passage
difficulty were constructed using the Rasch analyses. Two mixed-design repeated ANOVAs (3 conditions x
proficiency levels) were run for testing differences between the performances of participants across the four
conditions.
The results indicate that although there were no differences in two contrasts of conditions (the keyword-assisted
vs. the impromptu and the practiced vs. the keyword-assisted), other four contrasts were statistically significantly
different. Proficiency was also statistically significant but no interaction was found. What this means is that thefindings support some predictions derived from each of the two hypotheses and that neither of the hypotheses issuperior in explaining all the results. One notable finding is that the use of keyword list positively influences
performance /only/ when accompanied by practice. In other words, without practice, keyword list does not seemto benefit the oral summary test performance; it merely seems to provide psychological security.

27

第 4 室（A110 教室） Part I
新旧 TOEIC®テストの比較検証--４テスト・セット 800 問の受験データ分析
法月 健（静岡産業大学）
norizuki@ssu.ac.jp
伊藤 彰浩（西南学院大学）
島谷 浩（熊本大学）
2006 年 5 月に Test of English for International Communication®（以下、TOEIC®）は新しい様式に改定された。法月・伊藤・島谷・木下（2009）における分析（以下、分析１）結果から、新旧 TOEIC®の難
易度には有意差があり、表面上は従来通りの問題形式の部分にも、項目やテキストの特性に顕著な差が存在する可能性が示唆された。しかしながら、分析１で使用された TOEIC®は新旧それぞれ１テス
トずつで、すべての被験者に対して、旧→新の同一順でテストが実施されたため、その結果は、1) 個
別 TOEIC®の素点レベルでの難易度差、2) 各 TOEIC®の実施順に影響された可能性は否定できない。
そこで、1)、2)の問題点を解消するために、本研究では、異なる新旧 TOEIC®を１テストずつ分析に加えた。分析１で使用された新 TOEIC®を係留テスト(anchor test)として実施し、分析１の旧テストを含めて新旧それぞれ２テスト・セット（計４テスト・セットの 800 問）を比較する。2010 年 3 月～6月にかけて、日本の３大学の学生に、旧 TOEIC® (OT)、新 TOEIC® (NT)、係留テスト(AT)の 3 種類のテストを、A グループについては、OT→NT→AT の順で、B グループには、NT→OT→AT の順で実施した。まず、OT と NT を共通に受験した 93 名のデータを基に各テストの難易度を比較し、次にOT、NT、AT を共通に受験した 83 名のデータと上記の分析１で使用した 136 名のデータを使って、４テスト・セットの TOEIC®の難易度を共通の尺度に位置付け、テスト項目やテキストの特性と対比しながら、TOEIC®が日本の大学生にどのように機能しているかについて、議論していく。

28

第 4 室（A110 教室） Part II
ＥＦＬライティング・ルーブリックの信頼性と妥当性の検証
大年 順子（岡山大学）
otoshi-j@cc.okayama-u.ac.jp
金志 佳代子（兵庫県立大学）
久留 友紀子（愛知医科大学）
正木 美知子（大阪国際大学）
山西 博之（関西外国語大学）
ライティングの評価時にルーブリックを使用することは、公平で信頼性のある評価のために欠くことはできないと考えられるが、実際のルーブリックの信頼性・妥当性の検証についての研究事例は少ない。本研究は、当研究グループが開発した分析的なライティング評価表であるルーブリック 2009 の
量的及び質的な検証報告である。関東地方の公立大学の英語専攻の学生 30 人のパラグラフライティ
ングを、まず日本人教員 5 名が、5 つの項目（dimensions：内容・展開、構成、文法、語彙、綴り・句読点）からなるルーブリック 2009 を用いて 4 段階で評価した。全体スコアと項目間の、項目− 全体相関分析の結果、｢綴り・句読点｣においては高い相関は見られなかったものの、他の 4 項目については高い相関が見られた。次に、上記と同一のパラグラフライティングに対してネイティブ教員が総合的に評価した結果と、ルーブリック 2009 を用いて日本人教員が分析的に評価した結果を用いて、両者の相関分析を行った。これはネイティブ教員の総合的な評価結果を用いた、ルーブリック 2009の併存的妥当性検証である。その結果、ネイティブ教員の総合的評価と｢内容・展開｣、｢文法｣、及び｢語彙｣は中程度の統計的に有意な相関が示されたものの、｢構成｣と｢綴り・句読点｣においては有意な相関は認められなかった。｢綴り・句読点｣の評価については、上述の項目− 全体相関分析の結果と併せ、見直しが求められる項目であると判断された。一方、｢構成｣に関する日本人教員の項目− 全体相関は高かったことから、ネイティブ教員と日本人教員では｢構成｣の認識に差異がある可能性が示唆された。なお、日本人教員、ネイティブ教員とも、どのように評価を行ったかに関する回顧的なコメン
トを記述しており、そのコメントを質的な分析方法である KJ 法を用いて分析することで、量的な分析結果と合わせた検証を行っていく。

29

第 4 室（A110 教室） Part III
英語エッセイの評価に求められる教師の特性とは―指導経験，環境，及び言語能力を背景に―
長橋 雅俊（筑波大学大学院生）
pika07300@yahoo.co.jp
新学習指導要領（文部科学省, 2009）によると，高校英語の主要科目は「コミュニケーション英語」と「英語表現」に移行し，前者では４領域（読解，聴解，作文，発話）の学習が統合され，後者から生徒の英語発信力の育成が重視される。今後，現場教師は語彙や文構造だけでなく内容や構成を助言
し，自己表現への発想と態度の育成が一層求められるだろう。先行研究（例．Weigle, 1998; Wolfe, 1997）
では，評価者の学習や指導経験から作文採点の信頼や一貫性への影響を指摘しているが，経験の質的な違いや，資質充分と思しき量的な指標は言及されていない。
本研究では現職教師による英語作文の評価信頼性を分析し，一貫性と配点の異なる原因を①指導経験
年数，②勤務校による指導の実態，③言語判断力の３点から調べた。対象は経験年数と勤務校の生徒
習熟度が異なる高校の教師 10 名と，教職未経験で英語専攻の大学生 14 名であり，前後半 25 編ずつの生徒作文を６段階の分析的評価法（内容展開，構成，正確さ，全体）で採点した。また２回の採点
セッションの間には誤文訂正，パラグラフ整合性などを問う訓練タスクを設け，評価者個々で言語判
断力を測定した。
訓練タスクの応答と採点セッションの配点状況からラッシュモデル分析を行った結果，現職教師は３名を除き言語判断力が平均以上で，その測定値が高い教師からは判断の揺らぎを検出しなかった（infitMS < 1.30）。一方，大学生の言語判断力は比較的低く，特に測定値の低い者から infit MS の危険値を観測した。また，前後半で一貫性に欠ける評価者（mis-fit raters）は７名から３名に減り，訓練効果はみられたが，配点のきびしさにも勤務校の生徒習熟度や評価者の言語判断力が作用している可能性がみられた。当発表においては，データ採取及び分析の過程，そして上記結果の詳細を報告する。

30

第 4 室（A110 教室） Part IV
英語 Can-Do 調査分析に基づく TUFS 言語フレームワーク構築の試み
長沼 君主（東京外国語大学）
n.naganuma@tufs.ac.jp
工藤 洋路（東京外国語大学）
吉冨 朝子（東京外国語大学大学院）
2008 年度に設立された東京外国語大学英語学習支援センター(ELC)では、全学学生の英語学習支援のため、「英語自律学習支援プログラム」の提供を行っている。プログラムでは e ラーニングによるイ
ンプット学習に加えて、スピーキングやライティングなどのアウトプット学習機会も設け、総合的な学習サポートを行っている。入学時、1 年次末、2 年次末には TOEIC-IP テストを実施し、とりわけ、英語を主専攻語または副専攻語として履修している学習者には、一定量の授業外学習課題をレベルに応じて課している。
昨年度末には、2008 年度入学者を対象として、2 年次末までの 2 年間の学習成果を組み込んだポート
フォリオ評価により、CEFR に準拠した「TUFS 言語パスポート」の発行を行った。パスポートには
TOEIC テストスコアを参照としたリスニングとリーディングの CEFR のレベルが記載された他、ライティング評価テストおよびスピーキング評価テストも、CEFR の基準に基づいて実施され、スピー
キング・セッションやライティング・セッションの記録とともに記載された。
本研究ではこのような背景のもと、CEFR に準拠しつつも本学独自の言語フレームワークを開発するための基礎資料として、TOEIC-IP テストとともに実施された英語アカデミック Can-Do 調査(長沼・宮嶋、2006)の結果を分析し、より大学英語教育の文脈に適したフレームワークの開発への示唆を得ることを目的とする。Can-Do 調査結果は IRT を用い困難度の分析を行い、TOEIC テストスコアやラ
イティング評価テストおよびスピーキング評価テスト結果とともに、CERF レベルとの関連が分析された。こうした情報をもとに今後「TUFS 言語フレームワーク」の開発を行い、評価テストや自律学
習支援プログラムの改善につなげていきたい。

31

第 5 室（A111 教室） Part I
インハウス CAT の設計手法－実施済みテストデータを利用したシミュレーション－
秋山 實（東北大学大学院教育情報学教育部／株式会社ｅラーニングサービス）
akiyama@ei.tohoku.ac.jp / akiyama@e-learning-service.co.jp
アダプティブテスト（Computerized Adaptive Test）は、全員が同じ問題セットを受験する従来のテス
トに比べ、半分以下の時間で同程度の誤差で能力を測定でき、受験者の能力に近い問題が多く出題されるという点で優れている。
しかし、大規模な利用例が多く、個々の教育機関が独自に運用する CAT は実現しにくいと考えられてきた。筆者は、項目応答理論に基づいた CAT を、そのモデル（1/2/3 パラメータロジスティックモデル）、初期条件、項目選択基準、能力推定法、終了条件、アイテムバンクの規模、受験者集団の能力分布など、CAT の動作に影響を与える条件をシミュレーションによって決定し、最適な条件で設計することができるシミュレーションツールを開発した。
既に実施したテストの問題と受験データを利用してシミュレーションを行うことにより、現有リソー
スを生かしてどの程度の能力測定が可能かを知ることができる。これにより、インハウスの CAT を
無理なく構築し、徐々にアイテムバンクを充実させ、CAT の能力を強化することが可能となる。
本シミュレーションツールは、教育機関に公開するので、CAT の設計およびテスト理論の理解に活
用できる。

32

第 5 室（A111 教室） Part II
日本人 EFL 学習者におけるワーキングメモリ容量と記憶表象との関係：動詞分類課題による検証
高木 修一 (筑波大学大学院生)
ka_to_n@yahoo.co.jp
読解プロセスにおいて、読み手はテキストに基づいた心的表象を構築することが求められる。心的表
象は、単なるテキストの表面的な記憶ではなく、読み手が自らの背景知識を活用したり推論を働かせたりすることによって作られる (Zwaan & Radvansky, 1998)。心的表象はワーキングメモリ (WM) 内で構築され、既存の表象に新しい情報が統合される。そのため、読み手の WM 容量は心的表象を構築するプロセスにおいて重要な役割を担うと考えられる。イベント索引化モデルによると、心的表象には登場人物、時間、空間、因果関係、目的関係に関する 5 つの側面が含まれる。L1 と L2 の心的表象を比較した Zwaan and Brown (1996) において、L1 の心的表象の方がより精緻であり、多くの側面が保持されていることがわかった。すると、同じ EFL 学習者においても WM 容量が大きい学習者の方が、より精緻な表象が構築されるのではないかと考えられる。
そこで、本研究ではイベント索引化モデルの枠組みで使用されている動詞分類課題を使用し、日本人
EFL 学習者のワーキングメモリと心的表象の関係性を検証した。動詞分類課題とは、テキストを読解させた後にテキスト中に登場した動詞の原形を分類させる課題である。動詞は記憶の手がかりとしての役割を果たし、同じカテゴリとして分類された動詞が示す情報は読み手の心的表象の中で強く結びついているとされる (井関・川﨑,2006)。
協力者は英語版のリーディングスパンテスト (RST) に取り組んだ後、テキストを読み動詞分類課
題に解答した。RST によって測定された WM により協力者を 2 群に分類した。先行研究に従って動詞分類課題の結果をコード化し、協力者群ごとに重回帰分析を行った (Zwaan & Brown, 1996)。その結果、読み手の WM が記憶表象に与える影響が明らかになった。

33

第 5 室（A111 教室） Part III
スピーチにおける自己評価の妥当性：質的観点から
深澤 真（茨城県立竹園高等学校）
VES05671@nifty.ne.jp
本研究の目的は，スピーチにおける日本人高校生の自己評価の妥当性について質的観点よりその特徴を検証することである。深澤（2010）では，Messick（1996）の妥当性の枠組みに基づき，スピーチにおける自己評価の妥当性の検証を行った。その結果，量的観点からは，教員評価や相互評価ほどではないものの，自己評価にもある程度の妥当性が認められた。本研究はその結果に基づき，１）自己評価においてラッシュ・モデルで予想されない評価をした生徒には共通の特徴は見られるか，２）自己評価の妥当性について生徒はどのような意識を持っているか，３）生徒は，自己評価にはどのような影響があると考えているか，の３点について検証を行う。
実験は日本人高校生 52 名を対象に実施され，スピーチを行った後に発表者は自己評価，その他の生徒は相互評価，教員は教員評価を行った。全てのスピーチと評価の終了後，アンケート調査も実施した。主な分析方法としてラッシュ・モデル分析とアンケート分析を行った。
その結果，１）について，モデルで予期されない評価をした生徒は５名おり，それらの生徒には，熟達度の高い生徒は自己評価を低くつけ，熟達度の低い生徒は自己評価を高くつける傾向が見られた。
またアンケートから，自己評価の客観性や成績に考慮されるかどうかなど，予期せぬ評価をした生徒
は何らかの点において自己評価の妥当性に不安を持っていたこともわかった。２）については，自己評価をどちらかというと妥当な評価と考えている生徒が多いものの，評価が主観的であることや緊張のため妥当な評価が難しいと考えている生徒像も明らかとなった。３）については，自己評価を自分のスピーチの分析や次回に向けての反省材料として活用したり，自己評価をとおして英語力の必要性を再確認する機会としたりする傾向が見られた。

34

Workshop Information（ワークショップの情報）
A207 教室（A 棟 2 階）
分散分析―3 元配置デザインを中心に
司 会
講 師

印南 洋 （豊橋技術科学大学）
平井 明代（筑波大学）
伊藤 尚子（筑波大学大学院生）

日時：2010 年 9 月 10 日（金）15:00－17:00
場所：豊橋技術科学大学 A207 教室（A 棟 2 階）
参加費：500 円
定員：30 名 (申し込み順)
参加条件：SPSS で 1 元配置分散分析を行ったことがある方が望ましい
形式：
コンピュータのない教室で、講師のコンピュータを映したスクリーンを見ながら手順を学びます。
SPSS を使った実習の時間もありますので、SPSS をお持ちの方はそのコンピュータをご持参ください。お持ちでない方は、コンピュータを持参された方の隣に座っていただくか、前のスクリーンを見て手順を見ていただくことになります。
概要：
１．3 元配置分散分析の種類
２．3 元配置の下位検定方法
３．分析実習（1）対応あり×対応あり×対応なしの分析（2）対応あり×対応なし×対応なしの分析（一部、シンタックスを利用する方法を紹介する）
４．3 元配置分散分析の報告の仕方と効果量
５．質疑応答申し込み：
参加希望の方は、以下の情報とともに、8 月 29 日 (日) までに小泉宛にご連絡ください。ただし、定員に達していない場合には当日参加も可能です。
早めにご質問をいただきますと、ワークショップの内容に反映しやすくなります。たくさんのお申し込みをお待ちしております。
■ワークショップの申し込み時にお教えください。
(1) 氏名とご所属(2) 当日 SPSS を持参されますか。
持参できる・できない(3) 以下の分析を使ったことはありますか。あるものを挙げてください。
(a) 1 元配置分散分析(b) 2 元配置分散分析(c) 3 元配置分散分析(d) その他（分散分析に関係することで、具体的に）(4) 何かご質問はありますか？
(5) ワークショップに対して何かご要望がありましたらお書きください。
申し込みの宛先 (可能な方は e-mail をお使いください)
小泉 利恵 e-mail: koizumir@tokiwa.ac.jp 〒310-8585 茨城県水戸市見和 1-430-1 常磐大学国際学部
35

Access to the Conference Venue (Toyohashi University of Technology [TUT])
To JR Toyohashi Station

JR Toyohashi
Station36

会場へのアクセス

From JR Toyohashi Station to the Conference Venue (Toyohashi University of Technology)(1) Get out of
Toyohashi Stationfrom the east exit.

(2) Go downstairs and takea bus from Gate 2. Allbuses leaving from Gate 2
are bound for the university.
Get off at “Gikadai-mae” oruniversity. The trip takes
around 35 minutes.

豊橋駅から会場の大学まで
豊橋駅東口を出て、階段を下ります。2 番の乗り場でバスに乗ります。2 番の乗り場からのバスならばすべて会場校を経由します。約 35 分で「技科大前」に着きます（430 円）。

37Bus Timetable（バス時刻表）

Toyohashi
Station

University

University⇒⇒

Toyohashi
Station

After the conference, the shuttle bus will take you free of charge to the Toyohashi Green Hotel, which is at a
two-minute walking distance from the east gate of JR (Japan Railway) Toyohashi Station.
大会終了後、懇親会場の豊橋グリーンホテル（JR 豊橋駅近く）まで、無料の貸し切りバスを用意いたしましたのでご利用下さい。

38

Direction from the Bus Stop to Building A（学会が開かれる A 棟まで）
Follow the black arrow in the map. The venue is mere 100 meters away from the bus stop.（100 メートルほど、矢印どおりにお進みください。)
JLTA
Conference:
Building A

Bus Stop39

Building A, 1st floor

Room 108
The 3rd Presentation Room

Room 110
The
4th
Presentation
Room
Room A111
The
5th
Presentation
Room

Room A106
The 2nd Presentation
Room
Vending
Machine(自動販
売機)

In front of A105
Registration, Commercial
Exhibits

Room A109
Committee Meetings,
Headquarter

Room A105
LoungeRoom A114
The 1st Presentation
Room
Room A101
Opening&
Closing
Ceremony, Keynote Speech,
Symposium
Entrance40

Building A, 2nd floor

階段

階段
A207 教室
9 月 10 日
ワークショップ会場

階段41

Commercial Exhibits（展示協賛企業）
株式会社 e ラーニングサービス
e Learning Service
http://www.e-learning-service.co.jp/index.html(Commercial exhibits can be found in front of Room A105. A105 教室前で展示があります)

日本言語テスト学会 (JLTA) 第 14 回（2010 年度）全国研究大会発表要綱
Handbook of the 14th Annual Conference of the Japan Language Testing Association
発行日：2010 年 9 月 1 日
発行：日本言語テスト学会 (JLTA)
会長：浪田 克之介 （北海道大学名誉教授）
事務局 〒389-0813 長野県千曲市若宮758
TEL: 026-275-1964 FAX: 026-275-1970
E-mail: youichi@avis.ne.jp
編集：全国研究大会実行委員会
印刷：（有）ユニオン・プレス〒380-0815 長野県長野市鶴賀田町 2152
TEL: 026-235-6236 FAX: 026-235-6258

42