Koizumi Kaken 1st Research Meeting
Jointly organized by Japan Language
Testing Association (JLTA): 51st JLTA
Research Meeting
November 22 (Sunday)
2020 (10:00-17:00, Japan Time)
Theme: Improving scoring methods in English speaking
assessment (Part 1)
Supported by the Japan Society for the Promotion of Science (JSPS) KAKENHI,
Grant-in-Aid for Scientific Research (C), Grant Number 20K00894
1

Administrative information• During the lectures– Please mute sound and turn off video camera.
• Recording and slides– All of the sessions will be recorded. We will makethem accessible on YouTube (unlisted) to those whoregistered in advance. They will be available for onemonth. If you prefer not to appear in the video,
turn your video function off.
– Some slides will be available before or after the
meeting on the JLTA website:• http://jlta2016.sakura.ne.jp/?page_id=212

Administrative information• Break– Feel free to take a break from time to time yourself.
• Questions and comments– Please post your questions and comments below:• https://forms.gle/iaaH3T6GTTf1evhZ7– You can post them in the chat box on Zoom, but
priority is given to the website questions andcomments.
• Questionnaire– Please post your comments below:• https://forms.gle/8EtNzdtDoXdu6toy93

Kaken Project
Supported by the Japan Society for the Promotion ofScience (JSPS) KAKENHI, Grant-in-Aid for Scientific
Research (C), Grant Number 20K00894
Project: Development of scoring guidelines for
ensuring rater reliability in second language (L2) English
classroom speaking assessment (SA) at senior highschools in JapanMembers
Rie Koizumi (Chair: Juntendo University), Makoto Fukazawa(University of the Ryukyus), Yuichiro Yokouchi (HirosakiUniversity), Chihiro Inoue (Centre for Research in English
Language Learning and Assessment [CRELLA], University ofBedfordshire, U.K.)

4

Assumptions
・Kaken project focuses on classroom SA involving
teacher-raters.
・SA in classroom is an essential part ofeducation (e.g., Poehner & Inbar-Lourie, 2020).
・Formative functions of L2 classroom assessment-- Assessment for learning (AfL; Black & Wiliam, 2009),
learning-oriented assessment (Turner & Purpura,2016), and dynamic assessment (Leung, 2007)
・Summative use of L2 classroom assessment
・Classroom SA is not high-stakes.
・Typically used for formative and summative purposes,with more focus on the summative purpose (Bacquet, 2020)5.

Kaken Project• 1. Learn assessment theories and practices in theworld regarding the improvement of scoring methods.
– Current research meeting– Another meeting scheduled for February 14 (in Japanese)• 2. Investigate the current practices in Japan– SA is not conducted frequently in classroom in Japanese
secondary school classrooms.
– Tasks and rubrics are made at each school.
– Rater training sessions are not typically conducted. Rater
reliability is often not checked.

• 3. Create scoring guidelines and a website for ensuringrater reliability in senior high schools in Japan• 4. Examine the effectiveness of the guidelines and the6website

Assessment theories and practices
in Japan and the world• Aspects to consider when assessing L2 speaking

7

Task:
Presentation & Interview

8Pair talk9Group talk (e.g., by 4 students)

10

Rubric:
Analytic rubric (Koizumi & Yano, 2019)

Tasks and rubrics (Muñoz & Álvarez, 2010)• 1. Match test tasks with teaching objectives and tasks.
Create authentic tasks that cover a wide range ofcommunicative abilities.
• 2. Specify instructions and procedures to elicit
targeted abilities.
• 3. Create rubrics (evaluation criteria) that clearlyshow the tested ability and are easy for students andteachers to understand.
• They should have positive effects on learning and teaching.
• Related to validity– To what degree is test interpretation and use appropriate?
12Effects of raters• Rater reliability• Includes consistency across raters & within raters• Assumed, and not examined much in practices• Teacher-raters typically do not have rater training.
• They score alone.
• Inconsistency among teachers may lead to a lack ofscore comparability.
• Classroom SA does not need to have a high reliability,
but it should have a moderate reliability.
13

Typical rater training procedures• Prior discussion– Check the rubric and performance examples. Usevideos or recordings to mark separately. Discussratings to reduce discrepancies.
• During-test practice– Double rating (fully or partially).
• Post-test discussion– Discuss the discrepancy and identify reasons.
Modify the rubric. Decide the final ratings.

14

Rater training procedures and practicalconstraintsinJapan• Prior discussion– Check the rubric and performance examples. Use videos orrecordings to mark separately. Discuss ratings to reduce
discrepancies.
– In Japan, teachers cannot usually hold a long training
meeting.
• During-test practice– Double rating (fully or partially).
– In Japan, a double rating may not be possible.
• Post-test discussion– Discuss the discrepancy and identify reasons. Modify therubric. Decide the final ratings.
– In Japan, this may not be possible.
15

Koizumi & Watanabe (2020)• Public high school SA: 116 students, 2 to 9 raters• To what extent can rater reliability be maintained using asimple rubric without detailed rater training?
• Tasks: Individual presentation, paired role play, and twogroup discussions• Rubric: 3 criteria with 3 levels No intensive rater training• Data analyzed using many-facet Rasch measurement andgeneralizability theory• In general, raters scored similarly and consistently.
– Agreement: 49.5% to 80.7%• The number of raters required to maintain sufficientreliability (Φ = .70): one to four raters• Group discussion tasks required more raters.

16

Koizumi, Hatsuzawa, Isobe, &Matsuoka (2020)• Public high school SA: 232 students, 3 raters• To what extent can rater reliability be maintained using asimple rubric without detailed rater training?
• Tasks: Group discussion and debate• Rubric: 3 criteria with 3 levels• No intensive rater training• Data analyzed using many-facet Rasch measurement andgeneralizability theory• In general, raters scored similarly and consistently.
– Agreement: 72.9% to 81.6%• The number of raters required to maintain sufficientreliability (Φ = .70): one rater

17

Rater training at Japanese schools• With clear and simple rubrics, intensive training may
not be necessary in some contexts.
• Still, during-test practice may be necessary.
– Double rating (fully or partially).
– Discuss the ratings for the first few performances
and adjust the criteria during the discussion.
• More research is needed to identify contexts that
require simple or intensive training.
• Effective practices outside Japan can provide furtherclues.
18This research meeting• 10:45-12:00 Scoring spoken performance in largescale language testing programs in China– Lecturers: Jason Fan (University of Melbourne, Australia), Jin
Yan (Shanghai Jiao Tong University, China)• 13:00-14:15 Implementing and rating a new peer-topeer assessment of speaking skills in New Zealand– Lecturer: Martin East (University of Auckland, New Zealand)• 14:30-15:45 Teacher ‘rater’ training in Hong Kongand Australia: Different contexts, same problems?
– Lecturer: Chris Davison (University of New South Wales,Australia)• 16:00-17:00 Discussion (breakout session + Q&A)
19

References• Bacquet, J. N. (2020), Implications of summative and formative
assessment in Japan―A review of the current literature. International
Journal of Education & Literacy Studies, 8(2), 28–35.
http://dx.doi.org/10.7575/aiac.ijels.v.8n.2p.28• Black, P., & Wiliam, D. (1998). Assessment and classroom learning.
Assessment in Education: Principles, Policy & Practice, 5(1), 7–74.
https://doi.org/10.1080/0969595980050102• Eckes, T. (2019). Many-facet Rasch measurement: Implications forrater-mediated language assessment. In V. Aryadoust & M. Raquel (Eds.),
Quantitative data analysis for language assessment (Vol. I: Fundamentaltechniques; pp. 153–175). New York, NY: Routledge.
• Koizumi, R., Hatsuzawa, S., Isobe, R., & Matsuoka, K. (2020). Rater

reliability in classroom group discussion and debate in a Japanese seniorhigh school. Unpublished manuscript.

20

References• Koizumi, R., & Watanabe, A. (2020). Rater reliability in classroom
speaking assessment in a Japanese senior high school. Unpublishedmanuscript.
• Koizumi, R., & Yano, Y. (2019). Assessing students’ English presentation
skills using a textbook-based task and rubric at a Japanese senior highschool. SHIKEN, 23(1), 1–33. http://teval.jalt.org/node/87• Leung, C. (2007). Dynamic assessment: Assessment for and asteaching? Language Assessment Quarterly, 4(3), 257–278.
https://doi.org/10.1080/15434300701481127• Munóz, A. P., & Álvarez, M. E. (2010). Washback of an oral assessment
system in the EFL classroom. Language Testing, 27, 33–49. doi:
10.1177/0265532209347148• Turner, C. E., & Purpura, J. E. (2016). Learning-oriented assessment insecond and foreign language classrooms. In D. Tsagari & J. Banerjee(Eds.), Handbook of second language assessment (pp. 255–273). De
Gruyter.
21