Scoring spoken
performance in large-scalelanguage testing programsin China
Jason Fan, Language Testing Research Centre,
University of Melbourne
Yan Jin, Shanghai Jiao Tong University
1Overview
Rating and rater effects in L2 speaking assessment
Assessing English speaking in China
Rater training and moderation in large-scale testing programs in China
Summary and discussion
Future directions
2

Rating and rater effects in L2 speaking assessment• Raterqualifications• Rater training• Rating session setup• Rater
performancemoderationScores(McNamara, Knoch, & Fan, 2019, p. 93, adapted from McNamara, 1996, p. 86)

Decisions andconsequences3Knoch & Chapelle (2018): Rating processes within an argumentbased validation framework

Bridges that represent inferences linking components in performance assessment (Kane,Crooks, & Cohen, 1999, p. 9, cited in Chapelle, Enright, & Jamieson, 2008, p. 10)

4

Evaluation inference: Observations are evaluated using procedures that provide observed scores withintended characteristics
Warrant

Assumptions

Sources for backing

5Warrant

Assumptions

Sources for backing

Raters rate reliably attask level.
••••

Rater training
Rating scale
Rater qualifications
Design of rating
sessions

6

Generalisation inference: Observed scores are estimates of expected scores over the relevant parallelversions of tasks and test forms and across raters.
Warrant

Assumptions

Sources for backing• Design of rating session• Moderating rater
performance7

Questions about rating and raters
What tasks are used toassess speaking
proficiency?

What rating scales areused?

How are the raters
recruited?

What qualifications arerequired of raters?

What rater training isprovided?

How is a rating session
designed?

How is rater
performancemoderated?

Is there any researchthat has been conductedof rating quality?
8Assessing English speaking in China••••

Speaking test design and delivery
Direct or semi-direct method (live orcomputer-based)?
Task: implications for constructrepresentation, design of rating sessions,test washback, test validity and fairness…
Rating scales: analytic, holistic?
Rating practices: rater qualifications,rater training, rater moderation, etc.

9Speaking test design in four large-scale testing programs
The College English Test – Spoken English Test (CETSET4 and CET-SET6) → Prof Jin Yan
The speaking test of the Test for English Majors (TEM4and TEM8)The National English Matriculation Test (NMET –Shanghai Version)
The speaking subtest of the English Test forInternational Communication (ETIC)
10The speaking test of the Test for English Majors (TEM)• Target test takers: English major
undergraduates in universities in Chinamainland• Test population: 20,000 for TEM4 Oral and
10,000 for TEM8 Oral every year.
• Purpose: a curriculum-based test; examine
whether students meet the required levels ofEnglish language abilities as specified in theNational College English Teaching Syllabus forEnglish Majors (Jin & Fan, 2011)

TEM4

Compulsory for
English major
students

Foundational
stage

TEM4 Oral
TEM8

End of the 2nd year

AdvancedStageEnd of the 4th year

TEM8 Oral
11

TEM4 Oral tasks

Retelling: Retelling astory (300 words,R_3mins)

Presentation:
Talking on a topic
related to the story(P_3mins, R_3mins)

TEM8 Oral tasks

Interpreting: English to Chinese (150 words outof a speech with 300 words; 2-3 mins)

Interpreting: Chinese to English (200 charactersout of a speech with 400 characters; 2-3 mins)
Role-play:
Interacting between
two test takers(P_3mins; R_4mins)

Presentation: Presenting on a given topic(P_4mins; R_3mins)

12

Task II: Talking on a given topic

Task III: Role-playing

Describe a teacher of yours whom you findunusual.

Many high school graduates in China are goingoverseas for their college education. A friend ofyours is graduating this years and would like to ask
for your advice on whether it is a good idea for ahigh school graduate to go abroad to study.
Student A: You think this friend should go by allmeans, and you should try to convince yourpartner. Remember you should start theconversation.

13Rating scale for TEM4 Oral
Descriptors at four levels: distinction, good, pass, fail• Retelling: Can coherently tell the story• Presentation: Can speak fluently on the given topic with aclear presentation of the viewpoints; no unnecessary pauses

Task-specificholistic scale• Role-play: Can engage in communication based on the contextand role
• Pronunciation: accurate and clear pronunciation with natural
intonation• Grammar and vocabulary: accurate grammar with very few
mistakes; a wide range of lexical resources

Raters award five scoreson each of the fiveaspects in the scale
Test-based
analytic scale

14

The speaking test of the National Matriculation English Test(NMET) – Shanghai Version
Read aloud sentences (P_30s,R_15s)
Speaking

Read aloud a passage (P_60s,R_30s)
Ask questions about twosituations
Story telling based on given
pictures (P_60s, R_60s)

Listening andSpeaking

Make quick responses tosentences
Respond to questions after
hearing a passage (P_30s;R_30s; P_60s; R_60s)

The NMET assess students’ ability to• read sentences and passages with the correctintonations and tones based on the phenetic
knowledge and reading aloud skills learned• make enquiries in order to obtain theinformation needed based on the linguistic
notions and functions learned• describe, explain or comment on a person or
an incident.
• understand daily conversations and respondaccordingly• answer questions based on a listening
discourse and express personal views, feelings
or make comments
15

16

17

18

Rating criteria: task-specific holistic scale
Storytelling:
Read aloud (sentence):
Fluency, pronunciation, intonation, stress, pause

Task fulfilment, coherence, content relevance tothe pictures, grammar, vocabulary, pronunciation

Read aloud (passage):

Respond to questions:

Fluency, pronunciation, intonation, stress, pause

Task fulfilment, content relevance, language
expression

Ask questions:
Appropriate to the context, sentence structure

Answer questions:Task fulfilment, content relevance, language
expression
19Machine
ratingHumanrating• Human rating + automatedscoring• Automated scoring based onmachine learning• An average between human andautomated scoring results isadopted

20

The speaking subtest of the English Test for International
Communication (ETIC)• Developed by China Language Assessment (CLA), theEnglish Test for International Communication (ETIC)
assesses one’s ability to perform English language tasks
in international workplace contexts (Wang & Luo,2019).
• Since it made its official debut in 2016, it has attracted
over 90,000 test takers.
• ETIC is a certification system which acts as a benchmark
for employers to select employees who have a global
vision, master foreign languages kills, understandinternational rules, and excel at cross-culturalcommunication.
21Translation suiteMain suite• Basic• Intermediate• Advanced• Superior• Written translation• Consecutive
interpretation• Simultaneousinterpretation

22

ETIC speaking test (intermediate)
Task 3
Task 2
Task 1

Respond toquestions basedon listening
input (R_15s*5)

Respond toquestions basedon graph input(P_90s, R_15s*8)

Give an oralsummary of areading passage(P_90s, R_60s)

Task 4Present on a
topic in business(P_90s, R_60s)

23

ETIC speaking sample tasks (intermediate)

24

25

• Six levels ranging from A to E
• Test-based analytic rating scale with twodimensions: Content and language
performance• Content: relevance , task fulfilment andcoherence• Language performance: fluency, vocabulary,grammar, appropriacy

26

27

Rater training and moderation: ETIC
Rater recruitment and qualifications• Raters are recruited through the test provider’s professional networks in English teaching
and assessment in China• University English teachers with demonstratable experience in language assessment

Rater training• All raters need to be trained before they commence rating• A typical training session lasts for a half day28

Rater training: ETIC

Raters get familiar
with the tasks andthe rating criteriaGroup leaders train
raters in smallergroupsRaters go through 23 pre-selectedbenchmark samples
at each score level

Raters start practicerating and getfeedback

Raters start rating in
the online rating
systemGroup leaders
monitor rater
performance andprovide feedback

Raters discuss thebenchmark samples

29

Monitoring rater performance: ETIC
Every performance is double rated.
The system automatically identifies the discrepancies that exceed a pre-set limit.
The group leader makes the final judgement.
The online rating system facilitates the continuous monitoring of irregularities.

Rater effects and rating quality have been the focus of investigation for research
graduates and faculty staff.
30

Over to Prof Jin Yan to talk about the CET-SET…

31

Summary and discussion(McNamara, Knoch, & Fan, 2019, p. 93, adapted from McNamara, 1996, p. 86)

Test reliability, validity and fairness

Scores32

Speaking test design• Test developers’ commendable efforts to include speaking in language assessment → practicalconstraints → stakeholders’ perceptions of 1) the inadequate speaking ability of Chinese students; 2)
speaking ability has been underrepresented in English tests (Fan, 2018)• Computer-based semi-direct format → logistic challenges in delivering the speaking assessments to ahigh volume of test takers → facilitating the rating process which is typically implemented using an online
rating system

Validity andauthenticity ofspeaking tasks

Representation of
the speaking
constructs
33

Speaking test design• Innovative task designs to assess test takers’ communicative language ability → CET and TEM (computermediated communication) → task authenticity → To what extent can the discourses elicited by such tasks
and the processes of engaging with such communication simulate real-life language use in the TLU
domains?

34

Speaking test design• Curriculum-based speaking tests→ therelationship between a speaking test,
curriculum and real-life language use

TLU domain

Curriculum• Understanding the TLU domain (e.g.,
the ETIC speaking tasks –communicative ability in theinternational workplace contexts)

Test

35

Rating scale
Rating scale → the de facto construct (Knoch, 2011)• How is the rating scale constructed?
• To what extent are the speaking constructs represented in the rating scale?
• Should an analytic or holistic rating scale be used?
• How does a rating scale function in the rating process?
• Can raters use a rating scale reliably?
•What feedback can a rating scale generate to teachers and learners?

36

Rater training and moderation

Test
familiarisation

Orientation
to thestandard

Guidedratingpractice

Ratingpractice

Rater
moderation

See also Standards for Educational and Psychological Testing(AERA, APA, & NCME, 2014)37

Rater training and moderation
Warrant

Assumptions

Sources for backing• Rater training• Raters’ use of the ratingscale• Rater qualifications• Rating sessions• Rater characteristics

Evaluation inference

Evaluation inference

Raters rate reliably attask level.

38

Rater training and moderation• Rater consistency• The number of raters• Construct-irrelevantvariance• Resolving discrepancies

Assumptions

Sources for backing

Generalisation inference

Generalisation inference

Warrant39

Rater training and moderation• Using automated scoring + human scoring (e.g., CET-SET; NMET-SH)• Constrained task designs → ‘the dog wagging the tail’ vs ‘the tail wagging the dog’
• The scoring algorithm → opening ‘the black box’ to stakeholders → the potential
washback effects on teaching and learning practices

40

Future directions
Speaking test design: strike the opportune balance between authenticity, constructrepresentation, and practicality

Speaking constructs: exploring the constructs that are meaningful to and representativeof the TLU domain → domain analysis (e.g., Chapelle, Jamieson, & Enright, 2008)

Rating scale: analytic OR holistic → scale development based on solid research →washback effects on teaching and learning

41

Rating practices: use the argument-based approach (e.g., Knoch & Chapelle, 2018) andbest practice models in performance assessment (e.g., AERA, APA, & NCME, 2014) toevaluate and continuously improve rater training and moderation practice → reliability,validity and fairness
Score reporting and feedback: provide richer and diagnostic feedback on students’performance (Koizumi, et al., 2020) → the teaching and learning of speaking

Align with the international practice → set up programs supporting independent
research → feeding research findings into the continuous improvement of testing
practices42

43

Thank you
Jason Fan: jinsong.fan@unimelb.edu.au
Yan Jin: yjin@sjtu.edu.cn