Scoring spoken
performance in large-scalelanguage testing programsin China
Jason Fan, Language Testing Research Centre,
University of Melbourne
Yan Jin, Shanghai Jiao Tong University
1Overview
Rating and rater effects in L2 speaking assessment
Assessing English speaking in China
Rater training and moderation in large-scale testing programs in China
Summary and discussion
Future directions
2

The College English Test – Spoken English Test (CET-SET)
Test purpose: The CET is a curriculum-based test, designed with an agenda of promoting the teaching andlearning of English in tertiary settings through the implementation of the teaching syllabus (e.g., Jin, 2014;Zheng & Cheng, 2008).

CET

CET written test
• CET Band 4 (CET4)• CET Band 6 (CET6)• 20 million test takers everyyear

CET spoken English
Test (CET-SET)• CET-SET Band 4 (CET-SET4)• CET-SET Band 6 (CET-SET6)• 1 million test takers everyyear
3

CET-SET: A brief history
4

ASSESSING INTERACTIONALCOMPETENCE ONLINE:THE CASE OF THE CET-SET

CET-SET face-to-face since 1999
CET-SET computer-based since 2013

Warm-up
Read-aloud
Q&A
PresentationPaired discussion

Get to knoweach other (selfintroduction)

Read-aloud atext of c. 120
words (P_45s)

Answertwo related
questions (R_40s)

Present on agiven topic for 1
minute (P_45s)

Collaborative
discussion for 3
minutes (P_60s)

CET-SET4:
Test design6Get to know eachother (selfintroduction)

Warm-up
Q&A
Presentation

Paired discussion

Present on a given
topic for 1.5
minutes (P-60s)

Answer one
question (R-30s)

Discussion on acontroversial topic for 3
minutes

Follow-up Question

Answer a followup question(R-45s)

CET-SET6:
Test design
7Computer-based CET-SET
Main features• Virtual examiner (video)• Paired format, involving human interaction(non-face-to-face)• Talking to peer, not interviewer• Using a variety of input modes: video, audio
and verbal +• Output includes both monologue anddialogue• A combination of human and machinescoring

Computer-mediated communication(CMC)8CET-SET rating criteria
Linguistic accuracy

Language performance

Analytic rating scale

Linguistic complexity

Contribution to the discourse

Discourse management
Discourse management

Flexibility

Pragmatic knowledgeAppropriacy
9

CET-SET rating scale
Analytic scoring of test takers’
performances on all tasks
except reading aloudCriteria:
linguistic

Criteria:
discourse

Criteria:
pragmatic

5

descriptors

descriptors

descriptors

4

descriptors

descriptors

descriptors

Score Accuracy and range

5

•

••

3

descriptors

descriptors

descriptors

2

descriptors

descriptors

descriptors

1

descriptors

descriptors

descriptors

The response
demonstrates fairly
accurate use ofgrammar andvocabulary.
It includes a wide
range of lexical
resource andgrammaticalstructure.
Pronunciation isgood; L1 accent has
minimal effect on
intelligibility.

Response length andcoherence• The response iscoherent and cansustain sufficienttime. It may include
minor lapses attimes in the processof organizing ideas
and selecting words
but not affectcommunication.

Flexibility andappropriateness
The test taker can• speak with ease on
a range of topics
within differentcontexts.
• engage in discussion
actively.
• generally adjust
what he/she says tocontext, function
and purpose.

CET-SET rating scale•

Score Descriptors
5• Speech shows good pacing, pronunciation and intonation.
• Read aloud smoothly. There are rare repetitions and self-corrections.
• Content is complete.
4• Speech shows some mistakes in pacing, pronunciation and intonation but
only occasionally causes problems for the listener.
Automated scoring of test
• Read aloud relatively smoothly. There are few repetitions and selftakers’ performances oncorrections.
• Content is basically complete.

reading aloud3• The rating scale is used forhuman scoring, which
produces scored samples
2
for machine learning•
•
•
••1

•
•

Speech shows many mistakes in pacing, pronunciation and intonation,and causes listener effort.
Read aloud not smoothly. There are some repetitions and selfcorrections.
Content is minimally complete.
Speech shows major mistakes in pacing, pronunciation and intonation,and causes considerable listener effort.
There are frequent staccatos, repetitions and self-corrections in reading
aloud.
Content is not complete at all.
No descriptor available

11

CET-SET scoring methods
CET-SET Band 4
Automatedscoring
5 points

Humanscoring
15 points

Total=20
reported in
grades

Accuracy

Contribution

Flexibility

Complexity

Discoursemanagement

Appropriacy

5x1.2

5x1.0

5x0.8

=6=5=4

CET-SET Band 6
Humanscoring
15 points
12

CET-SET6 score reporting
Grade description (can-do descriptors)

A+A
B+
B
C+C

Able to use
English for
an in-depth
discussion
on topicsof generalinterest.

D13

Quality control
Rater recruiting:• university English language teachers with experiences of teaching oral English are required, but this
could be difficult, especially when the scale of the test has expanded to over one million a year.

Rating training• Bench-mark scripts and samples are selected after each test for rater training

Human scoring: double-blind rating (+arbitration)
Machine scoring: large data sets of human scores for machine learning
Research on automated scoring system as a check rater

Study 1: effects of rating scale on constructrepresentation

Test-based
analytic scoring

Test
Non-interactive
Read-aloudQ&A

Task-type-based
analytic scoring
interactive

Task-based
holistic scoring
presentation

Discussion(Zhang, X. 2018; 2019)15

Study 1: findings
Test-based analytic

Task-based analytic

Task-based holistic• Most accurate• Most reliable (doublerating)• The inevitable halo effect• The widest range ofperformances• Raters least confident• Repeated penalty on core
criteria• Raters most confident• Over lenient• Least reliable(single rating)• Repeated penalty on corecriteria

16

Study 1: findings
The type of rating scale used in scoring oral performance affects raters’
rating focus, process, quality and confidence;
The test-based analytic scoring which is currently used by the CET-SET hassome advantages over other rating scales;
Need for rater training so as to minimize rater bias
• Reduce halo-effect (for test-based analytic scoring)• Avoid repeated penalty on core criteria (for task-based holistic scoring)17

Study 2: validating the CET-SET
Automated Scoring
Jin, Wang, Zhang & Zhao, 202018

Accuracy of automated speech recognition
For reading aloud: accuracy is above 98%
For other tasks (question and answer, individual presentation,pair discussion), the accuracy is over 95%
The accuracy of automated speech recognition has met therequirement of automated scoring.
19

Human-machine correlation

20

Human-machine grade consistency

21

Humanmachine
gradeconsistency22

A comparison of human-machine grades

23

Criterialfeatures of test
takers’performances
at each level(based onscores byautomatedscoring system)

24

124Seniorsecondary

568
9English major& graduate
Professional

ADVANCED

7College nonEnglish major

INTERMEDIA
TE

3JuniorsecondaryBASIC

Elementary

Study 3:
Linking
CET-SET4 to
China’sStandards ofEnglish
Language Ability(Jie, W. 2018)25

Study 3: findings
Experts were highly consistent during
standard-setting: Cronbach
Alpha=0.93
FACETS
InfitMnsq = 0.63-1.46
Separation =2.58
Reliability = 0.87
Cut-scores:
logistic regression + mid-point analysis
CSE4-5: 12.1-13.6
CSE5-6: 15.8-17.3
26

Study 3: findings
The validity of alignment can be enhanced by designing an appropriate standardsetting plan and implementing it carefully. Key considerations:• Inviting experts with teaching and research experience• Number of experts (10-12) for standard-setting• Providing comprehensive and timely feedback to experts during the training session, so as
to ensure the accuracy and consistency of their judgments.
• Need for explaining the descriptors, and where necessary, adapting the descriptors so as toimprove their relevance to students’ performances.

Different scoring methods will result in different cut-scores;
More evidence is needed to support the alignment between CSE and CET-SET.
27

Future research of the CET-SET
Future research may focus on
Construct definition(interactional competence)
Quality control of human scoring(parameters for evaluating the quality of scoring)
Scoring methods (human and machine collaboration)
Score interpretation and reporting (more accurate and informative grade
descriptions, and hopefully, individualized feedback)28

Thank youSubtitle

Identifier first line
Second line