# Presentation: 017598

## Automated Scoring of Essays:Evaluating Score Validity

- P. Adam Kelly
- Florida State University
- April 13, 2001

## Purpose

- To establish whether the computer is capable of *scoring* essays similar to how experts *rate* essays.

## Purpose

- To establish whether the computer is capable of *scoring* essays similar to how experts *rate* essays.
- While computer-generated essay scores correlate highly with human expert scores, no empirical linkage of electronic processing to cognitive processing currently exists.

## Purpose

- To establish whether the computer is capable of *scoring* essays similar to how experts *rate* essays.
- While computer-generated essay scores correlate highly with human expert scores, no empirical linkage of electronic processing to cognitive processing currently exists.
- The essence this proposed linkage: construct validation of computer-generated scores, following the Messick (1995) procedure.

## Essay Use on Large-Scale Tests:Large and Getting Larger

- {Figure 1}

## Why not even more growth?

_*ESSAYS ARE TOO EXPENSIVE!*_
*(Cost of hiring and training raters; *
*	**	**time involved in scoring thousands** **	**    of essays;*
*	**	**	**difficulty in finding qualified raters)*

## A Solution: Computerized Scoring of Essays

- Inexpensive
- Reliable
- Accessible

## Is |S1 - S2| > 1 ?

- Is |S1 - S2| > 1 ?

- essay

- human rater score
-                         (S1)

- human reader score
- (S2)

- NO

- YES

- expert human rater score (S3)

- Final Score
-  =
- mode, or mean
- of closest

- Final Score
- =
- mean

## Is |S1 - S2| > 1 ?

- Is |S1 - S2| > 1 ?

- essay

- human rater score
-                         (S1)

- Computer-generated score
- (S2)

- NO

- YES

- expert human rater score (S3)

- Final Score
- =
- mean

- Final Score
-  =
- mode, or mean
- of closest

## A Big Question

- Are the scores generated by computers _*valid*_*?*

## Review of Previous Work

- The Four Leaders in the Field (and their models):
- Burstein/ETS®  (*e-rater*TM, regression)
- Page/Tru-Judge, Inc.  (PEGTM, regression)
- Landauer/Colorado - KAT (IEATM, LSA)
- Elliott/Vantage Learning (IntelliMetricTM)

## Review of Previous Work(validity-related studies)

- Burstein/ETS®  (*e-rater*TM):
-  Computer-to-human rater correlation studies (3)
-  Computer-to-other criterion correlation studies (2)
-  Factor analysis
-  Tree-based regression

## Review of Previous Work(validity-related studies)

- Page/Tru-Judge, Inc.  (PEGTM):
-  Computer-to-human rater correlation studies (2)
-  Factor analysis / factor correlation (“trait-scoring”)      (2)

## Review of Previous Work(validity-related studies)

- Construct validation of essay scores: Messick’s (1987) 2-by-2 matrix, elaborated in Messick (1995) as a	six-point “road map”

- {Figure 2}

## Messick’s (1995)Six Aspects of Construct Validity

- Messick’s (1995)Six Aspects of Construct Validity

- _“__Intrinsic”_
- Content
- Structural
- Substantive

- _“__Extrinsic”_
-   Generalizability
-   External
-   Consequential

## The Sample

- ~1,800 GRE Writing Assessment essays:
- 	“Issue” prompt type: ~600 essays on 3 prompts
- 			model-building & cross-validation
- 	“Argument” prompt type: ~1,200 essays on 3 prompts
- 			model-building & cross-validation

## Phase I: Content(Relevance and Representativeness)

- Factor analyzed the 59 *e-rater* “features” from three linguistic classes: discourse (rhetorical), syntactic (structural), and topical (content)
- Rank-ordered the factors, or “underlying characteristics” of writing; discussed this list with essay test development experts

## Factor Analysis

- Principal Component/Principal Factor Method	(Muraki, Lee, & Kim, 2000)
- Two models: “Issue” and “Argument” generic
- _	__PCA_: “Issue”:  5 components, 25 features, PVE = 51%	    “Argument”:  5  comp. , 21 features, PVE = 48%
- 	   _FA_: “Issue”:  25 features fitted to 6 factors          “Argument”:  21 features fitted to 6 factors
-        		Rotated by Promax (oblique) method

- {Tables 1 - 4}

## Factor Analysis

- Fits of the six-factor models:
- _Exploratory_:
- “Issue” model:  PVE =  68% of 25 features
- “Argument” model:  PVE = 76% of 21 features
- _Confirmatory_:
- “Issue” model:  PVE =  38% of 25 features
- “Argument” model:  PVE = 34% of 21 features

- {Table 5}

## Factor Analysis

- Indication of a “dynamic” relationship between syntactic features and discourse features in an essay (Muraki, et al., 2000):
- Redundancy of feature loadings > .50 across factors,
- Loadings highest for discourse features on one factor, for syntactic features on the other, and
- The _same_ features and factors are involved in both models ...

## The “Characteristics of Writing”(in ranked order of explanatory importance)

- Issue-1:  4 discourse*
- Issue-2:  1 discourse
- Issue-3:  3 syntactic*
- Issue-4:  2 discourse
- Issue-5:  3 content
- 	(one excluded)

- Argument-1:  4 discourse*
- Argument-2:  2 discourse
- Argument-3:  1 discourse
- Argument-4:  3 content
- Argument-5:  3 syntactic*
- Argument-6:  3 discourse

- {Table 6}

## Phase II: Structural(Reflexivity of the Task and Domain Structure)

- Built:
- Detailed scoring criteria
- “Feature-based” (focused holistic) rubrics,   one per scoring criterion
- Had test development experts review these, looking for evidence of engaging thought processes and judgments, *not just rote counting tasks*

## e-rater & Rubric Criteria

***e-rater***** & Rubric Criteria**

_***e-rater***__** Criteria**_
-  Discourse structure
-  Types of sentence    structure
-  Content analysis

_**Rubric Criteria**_
-  “Organization of      ideas”
-  “Syntactic variety”
-  “Vocabulary usage”

## What the test development experts said: 

- Several of the characteristics (i.e., Issue-2 /Argument-3 and Issue-4/Argument-6) would not be seen as valid by expert raters.
- It would be unusual for expert raters to think of some of the characteristics in isolation from the rest of the essay.
*Both of these statements proved true ...*

## Phase III: Substantive(Theories and Process Models)

- Introduced expert raters to “feature-based” scoring rubrics
- Used “talk-aloud” protocols to prompt and audio recording to capture evidence that raters’ thought processes reflect the scoring rubrics and *e-rater* scoring processes

## Who the expert raters were:

- Four “scoring leaders,” recognized for their rating expertise and prior participation in studies
- Faculty in English programs at Delaware Valley-area colleges
- Two “issue” raters, and two “argument” raters  (as designated by the researcher in advance)

## What each expert rater did:

- At home, a week in advance:
- Rated a sample of 110 essays using the usual holistic scoring guide for the GRE W. A.
- At ETS:
- Participated in an initial group interview, an individual “talk-aloud” rating exercise, an interactive rating verbalization, and a debriefing.

## What the expert raters said:

- In the initial group interviews:
- “Issue” raters disliked Issue-2 and Issue-4
- “Argument” raters disliked Arg.-3 and Arg.-6
*Why? Characteristics not legitimate for rating*
- All four raters had difficulty parsing Issue-1/Arg.-1 and Issue-5/Arg.-4 from the rest of the essay
- 	   *Why? Odd to think of content as separate from**     the discourse approach used*

## What the expert raters said:

- In the “talk-alouds”:
-  More “off-task” than “on-task” utterances -- a rater   was prompted to “keep talking” when silent
-  Indicators of “slippage” back into holistic rating   paradigms:* “oh, well, I’ll just give this a four” *
- In the interactive rating verbalization:
-  Expressions of* *understanding, some exasperation:   *“Oh, so *_*that’s*_* how we’re supposed to rate that.”*

## What the expert raters said:

- In the debriefing:
- All four raters agreed with the hypothesis of a “dynamic interplay” between syntax and some forms of discourse -- the “right” use of syntactic variety can “make or break” certain discourse strategies.
*All four raters asked to go home!*

## Phase III: Substantive(continued)

- Analyzed and reported all proportions of agreement and pairwise correlations of human rater and *e-rater* scores:
- cross-validation GRE essay samples
- interrater subsamples
- adjudicated essay subsamples
- Assessed convergent/discriminant evidence

- {Table 7}

## e-rater vs. expert raters:

- _The Good_:
- Issue-1/Arg.-1, the most “important” of the characteristics, also shows the best agreement, both *e-rater*-to-expert and expert-to-expert.
- 4 of 5 “issue” characteristics and 4 of 6 “argument” characteristics show agreement near the holistic proportion, and the ones far off were viewed *a priori* as invalid characteristics.

## e-rater vs. expert raters:

- _The (sort of) Bad_:
- The *e-rater*-to-expert correlations are lower, across both models, for all characteristics than they are for the holistic scoring.
- _Bright spot_: The *e-rater*-to-expert correlation for Issue-1/Arg.-1, the most important characteristic, isn’t too far off the holistic ...

## e-rater vs. expert raters:

- _The (pretty) Ugly_:
- Cohen’s kappa, a “chance-corrected measure of agreement” between raters (Fleiss, 1981, p.217), indicates poor agreement beyond chance for all 11 characteristics (i.e, randomly assigned scores might  agree as well or better than these) ...
- And it isn’t the raters’ fault: their interrater agreement and correlations are high for both  models, including on previously adjudicated essays.

## Implications of the Findings

- _Content representativeness evidence_ of “intrinsic” construct validity of *e-*rater scores:
- The features that “drive” *e-rater *are identifiable and constant, group into factors (“characteristics”), forming reasonably interpretable, parsimonious factor models (although the model fits aren’t spectacular).

## Implications of the Findings(continued)

- _Structural evidence_:
- (Most of) the factors (“characteristics”) that emerge from the factor analysis resemble targeted writing qualities in the GRE W. A. Scoring Guides -- as ETS Technologies has claimed.

## Implications of the Findings(continued)

- _Substantive evidence_: the most compelling, both *for* and *against* the validity of *e-rater* scores:
- _All four_ expert raters agreed: the “syntactic” and “content” characteristics are relevant, identifiable, and reflective of what a rater should look for
- To a lesser extent, they agreed on _Issue-1/Arg.-1_.

## Implications of the Findings(continued)

- However, they also agreed strongly : _Issue-2/Arg.-3_ and _Issue-4/Arg.-6_ inappropriate under *any* circumstances
- _Conclusion_:
- Simpler scoring models that leave out certain features likely more acceptable to raters
- Explanatory ability of such models even less than that of current models

## Limitations of the Study

- Low factor model-to-data fits:
- isolating characteristics takes them out of context, produces a “reverse synergy” in interpretation
- Lack of rater familiarity with characteristic-specific rubrics, scoring tasks:
- but expertise, adaptability of raters probably helped mitigate the lack of training, practice

## Limitations of the Study(continued)

- Characteristic-specific rubrics not pre-tested:
- although ETS test development experts helped, still noticeable problems with interpreting the rubrics, likely leading to scoring inconsistencies

## Future Directions

- A comprehensive follow-up, addressing the limitations cited, with larger samples
- Extend the study to GMAT W. A., Test of Written English, NAEP W. A.