# Presentation: 000998

## Further Evaluation of Automated Essay Score Validity

- P. Adam Kelly
- Houston VA Medical Center andBaylor College of Medicine
- http://people.bcm.tmc.edu/~pakelly
- Paper Repository: www.ncme.org

## It would appear that we have reached the limits of what is possible to achieve with computer technology.

*It would appear that we have reached the limits of what is possible to achieve with computer technology.*
- 	  – John von Neumann, computer scientist, 1949

## Research Questions

- How do automated essay scoring models behave when:
- The *level of specificity* of models (“generic” vs. “prompt-specific”) is varied;
- The essay *task type* (“discuss an issue” vs. “make an argument”) and *program type* (grad school admissions vs. grade school achievement) is varied; and
- The distributional assumptions of the independent and dependent variables are varied?
- What are the *consequences* of score  interpretations/uses, as stated by end users

## The six aspects of evidence in Messick’s (1995) unitary validity framework

- The six aspects of evidence in Messick’s (1995) unitary validity framework

**Notes:**
Representativeness Evidence
   Relevance of content the scoring model measures to the content domain for the trait “ graduate-level writing proficiency”

Structural Evidence
   Comparability of the scoring model’s network of constructs to the network of constructs in writing proficiency, as judged by experts

Substantive Evidence
Consistency of rater tasks/behaviors while scoring an essay, indicating rater engagement in cognitive processes germane to writing proficiency (“think-alouds,” interviews)

External Evidence
  Convergence of scores from presumed similar scoring models, and divergence of scores from presumed substantively different models  

Generalizability Evidence
  Consistency of scoring models across task and program types and through variations of model assumptions

Consequential Evidence
Documentation of effects on examinees, both perceived and real, of the interpretation and use of scores for stated purpose(s) 

## Essay Samples and Scoring Program

- ~1,800 GRE® Writing Assessment essays:
-  “Issue” task: ~600 essays on 3 prompts, scored by raters and by computer
-  “Argument” task: ~1,200 essays,     “ “               “ “            “ “           “ “
- ~ 900 National Assessment of Educational Progress (NAEP) writing assessment essays:
-       “Informative” task: ~450 essays, scored by raters and by computer
-          “Persuasive” task:  ~450 essays,        “ “           “ “           “ “
*e-rater**TM* (ETS Technologies, Inc.):
- _Linear regression model_: 59 variables, covering *content*, *rhetorical* structure, and *syntactic* structure “features” of essays:
- “Generic” models calibrated for multiple prompts, and “prompt-specific” models

## “In-Task” vs. “Out-of-Task” e-scores

- Using the GRE W. A. “Issue” generic model, generated “out-of-task” scores for ~900 “Argument” essays
- Using the GRE W. A. “Argument” generic model, generated “out-of-task” scores for ~400 “Issue” essays
- “Issue”: Proportions of agreement and correlations of “in-task” (correct) with “out-of-task” *e-*scores exceeded the statistics for “in-task” scores with rater scores (Kelly, 2001).
- “Argument”: Ditto.
_*Meaning*_*:* *Models may be somewhat invariant to task type*

## “In-Program” vs. “Out-of-Program” e-scores

- Using the GRE W. A. “Issue” generic model, generated “out-of-program” scores for ~450 NAEP “Informative” essays
- Using the GRE W. A. “Argument” generic model, generated “out-of-program” scores for ~450 NAEP “Persuasive” essays
- For both NAEP task types: Proportions of agreement and correlations of “in-program” (correct) with “out-of-program” *e-*scores fell well below the statistics for “in-program” *e*-scores with rater scores.
_*Meaning*_*: Strong evidence of discrimination between programs*

## Generic vs. Prompt-Specific e-scores

- 					   _Generic Scoring Model_:
- 					_“__Issue”_		_“Argument”_
- _Prompt-Specific Models_:
- Exact + adjacent agreement	   >.95		      >.90
- Correlation			   >.80		   .72 - .77
- These statistics are similar in magnitude to rater/*e-rater *agreement statistics presented in Kelly (2001).
_*Meaning*_*: Evidence supporting generalizability of e-scores from prompt-specific to generic models *

## “Modified Model” e-scores

*e-rater*’s linear regression module replaced with ordinal regression
- “Modified model” *e*-scores generated for GRE essays
- Both task types: Proportions of agreement remained roughly constant, but correlations increased noticeably
_*Meaning*_*: An ordinal regression model may improve the accuracy of e-scores, especially in the extremes of the score distribution (e.g., 5s and 6s)*

## Consequences of e-score interpretation/use

- How are the scores interpreted? Used? By whom?What are the implications of this?
- Interviewed graduate program admissions decision-makers: open-ended questions, by phone, recorded on audio tape
- The sample: 12 humanities, 18 social sciences, 28 business graduate faculty

## Examples of Responses ...

Humanities:
Not familiar with GRE W. A. or e-rater
Wouldn’t be inclined to use an essay test for admissions
Concerned that computer scoring could undervalue creativity and ethnically diverse writing styles/formats

Social Sciences:
Not familiar with GRE W. A. or e-rater
Essay test likely only used to assess English language proficiency
Less concerned about potential threat to creativity; some disciplines have rigid writing styles anyway

- _Humanities_:
- Not familiar with GRE W. A. or *e-rater*
- Wouldn’t be inclined to use an essay test for admissions
- Concerned that computer scoring could undervalue creativity and ethnically diverse writing styles/formats
- _Social Sciences_:
- Not familiar with GRE W. A. or *e-rater*
- Essay test likely only used to assess English language proficiency
- Less concerned about potential threat to creativity; some disciplines have rigid writing styles anyway

## Examples of Responses ...

Business:
Didn’t realize that a computer currently helps score GMAT W. A., or knew it but wasn’t affected by it

Rarely use GMAT W. A. scores, then only to assess English language proficiency

Concerned that computer scoring could marginalizeW. A., but (may) feel it is already useless

Meaning: Since scores largely discounted by users, the consequences of interpretation/use are nonexistent (at present, at least).

- _Business_:
- Didn’t realize that a computer currently helps score GMAT W. A., or knew it but wasn’t affected by it
- Rarely use GMAT W. A. scores, then only to assess English language proficiency
- Concerned that computer scoring could marginalizeW. A., but (may) feel it is already useless
_*Meaning*_*: Since scores largely discounted by users, the consequences of interpretation/use are nonexistent **(at present, at least).*

## Conclusions ... (this year and last)

- _Content representativeness evidence_: Variables that “drive” *e-rater *are identifiable and constant, group into factors forming reasonably interpretable, parsimonious factor models
- _Structural evidence_: (Most of) the factors resemble writing qualities listed in the GRE W. A. Scoring Guides – just as ETS Technologies has claimed
- _Substantive evidence_: Raters agreed that the “syntactic” and “content” factors are relevant, identifiable, and reflective of what a rater should look for, but were *highly* skeptical of others

## Conclusions ... (this year and last)

- _Correlational evidence_: Apparent strong discrimination of “in-program” from “out-of-program” essays; important for commercial applications across academic/professional fields
- _Generalizability evidence_: the use of less expensive “generic” models, trained only to the task type, not the prompt, appears to be supported
- _Consequential evidence_: Many graduate program admissions decision-makers do not use the GRE W.A. or GMAT W. A.; those that *do *use it mostly for diagnostic/remedial purposes (so the scores *matter*, but not for the reasons thought ...)

## “D**n this computer, I think that I shall sell it. It never does what I want it to do, only what I tell it!”