**Automated Scoring of Essays:\
Evaluating Score Validity**

P. Adam Kelly

Houston VA Medical Center and\
Baylor College of Medicine\*

March 1, 2002

Dissemination of this research is supported by the Department of
Veterans Affairs Health Services Research and Development Service.

\* The author completed this research while a doctoral student at
Florida State University.

**Acknowledgments**

This research is supported by the Graduate Record Examination Board and
its Research Committee. The author would like to thank the following
people: Tom Rochon, Executive Director of the GRE Program, and the other
members of the GRE Board, for their generous support of this study; Dr.
Don Powers, Educational Testing Service, for playing a key role in
obtaining the support of the GRE Board, providing a review of the first
draft, and serving as mentor throughout the project; Dr. Jill Burstein,
ETS Technologies, Inc., for providing the\
*e-rater* data necessary to carry out this research and reviewing the
first draft; and Mary Fowles, ETS, for recruiting the expert raters,
scheduling the face-to-face components of the project, editing and
re-editing the study materials, and reviewing the first draft.

Also, thanks to Cyndi Welsh for helpful comments in reviewing the study
materials, Cynthia McAllister for designing the scoring sheets and
managing correspondence, Slava Andreyev and Chi Lu for programming and
running the many *e-rater* models, Irv Katz for observing and providing
advice on the "talk-aloud" procedures, Ann Willard for helpful comments
on the first draft, and Profs. Donna Tobin, Maria Cimadevilla, Marian
Kerr, and Joyce Munro, for patiently and professionally carrying out the
rating tasks, at times rigorous, that were requested of them.

Thanks also to Dr. Albert Oosterhof, Florida State University, for
providing helpful advice and moral support as Major Professor, and to
Dr. Akihito Kamata, Dr. Richard Tate, and Dr. Myke Gluck, all of Florida
State University, for their help as members of the Dissertation
Committee.

**Abstract**

The purpose of this research is to establish, within the constraints of
the methods presented, whether the computer is capable of assigning a
score to an essay in much the same way that a rater does. That is, this
investigation attempts to elucidate what is actually going on within the
computer and within the mind of the rater, and describe to what degree
these processes parallel. The essence of how this parallelism is to be
revealed lies in the careful assessment of three "internal" aspects of
validity of computerized essay scores, following Messick's (1995)
six-faceted unitary validity model.

Factor analysis and the advice of essay test development experts are
used to guide the deconstruction of essay scoring models into subscore
models corresponding to writing characteristics within the essay. The
writing characteristics identified in this process are also used as the
basis for developing characteristic-specific scoring rubrics to be used
by raters. Fresh essay samples are then scored by raters, both
holistically and characteristic-by-characteristic. These same essay
samples are assigned both holistic scores and characteristic-wise
subscores by the computer scoring models. The degree of congruence of
computer and rater scoring processes, as evidenced by the proportion of
agreement and strength of pairwise correlation among the scores and
subscores, yields evidence of validity of the computer-generated scores.

# Table of Contents

# Introduction 1

# Background 3

**Procedures 6**

**Results 11**

**Discussion 41**

**References 47**

# Introduction

The ever-increasing quantity of large-scale testing, coupled with the
growing movement towards more direct forms of student assessment,
creates a huge burden on test scoring resources. Specifically, the rise
in constructed-response assessment leads to a greatly increased need to
evaluate many more responses efficiently. Figure 1 shows the growth in
volume of several major essay tests over the last decade. Many in
educational testing believe that the time has come for computerized
essay scoring tools, envisioned as reliability checks -- but *not*
replacements -- for expert raters.

Recent studies have shown a high degree of correlation between expert
rater scores and computer-generated scores from several computerized
scoring tools. While such evidence of reliability is essential to the
success of these tools, it is equally essential that the tools generate
scores that are credible in their own right. That is, in scoring an
essay, a computerized scoring tool should ideally engage in processes
that can be shown to parallel, if not exactly duplicate, what an expert
rater would do. The degree to which these tools are accepted will likely
depend on such parallels being demonstrated convincingly.

This study presents and evaluates validity evidence of the
interpretation and use of scores produced by computerized essay scoring
tools, following the test score validity reasoning of Messick (1987,
1995). Applied in the current context, the premise is that the
interpretation and relevance/utility of computer-generated essay scores
should be no different from the interpretation and relevance/utility of
expert rater-generated scores. Specifically, this study assesses whether
a claim can be reasonably made that a computerized essay scoring tool
takes

![](media/image1.wmf)**Figure 1.** Growth in Selected Essay Tests (in
thousands of essays), 1990-2000. Statistics provided by the American
Council on Education, The College Board, Educational Testing Service,
and the Graduate Management Admissions Council.

into account many of the same things that an expert rater does in
determining an essay score. At present, the apparent high correlation of
computer-generated scores with expert rater-generated scores remains
unexplained in that no empirical linkage of the electronic processing of
a computer to the cognitive processing of a human being has been
established for the essay scoring process.

The focus of this study is *e-rater*^™^, a computer-based essay scoring
system developed by Educational Testing Service^®^ (ETS^®^). In 1999,
*e-rater* was implemented as the "second rater" on essays from
administrations of the computer-based Graduate Management Admissions
Test^®^ (GMAT^®^). Each GMAT essay is scored by an expert rater and by
*e-rater*. If the two scores differ by more than one point, the essay is
sent to a second expert rater for adjudication. In addition to being
used for operational GMAT scoring, *e-rater* is also used in the
evaluation of essays submitted through Criterion^SM^ Online Writing
Evaluation Service, offered by ETS Technologies, Inc., a for-profit
subsidiary of ETS. *E-rater* is presently one of four computerized essay
scoring services available commercially; the others are Project Essay
Grade^™^ (PEG^™^), offered by Tru-Judge, Inc., Intelligent Essay
Assessor^™^ by Knowledge Assessment Technologies, and IntelliMetric^™^
by Vantage Learning.

## Background

The reasons for the move towards greater use of essays are evident.
Essays address cognitive constructs unique to written communication,
such sustaining a well-focused, relevant discussion, that may not be
plausibly measured with less direct item types (Powers, Burstein,
Chodorow, Fowles, & Kukich, 2000; Braun, 1988). Additionally, the notion
that essays *appear* more reflective of communication skills desirable
in an academic setting has spurred the growth of essay use. The issue is
not even so much whether it is *right* to use essays in a test, but
simply that essays *are* used, and their use is defended locally with a
variety of justifications.

There are many reasons given against the use of essays, as well,
including difficulty in defining the constructs measured in an essay
test and the score validity problems that arise from interrater
unreliability. A universal criticism, however, is the high cost of
scoring essay tests. As the number and complexity of essay tests
increase, the pressure increases on administrators of these tests to
get, train, and retain raters; on funding sources to cover the costs of
attracting the highest-quality raters; and on professors' and teachers'
time outside the classroom available for rating which, after all, is a
second job for most raters. Moreover, the situation may worsen as the
culture of direct assessment -- and the technology to promote it -- both
continue to grow (Messick, 1999).

A solution may be to have computers score essays. This offers several
advantages:

-   [Cost Savings]{.underline}: If used alone, a computerized scoring
    tool eliminates nearly all of the cost of scoring an essay. As
    currently used on the GMAT Analytical Writing Assessment, as a
    "reliability check" following an initial rating by an expert rater,
    it cuts the cost of scoring significantly.

-   [Reliability]{.underline}: A (high-quality) computer never loses its
    virtually incorruptible consistency in scoring.

```{=html}
<!-- -->
```
-   [Accessibility]{.underline}: As access to technology grows,
    computerized scoring of essays becomes increasingly available.

## A big question is, are the scores generated by computerized scoring tools valid? This question has been addressed in several ways to date: 

## At ETS, multiple studies have shown that *e-rater*-generated essay scores agree frequently -- between 88 and 97 percent of the time -- with scores produced on the same essays by expert raters, under a variety of conditions (e.g., Burstein, Kukich, Wolff, Lu, & Chodorow, 1998a, 1998b; Kaplan, Wolff, Burstein, Lu, Rock, & Kaplan, 1998; Burstein & Chodorow, 1999). The proportion of agreement has been shown to be particularly high when "agreement" is defined as either two identical scores or a difference of one point between scores, a commonly accepted definition. Other ETS research has suggested that *e-rater*-generated scores correlate nearly as well with non-test indicators of writing skill as expert rater scores do (Powers, et al., 2000), and that at least one free-response test-- the writing assessment portion of the National Assessment of Educational Progress (NAEP) -- provides evidence of some identifiable, construct-centered structure underlying *e-rater* scores (i.e., a tying of the scores to constructs like discourse, syntactic variety, and on-topic content) (Muraki, Lee, & Kim, 2000).

## Similar research has been undertaken with PEG: Correlation studies (e.g., Page, 1994; Page & Petersen, 1995) show a consistently higher correlation between a PEG score and the score of any single rater, or any mean of two or three raters, than any single rater or multiple-rater mean has with another; and trait-scoring studies (Page, Poggio, & Keith, 1997; Shermis, Koch, Page, Keith, & Harrington, 1999; Keith, 1999) provide evidence of relatedness between underlying traits of writing and the overall PEG score assigned to an essay.

Messick (1987) identifies and examines the interaction of evidential and
consequential bases of test score validity with proposed test score
interpretations and uses. He most succinctly represents his view of test
score validity with his 2x2 matrix, shown in Figure 2.

  ------------------ ------------------------- --------------------------
                     Test Interpretation       Test Use

  Evidential Basis   Construct Validity (CV)   CV + Relevance/Utility
                                               (R/U)

  Consequential      CV +\                     CV + R/U +VI +\
  Basis              Value Implications (VI)   Social Consequences
  ------------------ ------------------------- --------------------------

**Figure 2.** Messick's (1987) Construct Validity Matrix.

Messick (1995) elaborates on a procedure for investigating the validity
of test scores for particular interpretations and uses. He provides a
"road map" for his 2x2 matrix, in the form of six "distinct aspects \[of
construct validity\] to underscore issues and nuances that might
otherwise be downplayed or overlooked ... \[to\] function as general
validity criteria or standards for all educational or psychological
measurement" (Messick, 1995, p. 744). These six aspects are:

> [Content]{.underline}: Evidence of content relevance,
> representativeness, and technical quality (Lennon, 1956; Messick,
> 1989b);
>
> [Structural]{.underline}: Appraises the fidelity of the scoring
> structure to the structure of the construct domain at issue
> (Loevinger, 1957; Messick 1989b);
>
> [Substantive]{.underline}: Theoretical rationales for the observed
> consistencies in test responses, including process models of task
> performance (Embretson, 1983), along with empirical evidence that the
> theoretical processes are actually engaged by respondents in the
> assessment tasks;
>
> [External]{.underline}: Includes convergent and discriminant evidence
> from multitrait-multimethod comparisons (Campbell & Fiske, 1959), as
> well as evidence of criterion relevance and applied utility (Cronbach
> & Gleser, 1965);
>
> [Generalizability]{.underline}: Examines the extent to which score
> properties and interpretations generalize to and across population
> groups, settings, and tasks (Cook & Campbell, 1979; Shulman, 1970),
> including validity generalization of test criterion relationships
> (Hunter, Schmidt, & Jackson, 1982);
>
> [Consequential]{.underline}: Appraises the value implications of score
> interpretation as a basis for action as well as the actual and
> potential consequences of test use, especially in regard to sources of
> invalidity related to issues of bias, fairness, and distributive
> justice (Messick, 1980, 1989b).\
> (Messick, 1995, p. 745)

The investigation presented here covers the first three of these, the
"internal" aspects of validity, while a companion investigation,
currently in progress, covers the latter three "external" aspects. The
results of both investigations are to be reported in the researcher's
dissertation, scheduled for completion in mid-Summer, 2001.

## Procedures 

### All data, essay scoring rubrics, and e-rater models were provided by ETS and ETS Technologies, Inc. The study utilized existing Graduate Record Examination^®^ (GRE^®^) Writing Assessment essays written and scored during recent test administrations. The procedures were performed in three phases, paralleling the first three aspects of Messick's (1995) validation technique. Briefly, content relevance and representativeness were examined through the identification and assessment of the factor structure of each GRE Writing Assessment essay type. The results of this assessment were used in the construction of characteristic-specific holistic scoring rubrics. Reflectivity of the task and domain structure was assessed through comprehensive reviews, conducted by ETS experts in essay test development, of the characteristic-specific rubrics. These experts attended particularly to assessing whether the rubrics target the desired constructs, and not merely the performance of rote counting tasks. Lastly, the engagement of substantive theories and process models was evaluated using the results of "talk-aloud" protocols produced from recordings of expert raters rating of a sample of essays. These procedures are described in more detail in Table 1.

## [The Context: The GRE Writing Assessment]{.underline}: As specified by the GRE Writing Test Advisory Committee ([http://www.gre.org/stuwrit.html#description]{.underline}), the GRE Writing Assessment is designed to measure the ability to:

-   articulate complex ideas clearly and effectively;

-   examine claims and accompanying evidence;

-   support ideas with relevant reasons and examples;

-   sustain a well-focused, coherent discussion; and

-   control the elements of standard written English.

The GRE Writing Assessment consists of two tasks: a 45-minute "issue"
task that requires the examinee to present a perspective on an issue,
and a 30-minute "argument" task that requires analysis of an argument
presented. The two tasks are intended to complement each other, in that
the first task requires an examinee to construct an argument and the
second to critique an argument already made (GRE Website,
[http://www.gre.org/twotasks.html]{.underline}).

  ------------------------------------ ----------------------------------
  **Table 1\                           
  **Sequence of Procedures Followed in 
  Study.                               

  Messick's (1995) Aspect of Validity  Method of Manifestation in Study

  *Phase I:\                           Using information from relevant
  Content relevance and                previous studies, hypothesized a
  representativeness*                  factor structure of *e-rater*
                                       features for "Issue" and
                                       "Argument" tasks; ranked factors
                                       and features in order of
                                       decreasing effect on *e-rater*
                                       score and increasing skill
                                       complexity subsumed; tested fit of
                                       factor structures to
                                       cross-validation data, using both
                                       linear and hierarchical
                                       confirmatory factor modeling.

  *Phase II:\                          Using factors identified in Phase
  Reflectivity of the task and domain  I, drafted detailed scoring
  structures*                          criteria and built factor-specific
                                       scoring rubrics for each task;
                                       discussed the scoring rubrics with
                                       ETS test development experts,
                                       seeking evidence that raters will
                                       engage in the desired constructs,
                                       not in mere counting exercises;
                                       for each factor, created a
                                       corresponding factor-specific
                                       *e-rater* submodel; had ETS
                                       Technologies, Inc. staff evaluate
                                       whether each set of submodels
                                       adequately replicated the
                                       respective full model for each
                                       task.

  *Phase III:\                         For each task, had raters
  Substantive theories and process     participate in a recorded mock
  models*                              essay scoring session, including a
                                       think-aloud scoring exercise;
                                       produced verbal protocols from the
                                       audio recordings; examined whether
                                       the cognitive processes raters
                                       verbalized during the mock scoring
                                       session reflected both the
                                       processes implied in the
                                       factor-specific scoring rubrics
                                       and the processes presumed to be
                                       emulated by the corresponding
                                       *e-rater* features; had raters
                                       score cross-validation essays (*N*
                                       = 110), both holistically and by
                                       factor; computed and examined all
                                       score correlations.
  ------------------------------------ ----------------------------------

The GRE Writing Assessment essays are scored on a six-point scale, using
holistic scoring rubrics. Expert raters, who are college and university
faculty, usually in English, Communications, or a discipline within the
humanities (e.g., Rhetoric and Composition, English Literature,
Classics, Linguistics), are trained to evaluate GRE essays over a two-
to three-day period and then also participate in various rating norming
procedures throughout each scoring session. For example, one or more
times per day during a scoring session, raters take part in a
"rangefinding" procedure, in which all raters examine and discuss
exemplar essays for each of the six score levels. The intent of these
procedures is to ensure that all raters are looking at similar qualities
and keeping in mind certain issues as they rate essays. Currently, most
rating on the GRE Writing Assessment is done on-line from the rater's
home location, with instant, on-line support from a GRE scoring leader.

### [The Scoring Tool: E-rater]{.underline}: Briefly, e-rater uses natural language processing techniques to model the performance of expert raters. For each essay prompt, a sample of essays, previously scored by expert raters, is selected such that an adequate number of essays representing each score category is included. These essays are then used to "train" e-rater to score new, unscored essays (Burstein, Kukich, et al., 1998a). In its attempt to model expert raters, e-rater first uses several subroutines to extract a variety of features from an essay. In the "training" step, these features are used in combination to "postdict" the score previously assigned the essay by expert raters. The system is adjusted when necessary to provide the maximum agreement between the e-rater "postdicted" scores and the actual expert rater scores in the model-building sample. The system is then ready to score new essays.

Powers, et al. (2000) provide a short overview of *e-rater*, the main
portion of which is summarized here. Currently, the *e-rater* system is
built on a stepwise ordinary least squares (OLS) regression model.
*E-rater* focuses on three general classes of essay features:
*discourse*, indicated by various rhetorical features that are expected
to occur throughout an essay; *syntactic*, indicated by the structure of
sentences; and *content*, indicated by prompt-specific vocabulary
expected to be present in the essay. A total of 59 features are
"extractable," but in practice usually only the most predictive
features, as measured by their regression weights, are retained and used
for further scoring. The features used must be both predictive of expert
rater scores *and* analogous in some recognizable way to the
characteristics that expert raters are trained to consider. The OLS
regression weights for these features are applied to each new essay to
estimate a score. The estimated score is then rounded to the nearest
integer, from 0 to 6, in order to make its scale conform to that used by
expert raters.

[The Sample]{.underline}: The GRE Program provided a total sample of
1,794 GRE Writing Assessment essays, consisting of 620 essays written on
the "issue" prompt type, and 1,174 essays written on the "argument"
prompt type. ETS Technologies' staff divided the "issue" essay sample
into a model-building sample of 226 essays and a cross-validation sample
of 394 essays; the "argument" sample was divided into 251 model-building
essays and 923 cross-validation essays. All of these essays are
considered "operational," that is, written by actual graduate school
candidates under ETS' standard testing conditions, and previously scored
by two expert raters. (In operational scoring, the GRE Writing
Assessment is always scored by two expert raters; the implementation of
a computerized scoring tool is not under consideration.)

**Results**

[Phase I: Content Relevance and Representativeness]{.underline}

Ordered lists of factors hypothesized a priori indicated factor relative
importance largely consistent across tasks. Moreover, in both "Issue"
and "Argument" *e-rater* models, the factors jointly incorporated all
features considered to be salient to *e-rater* score prediction for each
task. The results of confirmatory factor analyses of the hypothesized
factor structures suggest that the "Issue" factor structure fits actual
*e-rater* data well, but the "Argument" factor structure does not. A
hierarchical factor structure was not supported for either task.

[Ordered Lists of Factors.]{.underline} Interpreting in the present
context Messick's (1995) emphasis on ensuring the relevance and
representativeness of skill-constructs measured in a test (scoring
algorithm) to the examinee population of interest, the researcher
examined a list of all 59 *e-rater* features and, using the sources of
information listed in Table 2, assembled the most salient features into
sets of factors reflecting the features' shared operational definitions.

A list of the hypothesized factors and associated features is provided
in Table 3. This list provides a heuristic comparison of the relative
influence each factor has, by virtue of its associated features, on
determination of an *e-rater* score. As indicated in the table, with the
exception of features RHE 7 in the "Issue" factor structure and RHE 5 in
the "Argument" factor structure, the same *e-rater* features are used to
model both tasks. Also, no feature loads on more than one factor, with
the exception of RHE 2 in the "Argument" factor structure, which loads
on Factors ARG-5 and ARG-6.

+-----------------------------------------+----------------------------+
| **Table 2**                             |                            |
|                                         |                            |
| Sources of Information Relevant to      |                            |
| Including Features in Hypothesized      |                            |
| Factor Structures.                      |                            |
+-----------------------------------------+----------------------------+
| [Source]{.underline}                    | [Feature Label             |
|                                         | (Pseudonym)]{.underline}   |
| *[The stepwise linear regression output |                            |
| for this study:]{.underline}*           | SYN 1, SYN 2; TOP 1; RHE   |
|                                         | 2\                         |
| \"Issue\" Generic Model:                | TOP 2, RHE 7               |
|                                         |                            |
| Large standardized regression           | SYN 1; TOP 3\              |
| coefficient (\>0.5) for: Smaller        | RHE 4, RHE 6, RHE 8        |
| standardized regression coefficients    |                            |
| for:                                    |                            |
|                                         |                            |
| \"Argument\" Generic Model:             |                            |
|                                         |                            |
| Large standardized regression           |                            |
| coefficient (\>0.5) for:\               |                            |
| Smaller standardized regression         |                            |
| coefficients for:                       |                            |
+-----------------------------------------+----------------------------+
| *[Studies identifying e-rater features  |                            |
| as important in score                   |                            |
| determination:]{.underline}*            |                            |
+-----------------------------------------+----------------------------+
| *Burstein, Braden-Harder, et al.        | SYN 1, SYN 2, SYN 3\       |
| (1998):*                                | TOP 1, TOP 2\              |
|                                         | RHE 1, RHE 3               |
+-----------------------------------------+----------------------------+
| *Burstein, Kukich, Wolff, Lu,           | SYN 1, SYN 2\              |
| Chodorow,\                              | TOP 1, TOP 2\              |
| Braden-Harder, et al. (1998):*          | RHE 3, RHE 7               |
+-----------------------------------------+----------------------------+
| *Burstein, Kukich, et al. (1998a):*     | SYN 1, SYN 2, SYN 3, SYN   |
|                                         | 4, SYN 5\                  |
|                                         | TOP 1, TOP 2\              |
|                                         | RHE 3, RHE 6, RHE 7, RHE 8 |
+-----------------------------------------+----------------------------+
| *Burstein, Kukich, et al. (1998b):*     | SYN 1, SYN 2, SYN 3, SYN   |
|                                         | 4, SYN 5\                  |
|                                         | TOP 1, TOP 2\              |
|                                         | RHE 3, RHE 8               |
+-----------------------------------------+----------------------------+
| *Burstein & Kaplan (1999):*             | SYN 1, SYN 2, SYN 4, SYN   |
|                                         | 5\                         |
|                                         | TOP 1, TOP 2               |
+-----------------------------------------+----------------------------+
| *Burstein (2001):*                      | SYN 1, SYN 2, SYN 4\       |
|                                         | TOP 1, TOP 2\              |
|                                         | RHE 1, RHE 3               |
+-----------------------------------------+----------------------------+
| *[Other recent studies using e-rater    |                            |
| features:]{.underline}*                 |                            |
+-----------------------------------------+----------------------------+
| *Sheehan (2000)* (GMAT prompts)*:*      | SYN 5; TOP 1, TOP 2\       |
|                                         | SYN 1; RHE 3, RHE 7        |
| Selection probability of \>.85 in       |                            |
| stepwise regression for:\               |                            |
| Selection probability of .20 - .85 for: |                            |
+-----------------------------------------+----------------------------+
| *Muraki et al. (2000)* (NAEP            | SYN 1 - SYN 5; TOP 1 - TOP |
| prompts)*:*                             | 3;\                        |
|                                         | RHE 1, RHE 2, RHE 5, RHE 7 |
| Factor loadings of \>0.5 for:           |                            |
+-----------------------------------------+----------------------------+

<table>
<colgroup>
<col style="width: 24%" />
<col style="width: 3%" />
<col style="width: 10%" />
<col style="width: 7%" />
<col style="width: 3%" />
<col style="width: 2%" />
<col style="width: 0%" />
<col style="width: 0%" />
<col style="width: 25%" />
<col style="width: 13%" />
<col style="width: 7%" />
</colgroup>
<tbody>
<tr class="odd">
<td colspan="11"><p><strong>Table 3</strong></p>
<p>Factors and Associated Features, Ordered by Decreasing Relative
Importance of Each in the Scoring Models.</p></td>
</tr>
<tr class="even">
<td colspan="4"><u>“Issue” Generic Model</u></td>
<td></td>
<td colspan="3"></td>
<td colspan="3"><u>“Argument” Generic Model</u></td>
</tr>
<tr class="odd">
<td>Factor</td>
<td colspan="3">Feature ∆<em>R</em><sup>2</sup></td>
<td></td>
<td></td>
<td colspan="3">Factor</td>
<td colspan="2">Feature ∆<em>R</em><sup>2</sup></td>
</tr>
<tr class="even">
<td>ISSUE-1</td>
<td colspan="2">SYN 1</td>
<td>.45</td>
<td colspan="3"></td>
<td colspan="2">ARG-1</td>
<td>SYN 1</td>
<td>.30</td>
</tr>
<tr class="odd">
<td></td>
<td colspan="2">SYN 2</td>
<td>.02</td>
<td colspan="3"></td>
<td colspan="2"></td>
<td>SYN 2</td>
<td>.02</td>
</tr>
<tr class="even">
<td></td>
<td colspan="2">SYN 3</td>
<td>**</td>
<td colspan="3"></td>
<td colspan="2"></td>
<td>SYN 3</td>
<td>**</td>
</tr>
<tr class="odd">
<td></td>
<td colspan="2"></td>
<td></td>
<td colspan="3"></td>
<td colspan="2"></td>
<td></td>
<td></td>
</tr>
<tr class="even">
<td>ISSUE-2</td>
<td colspan="2">TOP 1</td>
<td>.04</td>
<td colspan="3"></td>
<td colspan="2">ARG-2</td>
<td>TOP 3</td>
<td>.07</td>
</tr>
<tr class="odd">
<td></td>
<td colspan="2">TOP 2</td>
<td>.02</td>
<td colspan="3"></td>
<td colspan="2"></td>
<td>TOP 1</td>
<td>**</td>
</tr>
<tr class="even">
<td></td>
<td colspan="2">TOP 3</td>
<td>**</td>
<td colspan="3"></td>
<td colspan="2"></td>
<td>TOP 2</td>
<td>**</td>
</tr>
<tr class="odd">
<td></td>
<td colspan="2"></td>
<td></td>
<td colspan="3"></td>
<td colspan="2"></td>
<td></td>
<td></td>
</tr>
<tr class="even">
<td>ISSUE-3</td>
<td colspan="2">RHE 2</td>
<td>.05</td>
<td colspan="3"></td>
<td colspan="2">ARG-3</td>
<td>RHE 6</td>
<td>.03</td>
</tr>
<tr class="odd">
<td></td>
<td colspan="2">RHE 1</td>
<td>**</td>
<td colspan="3"></td>
<td colspan="2"></td>
<td>RHE 8</td>
<td>.02</td>
</tr>
<tr class="even">
<td></td>
<td colspan="2"></td>
<td></td>
<td colspan="3"></td>
<td colspan="2"></td>
<td>RHE 3</td>
<td>**</td>
</tr>
<tr class="odd">
<td></td>
<td colspan="2"></td>
<td></td>
<td colspan="3"></td>
<td colspan="2"></td>
<td></td>
<td></td>
</tr>
<tr class="even">
<td>ISSUE-4</td>
<td colspan="2">RHE 7</td>
<td>.02</td>
<td colspan="3"></td>
<td colspan="2">ARG-4</td>
<td>RHE 4</td>
<td>.02</td>
</tr>
<tr class="odd">
<td></td>
<td colspan="2">RHE 3</td>
<td>**</td>
<td colspan="3"></td>
<td colspan="2"></td>
<td></td>
<td></td>
</tr>
<tr class="even">
<td></td>
<td colspan="2">RHE 6</td>
<td>**</td>
<td colspan="3"></td>
<td colspan="2"></td>
<td></td>
<td></td>
</tr>
<tr class="odd">
<td></td>
<td colspan="2">RHE 8</td>
<td>**</td>
<td colspan="3"></td>
<td colspan="2"></td>
<td></td>
<td></td>
</tr>
<tr class="even">
<td></td>
<td colspan="2"></td>
<td></td>
<td colspan="3"></td>
<td colspan="2"></td>
<td></td>
<td></td>
</tr>
<tr class="odd">
<td>ISSUE-5</td>
<td colspan="2">RHE 4</td>
<td>**</td>
<td colspan="3"></td>
<td colspan="2">ARG-5</td>
<td>RHE 1</td>
<td>**</td>
</tr>
<tr class="even">
<td></td>
<td colspan="2"></td>
<td></td>
<td colspan="3"></td>
<td colspan="2"></td>
<td>RHE 2</td>
<td>**</td>
</tr>
<tr class="odd">
<td></td>
<td colspan="2"></td>
<td></td>
<td colspan="3"></td>
<td colspan="2"></td>
<td></td>
<td></td>
</tr>
<tr class="even">
<td></td>
<td colspan="2"></td>
<td></td>
<td colspan="3"></td>
<td colspan="2">ARG-6</td>
<td>RHE 2</td>
<td>**</td>
</tr>
<tr class="odd">
<td></td>
<td colspan="2"></td>
<td></td>
<td colspan="3"></td>
<td colspan="2"></td>
<td>RHE 5</td>
<td>**</td>
</tr>
<tr class="even">
<td colspan="2"><p><em>R</em><sup>2</sup> OF MODEL</p>
<p><em>n</em></p></td>
<td></td>
<td><p>.65</p>
<p>226</p></td>
<td colspan="3"></td>
<td colspan="2"></td>
<td></td>
<td><p>.69</p>
<p>251</p></td>
</tr>
</tbody>
</table>

*Note:* All ∆*R*^2^ estimates are significant at the α = .01 level.
These were computed as follows:

${\Delta R}^{2}(X_{j}) = \left( \frac{1 - R^{2}}{n - k - 1} \right)t_{j}^{2}$,

where *X~j~* is the *j*th feature in the final regression model, *R^2^*
is a goodness-of-fit statistic reported for the final model, *n* is the
number of essays represented in the data set, k is the number of
features in the final model, and *t~j~* is the *t*-statistic for the
*j*th feature (Tate, 1998, p. 87).

\*\* indicates feature excluded during stepwise regression.

The incremental proportion of variance in *e-rater* scores explained
uniquely by each feature appears consistent across tasks for Factors
ISSUE-1/ARG-1, ISSUE-2 /ARG-2, ISSUE-4/ARG-3, and ISSUE-5/ARG-4 (a
compound label signifies that the two factors named are identical across
tasks). This pattern suggests a substantial construct overlap inherent
in the two tasks, albeit with a slightly varying emphasis across tasks.

Although the features associated with Factor ARG-6 are scarcely
mentioned in previous studies, this factor was included in the proposed
factor structures to enable this study to perhaps provide some insight
as to the figurative costs and benefits, from the perspective of trained
expert raters, of maintaining these particular features in the current
*e-rater* feature set. Shortly after the hypothesized factor structures
were created, it was determined that, since there was comparatively even
less empirical justification for including Factor ARG-6 in the "Issue"
factor structure than in the "Argument" one, this factor would be
exclusive to the "Argument" factor structure.

Messick (1995) advocated further arranging the relevant performance
constructs and their assessment task indicators into a functional
schema, "to insure that all important parts of the construct domain are
covered" (p. 745). Such a schema elucidates the balance of credit
awarded for competency in each proficiency claimed to be measured.
Messick elaborated:

> Functional importance can be considered in terms of what people
> actually do in the performance domain ... but also in terms of what
> characterizes and differentiates expertise in the domain, which would
> usually emphasize different tasks and processes (p. 745).

This conception of a progressive ordering of competencies stems from
Brunswik's (1956) proposition of an "ecological validity" of perceptual
cues, the demonstration of which hinges on an organization of cues that
not only corroborates the constructs assumed to be involved in a
performance but also replicates the occurrence and ordering of those
constructs in the "natural environment" of the performance. Brunswik
suggested, "any fairly consistent rapport ... with distal layers of the
environment presupposes the existence of proximal sensory cues ... to
serve as mediators of the relationship" (p. 48).

Elaborating on the role of cues, Brunswik (1956) opined that "their
inherent potentialities as representatives of a more distal factor
should be mirrored within the response system ... by the actual
effectiveness of the cue in establishing ... the more distal variable"
(p. 50). On the ordering of these cues, Brunswik continued, "the
rank-order of utilization in what may be called the ... 'cue family
hierarchy' should be the same as the order of their ecological validity.
In consequence, the two aspects are best studied together" (p.50).

Adapting Brunswik's concept, an "ecological hierarchy" of constructs and
associated mediating cues of writing skill was developed specifically
for writing assessments for graduate program admissions. This hierarchy
is displayed as Figure 3. Lowest on the skill-construct pyramid is
proper usage of grammar, punctuation and spelling. This class of writing
skills is considered the most basic of all, and proficiency with these
related skills is prerequisite to proficiency with any of the classes of
skills higher on the pyramid. The reader is reminded that the *­e-rater*
system does not measure this class of writing skills explicitly; only to
the extent that, for example, a misspelled word might not be recognized
as being on-topic or an improperly punctuated sentence

**Figure 3.** An ecological hierarchy of writing skills measured by
writing assessments.

might interfere with the system's detection of a particular rhetorical
strategy, might a writer's difficulty with these most basic skills
affect the *e-rater* score.

Next in skill-construct ascent is coverage of topic-related content. Of
the writing skills measured by *e-rater*, this is presumed to be the
least complex. The construct is represented in the *e-rater* models by
Factor ISSUE-2/ARG-2; the proficiency of a writer with respect to this
skill is estimated by the three topical *e-rater* features constituting
ISSUE-2/ARG-2. As reasoned above, grammar and spelling may be considered
implicitly subsumed in measures of topic-related content coverage.

Execution of basic rhetorical strategies, such as the introduction,
support and conclusion of a coherent argument, comprises the third level
of skill-constructs in the hierarchy. The constructs are represented in
the *e-rater* "Issue" model by Factors ISSUE-3 and ISSUE-5,and in the
"Argument" model by Factors ARG-4 and ARG-5; the rhetorical *e-rater*
features corresponding to these factors measure proficiency in this
class of skills.

At the top of the pyramid is execution of advanced rhetorical and
syntactic strategies, such as the making of nested or parallel arguments
or the use subordination. Another example of an advanced strategy,
postulated by Muraki et al. (2000), is the incorporation of certain
types of arguments and syntactic structures that interact dynamically in
an essay. The constructs are represented in the *e-rater* "Issue" model
by Factors ISSUE-1 and ISSUE-4, and in the "Argument" model by Factors
ARG-1, ARG-3 and ARG-6; proficiency in the skills is measured by the
respective *e-rater* features.

In support of a hierarchy of competencies such as that proposed here,
Breland and Jones (1982) offer the following insight:

> (L)ogical organization, rhetorical structuring, fluency, and control
> over style and tone, are different from knowledge of grammar and
> punctuation, spelling, capitalization, etc. English teachers tend to
> think of the former as "higher" or at least more sophisticated skills
> that are acquired more slowly than the latter and only after much
> practice. It is these higher skills that the essay ... is designed to
> test, albeit only in a general way (p. 3).

Table 4 presents the list of factors and associated *e-rater* features
in increasing order of complexity of the competency represented, as
proposed in Figure 3.

[Confirmatory Factor Analysis.]{.underline} All *e-rater* features are
measured as integers. The data for all but the three topical features
are counts of feature occurrence; the topical feature data are
ordinal-scale scores, ranging from 0 to 6. A preliminary examination of
univariate descriptive statistics for the "Issue" cross-validation
sample data indicated no violation of the normality assumption for
syntactic feature SYN 1 or for any of the three topical features. The
data distributions appeared to be relatively dense across the range of
values, skewness and kurtosis were nominal, and evidence of central
tendency was seen. However, the "Issue" data for the two other syntactic
features, as well as for all of the rhetorical features included in the
"Issue" factor structure, exhibited a non-negligible degree of
nonnormality. Skewness and kurtosis coefficients for each of these
features exceeded the ranges conventionally cited as supportive of the
normality assumption (Huck, 2000).

A similar pattern of univariate distributions emerged from the
"Argument" cross-validation sample statistics: the data for syntactic
features SYN 1 and SYN 2, all three topical features, and rhetorical
features RHE 1, RHE 3 and RHE 5 showed no violation of the normality
assumption, while there appeared to be slight to marked nonnormality in
the data for syntactic feature SYN 3 and the five

<table style="width:100%;">
<colgroup>
<col style="width: 22%" />
<col style="width: 2%" />
<col style="width: 9%" />
<col style="width: 6%" />
<col style="width: 0%" />
<col style="width: 3%" />
<col style="width: 2%" />
<col style="width: 0%" />
<col style="width: 23%" />
<col style="width: 12%" />
<col style="width: 15%" />
</colgroup>
<tbody>
<tr class="odd">
<td colspan="11"><p><strong>Table 4</strong></p>
<p>Factors and Associated Features, Ordered by Increasing Level of
Writing Skill Complexity.</p></td>
</tr>
<tr class="even">
<td colspan="5"><u>“Issue” Generic Model</u></td>
<td></td>
<td colspan="2"></td>
<td colspan="3"><u>“Argument” Generic Model</u></td>
</tr>
<tr class="odd">
<td>Factor</td>
<td colspan="4">Feature ∆<em>R</em><sup>2</sup></td>
<td></td>
<td colspan="2"></td>
<td>Factor</td>
<td colspan="2">Feature ∆<em>R</em><sup>2</sup></td>
</tr>
<tr class="even">
<td colspan="11"><u>LEVEL 1: “Grammar, Punctuation &amp; Spelling
Proficiency”</u></td>
</tr>
<tr class="odd">
<td colspan="11">(Not explicitly measured by <em>e-rater</em>)</td>
</tr>
<tr class="even">
<td colspan="11"><u>LEVEL 2: “Topic-Related Content Coverage”</u></td>
</tr>
<tr class="odd">
<td>ISSUE-2</td>
<td colspan="2">TOP 1</td>
<td colspan="2">.04</td>
<td colspan="2"></td>
<td colspan="2">ARG-2</td>
<td>TOP 3</td>
<td>.07</td>
</tr>
<tr class="even">
<td></td>
<td colspan="2">TOP 2</td>
<td colspan="2">.02</td>
<td colspan="2"></td>
<td colspan="2"></td>
<td>TOP 1</td>
<td>**</td>
</tr>
<tr class="odd">
<td></td>
<td colspan="2">TOP 3</td>
<td colspan="2">**</td>
<td colspan="2"></td>
<td colspan="2"></td>
<td>TOP 2</td>
<td>**</td>
</tr>
<tr class="even">
<td colspan="11"><u>LEVEL 3: “Basic Rhetorical Strategies”</u></td>
</tr>
<tr class="odd">
<td>ISSUE-3</td>
<td colspan="2">RHE 2</td>
<td colspan="2">.05</td>
<td colspan="2"></td>
<td colspan="2">ARG-4</td>
<td>RHE 4</td>
<td>.02</td>
</tr>
<tr class="even">
<td></td>
<td colspan="2">RHE 1</td>
<td colspan="2">**</td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
<td></td>
</tr>
<tr class="odd">
<td></td>
<td colspan="4"></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td colspan="2"></td>
</tr>
<tr class="even">
<td>ISSUE-5</td>
<td colspan="2">RHE 4</td>
<td colspan="2">.02</td>
<td colspan="2"></td>
<td colspan="2">ARG-5</td>
<td>RHE 1</td>
<td>**</td>
</tr>
<tr class="odd">
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td>RHE 2</td>
<td>**</td>
</tr>
<tr class="even">
<td colspan="11"><u>LEVEL 4: “Advanced Rhetorical and Syntactic
Strategies”</u></td>
</tr>
<tr class="odd">
<td>ISSUE-1</td>
<td colspan="2">SYN 1</td>
<td>.45</td>
<td colspan="3"></td>
<td colspan="2">ARG-1</td>
<td>SYN 1</td>
<td>.30</td>
</tr>
<tr class="even">
<td></td>
<td colspan="2">SYN 2</td>
<td>.02</td>
<td colspan="3"></td>
<td colspan="2"></td>
<td>SYN 2</td>
<td>.02</td>
</tr>
<tr class="odd">
<td></td>
<td colspan="2">SYN 3</td>
<td>**</td>
<td colspan="3"></td>
<td colspan="2"></td>
<td>SYN 3</td>
<td>**</td>
</tr>
<tr class="even">
<td></td>
<td colspan="3"></td>
<td colspan="3"></td>
<td colspan="2"></td>
<td colspan="2"></td>
</tr>
<tr class="odd">
<td>ISSUE-4</td>
<td colspan="2">RHE 7</td>
<td>.02</td>
<td colspan="3"></td>
<td colspan="2">ARG-3</td>
<td>RHE 6</td>
<td>.03</td>
</tr>
<tr class="even">
<td></td>
<td colspan="2">RHE 3</td>
<td>**</td>
<td colspan="3"></td>
<td colspan="2"></td>
<td>RHE 8</td>
<td>.02</td>
</tr>
<tr class="odd">
<td></td>
<td colspan="2">RHE 6</td>
<td>**</td>
<td colspan="3"></td>
<td colspan="2"></td>
<td>RHE 3</td>
<td>**</td>
</tr>
<tr class="even">
<td></td>
<td colspan="2">RHE 8</td>
<td>**</td>
<td colspan="3"></td>
<td colspan="2"></td>
<td></td>
<td></td>
</tr>
<tr class="odd">
<td></td>
<td colspan="2"></td>
<td></td>
<td colspan="3"></td>
<td colspan="2"></td>
<td></td>
<td></td>
</tr>
<tr class="even">
<td></td>
<td colspan="2"></td>
<td></td>
<td colspan="3"></td>
<td colspan="2">ARG-6</td>
<td>RHE 2</td>
<td>**</td>
</tr>
<tr class="odd">
<td></td>
<td colspan="2"></td>
<td></td>
<td colspan="3"></td>
<td colspan="2"></td>
<td>RHE 5</td>
<td>**</td>
</tr>
<tr class="even">
<td colspan="2"><p><em>R</em><sup>2</sup> OF MODEL</p>
<p><em>n</em></p></td>
<td></td>
<td><p>.65</p>
<p>226</p></td>
<td colspan="3"></td>
<td colspan="2"></td>
<td></td>
<td><p>.69</p>
<p>251</p></td>
</tr>
</tbody>
</table>

*Note:* All ∆*R*^2^ estimates are significant at the α = .01 level.

\*\* indicates feature excluded during stepwise regression.

other rhetorical features. One reasonable explanation for the
nonnormality exhibited in the data is that occurrences of certain
features are rare, so that those occurrences that are reported
constitute a data distribution with near-zero variability. This
"sparseness" of data was believed not to be a major concern for this
study since the data for the features most salient to an *e-rater* score
did not exhibit this condition.

A visual inspection of bivariate scatterplots of *e-rater* score on each
successive feature in the final regression model revealed no readily
apparent evidence of extreme influence on the scores. Most cases fell
within one standard deviation of the regression mean conditioned on each
score level, and the variability of the conditioned standard deviations
appeared moderate across all scatterplots. What potentially influential
observations were seen occurred at the extremes of the distributions.
However, since ETS Technologies, Inc. was unable to provide *e-rater*
regression residual data for these essays, a further diagnosis of
heteroskedasticity was not possible.

In the cross-validation samples made available for this study, only 2 of
394, or 0.6%, of "Issue" essays and 24 of 923, or 2.5%, of "Argument"
essays had been assigned field scores of "1" or "2." The quantity of
sample essays assigned a "6" field score was greater but still
comparatively small: 52 of 394, or 13.2%, of "Issue" essays and 106 of
923, or 11.5%, of "Argument" essays. This notable dearth of essays at
both extremes of the score scale may limit the usefulness of empirical
findings arising from these samples, in that such findings may not hold
in the score scale extremes of the larger pool of GRE Writing Assessment
essays. For example, it has been acknowledged informally (Muraki et al.,
2000; J. C. Burstein, personal communication, March 15, 2001) that
*e-rater* models are less stable at the extremes of the six-point score
scale, producing scores at the extremes that diverge from rater-assigned
scores more often than do those nearer the midpoint of the scale. Since
a different sample of essays is used each time *e-rater* is run, the
relative scarceness of extreme-score essay data in each sample is a
likely culprit.

[Confirmatory Factor Modeling:]{.underline} Table 5 summarizes the
model-to-data fit statistics for the linear as well as the hierarchical
confirmatory factor models described herein. The standardized factor
loadings and squared multiple correlations for the "Issue" and
"Argument" linear confirmatory models are provided in Tables 6 and 7,
respectively, and the inter-factor correlation matrix for each model is
shown in Table 4.9. Both the linear and the hierarchical factor models
were constructed with LISREL 8.30 (Jöreskog &Sörbom, 1999), a structural
equation modeling program. An input file written in the SIMPLIS Command
Language (Jöreskog & Sörbom, 1993) instructed LISREL to execute each
confirmatory factor model.

Possibly due to redundancies among several of the *e-rater* features,
the covariance matrices obtained for the "Issue" and "Argument"
cross-validation samples were not positive definite. Attempts to achieve
convergent confirmatory models using maximum likelihood (ML) model
parameter estimation were unsuccessful. Ultimately, convergence for both
the "Issue" and "Argument" models was achieved with unweighted least
squares (ULS) model parameter estimation.

+-----------------------------+--------------------+--------------------+
| # Table 5                   |                    |                    |
|                             |                    |                    |
| Confirmatory Factor Model   |                    |                    |
| Fit Statistics              |                    |                    |
| (cross-validation samples). |                    |                    |
+-----------------------------+--------------------+--------------------+
| Fit Statistic               | [                  | [Arg               |
|                             | Issue]{.underline} | ument]{.underline} |
|                             |                    |                    |
|                             | (*n* = 394)        | (*n* = 923)        |
+-----------------------------+--------------------+--------------------+
| [Linear                     |                    |                    |
| Models:]{.underline}        |                    |                    |
+-----------------------------+--------------------+--------------------+
| χ^2^ (*df*~Issue~ = 55,     | 205                | 631                |
| *df*~Argument~ = 53)        |                    |                    |
+-----------------------------+--------------------+--------------------+
| RMSEA                       | .08                | .11                |
+-----------------------------+--------------------+--------------------+
| Adjusted Goodness-of-Fit    | .92                | .88                |
| Index                       |                    |                    |
+-----------------------------+--------------------+--------------------+
| [Hierarchical               |                    |                    |
| Models:]{.underline}        |                    |                    |
+-----------------------------+--------------------+--------------------+
| χ^2^ (*df*~Issue~ = 62,     | 361                | 1737               |
| *df*~Argument~ = 76)        |                    |                    |
+-----------------------------+--------------------+--------------------+
| RMSEA                       | .11                | .17                |
+-----------------------------+--------------------+--------------------+
| Adjusted Goodness-of-Fit    | .89                | .79                |
| Index                       |                    |                    |
+-----------------------------+--------------------+--------------------+

+-----------------------+-----------------------+----------------------+
| ## Table 6            |                       |                      |
|                       |                       |                      |
| Standardized Factor   |                       |                      |
| Loadings and Squared  |                       |                      |
| Multiple Correlations |                       |                      |
| for "Issue" Linear    |                       |                      |
| Confirmatory Factor   |                       |                      |
| Model                 |                       |                      |
| (cross-validation     |                       |                      |
| sample, *n* = 394).   |                       |                      |
+-----------------------+-----------------------+----------------------+
| Factor and Feature(s) | Factor Loading        | *R*^2^               |
+-----------------------+-----------------------+----------------------+
| ISSUE-1               |                       |                      |
+-----------------------+-----------------------+----------------------+
| SYN 1                 | .64                   | .42                  |
+-----------------------+-----------------------+----------------------+
| SYN 2                 | .61                   | .37                  |
+-----------------------+-----------------------+----------------------+
| SYN 3                 | .77                   | .59                  |
+-----------------------+-----------------------+----------------------+
|                       |                       |                      |
+-----------------------+-----------------------+----------------------+
| ISSUE-2               |                       |                      |
+-----------------------+-----------------------+----------------------+
| TOP 1                 | .92                   | .84                  |
+-----------------------+-----------------------+----------------------+
| TOP 2                 | .48                   | .23                  |
+-----------------------+-----------------------+----------------------+
| TOP 3                 | .70                   | .48                  |
+-----------------------+-----------------------+----------------------+
|                       |                       |                      |
+-----------------------+-----------------------+----------------------+
| ISSUE-3               |                       |                      |
+-----------------------+-----------------------+----------------------+
| RHE 1                 | .81                   | .66                  |
+-----------------------+-----------------------+----------------------+
| RHE 2                 | .55                   | .31                  |
+-----------------------+-----------------------+----------------------+
|                       |                       |                      |
+-----------------------+-----------------------+----------------------+
| ISSUE-4               |                       |                      |
+-----------------------+-----------------------+----------------------+
| RHE 3                 | .91                   | .84                  |
+-----------------------+-----------------------+----------------------+
| RHE 6                 | .76                   | .58                  |
+-----------------------+-----------------------+----------------------+
| RHE 7                 | .49                   | .24                  |
+-----------------------+-----------------------+----------------------+
| RHE 8                 | .37                   | .14                  |
+-----------------------+-----------------------+----------------------+
|                       |                       |                      |
+-----------------------+-----------------------+----------------------+
| ISSUE-5               |                       |                      |
+-----------------------+-----------------------+----------------------+
| RHE 4                 | 1.00                  | 1.00                 |
+-----------------------+-----------------------+----------------------+

+-----------------------+-----------------------+----------------------+
| ## Table 7            |                       |                      |
|                       |                       |                      |
| Standardized Factor   |                       |                      |
| Loadings and Squared  |                       |                      |
| Multiple Correlations |                       |                      |
| for "Argument" Linear |                       |                      |
| Confirmatory Factor   |                       |                      |
| Model                 |                       |                      |
| (cross-validation     |                       |                      |
| sample, *n* = 923).   |                       |                      |
+-----------------------+-----------------------+----------------------+
| Factor and Feature(s) | Factor Loading        | *R*^2^               |
+-----------------------+-----------------------+----------------------+
| ARG-1                 |                       |                      |
+-----------------------+-----------------------+----------------------+
| SYN 1                 | .67                   | .45                  |
+-----------------------+-----------------------+----------------------+
| SYN 2                 | .39                   | .15                  |
+-----------------------+-----------------------+----------------------+
| SYN 3                 | .81                   | .66                  |
+-----------------------+-----------------------+----------------------+
|                       |                       |                      |
+-----------------------+-----------------------+----------------------+
| ARG-2                 |                       |                      |
+-----------------------+-----------------------+----------------------+
| TOP 1                 | .88                   | .76                  |
+-----------------------+-----------------------+----------------------+
| TOP 2                 | .85                   | .76                  |
+-----------------------+-----------------------+----------------------+
| TOP 3                 | .88                   | .76                  |
+-----------------------+-----------------------+----------------------+
|                       |                       |                      |
+-----------------------+-----------------------+----------------------+
| ARG-3                 |                       |                      |
+-----------------------+-----------------------+----------------------+
| RHE 3                 | .87                   | .75                  |
+-----------------------+-----------------------+----------------------+
| RHE 6                 | .75                   | .56                  |
+-----------------------+-----------------------+----------------------+
| RHE 8                 | .27                   | .06                  |
+-----------------------+-----------------------+----------------------+
|                       |                       |                      |
+-----------------------+-----------------------+----------------------+
| ARG-4                 |                       |                      |
+-----------------------+-----------------------+----------------------+
| RHE 4                 | 1.00                  | 1.00                 |
+-----------------------+-----------------------+----------------------+
|                       |                       |                      |
+-----------------------+-----------------------+----------------------+
| ARG-5                 |                       |                      |
+-----------------------+-----------------------+----------------------+
| RHE 1                 | .71                   | .45                  |
+-----------------------+-----------------------+----------------------+
| RHE 2                 | .42                   | .27                  |
+-----------------------+-----------------------+----------------------+
|                       |                       |                      |
+-----------------------+-----------------------+----------------------+
| ARG-6                 |                       |                      |
+-----------------------+-----------------------+----------------------+
| RHE 2                 | .36                   | .27                  |
+-----------------------+-----------------------+----------------------+
| RHE 5                 | .74                   | .45                  |
+-----------------------+-----------------------+----------------------+

Wothke (1993) describes ULS as "a very robust estimation technique that
imposes few additional requirements on the model and the sample
covariance matrices" (p. 272), and Bryant and Yarnold (1995) indicate
that ULS is an acceptable substitute for ML if all variables included in
the analysis are scaled on the same metric. Since only the topical
content features are scaled differently -- scores ranging from 1 to 6,
rather than counts of occurrence -- and since this scale may be viewed
alternatively as a measure of the "frequency of on-topicness" of an
essay, a plausible argument may be made that the data comply with the
single-metric requirement of ULS parameter estimation.

The fit indexes in Table 5 provide a small degree of support for fit of
the "Issue" model to the data, but no support for "Argument" model fit.
The ratio of chi-square to degrees of freedom of approximately 3.7 for
the "Issue" model and 11.9 for the "Argument" model exceed the threshold
of 3.0 conventionally regarded as indicating acceptable model fit (Tate,
2000). The root mean square error of approximation (RMSEA) of .08 for
the "Issue" model and .11 for the "Argument" model also indicate lack of
model fit. However, the adjusted goodness-of-fit (AGFI) index of .92 for
the "Issue" model exceeds "the agreed-upon cutoff for overall fit
indexes" of .90 (Hoyle & Panter, 1995, p. 164); the AGFI of .88 for the
"Argument" model does not.

A variety of model modification indexes were suggested for the linear
confirmatory factor models, but none of these, if implemented, would
maintain the conceptual integrity of the hypothesized factor structures;
that is, each modification index suggested a path that would have
connected conceptually unrelated features. Therefore, no modifications
to the original linear confirmatory models were implemented. Table 10
shows Factor ISSUE-1 to be highly correlated with ISSUE-4 and Factor
ARG-1 to be highly correlated with ARG-3. While the results of the
analysis of this particular data set suggest that the empirical fidelity
of the hypothesized factor structures might improve if these factor
pairs were to be combined into single factors, the overriding conceptual
framework of the factor structures mandates that these factors remain
separate. Still, the researcher found the magnitude of the correlation
intriguing, and decided to pursue combination of these factors in the
hierarchical confirmatory factor modeling procedure, in which a "g"
factor of writing ability was tested for fit to the data. (The reader
should note that, since *e-rater* does not explicitly measure the "Level
1" constructs shown in Figure 3, it is impossible to model the
hierarchical structure shown in Table 4.)

+-----------------------+-----------------------+----------------------+
| **Table 8**           |                       |                      |
|                       |                       |                      |
| Standardized Factor   |                       |                      |
| Loadings and Squared  |                       |                      |
| Multiple Correlations |                       |                      |
| for "Issue"           |                       |                      |
| Hierarchical          |                       |                      |
| Confirmatory Factor   |                       |                      |
| Model                 |                       |                      |
| (cross-validation     |                       |                      |
| sample, *n* = 394).   |                       |                      |
+-----------------------+-----------------------+----------------------+
| Factor and Feature(s) | Factor Loading        | *R*^2^               |
+-----------------------+-----------------------+----------------------+
| [Level                |                       |                      |
| II]{.underline}       |                       |                      |
+-----------------------+-----------------------+----------------------+
| "g"                   |                       |                      |
+-----------------------+-----------------------+----------------------+
| ISSUE-1&4             | .76                   | \--                  |
+-----------------------+-----------------------+----------------------+
| ISSUE-2               | .28                   | \--                  |
+-----------------------+-----------------------+----------------------+
| ISSUE-3&5             | .51                   | \--                  |
+-----------------------+-----------------------+----------------------+
| [Level I]{.underline} |                       |                      |
+-----------------------+-----------------------+----------------------+
| ISSUE-1&4             |                       |                      |
+-----------------------+-----------------------+----------------------+
| SYN 1                 | .66                   | .43                  |
+-----------------------+-----------------------+----------------------+
| SYN 2                 | .60                   | .37                  |
+-----------------------+-----------------------+----------------------+
| SYN 3                 | .76                   | .57                  |
+-----------------------+-----------------------+----------------------+
| RHE 3                 | .92                   | .84                  |
+-----------------------+-----------------------+----------------------+
| RHE 6                 | .77                   | .59                  |
+-----------------------+-----------------------+----------------------+
| RHE 7                 | .48                   | .23                  |
+-----------------------+-----------------------+----------------------+
| RHE 8                 | .36                   | .13                  |
+-----------------------+-----------------------+----------------------+
|                       |                       |                      |
+-----------------------+-----------------------+----------------------+
| ISSUE-2               |                       |                      |
+-----------------------+-----------------------+----------------------+
| TOP 1                 | .71                   | .51                  |
+-----------------------+-----------------------+----------------------+
| TOP 2                 | .66                   | .43                  |
+-----------------------+-----------------------+----------------------+
| TOP 3                 | .72                   | .52                  |
+-----------------------+-----------------------+----------------------+
|                       |                       |                      |
+-----------------------+-----------------------+----------------------+
| ISSUE-3&5             |                       |                      |
+-----------------------+-----------------------+----------------------+
| RHE 1                 | .83                   | .69                  |
+-----------------------+-----------------------+----------------------+
| RHE 2                 | .60                   | .36                  |
+-----------------------+-----------------------+----------------------+
| RHE 4                 | -.14                  | .02                  |
+-----------------------+-----------------------+----------------------+

The standardized factor loadings and error variances for the "Issue" and
"Argument" hierarchical models are provided in Tables 8 and 9,
respectively, and the inter-factor correlation matrix for each model is
shown in Table 10. The within-factor feature groupings shown in the
original factor structure and specified in the linear confirmatory
factor models were preserved in the hierarchical confirmatory factor
model specifications. For the "Issue" hierarchical model, Factors
ISSUE-1 and ISSUE-4 were folded into a common Factor "ISSUE-1&4," and
ISSUE-3 and

+-----------------------+-----------------------+----------------------+
| **Table 9**           |                       |                      |
|                       |                       |                      |
| Standardized Factor   |                       |                      |
| Loadings and Squared  |                       |                      |
| Multiple Correlations |                       |                      |
| for "Argument"        |                       |                      |
| Hierarchical          |                       |                      |
| Confirmatory Factor   |                       |                      |
| Model                 |                       |                      |
| (cross-validation     |                       |                      |
| sample, *n* = 923).   |                       |                      |
+-----------------------+-----------------------+----------------------+
| Factor and Feature(s) | Factor Loading        | *R*^2^               |
+-----------------------+-----------------------+----------------------+
| [Level                |                       |                      |
| II]{.underline}       |                       |                      |
+-----------------------+-----------------------+----------------------+
| "g"                   |                       |                      |
+-----------------------+-----------------------+----------------------+
| ARG-1&3               | .78                   | \--                  |
+-----------------------+-----------------------+----------------------+
| ARG-2                 | .12                   | \--                  |
+-----------------------+-----------------------+----------------------+
| ARG-4,5&6             | .77                   | \--                  |
+-----------------------+-----------------------+----------------------+
| [Level I]{.underline} |                       |                      |
+-----------------------+-----------------------+----------------------+
| ARG-1&3               |                       |                      |
+-----------------------+-----------------------+----------------------+
| SYN 1                 | .69                   | .43                  |
+-----------------------+-----------------------+----------------------+
| SYN 2                 | .34                   | .10                  |
+-----------------------+-----------------------+----------------------+
| SYN 3                 | .82                   | .58                  |
+-----------------------+-----------------------+----------------------+
| RHE 3                 | .81                   | .73                  |
+-----------------------+-----------------------+----------------------+
| RHE 6                 | .72                   | .56                  |
+-----------------------+-----------------------+----------------------+
| RHE 8                 | .25                   | .07                  |
+-----------------------+-----------------------+----------------------+
|                       |                       |                      |
+-----------------------+-----------------------+----------------------+
| ARG-2                 |                       |                      |
+-----------------------+-----------------------+----------------------+
| TOP 1                 | .88                   | .77                  |
+-----------------------+-----------------------+----------------------+
| TOP 2                 | .85                   | .73                  |
+-----------------------+-----------------------+----------------------+
| TOP 3                 | .88                   | .77                  |
+-----------------------+-----------------------+----------------------+
|                       |                       |                      |
+-----------------------+-----------------------+----------------------+
| ARG-4,5&6             |                       |                      |
+-----------------------+-----------------------+----------------------+
| RHE 1                 | .31                   | .10                  |
+-----------------------+-----------------------+----------------------+
| RHE 2                 | .41                   | .16                  |
+-----------------------+-----------------------+----------------------+
| RHE 4                 | .10                   | .01                  |
+-----------------------+-----------------------+----------------------+
| RHE 5                 | .59                   | .31                  |
+-----------------------+-----------------------+----------------------+

ISSUE-5 were folded into a Factor "ISSUE-3&5." Similarly, Factors ARG-1
and ARG-3 and Factors ARG-4, ARG-5, and ARG-6 were folded into common
Factors "ARG-1&3" and "ARG-4,5&6," respectively. Factors ISSUE-1&4 and
ARG-1&3 were intentionally specified as such to permit an investigation
of the high correlations seen between the original factors in each
factor structure. Factor "g" was modeled as a Level II hierarchical
factor, loaded on by all other factors in the model. The structure of
these models reflects the design Mulaik and Quartetti (1997) refer

+-----------+-----------+-----------+-----------+-----------+-----------+
| **Table   |           |           |           |           |           |
| 10**      |           |           |           |           |           |
|           |           |           |           |           |           |
| Int       |           |           |           |           |           |
| er-Factor |           |           |           |           |           |
| Cor       |           |           |           |           |           |
| relations |           |           |           |           |           |
| for       |           |           |           |           |           |
| Linear    |           |           |           |           |           |
| and       |           |           |           |           |           |
| Hie       |           |           |           |           |           |
| rarchical |           |           |           |           |           |
| Con       |           |           |           |           |           |
| firmatory |           |           |           |           |           |
| Factor    |           |           |           |           |           |
| Models    |           |           |           |           |           |
| (cross-v  |           |           |           |           |           |
| alidation |           |           |           |           |           |
| samples). |           |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
|           | ##        |           |           |           |           |
|           | #### Line |           |           |           |           |
|           | ar Models |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
|           | #         |           |           |           |           |
|           | ##### Iss |           |           |           |           |
|           | ue (13 fe |           |           |           |           |
|           | atures, * |           |           |           |           |
|           | N* = 394) |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| Factor    | [IS       | [IS       | [IS       | [IS       |           |
|           | SUE-1]{.u | SUE-2]{.u | SUE-3]{.u | SUE-4]{.u |           |
|           | nderline} | nderline} | nderline} | nderline} |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| ISSUE-2   | .01       |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| ISSUE-3   | .42       | .22       |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| ISSUE-4   | .99       | -.19      | .31       |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| ISSUE-5   | .12       | -.11      | -.33      | .16       |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
|           | #####     |           |           |           |           |
|           | ## Argume |           |           |           |           |
|           | nt (14 fe |           |           |           |           |
|           | atures, * |           |           |           |           |
|           | N* = 923) |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| [F        | [         | [         | [         | [         | [         |
| actor]{.u | ARG-1]{.u | ARG-2]{.u | ARG-3]{.u | ARG-4]{.u | ARG-5]{.u |
| nderline} | nderline} | nderline} | nderline} | nderline} | nderline} |
+-----------+-----------+-----------+-----------+-----------+-----------+
| ARG-2     | -.05      |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| ARG-3     | .93       | .20       |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| ARG-4     | .15       | -.06      | .24       |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| ARG-5     | .35       | .29       | .14       | -.57      |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| ARG-6     | .40       | -.06      | .64       | .26       | -.13      |
+-----------+-----------+-----------+-----------+-----------+-----------+
|           | ###### H  |           |           |           |           |
|           | ierarchic |           |           |           |           |
|           | al Models |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
|           | #         |           |           |           |           |
|           | ##### Iss |           |           |           |           |
|           | ue (13 fe |           |           |           |           |
|           | atures, * |           |           |           |           |
|           | N* = 394) |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| [F        | [ISSU     | [IS       |           |           |           |
| actor]{.u | E-1&4]{.u | SUE-2]{.u |           |           |           |
| nderline} | nderline} | nderline} |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| ISSUE-2   | -.13      |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| ISSUE-3&5 | .32       | .14       |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
|           | #####     |           |           |           |           |
|           | ## Argume |           |           |           |           |
|           | nt (14 fe |           |           |           |           |
|           | atures, * |           |           |           |           |
|           | N* = 923) |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| [F        | [AR       | [         |           |           |           |
| actor]{.u | G-1&3]{.u | ARG-2]{.u |           |           |           |
| nderline} | nderline} | nderline} |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| ARG-2     | .09       |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| ARG-4,5&6 | .60       | .15       |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+

to as the "complete general hierarchical model" (p. 203). ULS model
parameter estimation was used.

The model fit statistics shown in Table 5 offer no empirical support for
a hierarchical arrangement for either of the hypothesized factor
structures: for the "Issue" and "Argument" hierarchical models,
respectively, the χ^2^/*df* of 5.8 and 22.8, the RMSEA of .11 and .17,
and the AGFI of .89 and .79 all fail to cross the conventional
thresholds for good model-to-data-fit. As earlier, model modifications
suggested for the hierarchical confirmatory models would have violated
the conceptual design of the factor structures and, therefore, were
disregarded.

The merger of highly correlated factors into a single factor, while
reducing the redundancy apparent in each linear confirmatory factor
model, does not appear to have had a positive effect on model-to-data
fit. To verify this conclusion, the "Issue" linear confirmatory model
was modified to model Factors "ISSUE-1" and "ISSUE-4" as a single
factor. The results indicated no improvement in fit; χ^2^/*df* actually
increased from the original model level, and neither the RMSEA nor the
AGFI changed substantially.

######## Phase II: Reflectivity of the Task and Domain Structures

Upon reviewing the five "Issue" factors and six "Argument" factors
identified in Phase I, ETS experts in essay test development predicted
that several factors from each model would get a grim reception from
raters. On the positive side, these experts found all 11 factors to be
operationally definable and, following a series of edits, concurred that
the scoring rubrics drafted around these factors explained the factors
adequately.

[Reviewing the "Underlying Characteristics" of Writing.]{.underline} The
"reasonableness review" conducted by ETS essay test development experts
produced mixed results: while the ETS experts generally concurred that
all 11 could be adequately described in terms understandable to a rater,
and therefore suggested that all 11 be retained for further study, the
experts predicted that several of the factors would be seen by raters as
representing invalid measures of writing ability. For example, they
pointed out that several of the factors represented characteristics of
an essay that ETS raters never evaluate. More detail on this is provided
in the next section.

[Reviewing the Factor-Specific Scoring Rubrics.]{.underline} The test
development experts, both of whom regularly assist in the development of
essay prompts, scoring rubrics and supplemental materials for the GRE
Writing Assessment and other essay test programs, provided invaluable
feedback on the factor-specific scoring rubrics. Upon examining the
first drafts of the rubrics, the test development experts suggested a
number of clarifying revisions, which were incorporated immediately.
Table 11 lists the suggestions they made.

The test development experts agreed that the revised rubrics would
likely engage raters in a search for qualities as well as quantities of
factors. In particular, the phrasing that had predominated in the first
drafts of the rubrics had emphasized, in the opinions of the experts,
mainly quantitative aspects of the factors. However, the experts noted
that the operational definition of several factors, particularly
ISSUE-3/ARG-5 and ISSUE-5/ARG-4, made prompting of anything other than
the counting of occurrences of the subsumed features a difficult
proposition. Overall, the test development experts believed that, in
most instances, the tendency

+-------------------------------+----+---------------------------------+
| ######### Table 11            |    |                                 |
|                               |    |                                 |
| Initial Revisions to Scoring  |    |                                 |
| Rubrics (made on              |    |                                 |
| recommendation of test        |    |                                 |
| development experts).         |    |                                 |
+-------------------------------+----+---------------------------------+
| [ISSUE-1]{.underline}         |    | [ARG-1]{.underline}             |
+-------------------------------+----+---------------------------------+
| -   Use synonyms for          |    | -   Same as ISSUE-1             |
|     syntactic variety:        |    |                                 |
|     "subordination,"          |    |                                 |
|     "parallelism,"            |    |                                 |
|     "parenthetical phrases,"  |    |                                 |
|     "word choice"             |    |                                 |
+-------------------------------+----+---------------------------------+
| [ISSUE-2]{.underline}         |    | [ARG-2]{.underline}             |
+-------------------------------+----+---------------------------------+
| -   Incorporate "well-chosen  |    | -   Same as ISSUE-2             |
|     examples" as an example   |    |                                 |
|     of on-topic content       |    |                                 |
+-------------------------------+----+---------------------------------+
| [ISSUE-3]{.underline}         |    | [ARG-3]{.underline}             |
+-------------------------------+----+---------------------------------+
| -   Remove reference to use   |    | -   Same as ISSUE-4             |
|     of "detail," as this      |    |                                 |
|     relates to argument       |    |                                 |
|     development, not          |    |                                 |
|     introduction of arguments |    |                                 |
+-------------------------------+----+---------------------------------+
| [ISSUE-4]{.underline}         |    | [ARG-4]{.underline}             |
+-------------------------------+----+---------------------------------+
| -   Incorporate the term      |    | -   Same as ISSUE-5             |
|     "logical cohesiveness" as |    |                                 |
|     an organizational         |    |                                 |
|     component of argument     |    |                                 |
|     development               |    |                                 |
+-------------------------------+----+---------------------------------+
| [ISSUE-5]{.underline}         |    | [ARG-5]{.underline}             |
+-------------------------------+----+---------------------------------+
| -   Restate "uses ...         |    | -   Same as ISSUE-3             |
|     transitional phrases" to  |    |                                 |
|     "uses transitions and     |    |                                 |
|     other signposts"          |    |                                 |
+-------------------------------+----+---------------------------------+
|                               |    | [ARG-6]{.underline}             |
+-------------------------------+----+---------------------------------+
|                               |    | -   Use the term "persuasive    |
|                               |    |     point" interchangeably with |
|                               |    |     "argument," to clarify the  |
|                               |    |     concept of an "argument"    |
+-------------------------------+----+---------------------------------+

for raters to count occurrences of certain features, at least roughly if
not exactly, would be unavoidable.

The test development experts were prescient on this point. Ultimately,
it proved necessary to encourage the raters verbally, on multiple
occasions, to score essays holistically with respect to the qualities
implied in the rubrics, rather than permit the sheer frequency of
occurrence of the features suggested in the rubrics to dominate the
scoring process.

[Reviewing the Factor-Specific *e-rater* Submodels.]{.underline} One
concern voiced by ETS Technologies, Inc. staff as they prepared the
requested factor-specific *e-rater* submodels was that these submodels
would be driven by feature regression weights computed during the
original, operational scoring of the essays and, therefore, perhaps fail
to reflect scoring tendencies of the particular group of raters
participating in this study.

There were two implications arising from this. First, since the
submodels would use a "forced" (non-stepwise) regression procedure, it
was suggested that if a given feature that would normally drop from the
regression during the stepwise procedure were to instead remain, its
presence could perhaps bias the submodel score computed. Second, all
submodel scores generated would be based on original holistic scores
assigned by someone other than the raters participating in this study.

Ultimately, it was surmised that the net effect of "falsely retained"
features on the submodel scores, if any, would most likely be small,
since the regression weights of such low-loading features would also
most likely be small. It was further reasoned that the existing
regression weights, while not specifically representative of the
participating raters, were nevertheless based on the work of many raters
over a considerable period of time. These "long-range rater mean"
regression weights would most likely perform as well as weights
generated exclusively from holistic scores assigned by the participating
raters, particularly since the number of sample essays that the raters
scored was insufficient to have generated stable regression
coefficients.

[Fidelity Check:]{.underline} For each task, interrater subsample score
correlations, correlations of two-rater means with field scores over the
20-essay interrater subsamples, and agreement statistics of
rater-assigned scores with field scores over the 110-essay samples, all
computed in Phase III, provided empirical evidence of the scoring
consistencies of the participating raters. This evidence influenced the
tenability of the assumption that the four participating raters did not
differ substantively from the larger population of raters.

[Phase III: Substantive Theories and Process Models]{.underline}

In this phase, the question to be answered was, "is the study addressing
the right things?" That is, the study probed whether the cognitive
processes raters engage during essay rating actually reflect *both* the
processes identified as the intended targets, in Phase II, *and* the
processes assumed to be emulated by *e-rater*.

The newly written characteristic-specific scoring rubrics were used by a
group of four expert raters, along with the original holistic rubrics,
on a fresh sample of GRE Writing Assessment essays taken from the
cross-validation essay samples discussed earlier. Each expert rater
rated a sample of 110 essays twice, using on each essay first the
holistic rubric normally used to score GRE Writing Assessment essays
operationally and then the characteristic-specific rubrics designed for
this study. In order to expedite the process, the initial ratings were
done by these raters at home. While it is generally preferable to have
raters interacting on-site in a "conference-type" atmosphere when
rating, as this has been shown to have a small, positive effect on
rating reliability comparative to remote-site rating (Breland & Jones,
1988), this particular group of expert raters, quite familiar with the
essay types and scoring rubrics, were believed by ETS staff to be
capable of producing reliable ratings regardless of their location.

Several weeks later, the group arrived at ETS, two raters on each of two
consecutive days, to score their same samples of essays using the
characteristic-specific rubrics. The raters were selected specifically
on account of their expertise in GRE essay scoring and their previous
participation in research activities with the GRE program. These four
raters, known as "scoring leaders," are all currently faculty in English
programs at colleges and universities in the Delaware Valley area. ETS
Test Development staff have a high degree of confidence in the abilities
of these raters, a confidence which proved critical as the challenging
procedures that follow were carried out.

Upon their arrival at ETS on the day of their face-to-face participation
in the project, each rater was provided a thorough introduction to the
research and the specific activities scheduled to take place that day.
The characteristic-specific scoring rubrics were then presented. This
was the first time any of the raters has seen these rubrics. All
procedures, except as noted, were recorded on audio tape. The first
procedure was a roughly 30-minute semi-structured group interview, in
which the raters were asked about their first impressions of the
characteristic-specific rubrics. Specifically, they were asked the
following questions about each rubric in turn:

-   Do you find the rubric to be interpretable, meaningful, and useable
    to you? That is, do you feel, on first impression, that you will
    even be able to identify the characteristic that this rubric calls
    for you to evaluate in an essay?

-   Do you find the characteristic targeted by the rubric a
    *justifiable* trait of an essay, worthy of being evaluated
    individually?

-   Is it possible for you to conceive, on first impression, how you
    would go about distinguishing among the six possible score
    categories with respect to the characteristic identified in this
    rubric?

Almost immediately, some strong impressions emerged from the raters on
rubrics for both the "issue" and "argument" essay types. For both essay
types, the raters had problems rationalizing the legitimacy of looking
for two of the characteristics: ISSUE-2/ARGUMENT-3 and
ISSUE-4/ARGUMENT-6 (the compound label indicates that a characteristic
appears in both essay types). All four raters indicated that it was
unusual for them to look at such characteristics in isolation, and they
all questioned the legitimacy of scoring an essay exclusively for such
characteristics, even when the characteristics are to be taken as part
of a larger composite scoring of the essay. Also, they found the very
prospect of looking for these characteristics to be something unfamiliar
to them; adjectives like "unnatural" and "discomforting" were heard in
their description of the impending task they would perform.

Two other characteristics, ISSUE-1/ARGUMENT-1 and ISSUE-5/ARGUMENT-4,
were perceived by all four raters as difficult to parse into separate,
ratable qualities as, by their account, it was odd for them to think in
terms of separating the content of the essay from the type of discourse
used. In fact, it emerged later each day that the raters typically
considered content to be relevant to an essay only inasmuch as it
relates to the positions being taken, and that idle mention of
prompt-related material would not normally constitute "content" for
them. (This issue presents one example of the conceptual "retraining"
that had to be undertaken, by both the researcher and the raters, in
order to help the raters to see the task as the researcher construed
it.)

Interestingly, a concept introduced by Muraki et al. (2000), the dynamic
relationship hypothesized between syntax and discourse structures in the
construction of an essay, did not manifest itself as much in the initial
discussions (although there is some evidence that it emerged in a later
procedure). That is, the raters did not report anticipating difficulty
in separating the syntactic structure of the essay from the type of
discourse used, although several raters agreed that these qualities are
highly associated in some instances.

To examine in greater detail whether the expert raters were following
processes that focus implicitly in some way on the features analyzed by
*e-rater*, each rater was asked to "talk aloud" while rating essays.
Using a procedure outlined by Ericsson and Simon (1980) as a guide, each
rater was instructed to verbalize all her thoughts as she rated two
essays, using each of the characteristic-specific rubrics in turn.
Intermittently, she was stopped by the researcher and asked to recall
what thought processes had just gone through her mind during the most
recent rating. This exercise was performed initially under the
observation of a resident expert in "talk-aloud" procedures at ETS. The
four "talk-aloud" sessions produced several hours of audio tape, which
were then converted into coded protocols, or bullet-like summary
transcriptions of what was said by the rater during each session.

Several issues emerged from a review of the protocols. First, the
protocols suggest that the essay rating process is highly complex and
interactive with many components of the rater's cognitive processing
structure, to the point that it is essentially imbedded in the
processing pattern. That is, the data suggest that, even with
considerable advance discussion (and, in the case of the two raters who
sat for the "talk-aloud" session in the afternoon, a half-day of
hands-on experience), raters may have had some difficulty adapting to
the specific requirements of the characteristic-specific rubrics,
tending to slip back into their more familiar holistic scoring paradigm,
even as they *tried* to accommodate the characteristic-specific scoring
task presented. Evidence of this includes statements suggesting
reversion due to frustration, such as "oh, well, I'll give this a four,"
and statements suggesting uncertainty of reference: "well, I *think*
this is a three."

Second, not surprisingly, the protocols suggest dramatically differing
personal styles of reaching a scoring conclusion. Not only did the
amount of time to produce a score vary widely across both raters and
rubrics, but the engaged processes often seemed incongruous with the
task presented. This is not an unusual finding in "talk-aloud"
protocols, however, since the context-lending descriptions of the
processes are absent; the participants had not had more than minimal
prior practice and, as such, had not thoroughly processed how to "keep
talking" throughout the session.

There was some evidence that raters were looking for words, phrases,
transitional terms, and other indicators of discourse, syntax and
content. Occasionally, there was reiteration of terms used in the
rubrics, indicating that the rater was "mindful of," if not actually
looking for, evidence of the characteristic of interest. Many of these
indicators were recognizable as features that *e-rater* targets; others
were terms that likely co-occur with one or more features. Overall,
while it is largely the case that the raters were *not* actually
counting occurrences of indicator cues representing *e-rater* features,
they were tracing qualities that incorporate such features. The degree
to which this was consistently done, however, is impossible to determine
more precisely from the protocols.

Next, each rater participated in what Ericsson and Simon (1980) refer to
as a "social verbalization" exercise, in which the rater scores an essay
interactively with the researcher and, in the final step, with another
rater as well. This exercise proved to be most useful as a training
process and, in retrospect, probably would have been more beneficial to
all four raters if it had been conducted at the beginning of the day. In
this exercise, the raters finally "learned" what the researcher had in
mind as the right way to score each characteristic, and compared this to
the process they had each developed independently for scoring each
characteristic up to that point. Several of the raters indicated, after
this exercise, that they had significantly changed their understanding
of what was being sought by one or more of the characteristics
(particularly the topic-relevant content characteristic) and, therefore,
should revisit some of the scorings they had done earlier in the day.
Each rater that indicated the inclination to do this also felt that she
could do so in a consistent manner, without biasing either her earlier
scoring or the scoring yet to be done.

Each day concluded with a short debriefing session, preceded by a
"normalization" period of silent, independent scoring intended to get
each rater "into the groove." At this point, several of the raters
mentioned the interplay between syntax and certain forms of discourse.
Specifically, when an essay writer would make a certain type of
assertion in the essay, the raters would expect to see the associated
use of certain types of syntactic structures. The absence of such syntax
in such an instance would render the assertion superficial. While essays
with and without such syntactic variety were both seen, clearly the
essays containing the syntactic variety associated with that type of
discourse were viewed by the raters as superior.

*E-rater* was later run on each essay in the cross-validation samples,
both in its original generic form and its subscore-models form, for each
essay prompt type. The *e-rater*-to-expert rater proportions of
agreement and correlations are presented in Table 12, along with
estimates of Cohen's Kappa, a "chance-corrected measure of agreement"
(Fleiss, 1981, p. 217) that is commonly reported in results of ratings.
Additionally, the correlations of *e-rater* scores between the three
individual prompt-specific models and the generic model for each essay
type are presented as a note to each section of Table 12.

+----------------------+-------+-------+-------+-------+-------+-------+
| **Table 12**         |       |       |       |       |       |       |
|                      |       |       |       |       |       |       |
| Agreement of         |       |       |       |       |       |       |
| *e-rater* Scores     |       |       |       |       |       |       |
| with Rater-Assigned  |       |       |       |       |       |       |
| Scores: "Issue"      |       |       |       |       |       |       |
| Generic Models (*N*  |       |       |       |       |       |       |
| = 200 essays).       |       |       |       |       |       |       |
+----------------------+-------+-------+-------+-------+-------+-------+
| ## [Cor              | ## [  |       |       |       |       |       |
| responding *e-rater* | Rater |       |       |       |       |       |
|  score:]{.underline} | -Assi |       |       |       |       |       |
|                      | gned  |       |       |       |       |       |
|                      | Score |       |       |       |       |       |
|                      | s:]{. |       |       |       |       |       |
|                      | under |       |       |       |       |       |
|                      | line} |       |       |       |       |       |
|                      |       |       |       |       |       |       |
|                      | #     |       |       |       |       |       |
|                      | #  [H |       |       |       |       |       |
|                      | olist |       |       |       |       |       |
|                      | ic]{. |       |       |       |       |       |
|                      | under |       |       |       |       |       |
|                      | line} |       |       |       |       |       |
|                      |  [ISS |       |       |       |       |       |
|                      | UE-1] |       |       |       |       |       |
|                      | {.und |       |       |       |       |       |
|                      | erlin |       |       |       |       |       |
|                      | e} [I |       |       |       |       |       |
|                      | SSUE- |       |       |       |       |       |
|                      | 2]{.u |       |       |       |       |       |
|                      | nderl |       |       |       |       |       |
|                      | ine}  |       |       |       |       |       |
|                      | [ISSU |       |       |       |       |       |
|                      | E-3]{ |       |       |       |       |       |
|                      | .unde |       |       |       |       |       |
|                      | rline |       |       |       |       |       |
|                      | } [IS |       |       |       |       |       |
|                      | SUE-4 |       |       |       |       |       |
|                      | ]{.un |       |       |       |       |       |
|                      | derli |       |       |       |       |       |
|                      | ne} [ |       |       |       |       |       |
|                      | ISSUE |       |       |       |       |       |
|                      | -5]{. |       |       |       |       |       |
|                      | under |       |       |       |       |       |
|                      | line} |       |       |       |       |       |
+----------------------+-------+-------+-------+-------+-------+-------+
| Exact agreement with | #     | #     | ##    | #     | #     | #     |
| one or both raters   | # .46 | # .40 |   .35 | # .45 | # .46 | # .34 |
+----------------------+-------+-------+-------+-------+-------+-------+
| Adjacent agreement   | #     | #     | #     | #     | #     | #     |
| with both raters     | # .40 | # .44 | # .52 | # .40 | # .42 | # .48 |
+----------------------+-------+-------+-------+-------+-------+-------+
| Total:               | #     | #     | #     | #     | #     | #     |
| Exact-plus-adjacent  | # .86 | # .84 | # .87 | # .85 | # .88 | # .82 |
| agreement            |       |       |       |       |       |       |
+----------------------+-------+-------+-------+-------+-------+-------+
| ## Kappa             | #     | #     | ##    | #     | #     | ##    |
|                      | # .24 | # .12 |  -.02 | # .04 | # .23 |  -.01 |
+----------------------+-------+-------+-------+-------+-------+-------+
| ## Correlation with  | #     | #     | ##    | #     | #     | ##    |
| two-rater mean score | #  .5 | #  .3 | .18\* | #  .2 | #  .4 |  -.01 |
|                      | 6\*\* | 2\*\* |       | 6\*\* | 4\*\* |       |
+----------------------+-------+-------+-------+-------+-------+-------+
| ##                   | #     | #     | #     | #     | #     | ##    |
|  Squared Correlation | # .31 | # .10 | # .03 | # .07 | # .19 | \<.01 |
+----------------------+-------+-------+-------+-------+-------+-------+
| [Interrater          | ##    | ##    | ##    | ##    | #     | ##    |
| subsample (*n* =     |  1.00 |  1.00 |  1.00 |  1.00 | # .95 |   .95 |
| 20):]{.underline}    |       |       |       |       |       |       |
|                      |       |       |       |       |       |       |
| Exact-plus-adjacent  |       |       |       |       |       |       |
| agreement:           |       |       |       |       |       |       |
+----------------------+-------+-------+-------+-------+-------+-------+
| ## Kappa             | ##    | ##    | ##    | #     | #     | ##    |
|                      |   .83 |   .77 |   .79 | # .76 | # .70 |   .64 |
+----------------------+-------+-------+-------+-------+-------+-------+
| Correlation          | #     | #     | #     | #     | #     | ##    |
|                      | #  .9 | #  .8 | #  .8 | #  .8 | #  .8 | .66\* |
|                      | 5\*\* | 7\*\* | 9\*\* | 3\*\* | 0\*\* |       |
+----------------------+-------+-------+-------+-------+-------+-------+

^\*^ statistically significant at the α = .05 level.

^\*\*^ statistically significant at the α = .01 level.

+--------------------+------+------+------+------+------+------+------+
| **Table 4.14.**    |      |      |      |      |      |      |      |
| Agreement of       |      |      |      |      |      |      |      |
| *e-rater* Scores   |      |      |      |      |      |      |      |
| with               |      |      |      |      |      |      |      |
| Rater-Assigned     |      |      |      |      |      |      |      |
| Scores: "Argument" |      |      |      |      |      |      |      |
| Generic Models     |      |      |      |      |      |      |      |
| (*N* = 200         |      |      |      |      |      |      |      |
| essays).           |      |      |      |      |      |      |      |
+--------------------+------+------+------+------+------+------+------+
| ## [Corresp        | ##   |      |      |      |      |      |      |
| onding *e-rater* s | [Rat |      |      |      |      |      |      |
| core:]{.underline} | er-A |      |      |      |      |      |      |
|                    | ssig |      |      |      |      |      |      |
|                    | ned  |      |      |      |      |      |      |
|                    | Scor |      |      |      |      |      |      |
|                    | es:] |      |      |      |      |      |      |
|                    | {.un |      |      |      |      |      |      |
|                    | derl |      |      |      |      |      |      |
|                    | ine} |      |      |      |      |      |      |
|                    |      |      |      |      |      |      |      |
|                    | ##   |      |      |      |      |      |      |
|                    |   [H |      |      |      |      |      |      |
|                    | olis |      |      |      |      |      |      |
|                    | tic] |      |      |      |      |      |      |
|                    | {.un |      |      |      |      |      |      |
|                    | derl |      |      |      |      |      |      |
|                    | ine} |      |      |      |      |      |      |
|                    |  [AR |      |      |      |      |      |      |
|                    | G-1] |      |      |      |      |      |      |
|                    | {.un |      |      |      |      |      |      |
|                    | derl |      |      |      |      |      |      |
|                    | ine} |      |      |      |      |      |      |
|                    |  [AR |      |      |      |      |      |      |
|                    | G-2] |      |      |      |      |      |      |
|                    | {.un |      |      |      |      |      |      |
|                    | derl |      |      |      |      |      |      |
|                    | ine} |      |      |      |      |      |      |
|                    |  [AR |      |      |      |      |      |      |
|                    | G-3] |      |      |      |      |      |      |
|                    | {.un |      |      |      |      |      |      |
|                    | derl |      |      |      |      |      |      |
|                    | ine} |      |      |      |      |      |      |
|                    |  [AR |      |      |      |      |      |      |
|                    | G-4] |      |      |      |      |      |      |
|                    | {.un |      |      |      |      |      |      |
|                    | derl |      |      |      |      |      |      |
|                    | ine} |      |      |      |      |      |      |
|                    |  [AR |      |      |      |      |      |      |
|                    | G-5] |      |      |      |      |      |      |
|                    | {.un |      |      |      |      |      |      |
|                    | derl |      |      |      |      |      |      |
|                    | ine} |      |      |      |      |      |      |
|                    |  [AR |      |      |      |      |      |      |
|                    | G-6] |      |      |      |      |      |      |
|                    | {.un |      |      |      |      |      |      |
|                    | derl |      |      |      |      |      |      |
|                    | ine} |      |      |      |      |      |      |
+--------------------+------+------+------+------+------+------+------+
| Exact agreement    | ##   | ##   | ##   | ##   | ##   | ##   | ##   |
| with one or both   |  .57 |  .43 |  .30 |  .32 |  .31 |  .40 |  .29 |
| raters             |      |      |      |      |      |      |      |
+--------------------+------+------+------+------+------+------+------+
| Adjacent agreement | ##   | ##   | ##   | ##   | ##   | ##   | ##   |
| with both raters   |  .36 |  .50 |  .59 |  .62 |  .49 |  .52 |  .50 |
+--------------------+------+------+------+------+------+------+------+
| Total:             | ##   | ##   | ##   | ##   | ##   | ##   | ##   |
| E                  |  .93 |  .93 |  .89 |  .94 |  .80 |  .92 |  .79 |
| xact-plus-adjacent |      |      |      |      |      |      |      |
| agreement          |      |      |      |      |      |      |      |
+--------------------+------+------+------+------+------+------+------+
| ## Kappa           | ##   | ##   | ##   | ##   | ##   | ##   | ##   |
|                    |  .40 |  .21 |  .05 |  .10 |  .01 |  .11 |  .05 |
+--------------------+------+------+------+------+------+------+------+
| ## C               | ##   | ##   | ##   | ##   | ##   | ##   | ##   |
| orrelation with tw |  .74 |  .39 |  .19 |  .49 | -.08 |  .39 |  .36 |
| o-rater mean score | \*\* | \*\* | \*\* | \*\* |      | \*\* | \*\* |
+--------------------+------+------+------+------+------+------+------+
| ## S               | ##   | ##   | ##   | ##   | #    | ##   | ##   |
| quared Correlation |  .55 |  .15 |  .04 |  .25 | #  \ |  .15 |  .13 |
|                    |      |      |      |      | <.01 |      |      |
+--------------------+------+------+------+------+------+------+------+
| [Interrater        | ##   | ##   | ##   | ##   | ##   | ##   | ##   |
| subsample (*n* =   |  .95 |  .85 |  .95 |  .85 |  .70 |  .95 |  .85 |
| 20):]{.underline}  |      |      |      |      |      |      |      |
|                    |      |      |      |      |      |      |      |
| E                  |      |      |      |      |      |      |      |
| xact-plus-adjacent |      |      |      |      |      |      |      |
| agreement          |      |      |      |      |      |      |      |
+--------------------+------+------+------+------+------+------+------+
| ## Kappa           | ##   | ##   | ##   | ##   | ##   | ##   | ##   |
|                    |  .69 |  .57 |  .61 |  .58 |  .37 |  .61 |  .55 |
+--------------------+------+------+------+------+------+------+------+
| Correlation        | ##   | ##   | ##   | ##   | #    | ##   | ##   |
|                    |  .81 |  .72 |  .78 |  .75 | #  . |  .77 |  .67 |
|                    | \*\* | \*\* | \*\* | \*\* | 51\* | \*\* | \*\* |
+--------------------+------+------+------+------+------+------+------+

^\*^ statistically significant at the α = .05 level.

^\*\*^ statistically significant at the α = .01 level.

**Discussion**

Four topics are discussed here: the *implications* that the findings
have on the research question asked; the *limitations* of the study
arising from both gaps in theory and shortcomings of the implementations
of the methods proposed; the *educational importance* of the research;
and *potential extensions* of the research in the future.

[Implications of the Findings]{.underline}: This investigation sought
evidence for *e-rater* score "internal" validity from three sources: the
underlying factor structure of the *e-rater* scoring models; the degree
of correlation between expert rater scores and *e-rater* scores, both
holistically and on specific characteristics of writing that both expert
raters and *e-rater* attend to in an essay; and the nature of the
testimonials provided by expert raters as they examine essays,
holistically as well as by characteristic. The first of these sources,
the underlying factor structure of the *e-rater* scoring models,
provided some evidence that *e-rater* counts features that do in fact
signal the presence of desirable writing characteristics, such as
topic-relevant content, syntactic variety, and skillful use of discourse
in an essay. Evidence of this lies in the close parallel between the way
that ETS Test Development experts and the GRE Writing Assessment
holistic scoring guides define these characteristics and the way the
corresponding *e-rater* features group in order under analogous
characteristics. That is, the evidence gathered supports the statement
by ETS Technologies, Inc. that *e-rater* is designed to reflect the
writing qualities specified in the GRE Writing Assessment holistic
scoring guides. Obviously, *e-rater* does not *read* an essay, so it
cannot "look for" or "evaluate" writing qualities. However, *e-rater*
can, and does in some instances, detect evidentiary traces, the
proverbial "breadcrumbs in the path," that signal these qualities, using
its own version of the characteristics.

The patterns of score proportions of agreement and correlations, as
reported in Table 7, are a second source of evidence for *e-rater* score
"internal" validity. The factor analysis results had implied that the
first factor in each model, named characteristic ISSUE-1/ARG-1, having
the highest feature loadings, on average, and explaining by far the
largest percentage of variance in the data (before rotation), would
drive the *e-rater* score for either essay type. This appears to be
borne out in Table 7; no other characteristic produced scores for either
essay type that agree as often or correlate as highly between expert
raters and *e-rater*. The syntactic variety characteristic,
ISSUE-3/ARG-5, produced the second-highest correlation of all for both
essay types and the highest kappa of all the "argument" characteristics.
However, the topic-relevant content characteristic, ISSUE-5/ARG-4, did
not fare as well, even though the researcher had presumed beforehand
that identifying on-topic content in an essay would be a fairly
straightforward task. The difficulties several raters cited in
dissociating essay *content* from the rater's interpretation of the
writer's *intent*, as mentioned earlier, might be to blame for this.

The transcripts of raters' recorded conversations with the researcher
provide the most compelling evidence both for and against the "internal"
validity of *e-rater* scores; that is, their construct relevance and
content representativeness, their reflection of the task and domain
structures engaged in scoring an essay, and their agreement with the
process models used by experts to generate essay scores. Specifically,
all four raters agreed that the syntactic variety characteristic,
ISSUE-3/ARG-5, and the topic-relevant content characteristic,
ISSUE-5/ARG-4, are relevant, identifiable in an essay, reflective of
what a rater should look for in an essay, and either explicitly or
implicitly parts of their processing schemas when rating an essay. To a
lesser extent, the raters viewed the principal discourse characteristic,
ISSUE-1/ARG-1, the same way, although they had reservations about the
efficacy of assessing this characteristic in isolation from the other
qualities of the essay. Conversely, all of the raters viewed two of the
characteristics, ISSUE-2/ARG-3 and ISSUE-4/ARG-6, as being inappropriate
with respect to any aspect of "internal" score validity.

Taken together, the three sources present a "mixed bag" of evidence for
and against the "internal" validity of *e-rater* essay scores. The
strongest evidence for "internal" validity comes from the same source as
the strongest evidence against it: the raters themselves. From the
raters' perspective, simpler scoring models that leave out certain
features of the current *e-rater* models would likely be substantially
more likely to be accepted by expert raters, and particularly those who
are college faculty in the liberal arts and sciences or the humanities,
high school English teachers, or others in related professions. The
statistics arising from the present study suggest that simpler *e-rater*
models might do just as well at agreeing with the scores of expert
raters, although the proportion of total variance in the expert rater
scores explained by the *e-rater* scores might decrease from an already
modest level.

[Limitations of the Study]{.underline}: The first limitation encountered
was the limited amount of data available for the study. Since its
inception in October, 1999, the GRE Writing Assessment examinee
population has grown slowly, so the number of essays available (for a
selected set of prompts) for study purposes was relatively small. While
the sample sizes available for model building and cross-validation meet
the minimums suggested by Stevens (1996), larger samples might have
improved the agreement between the initial and cross-validation models.

## The relatively low model-to-data fits suggest that the explanatory power of the hypothesized factor structures does not go far in helping to answer the score validity question. The researcher postulates that this is largely due to what Muraki et al. (2000) found and what the correlations reported in the present study suggest, namely, that there is such a high level of dynamic interaction between writing characteristics that any attempt to isolate them, essentially taking them out of their natural context, produces a sort of "reverse synergy," yielding parts that are far less useful individually than they are together. This may well be an insurmountable limitation to this kind of investigation.

Another limitation to the study arises from the lack of adequate
training and practice time the four raters received prior to performing
the characteristic-specific scoring tasks. Optimally, raters involved in
a study of this kind would be introduced to the tasks and materials well
in advance and given ample opportunity to practice on exercise essays
and to ask questions. Unfortunately, in the present study, the training
and learning were going on as the actual study samples were scored.
Fortunately, ETS Test Development staff took care to select experienced
raters who were up to the tasks. The high level of expertise and
adaptability of these raters likely mitigated the task-specific
inexperience limitation posed here.

Still another limitation of the present study is that the
characteristic-specific scoring rubrics developed for this study, while
modeled after existing GRE Writing Assessment holistic scoring rubrics,
were not adequately pre-tested before being used in the study. Again,
the limitation is partially mitigated by the high level of expertise
employed in the development of the rubrics, which included the active
participation of two leading essay test development professionals.
However, the tentativeness expressed by these experts with respect to
some of the rubrics, combined with the unforeseen need to make
considerable revisions to the rubrics following the first day of the
face-to-face portion of the project, lead the researcher to believe that
there may have been nontrivial inaccuracies in the rubrics, resulting in
inconsistent application of the rubrics across raters. While the various
discussion exercises described earlier were intended partly to address
this problem, it is still likely that inconsistencies in using the
rubrics contributed substantially to the low correlations produced in
the characteristic-specific scorings.

## [Educational Importance of the Research]{.underline}: Commencing with the work of Carlson and Ward (1988), ETS has for many years pursued the development of computer-based free-response scoring tools. As reducing by half the number of expert rater scorings required may produce considerable cost savings for essay testing programs, the appeal of these tools is certain to grow, and perhaps even extend beyond testing organizations. The present study represents a first attempt at establishing a protocol for demonstrating the "internal" validity of computer-assisted essay scores. (The term "computer-assisted" reflects the current practice with the GMAT of using *e-rater* strictly as a reliability check for one expert rater score; when that one expert rater and *e-rater* disagree, a second expert rater is called upon to adjudicate the discrepancy. Scoring of the GRE Writing Assessment is not computer-assisted.)

[Potential Extensions of the Research]{.underline}: As the pool of
available GRE Writing Assessment essays grows, a more comprehensive
follow-up effort, addressing all or most of the limitations cited
earlier, could produce results that confirm and significantly augment
the findings presented in the present study. Also, it will become
possible to investigate the behavior of expert rater and *e-rater*
scores under specific anomalous circumstances, such as with "1" and "6"
essays alone, with adjudicated essays, and with contrasting samples of
operational essays and essays commissioned especially for a study.
Separately, additional research into parallels between the cognitive
processing of expert raters and the electronic processing of *e-rater*
may provide further insight to strategies for improving *e-rater* from
an "internal" score validity perspective.

**References**

Braun, H. I. (1988). Understanding scoring reliability: Experiments in
calibrating essay readers. *Journal of Educational Statistics, 13* (1),
1-18.

Breland, H. K. (1983). *The direct assessment of writing skill: A
measurement review* (Research Report 83-32). Princeton, NJ: Educational
Testing Service.

Breland, H. K., & Jones, R. J. (1988). *Remote scoring of essays*
(Research Report 88-04). Princeton, NJ: Educational Testing Service.

Burstein, J. C., & Chodorow, M. (1999, June). *Automated essay scoring
for nonnative English speakers*. Paper presented at the Joint Symposium
of the Association of Computational Linguistics and the International
Association of Language Learning Technologies, Workshop on
Computer-Mediated Language Assessment and Evaluation of Natural Language
Processing, College Park, MD.

Burstein, J. C., Kukich, K., Wolff, S. E., Lu, C., & Chodorow, M.
(1998a, April). *Computer analysis of essays.* Paper presented at the
annual meeting of the National Council on Measurement in Education, San
Diego, CA.

Burstein, J. C., Kukich, K., Wolff, S. E., Lu, C., & Chodorow, M.
(1998b, August). *Enriching automated scoring using discourse marking*.
Paper presented at the Workshop on Discourse Relations and Discourse
Marking at the annual meeting of the Association of Computational
Linguistics, Montréal, Canada.

Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant
validation by the multitrrait-multimethod matrix. *Psychological
Bulletin, 56,* 81-105.

Carlson, S. B., & Ward, W. C. (1988). *A new look at formulating
hypothesis items* (GRE Board Professional Report 85-14P). Princeton, NJ:
Educational Testing Service.

Cook, T. D., & Campbell, D. T. (1979). *Quasi-experimentation: Design
and analysis issues for field settings.* Boston: Houghton-Mifflin.

Cronbach, L. J., & Gleser, G. C. (1965). *Psychological tests and
personnel decisions* (2nd ed.). Urbana, IL: University of Illinois
Press.

Embretson, S. (1983). Construct validity: Construct representation
versus nomothetic span. *Psychological Bulletin, 93,* 179-197.

Ericsson, K. A., & Simon, H.A. (1993). *Protocol analysis: Verbal
reports as data* (Rev. ed.). Boston: MIT Press.

Fleiss, J. L. (1981). *Statistical methods for rates and proportions*
(2nd ed.). New York: Wiley.

Hunter, J. E., Schmidt, F. L., & Jackson, C. B. (1982). *Advanced
meta-analysis: Quantitative methods for cumulating research findings
across studies*. Newbury Park, CA: Sage.

Kaplan, R. M., Wolff, S. E., Burstein, J. C., Lu, C., Rock, D. A., &
Kaplan, B. A. (1998). *Scoring essays automatically using surface
features* (ETS Research Report 98-39)*.* Princeton, NJ: Educational
Testing Service.

Keith, T. Z. (1999, April). *Newest evidence for construct validity in
PEG.* Paper presented at the annual meeting of the National Council on
Measurement in Education, Montréal, Canada.

Lennon, R. T. (1956). Assumptions underlying the use of content
validity. *Educational and Psychological Measurement, 16,* 294-304.

Loevinger, J. (1957). Objective tests as instruments of psychological
theory. *Psychological Reports, 3,* 635-694.

Messick, S. (1987). The once and future issues of validity: Assessing
the meaning and consequences of measurement. In H. Wainer & H. Braun
(Eds.), *Test validity* (pp. 33-45). Mahwah, NJ: Erlbaum.

Messick, S. (1989b). Validity. In Linn, R. (Ed.), *Educational
measurement* (3rd ed.). Washington, DC: American Council on Education.

Messick, S. (1995). Validity of psychological assessment: Validation of
inferences from persons' responses and performances as scientific
inquiry into score meaning. *American Psychologist, 50* (9), 741-749.

Messick, S. (1999). Technology and the future of higher education
assessment. In Messick, S. (Ed.), *Assessment in higher education:
Issues of access, student development, and public policy* (pp. 245-254).
Mahwah, NJ: Erlbaum.

Muraki, E., Lee, Y., & Kim, R. M. (2000). *Factor analysis of NAEP
e-rater data*. Unpublished manuscript.

Page, E. B. (1994). Computer grading of student prose, using modern
concepts and software. *Journal of Experimental Education, 62* (2),
127-142.

Page, E. B., & Petersen, N. S. (1995). The computer moves into essay
grading: Updating the ancient test. *Phi Delta Kappan, 76* (7), 561-565.

Page, E. B., Poggio, J. P., & Keith, T. Z. (1997, March). *Computer
analysis of student essays: Finding trait differences in student
profile*. Paper presented at the annual meeting of the American
Educational Research Association, Chicago, IL.

Powers, D. E., Burstein, J. C., Chodorow, M., Fowles, M. E., & Kukich K.
(2000). *Comparing the validity of automated and human essay scoring*
(GRE Board Research Report 98-08aR). Princeton, NJ: Educational Testing
Service.

Shermis, M. D., Koch, C. M., Page, E. B., Keith, T. Z., & Harrington, S.
(1999, April). *Trait ratings for automated essay grading.* Paper
presented at the annual meeting of the National Council on Measurement
in Education, Montréal, Canada.

Shulman, L. S. (1970). Reconstruction of educational research. *Review
of Educational Research, 40,* 371-396.

Stevens, J. (1996). *Applied multivariate statistics for the Social
Sciences* (3rd Ed.). Mahwah, NJ: Erlbaum.