System and method for computer-based automatic essay scoring

A method of grading an essay using an automated essay scoring system is provided. The method comprises the automated steps of (a) parsing the essay to produce parsed text, wherein the parsed text is a syntactic representation of the essay, (b) using the parsed text to create a vector of syntactic features derived from the essay, (c) using the parsed text to create a vector of rhetorical features derived from the essay, (d) creating a first score feature derived from the essay, (e) creating a second score feature derived from the essay, and (f) processing the vector of syntactic features, the vector of rhetorical features, the first score feature, and the second score feature to generate a score for the essay. The essay scoring system comprises a Syntactic Feature Analysis program which creates a vector of syntactic features of the electronic essay text, a Rhetorical Feature Analysis program which creates a vector of rhetorical features of the electronic essay text, an EssayContent program which creates a first Essay Score Feature, an ArgContent program which creates a second Essay Score Feature, and a scoring engine which generates a final score for the essay from the vector of syntactic features, the vector of rhetorical features, the first score feature, and the second score feature.

FIELD OF THE INVENTION
 This invention generally relates to the field of computer-based test
 scoring systems, and more particularly, to automatic essay scoring
 systems.
 BACKGROUND OF THE INVENTION
 For many years, standardized tests have been administered to examinees for
 various reasons such as for educational testing or for evaluating
 particular skills. For instance, academic skills tests, e.g., SATs, LSATs,
 GMATs, etc., are typically administered to a large number of students.
 Results of these tests are used by colleges, universities and other
 educational institutions as a factor in determining whether an examinee
 should be admitted to study at that particular institution. Other
 standardized testing is carried out to determine whether or not an
 individual has attained a specified level of knowledge, or mastery, of a
 given subject. Such testing is referred to as mastery testing, e.g.,
 achievement tests offered to students in a variety of subjects, and the
 results are used for college credit in such subjects.
 Many of these standardized tests have essay sections. These essay portions
 of an exam typically require human graders to read the wholly unique essay
 answers. As one might expect, essay grading requires a significant number
 of work-hours, especially compared to machine-graded multiple choice
 questions. Essay questions, however, often provide a more well-rounded
 assessment of a particular test taker's abilities. It is, therefore,
 desirable to provide a computer-based automatic scoring system.
 Typically, graders grade essays based on scoring rubrics, i.e.,
 descriptions of essay quality or writing competency at each score level.
 For example, the scoring guide for a scoring range from 0 to 6
 specifically states that a "6" essay "develops ideas cogently, organizes
 them logically, and connects them with clear transitions." A human grader
 simply tries to evaluate the essay based on descriptions in the scoring
 rubric. This technique, however, is subjective and can lead to
 inconsistent results. It is, therefore, desirable to provide an automatic
 scoring system that is accurate, reliable and yields consistent results.
 Literature in the field of discourse analysis points out that lexical
 (word) and structural (syntactic) features of discourse can be identified
 (Mann, William C. and Sandra A. Thompson (1988): Rhetorical Structure
 Theory: Toward a functional theory of text organization, Text 8(3),
 243-281) and represented in a machine, for computer-based analysis (Cohen,
 Robin: A computational theory of the function of clue words in argument
 understanding, in "Proceedings of 1984 International Computational
 Linguistics Conference." California, 251-255 (1984); Hovy, Eduard, Julia
 Lavid, Elisabeth Maier, Vibhu Nettal and Cecile Paris: Employing Knowledge
 Resources in a New Text Planner Architecture, in "Aspects of Automated NL
 Generation," Dale, Hony, Rosner and Stoch (Eds), Springer-Veriag Lecture
 Notes in Al no. 587, 57-72 (1992); Hirschberg, Julia and Diane Litman:
 Empirical Studies on the Disambiguation of Cue Phrases, in "Computational
 Linguistics" (1993), 501-530 (1993); and Vander Linden, Keith and James H.
 Martin: Expressing Rhetorical Relations in Instructional, Text: A Case
 Study in Purpose Relation in "Computational Linguistics" 21(1), 29-57
 (1995)).
 Previous work in automated essay scoring, such as by Page, E. B. and N.
 Petersen: The computer moves into essay grading: updating the ancient
 test. Phi Delta Kappa; March, 561-565 (1995), reports that predicting
 essay scores using surface feature variables, e.g., the fourth root of the
 length of an essay, shows correlations as high as 0.78 between a single
 human rater (grader) score and machine-based scores for a set of PRAXIS
 essays. Using grammar checker variables in addition to word counts based
 on essay length yields up to 99% agreement between machine-based scores
 that match human rater scores within 1 point on a 6-point holistic rubric.
 These results using grammar checker variables have added value since
 grammar checker variables may have substantive information about writing
 competency that might reflect rubric criteria such as, essay is free from
 errors in mechanics, usage and sentence structure.
 SUMMARY OF THE INVENTION
 A method of grading an essay using an automated essay scoring system is
 provided. The method comprises the steps of (a) parsing the essay to
 produce parsed text, wherein the parsed text is a syntactic representation
 of the essay, (b) using the parsed text and discourse-based heuristics to
 create a vector of syntactic features derived from the essay, (c) using
 the parsed text to create a vector of rhetorical features derived from the
 essay, (d) creating a first score feature derived from the essay, (e)
 creating a second score feature derived from the essay, and (f) processing
 the vector of syntactic features, the vector of rhetorical features, the
 first score feature, and the second score feature to generate a score for
 the essay.
 In a preferred embodiment, the method further comprises the step of (g)
 creating a predictive feature set for the test question, where the
 predictive feature set represents a model feature set for the test
 question covering a complete range of scores of a scoring guide for the
 test question, wherein in step (f), a scoring formula may be derived from
 the predictive feature set and the score for the essay may be assigned
 based on the scoring guide. Preferably, a batch of original essays, which
 are essays of a known score to a test question, are used in accordance
 with the model feature of the invention to create the predictive feature
 set. Creating the predictive feature set in this manner comprises the
 steps of repeating steps (a) through (f) for the batch of original essays
 and processing the vector of syntactic features, the vector of rhetorical
 features, the first score feature, and the second score feature for each
 original essay using a linear regression to generate the predictive
 feature set for the test question.
 Preferably, each essay is already in the form of electronic essay text as
 in the case with on-line essay testing. If this is not the case, however,
 then the method of the present invention further comprises the step of
 converting the essay into the form of electronic essay text.
 A computer-based automated essay scoring system for grading an essay also
 is provided. The essay scoring system comprises a Syntactic Feature
 Analysis program which creates a vector of syntactic features of the
 electronic essay text, a Rhetorical Feature Analysis program which creates
 a vector of rhetorical features of the electronic essay text, an
 EssayContent program which creates a first Essay Score Feature, an
 ArgContent program which creates a second Essay Score Feature, and a score
 generator which generates a final score for the essay from the vector of
 syntactic features, the vector of rhetorical features, the first score
 feature, and the second score feature.
 In a preferred embodiment, the essay scoring system further comprising a
 parser for producing a syntactic representation of each essay for use by
 the Syntactic Feature Analysis program and the Rhetorical Feature Analysis
 program. In another preferred embodiment, the essay scoring system further
 comprising a Stepwise Linear Regression program which generates a
 predictive feature set representing a model feature set that is predictive
 of a range of scores for the test question which is provided to the
 scoring engine for use in assessing the final score for the essay.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
 A computer-based system designed to automatically score essay responses is
 described herein. Solely for illustrative purposes, the following
 description of the invention focuses on the standardized GMAT Analytical
 Writing Assessments: (a) Analysis of an Argument (Argument essays) and (b)
 Analysis of an Issue (Issue essays) item types, examples of which are
 shown in Appendix A1 and Appendix A2, respectively. The system of the
 present invention, named e-rater as in Electronic Essay Rater,
 automatically analyzes several features of an essay and scores the essay
 based on the features of writing as specified in holistic scoring rubrics
 (descriptions of essay quality or writing competency at each score level
 of a 6-point scoring guide used by several standardized exams such as the
 GMAT, with 6 being the best score).
 The present system automatically rates essays using features that reflect
 the 6-point holistic rubrics used by human raters to assign scores to
 essay responses. E-rater is completely automated so that it can be quickly
 moved into an operationally-ready mode and uses rubric-based features to
 evaluate essay responses, such as rhetorical structure, vocabulary and
 syntactic features.
 E-rater uses a hybrid feature methodology. It incorporates several
 variables that are derived statistically, or extracted through Natural
 Language Processing (NLP) techniques. As described in this specification,
 e-rater uses four sets of critical feature variables to build the final
 linear regression model used for predicting scores, referred to as
 predictor variables. All predictor variables and counts of predictor
 variables are automatically generated by several independent computer
 programs. For argument and issue essay types, all relevant information
 about the variables are introduced into a stepwise linear regression in
 order to evaluate the predictive variables, i.e., the variables that
 account for most of the variation between essays at different score
 intervals. Variables included in e-rater's final score prediction model
 for argument and issue essays are: (a) structural features, (b) rhetorical
 structure analyses, (c) content vector analyses, and (d) content vector
 analyses by argument (argument vector analyses). A conceptual rationale
 and a description of how each variable is generated is described below.
 A. Structural Features
 The scoring guides for both argument and issue essays indicate that one
 feature used to rate an essay is "syntactic variety." Syntactic structures
 in essays can be identified using NLP techniques. In the present
 invention, all sentences in the essay responses are parsed. The parser
 takes a sentence string as input and returns a syntactically analyzed
 version of a sentence, as illustrated in Table 1. Examination of syntactic
 structures in an essay response yields information about the "syntactic
 variety" in the essay. For example, information about what types of
 clauses or verb types can reveal information about "syntactic variety." In
 Table 1, DEC is a declarative sentence, NP is a Noun phrase, AJP is an
 adjective phrase, ADJ is an adjective, NOUN is a noun, PP is a
 prepositional phrase, PREP is a preposition, INFCL is an infinitive
 clause, DETP is a determiner phrase, and CHAR is a character.
 TABLE 1
 Syntactic Parse for a Sentence from an Issue Essay
 Young people often express discontent with taxation levels
 to support the aged.
 DEC1 1NP1 AJP1 ADJ1* "Young"
 NOUN1* "people"
 AVPI ADV1* "often"
 VERB1* "express"
 NP2 NOUN2* "discontent"
 PP1 PP2 PREP1* "with"
 NP3 NOUN3* "taxation"
 NOUN4* "levels"
 INFCL1 INFTO1 PREP2* "to"
 VERB2* "support"
 NP4 DETP1 ADJ2* "the"
 NOUN5* "aged"
 CHAR1 "."
 A program for examining syntactic structure was run on approximately 1,300
 essays. The program counted the number of complement clauses, subordinate
 clauses, infinitive clauses, relative clauses and the subjunctive modal
 auxiliary verbs such as would, could, should, might and may, for each
 sentence in an essay. A linear regression analysis then selected the
 variables in Table 2 as predictive variables for the final score
 prediction model. By using these predictive variables, a vector of
 syntactic counts (42 in FIG. 1) for each essay is generated and is used by
 e-rater is the final scoring.
 TABLE 2
 Grammatical Structural Variables Used in e-rater to
 Predict Essay Scores
 Argument Essays Total Number of Modal Auxiliary
 Verbs
 Ratio of Complement Clauses Per
 Sentence
 Issue Essays Total Number of Infinitive
 Clauses
 Total Number of Modal Auxiliary
 Verbs/Paragraph
 B. Rhetorical Structure Analysis
 In both argument and issue essays, the scoring guides indicate that an
 essay will receive a score based on the examinee's demonstration of a
 well-developed essay. For the argument essay, the scoring guide states
 specifically that a "6" essay "develops ideas cogently, organizes them
 logically, and connects them with clear transitions." For the issue essay,
 a "6" essay "develops a position on the issue with insightful reasons . .
 . " and the essay "is clearly well-organized."
 Language in holistic scoring guides, such as "cogent", "logical,"
 "insightful," and "well-organized" have "fuzzy" meaning because they are
 based on imprecise observation. Methods of "fuzzy logic" can be used to
 automatically assign these kinds of "fuzzy" classifications to essays.
 This part of the present invention identifies the organization of an essay
 through automated analysis of the rhetorical (argument) structure of the
 essay.
 The linguistic literature about rhetorical structure (Cohen (1984), Hovy et
 al. (1992), Hirschberg and Litman (1993), and Vander Linden and Martin
 (1995)) point out that rhetorical (or discourse) structure can be
 characterized by words, terms and syntactic structures. For instance,
 words and terms that provide "clues" about where a new argument starts, or
 how it is being developed are discussed in the literature as "clue words."
 Conjunctive relations from Quirk, Randolph, Sidney Greenbaum, Geoffrey
 Leech, and Jan Svartik: A Comprehensive Grammar of the English Language,
 Longman, N.Y. (1985) including terms such as, "In summary" and "In
 conclusion," are considered to be clue words and are classified as
 conjuncts used for summarizing. Clue words such as "perhaps," and
 "possibly" are considered to be "belief" words used by a writer to express
 a belief in developing an argument in the essay. Words like "this" and
 "these" may often be used to flag that the writer has not changed topics
 (Sidner, Candace: 1986, Focusing in the Comprehension of Definite
 Anaphora, in "Readings in Natural Language Processing," Barbara Grosz,
 Karen Sparck Jones, and Bonnie Lynn Webber (Eds.), Morgan Kaufmann
 Publishers, Los Altos, Calif., 363-394). It also was observed that in
 certain discourse contexts, structures such as infinitive clauses (INFCL)
 mark the beginning of a new argument, e.g., "To experimentally support
 their argument, Big Boards (INFCL) would have to do two things."
 One part of the present invention is an automated argument partitioning and
 annotation program (APA). APA outputs a file for each essay after it is
 partitioned into argument units. In addition, APA outputs a second file in
 which each sentence in an essay is annotated with word, term or structure
 classifications that denote argument structure.
 A specialized dictionary (lexicon) is used by APA to identify relevant clue
 words and terms. The lexicon used by e-rater is displayed in Appendix B1.
 APA's heuristics select the dictionary-based clue words, terms, and
 non-lexical structures. Descriptions of the rules used by APA appear in
 Appendix B2. The rules that APA uses to partition and annotate arguments
 specify syntactic structure and the syntactic contexts in which clue words
 contribute to argument structure. APA uses parsed essays to identify
 syntactic structures in essays. Essays have been syntactically parsed and
 each sentence in the essay has a syntactic analysis. Table 3 illustrates
 original essay text and the text output by APA with corresponding argument
 partitioning annotations, where wording in the argument-unit annotations
 has been revised for comprehensibility.
 TABLE 3
 Example of Automated Argument Partitions and
 Annotations (APA)
 Essay Text Argument Partitioned/Annotated Essay Text
 " . . . Another problem with Another problem with the argument is found
 the argument is found in in the evidence used to support it.
 the evidence used to
 support it. Big Boards &gt;Start Argument at Parallel Word: Another
 takes responsibility for
 increasing River City's Big Boards takes responsibility for
 awareness of the marathon increasing River City's awareness of the
 runner over the three- marathon runner over the three-month
 month period by putting period by putting her name on billboards,
 her name on billboards, but they also stated that there was
 but they also stated that "extensive national publicity" of the
 there was "extensive woman during that time.
 national publicity" of
 the woman during that &gt; Argument Development at Contrast Word:
 time. The 30% increase but
 in awareness of this &gt; Argument Development at Complement
 woman could have been a Clause: that . . .
 result of the national
 publicity. Big Boards The 30% increase in awareness of this
 did not necessarily have woman could have been a result of the
 anything to do with River national publicity.
 City learning the woman's
 name - they may have &gt; Speculation at Auxiliary Word: could
 learned it from TV or
 magazines or Big Boards did not necessarily have
 newspapers . . . " anything to do with River City learning
 the woman's name - they may have learned
 it.
 Table 4 shows the rhetorical variables used by e-rater for predicting
 scores. By using these predictive variables, a vector of rhetorical
 feature counts (54 in FIG. 1) for each essay is generated and used by
 e-rater in final scoring.
 TABLE 4
 Rhetorical Structure Variables Used to Predict
 Scores for Issue and Argument Essays
 Argument Essays Total Occurrences of independent
 Arguments in the Final Paragraph of the
 Essay
 Total Occurrences of Subjunctive Modal
 Verbs in the Final Paragraph of the Essay
 Total Occurrences of Parallel Words
 that Begin an Independent Argument
 Total Occurrences of Argument
 Development Using Belief Words
 Issue Essays Total Occurrences of Arguments Starting
 With a Summary Phrase
 Total Occurrences of Arguments Starting
 With an Infinitive Clause
 Total Occurences of Argument Starting
 With a Subordinate Clause
 Total Occurrences of Argument
 Development Using an Infinitive Clause
 Total Occurrences of Argument
 Development Using a Belief Word
 Total Number of Independent Arguments
 in the Body of the Essay, Excluding The
 First And Final Arguments/Paragraph
 C. Content Vector Analysis
 The scoring rubric suggests that certain ideas are expected in an essay by
 stating that the essay "effectively supports the main points of the
 critique" for argument essays and "explores ideas and develops a position
 on the issue with insightful reasons and/or persuasive examples" for the
 issue essays. Content vector (CV) analysis is a statistical weighting
 technique used to identify relationships between words and documents. With
 regard to the approximate specifications in the rubric about essay
 content, CV analysis can be used to identify vocabulary (or content words)
 in essays that appear to contribute to essay score.
 Assigning one of six scores to a GMAT essay is a standard type of
 classification problem. Statistical approaches to classification define
 each class (score) by the distribution of characteristics found in labeled
 training examples. Then, each test essay is analyzed, and its distribution
 is compared to that of the known classes. The class which best matches the
 test essay is selected.
 For text, the characteristics may be physical (the number or length of
 words, sentences, paragraphs, or documents), lexical (the particular words
 that occur), syntactic (the form, complexity, or variety of
 constructions), rhetorical (the number or type of arguments), logical (the
 propositional structure of the sentences), or a combination of these.
 Standard CV analysis characterizes each text document (essay) at the
 lexical (word) level. The document is transformed into a list of
 word-frequency pairs, where frequency is simply the number of times that
 the word appeared in the document. This list constitutes a vector which
 represents the lexical content of the document. Morphological analysis can
 optionally be used to combine the counts of inflectionally-related forms
 so that "walks," "walked," and "walking" all contribute to the frequency
 of their stem, "walk." In this way, a degree of generalization is realized
 across morphological variants. To represent a whole class of documents,
 such as a score level for a set of essays, the documents in the class are
 concatenated and a single vector is generated to represent the class.
 CV analysis refines this basic approach by assigning a weight to each word
 in the vector based on the word's salience. Salience is determined by the
 relative frequency of the word in the document (or class) and by the
 inverse of its frequency over all documents. For example, "the" may be
 very frequent in a given document, but its salience will be low because it
 appears in all documents. If the word "pterodactyl" appears even a few
 times in a document, it will likely have high salience because there are
 relatively few documents that contain this word.
 A test essay is compared to a class by computing a cosine correlation
 between their weighted vectors. The cosine value is determined by the
 following equation:
EQU cos=.SIGMA.(a.sub.i *
 b.sub.i)/sqrt(.SIGMA.(a.sub.i.sup.2)*.SIGMA.(b.sub.i.sup.2))
 where a.sub.i is the frequency of word "i" in document "a" and b.sub.i is
 the frequency of word "i" in document "b." The larger the value of the
 correlation, the closer the test essay is to the class. The class which is
 closest to the test essay is selected and designated "Essay Score Feature
 A" (22 in FIG. 1). These steps are summarized below.
 Vector construction for each document (or class):
 Extract words from document (or combined documents)
 Apply morphological analysis (optional)
 Construct frequency vector
 Assign weights to words to form weighted vector
 Compute cosine correlation between test essay vector and the vector of each
 class
 Select class with highest correlation
 As discussed in the next section, CV analysis can also be applied to units
 of text smaller than essays. For example, it can be used to evaluate the
 rhetorical arguments within an essay. In this case, each argument is
 treated like a mini-document and is compared to the classes independently
 of the other arguments. The result is a vector of classes (scores), one
 for each argument in the essay.
 E-rater uses a CV analysis computer program which automatically predicts
 scores for both argument and issue essays. The scores assigned by the CV
 analysis program are used as a predictor variable for the set of argument
 essays.
 D. Argument-Content Vector Analysis
 An important goal of this invention is to be able to predict essay scores
 based on "what the writer says." CV analysis, as it is used above,
 identifies word associations over the essay as a whole. It looks at words
 randomly in the essay. Although this tells the reader something about
 possible essay content, it is important to capture words in a more
 structured way, so that topic may be identified using closely clustered
 word groupings.
 The scoring rubric specifies that relevant essay content (or relevant words
 used in an essay) should be well organized and should address relevant
 content. Therefore, a revised version of the content vector analysis
 program was implemented and run on the "argument partitioned" training
 essays for argument and issue essays.
 Another content similarity measure, ArgContent, is computed separately for
 each argument in the test essay and is based on the kind of term weighting
 used in information retrieval. For this purpose, the word frequency
 vectors for the six score categories, described above, are converted to
 vectors of word weights. The weight for word "i" in score category "s" is:
EQU w.sub.i,s =(freq.sub.i,s /max_freq.sub.s)*log(n_essays.sub.total
 /n_essays.sub.i)
 where freq.sub.i,s is the frequency of word "i" in category "s,"
 max_freq.sub.s is the frequency of the most frequent word in category "s"
 (after a stop list of words has been removed), n_essays.sub.total is the
 total number of training essays across all six categories, and
 n_essays.sub.i is the number of training essays containing word "i."
 The first part of the weight formula represents the prominence of word "i"
 in the score category, and the second part is the log of the word's
 inverse document frequency (IDF). For each argument "a" in the test essay,
 a vector of word weights is also constructed. The weight for word "i" in
 argument "a" is:
 w.sub.i,a =(freq.sub.i,a /max_freq.sub.a)*log(n_essays.sub.total
 /n_essays.sub.i)
 where freq.sub.i,a is the frequency of word "i" in argument "a," and
 max_freq.sub.a is the frequency of the most frequency word in "a" (once
 again, after a stop list of words has been removed). Each argument (as it
 has been partitioned) is evaluated by computing cosine correlations
 between its weighted vector and those of the six score categories, and the
 most similar category is assigned to the argument. As a result of this
 analysis, e-rater has a set of scores (one per argument) for each test
 essay. The final score is then calculated as an adjusted mean of the set
 of scores, represented as ArgContent:
EQU ArgContent=((arg_scores+n_args)/(n_args+1)
 This final score output is designated "Essay Score Feature B" (62 in FIG.
 1).
 E. The e-rater System Overview
 FIG. 1 shows a functional flow diagram for a preferred embodiment of the
 e-rater system of the present invention. The first step in automatically
 scoring an essay is creating a model feature set, i.e., a model feature
 set used to predict scores at each score point of the scoring rubric. The
 system starts with a batch of approximately 250-300 original electronic
 essay text responses (essays already having a known score). Each original
 electronic essay text is evaluated by EssayContent to perform Content
 Vector Analysis (as described in Section C above) and to generate "Essay
 Score Feature A" and is also parsed by the parser 30 to produce a
 "syntactic" representation of each essay response, denoted as parsed essay
 text 32.
 Syntactic Feature Analysis 40 (program clause.c) then processes the parsed
 essay text 32 to extract syntactic information (as described above in
 Section A entitled "Structural Features") and creates a vector of
 syntactic feature counts 42 for each syntatic feature considered by
 e-rater. Rhetorical Feature Analysis 50 (program gmat.c) also processes
 the parsed essay text 32 (as described above in Section B entitled
 "Rhetorical Structure Analysis") to generate annotated text 52, which
 includes a vector of rhetorical feature counts 54 and text partitioned
 into independent arguments 56. This argument partitioned text 56 is then
 evaluated by ArgContent to perform Argument-Content Vector Analysis
 (Section D above) to produce "Essay Score Feature B" 62.
 The vector of syntactic features 42, the vector of rhetorical features 54,
 Essay Score Feature A 22, and Essay Score Feature B 62 are then fed
 (depicted by the phantom arrows) into a stepwise linear regression 70,
 from which a "weighted" predictive feature set 72 is generated for each
 test question using the batch of sample data. The set of weighted
 predictive features define the model feature set for each test question.
 The steps just described above up to the linear regression 70 are then
 performed for a score to be predicted for each actual essay response. The
 vector of syntactic features 42, the vector of rhetorical features 54,
 Essay Score Feature A 22, and Essay Score Feature B 62 for each response
 are then fed (depicted by the solid arrows) into the score calculation
 program 80 associated with the model answer for the test question with
 which the essay is associated and a Final Score 90 between 0 and 6 is
 generated.
 It will be appreciated by those skilled in the art that the foregoing has
 set forth the presently preferred embodiment of the invention and an
 illustrative embodiment of the invention, but that numerous alternative
 embodiments are possible without departing from the novel teachings of the
 invention. All such modifications are intended to be included within the
 scope of the appended claims.
 APPENDIX A1: ANALYSIS OF AN ARGUMENT ITEM
 ANALYSIS OF AN ARGUMENT
 Time--30 minutes
 Directions: In this section you will be asked to write a critique of the
 argument presented below. You are not being asked to present your own
 views on the subject.
 Read the argument and the instructions that follow it, and then make any
 notes in your test booklet that will help you plan your response. Begin
 writing your response on the separate answer document. Make sure that you
 use the answer document that goes with this writing task.
 The following is from a campaign by Big Boards, Inc., to convince companies
 in River City that their sales will increase if they use Big Boards
 billboards for advertising their locally manufactured products.

"The potential of Big Boards to increase sales of your products can be seen
 from an experiment we conducted last
 year. We increased public awareness of the name of the current national
 women's marathon champion by
 publishing her picture and her name on billboards in River City for a
 period of three months. Before this time,
 although the champion had just won her title and was receiving extensive
 national publicity, only five percent of
 15,000 randomly surveyed residents of River City could correctly name the
 champion when shown her picture;
 after the three-month advertising experiment, 35 percent of respondents
 from a second survey could supply her
 name."
 Discuss how well reasonsed you find this argument. In your discussion be
 sure to analyze the line of reasoning and
 the use of evidence in the argument. For example, you may need to consider
 what questionable assumptions
 underlie the thinking and what alternative explanations or counterexamples
 might weaken the conclusion. You can
 also discuss what sort of evidence would strengthen or refute the argument,
 what changes in the argument would
 makes it more sound and persuasive, and what, if anything, would help you
 better evaluate its conclusion.
 NOTES
 Use the space below or on the facing page to plan your response. Any
 writing on these pages will not be evaluated.
 STOP
 YOU FINISH BEFORE TIME IS CALLED, YOU MAY CHECK YOUR WORK ON THIS SECTION
 ONLY. DO NOT TURN TO ANY OTHER SECTION IN THE TEST.
 APPENDIX A2: ANALYSIS OF AN ISSUE ITEM
 ANALYSIS OF AN ISSUE
 Time--30 minutes
 Directions: In this section you will need to analyze the issue presented
 below and explain your views on it The question has no "correct" answer.
 Instead, you should consider various perspectives as you develop your own
 position on the issue.
 Read the statement and the instructions that follow it, and then make any
 notes in your test booklet that will help you plan your response. Begin
 writing your response on the separate answer document. Make sure that you
 use the answer document that goes with this writing task.

"Everywhere, it seems, there are clear and positive signs that people are
 becoming more respectful of one another's differences."
 In your opinion, how accurate is the view expressed above? Use reasons
 and/or examples from your own experience, observations, or reading to
 develop your position.
 NOTES
 Use the space below or on the facing page to plan your response. Any
 writing on these pages will not be evaluated.
 STOP
 IF YOU FINISH BEFORE TIME IS CALLED, YOU MAY CHECK YOUR WORK ON THIS
 SECTION ONLY. DO NOT TURN TO ANY OTHER SECTION IN THE TEST.
 APPENDIX B1
 Lexicon (CLUELEX) used by e-rater
 argument/content#ALTERNATIVE::.or, either
 argument development class#BELIEF_word:: certainly, clearly, obviously,
 plainly, possibly, perhaps,
 potentially, probably, fortunately, generally, maybe, presumably, unless,
 albeit, luckily, unluckily, normally, for
 sure, apparently, herein, likely, surely, ideally, undoubtedly, naturally
 argument development class#BELIEF_phrase:: for_certain, for_sure,
 of_course, to_some-extent, above_all,
 if_only, in_order_to, in_order_for, so_that, so_as_to
 argument initialization class#CLAIM_N:: argument, arguments, assumption,
 assumptions, claim, claims, issue,
 issues; evidence, evidences, idea, ideas, flaw, flaws, study, studies,
 point, points, position, positions, leap_of_logic,
 conclusion, conclusions, emission, generalization, indication, indications,
 deduction, passage, factor, factors,
 problem, problems, statement, statements, discussion, discussions,
 question, questions, example, examples,
 counterexample, counterexamples, reason, reasons
 argument initialization class#CLAIM_phraseRO:: I, we
 argument initialization class#CLAIM THAT:: that
 argument development class#CONTRAST_word:: othervise, conversely, however,
 nonetheless,
 though, yet, meanwhile, while, but, instead, although, still,
 notwithstanding, anyway, unlike
 argument development class#CONTRAST_phrase:: on_the_contrary, in_contrast,
 by_comparison, in_any_case,
 at_any_rate, in_spite_of, rather_than, on_the_other-hand, even_wordhen,
 even_if, even_though, even_wordith,
 apart_from, instead_of
 argument development class#DETAIL_word:: if, specifically, particularly,
 when, namely
 argument development class#DETAIL_phrase:: for_example, for_instance, e.g.,
 in_this_case, in_that_case,
 such_that, as_well_as, in-that, such_as, about_how, in_addition,
 in_addition_to
 argument development class#DISBELIEF_word:: unfortunately
 argument development class#EVIDENCE_word:: since, because, actually
 argument development class#EVIDENCE_phrase:: in_fact, after_all,
 as_a_matter_of_fact, because_of
 argument development class#INFERENCE_word:: accordingly, consequently,
 hence, thus, ultimately, so,
 thereby, then, therefore, following, after, afterward, afterwards
 argument development class#INFERENCE_phrase:: as_a_consequence,
 as_a_result, if_so, if_not, as_such,
 according_to, in_turn, right_after
 argument initialization class#ALLEL_word:: firstly, essentially,
 additionally, first, second, another, third,
 secondly, thirdly, fourth, next, finally, final, last, lastly, moreover,
 too, also, likewise, similarly, initially, further,
 furthermore
 argument initialization class#ALLEL_phrase:: first_of all,
 in_the-first_place,
 for_one_thing, for_a_start, second_of_all, many_times, more_importnatly,
 most_importantly
 argument development class#REFORMULATION word:: alternatively
 argument development class#REFORMULATION_phrase:: that_is, in_other words,
 i.e., briefly
 argument development class#RHETORICAL word:: ?, suppose, supposedly,
 supposing
 argument development class#RHETORICAL_phrase:: what if
 argument initialization class#SUMMARY_word: altogether, overall
 argument initialization class#SUMMARY_phrase:: all_in_all, in_conclusion,
 in_sum, in_summary,
 in_summation, in_short, on_the_whole
 arg auxiliary_verb#SPECULATE_word::might, may, should, could, would
 argument initialization class#TRANSITION_phrase::let us
 APPENDIX B2
 Rules Used By e-rater
 I. "AFTER" RULE
 A. Extracts "after", "after", and "afterwards" if they occur
 sentence initially as
 conjunction.
 II. "ALSO" RULE
 A. Contrains argument extraction for "also", classified in the
 lexicon as arg-init#Parallel,
 and for additional adverbs classified as arg_dev#Belief such
 that all are extracted if
 they appear in sentence initial position or if they modify the
 main verb of the sentence
 (defined as the first verb that occurs in the second column of
 the parse tree).
 III. LEXICALLY-BASED RULE FOR BEGINNING AN ARGUMENT
 a. Constrains the extraction of nouns and pronouns classified as
 arg-init#CLAIM words
 in the lexicon to main clause subject NPs and in sentences
 beginning with "There", to
 the position after a form of the verb "to be".
 IV.
 a. Controls the extraction and labeling of Nouns in arg_init
 position that are modified by
 "this" or "these" that are labeled arg_dev#SAME_TOPIC when
 they occur in the
 second or later sentence of a paragraph.
 b. If "This", "These" or "It" occur as a pronoun in the first
 noun phrase of the parse tree of
 sentences that are not paragraph-initial, they are output with
 the label
 arg_dev#SANE_TOPIC. This label is generated dynamically.
 "This", "these" and "it"
 are not stored in the lexicon.
 V. "BUT" RULE
 A. Extracts "but" if it is labeled as a conjunction.
 VI. COMPLEMENT CLAUSE RULE
 A. Extracts complement clauses introduced by "that" as well as
 complement clauses that
 do not begin with "that."
 B. Labels complement clause as arg_init#CLAIM_THAT* when it is
 the first or only
 sentence of a paragraph, otherwise it is labeled as
 arg_dev#CLAIM_THAT*
 C. Extracts the conjunction "that" if it occurs in a complement
 clause, or a complement
 clause not introduced by "that" under the following
 conditions:
 1. the complement clause is not embedded in another COMPCL or
 SUBCL
 2. the complement clause is not further embedded than the
 third column of the
 parse tree
 VII. "SUBORDINATE CLAUSE" RULE FOR BEGINNING AN ARGUMENT
 A. If the very first sentence of a paragraph begins with a
 subordinate clause, extract the
 noun or pronoun from the main clause NP and consider it to be
 the beginning of a new
 argument. The noun or pronoun extracted is labeled
 arg_init#D-SPECIFIC if it is not
 listed in the lexicon.
 VIII. "FIRST" RULE
 A. Constrains words listed in lexicon that are classified as
 arg_init#Parallel words.
 B. All words of this category in sentence initial position are
 extracted (cf ALSO RULE).
 C. If the word is not sentence-initial one of the following
 conditions must be satisfied.
 1. It must be in the first constituent of the parse tree,
 provided that the first
 constituent is not a subordinate clause and that it is not
 further embedded in
 the parse tree than the third column.
 2. It must be the first NP following a sentence-initial
 adverb.
 3. If the first constituent is the pronoun "I" followed by a
 verb, then the
 "FIRST" item must be immediately following the verb.
 IX. "FURTHER" RULE
 A. Extracts "further" "overall" or "altogether" if they occur
 sentence-initially and do not
 modify another constituent.
 X. INFINITIVE CLAUSE RULE
 A. Extracts an infinitival clause that is not further embedded
 than the third column of the
 parse tree and either follows or precedes the main verb of the
 sentence. The clause is
 not embedded in a subordinate clause or a complement clause.
 Infinitival clauses that
 are extracted are labeled as arg_init#To-INFL if it is the
 first or only sentence of a
 paragraph, otherwise arg_dev#To_INFL.
 XI. RULE FOR BEGINNING AN ARGUMENT AT A NEW AGRAPH
 A. If a paragraph has no lexical or structural "argument
 initializations" then a label
 arg_init#NEW_AGRAPH is applied.
 XII. "OR" RULE
 A. Extracts the conjunctions "or" and "either" when they occur in
 the second column of
 the parse tree, and the node immediately following the
 conjunction is a verb phrase.
 XIII. ALLEL TERM RULE
 A. Prevents the extraction of arg_init#Parallel lexical entries
 terms if they modify a verb
 or a noun at any level of embedding. (cf also FIRST.DOC)
 XIV. "SHOULD" RULE
 A. The words, would, should, might, may, and could are be picked
 up for each essay. They
 are classified as arg_aux#SPECULATE in the lexicon.
 B. These words occur in parse trees in the structure
 C. AUXP VERB " "
 XV. "SO" RULE
 A. Extracts the conjunction so if it occurs initially in a
 subordinate clause or if it is a
 sentence-initial adverb.
 XVI. "THEN" RULE
 A. Extracts "then" if it occurs as an adverb or a conjunction
 that is not further embedded
 than the second column of the parse tree.
 XVII. VERBING RULE
 A. Extracts sentence-initial nouns and verbs ending in "-ing", as
 well as "-ing" verbs that
 immediately follow a prepositional phrase or an adverb that is
 in the second column of
 a parse tree. These extracted "-ing" words are labeled as
 arg_init#CLAIM_Ving if in
 the first or only sentence of a paragraph, and
 arg_dev#CLAIM_Ving otherwise.
 B. If the base form of the verb is "do", then the label will be
 arg_dev#lnference.
 XVIII. "WHEN" RULE
 A. Extracts all occurrences of "when" in the following structure
 I. ABBCL*CONJUNCTION PHRASE* CONJUNCTION* "when"
 if this structure occurs no further embedded than the fourth
 column of the parse.
 XIX. "WHILE" RULE
 A. Extracts "while" under the following conditions.
 1. It is the first constituent of a sentence
 2. It is a conjunction in a subordinate clause that is not
 further embedded than the
 third column.