This study will provide a descriptive analysis of the processes raters go through as they rate ESL students' writing. Data analysis is also proceeding in stages. Patterns of interactions in the training and scoring sessions have been identified via the analysis of field notes and videotapes. The think-aloud protocols have been analyzed on the basis of a scoring rubric made up of categories representing raters' comments. Interview data are currently being analyzed.
Preliminary results show that raters approach the process of scoring from individual perspectives.
The comments they make are varied and are not limited by the features listed on the analytic scoring guidelines. In fact, as the interviews have illustrated, the raters appear to be interpreting the scoring guidelines differently so that what is important to one rater in judging the quality of the essay's content may not be important to another. Interesting also has been the preliminary descriptive look at the discourse of individual raters. Although they maintain a certain air of individuality, they do have styles that are similar to those of other raters, particularly when they have teaching status and degree of experience as composition raters in common. To assume that raters are focusing on the same features when they come up with the same or a similar score on an essay does appear to be presumptuous.