VALIDATING A WRITING TEST FOR GRADUATE ADMISSIONS
Authors: Powers Donald E. , Fowles Mary E. , Boyles Kelli 
Published: August 2014

Proficiency in writing is important to success in many fields of graduate study. Measuring this construct, however, involves facing difficult challenges and criticisms from various perspectives. Among them is a concern that the writing skill that students demonstrate in an assessment situation may not relate fully enough to the writing competencies used in other contexts. This study was designed to reveal some of the facets of writing proficiency that different kinds of timed writing exercises might reflect. The approach taken was to examine how performance on these exercises correlates with performance on the following nontest indicators of writing skill: (a) self‐assessments of writing skills, (b) self‐reports of writing‐related accomplishments and relevant undergraduate grades, and (c) independently graded samples of writing, such as course‐related writing assignments and personal statements written in conjunction with application to graduate school. The writing exercises investigated in this study were of several kinds: composing a relatively traditional expository essay, writing somewhat briefer responses involving either explaining the meaning of a quotation or presenting ideas in a specified format, and revising a poorly written composition. These exercises were selected by university faculty members serving on a GRE Writing Committee.

A geographically diverse sample of 470 undergraduate students, all of whom had taken the GRE General Test (and who represented the GRE test‐taking population reasonably well), were recruited to participate in the study. Subgroups of the study sample tried various combinations of the writing tasks.

The assorted nontest indicators of writing proficiency correlated with one another in differing degrees. None, however, appeared to be (statistically speaking) completely redundant with the others, suggesting that, in all likelihood, each reflected a somewhat different aspect of writing skill. The alternative writing exercises, in turn, exhibited distinctly different patterns of correlations with the individual indicators and with several composite indices based on logically determined subsets of indicators. By virtue of these distinct correlational patterns, the exercises appeared to have differential utility for a GRE writing test. A traditional issue‐based expository essay appeared to be the most promising candidate. An exercise involving the revision of a composition also appeared to have potential. Two other experimental tasks, however, did not, from their relationships with the writing indicators, seem as defensible.

Limitations of the study are discussed – principally the shortcomings of the nontest indicators as criteria for validating the assessment of writing proficiency. Despite the limitations, the results of the study are believed to have provided a reasonably good beginning to accumulating the kind of evidence required by professional standards for educational and psychological testing. Additional information will be required, however, about the validity of the inferences to be drawn and the decisions to be made about the examinees who will eventually take the GRE writing test.