The Burden of Proof: Validity as Improvement of Instructional Practice:

In a quiet but profound way, the work of Samuel Messick (1980, 1988, 1994) has revolutionized how educational researchers establish claims for the validity of assessment procedures. Messick's framework for understanding validity was initially applied to the use of intelligence or achievement tests for placement of students into special education or remedial programs. We believe that his framework has equally exciting, challenging, and powerful implications for improving instructional practice for children with disabilities. The purpose of this article is to explore these implications in mathematics assessment and instruction in special education. Recently, three state-of-the-art innovations in special education mathematics assessment were described in Exceptional Children (Fuchs, Fuchs, & Hamlett, 1994; Gerber, Semmel, & Semmel, 1994; Woodward & Howard, 1994) and the Journal of Special Education Technology (Woodward & Carnine, 1993). All three innovations view assessment as a means to improve mathematics instruction for students with learning disabilities. Yet all three take different paths to achieving this end. In one case, expert systems identify errors and develop hypotheses concerning underlying misconceptions in subtraction (Woodward & Howard, 1994). In a second, student responses are analyzed and problem types generated where additional instruction is required (Fuchs et al., 1994). In the third case, instantaneous prompting and graphical elaboration is provided to prevent errors and to help a student solve a multidigit multiplication problem (Gerber et al., 1994). In this article, we discuss these three innovative projects using Messick's (1988) framework as a tool for understanding what really is entailed in claims made about "the appropriateness, meaningfulness, and usefulness" (p. 35) of inferences drawn from assessment data. In analyzing these three approaches, we discuss not only current thinking on assessment, but also changing views on how to improve special education math instruction. Messick argued convincingly that validity comprises not only technical and statistical concerns, but also social and instructional ones. It is incumbent on researchers involved in assessment to do the following: * Explain instructional theory underlying assessment. * Explain in detail expected uses of assessment data by teachers. * Present data on how teachers actually use the assessment procedure. * Demonstrate the consequences--both positive and negative, both intended and unintended--of use on student learning. Whereas commonly used validity indexes (concurrent, predictive, construct) remain interesting and important, they are not sufficient justification for use. They are what Messick calls the first face of validity, or evidential interpretation of the meaning(s) of assessment data; they indicate that the underlying constructs have some theoretical basis (see Figure 1, left column). Validity also entails both persuasive, logical arguments supporting use, (Figure 1, right column), and empirical data demonstrating the consequences of that assessment, i.e., that it does in fact help guide instruction to enhance student learning. A HISTORICAL INTERLUDE For many years, the key to successful reading instruction for students with learning disabilities was believed to lie in an understanding of auditory and visual processing deficits. Kavale (1981) conducted a comprehensive meta-analysis of extant research and concluded that a significant correlation existed between auditory processing skills and reading abilities. In Messick's terminology, he provided evidence of construct validity to support test interpretation. However, Kavale and Mattson (1983) subsequently found that training in auditory skills did not improve reading ability. In Messick's terminology, no evidence existed to support use. More recently, Adams (I 990) noted that a set of variables, labeled phonemic or phonological awareness, were consistently predictive of subsequent reading performance. …