Psychometrics of Direct Observation.

Abstract. Direct observation plays an important role in the assessment practices of school psychologists and in the development of evidence-based practices in general and special education. The defining psychometric features of direct observation are presented, the contributions to assessment practice reviewed, and a specific proposal is offered for evaluating the psychometric merit of direct observation in both practitioner developed and commercial/research specific applications. This article allows readers to determine if a specific direct observation system is credible, offers psychometric merit, and should be considered as part of a multimethod-multisource assessment. ********** Direct observation is one of the most frequently used assessment procedures by school psychologists. A survey of more than 1,000 school psychology practitioners (Wilson & Reschly, 1996) found that of the 26 different types of assessment instruments listed across seven different assessment categories, direct observation methods were ranked highest in terms of frequency of use. Overall, practitioners report that they conduct more than 15 behavioral observations of student behavior during the course of a typical month. Of interest, however, although such assessment techniques constitute a good deal of practitioners' assessment practice, rarely are direct observation or direct observation systems evaluated with respect to the psychometric properties and characteristics demanded of other assessment techniques (e.g., cognitive ability scales, academic achievement instruments, behavior rating scales). Indeed, of the variety of forms of direct observation reported in practice, the most common (e.g., anecdotal, narrative, descriptive) fall far short with respect to the level of psychometric rigor expected by governing bodies (e.g., American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1999) or legal standards (IDEA, 1997). Historically, the applicability of psychometric principles to direct observation has been equivocal. Whereas some have suggested that classical psychometric concepts based on differences between persons is antithetical to an assessment methodology that focuses on behavior and its variation within individuals (Nelson, Hay, & Hay, 1977), others have suggested that the differences between traditional test development and behavioral assessment are primarily conceptual, not methodological, and as such, reliability and validity considerations apply (Cone, 1988). At the heart of this divide lie fundamental differences regarding the assumptions of behavior (e.g., psychometric assumptions of consistency and stability not being shared by those with a more behaviorally oriented perspective), the level of analysis (group versus individual), and the model of causality (intrasubject structural causality versus setting-subject functional causality) (Silva, 1993). The purpose of this article is to provide readers with an introduction and overview to psychometric theories of measurement as they apply to direct observation, examine various ways to evaluate the reliability of direct observation data, and review the types of validity evidence that apply to direct observation systems. Finally, quality indicators are suggested as a standard for determining whether direct observation systems should be considered for use. Contrasting Theories and the Measurement of Behavior Measurement can be described as the process of obtaining a numerical description of the degree to which an individual possesses a particular characteristic (Gresham & Carey, 1988; Linn & Gronlund, 2000; Nunnally & Bernstein, 1994). In the case of direct observation, this amounts to the quantification of observable phenomenon in measurable units as represented by an operational definition that defines the behavioral parameters with respect to both time and frequency (Johnston & Pennypacker, 1993). …