The generalizability of systematic direct observations across time and setting: A preliminary investigation of the psychometrics of behavioral observation

Abstract. This study examined the generalizability of systematic direct observation across setting and time. Participants included 14 students from an intact inclusionary fifth grade classroom. On-task/off-task behavior was directly observed using momentary time-sampling recording, twice a day, for 10 school days. Using Generalizability (G) theory, results suggested that of the total amount of variance present, individual variation among participants accounted for 62% of the observed variance and unexplained error 24%. Although adequate levels of interobserver agreement were attained, resulting reliability coefficients of the observational procedures were in the low .60s. Results are discussed in terms of both behavioral and classic measurement theory and implications for practice. ********** Direct classroom observation is one of the most widely used assessment procedures by school psychologists. In a survey of more than 1,000 school psychology practitioners, Wilson and Reschly (1996) found that of the 26 different types of assessment instruments listed across seven different assessment categories, structured observation methods were ranked highest in terms of frequency of use. Overall, practitioners report that they conduct more than 15 behavioral observations of student behavior during the course of a typical month. When observing student behavior within the classroom, practitioners generally rely on one of two observational techniques. One form, naturalistic/descriptive observation, refers to observational procedures where the observer enters specific situations (e.g., a classroom) and observes the behavior of a targeted student with no predetermined behaviors in mind. Here, the most common way of recording observations is to keep an anecdotal record of the behaviors that seem important to the observer. In summarizing the information, the observer provides a complete description of the many behaviors and the context in which they occurred (Hintze, Volpe, & Shapiro, 2002). The other, systematic direct observation, refers to the observation of behavior that is prompted or explicitly elicited by a predetermined set of environmental stimuli (i.e., test behavior) (Salvia & Ysseldyke, 2004). In contrast to naturalistic approaches, systematic direct observation is distinguished by five characteristics (Salvia & Ysseldyke, 2004). First, the goal of observation is to measure specific behaviors. Second, the behaviors being observed have been operationally defined a priori in a precise manner. Third, observations are conducted under standardized procedures and are highly objective in nature. Fourth, the times and places for observation are carefully selected and specified. Finally, scoring and summarizing of data are standardized and do not vary from one observer to another. Reliability of Systematic Direct Observation The applicability of reliability to direct observation has been equivocal, with some suggesting that classical psychometric concepts based on differences between persons are irrelevant to an assessment methodology that focuses on behavior and its variation within individuals (Nelson, Hay, & Hay, 1977), and others suggesting that the differences between traditional and behavioral assessment are primarily conceptual, not methodological, and as such reliability and validity considerations apply (Cone, 1981, 1988). Traditionally, behavioral assessors and researchers have approached the issue of reliability using accuracy and interobserver agreement as substitutes (Johnston & Pennypacker, 1993). Specifically, accuracy refers to the extent to which observed values of behavior approximate the "true" state of the behavior as it actually occurs. When the true state of the behavior is known, comparing the observed values to the true values derives accuracy of measurement. In this case, the behavioral notion of accuracy is the same as reliability in the classic measurement case, where reliability is represented by the extent to which an observed score represents the true score (Cone, 1977). …