Quantifying Context in Assessment: Capturing the Effect of Base Rates on Teacher Referral and a Problem-Solving Model of Identification.

Abstract. The purpose of this article was to examine the effect of base rate occurrence of race, sex, and student achievement on the accuracy of a problem-solving model of assessment and teacher referral. All students in first and second grade (n = 182) at a participating school were exposed to four screening measures. Students who performed poorly on at least one of the screening measures participated in a more thorough assessment process including individual curriculum-based assessment with individual intervention. The predictive accuracy (e.g., sensitivity, specificity) of the problem-solving model and teacher referral was examined using the more thorough assessment process as the standard for comparison. The degree to which the problem-solving model and teacher referral correctly identified male and female children, children of minority and nonminority ethnicity, and children in high-achieving and low-achieving classrooms was specified. Further, the efficiency of the problem-solving model and teacher referral was examined by comparing identification relative to base rate occurrence of problems in the screened population. ********** One of the stated goals of the current educational reform movement is to identify and provide with effective intervention those children who are struggling with required academic tasks prior to a protracted failure experience. Powerful screening methods are needed to accomplish this goal. The current identification system relies heavily on teacher referral. The teacher typically serves as the "screening device" at the school level, identifying who will and will not be discussed at the school level meeting where it is determined who will and will not be evaluated. Decades of research show that human perception is limited, fallible, and subject to errors of immediacy, selectivity, and attribution (Macmann & Barnett, 1999; Meehl, 1986; Nisbett & Ross, 1980). Numerous reports have suggested that psychologists, physicians, and other professionals are not immune to the problem of inaccurate perception that can result from the context in which the problem is observed (Gould, 1981). School-based professionals who travel from high-performing schools to low-performing schools have long observed the tendency of the "low" children to be referred by their teachers regardless of their absolute level of achievement. Hence, the low fifth graders in the low-performing school may function 2 years below grade level, whereas the low children in the high-performing school may perform only a half-year below grade level. In such cases it may be the performance of those referred relative to the performance of the group that results in a teacher's perception that the child is different and then is labeled a concern that warrants referral. The decision to refer may be based on temporally proximal events that are most salient to the teacher but are not necessarily causally related to poor performance. Just as the child's performance occurs within the context of a classroom, school, and community, so does the decision-making process. If one assumes that context is critical in determining whether deficits exist that merit further attention, then the primary question concerns how best to measure context. Whereas teachers have the benefit of understanding the instructional context in which the child's performance problem occurs, relying on teacher nomination to identify and validate performance problems may lead to errors in identification for several reasons. For teacher nomination to be an accurate screening source, the stimuli governing teacher referral would need to be stable across children, settings, and teachers. Further, teacher response (e.g., prereferral assistance, referral) to these particular stimuli would need to be similarly stable across children, settings, and teachers. Such stability is highly unlikely; hence, despite teachers' opportunities to evaluate child performance within the instructional context, relying on teacher referral to identify potentially serious performance problems is likely to lead to inaccuracies in identification. …