The generalizability of CBM survey-level mathematics assessments: Just how many samples do we need?

Abstract. This study examined the extent to which CBM survey-level mathematics assessment varies as a function of type of survey-level assessment used (i.e., single-skill and multiple-skill math probes) and the generalizability of results both within and across assessment methodologies. Participants included 67 students enrolled in first through fifth grades. All students were assessed using grade-appropriate single- and multiple-skill CBM math survey-level probes. Findings indicate that single- and multiple-skill survey level CBM math assessments do indeed measure two distinct constructs. Results of a generalizability analysis suggest that for single-skill survey-level CBM math assessments the administration of one single-skill math probe provides information of sufficient quality to make both criterion- and norm-referenced decisions. Similar results were also observed for multiple-skill probes. However, given variations in performance the administration of three multiple-skill math probes as part of a surve y-level CBM math assessment continues to be recommended. ********** Curriculum-based measurement (CBM) is a standardized set of measurement procedures that can be used to evaluate performance outcomes in the basic academic skill areas of reading, spelling, mathematics calculation, and written expression (Deno, 1985; Deno & Fuchs, 1987; Fuchs & Deno, 1991; Shinn, 1989a). As dynamic indicators of basic skill development (DIBS; Shinn & Bamonto, 1998), curriculum-based measures can summarize and inform decision making for use in screening, eligibility determination, instructional planning, and program evaluation (Deno, 1986). Strong research support exists for the use of CBM in educational decision making (see Good & Jefferson, 1998; and Marston, 1989 for reviews). Used in a larger problem-solving model (Deno, 1986; Shinn, Nolet, & Knutson, 1990), CBM assessment activities vary as a function of the type of assessment question asked. For example, for questions involving problem identification and certification, summative survey-level measurement and evaluation activities are used to decide if a performance discrepancy exists between what is observed and expected within any one or more academic basic skill areas and whether or not observed discrepancies warrant the implementation of specialized instructional practices. Second, in a formative manner CBM progress monitoring techniques can be used to evaluate the effects of instructional changes and guide decision making regarding program modification and termination issues (Shinn et al., 1990). In CBM survey-level assessment, standardized measurement procedures involve the administration of three parallel form curriculum-based measures in the content area of concern (i.e., reading, spelling, mathematics, written expression) with the median score across the three parallel forms serving as the summary datum for performance at any particular level. For example, a third-grade student suspected of exhibiting reading difficulties would first be administered three randomly selected curriculum-based reading measures from the third-grade material in which he or she was being instructed. The median performance across the three reading measures would then be used to compare the student's observed performance to expected or peer-referenced performance (e.g., average performance of other third-grade students) and a decision would be made as to whether there was a discrepancy between the two. If a discrepancy was indeed evident, sets of three randomly selected passages from successively easier reading material (e .g., second- and first-grade material) used for instruction would be administered in an attempt to describe the magnitude of the discrepancy from typical third-grade performance and determine the level of instructional modification needed to ameliorate the reading difficulty. …