Table 2 summarizes the mathematics strands, materials, analysts, and benchmarks that were used in the study. Team members were encouraged to consult with each other and to ask questions of the director. They were asked not to consult or communicate with the members of other teams, especially the team that was analyzing the same set of materials. With the exception of two or three analysts, the ratings were completed within the month, with the remainder being completed within two more weeks. All of the reports were complete and useable in the study.
Shapes can match exactly or have the same shape in different sizes.
There are 24 criteria across the seven instructional clusters. Overall, six separate ratings were done for each of these criteria on each of the two materials, resulting in 288 ratings. The results are summarized in Table 3. Therefore, the overall percentage agreement was 88.2 percent. The percentage agreement on the two materials differed considerably.