These sets include responses from all activities to be scored, and they are formatted to resemble the portion of the Answer Book the team will score. Each team scores the open-ended student responses and assigns the appropriate score point on a customized scan sheet. At grades 3 and 5, where most teachers work across subject areas, teachers score various subject areas. Scoring Coordinators receive five days of training from the test contractor in preparation for training readers on their teams, monitoring readers for quality and production during the scoring process, and administering scoring in concert with the test contractor's staff. Team Leaders, who assist Scoring Coordinators, receive three days of training.
Scorers receive two to three days of training.
As inconsistencies and inaccuracies are detected, Scoring Coordinators and Team Leaders hold discussions with the team and assist individual readers to improve accuracy. Accuracy sets are administered on two or three mornings each week to determine whether teams of readers maintain appropriate levels of accuracy during the scoring process. Readers who score below 70 percent on any accuracy set receive additional training immediately from the Scoring Coordinator or the Team Leader and are released from retraining only after the scoring problems are resolved. If the scoring problems are not resolved, the reader is dismissed from the scoring project. In spot checking, a Scoring Coordinator or Team Leader scores the same booklet scored by a reader to estimate a reader's overall accuracy, to identify items a reader is having difficulty with, or to ascertain specific items that are causing individual readers to perform poorly on check sets or accuracy sets.