Because several thousand American students take the exam each year, statistics about the exam outcome provide a broad base of information about the performance of novice computer science students. Anonymous data obtained from Educational Testing Services provided demographics and responses to individual multiple-choice questions for every person who had taken one of the exams under study. In order to provide verifiable results, a procedure was developed to classify each APCS multiple-choice question for the strength of its relationship to logic. This study adapted the procedure to allow judges to rate each multiple-choice question according to how strongly it was related to the subdomain of logic. Having the combined opinion of this number of judges helped establish the content validity of the classification results.
Content analysis depends on having a set of criteria against which content can be measured.
When carrying out the content analysis procedure, the content under consideration must be categorized based on these criteria. For this study, the criteria were derived from the taxonomy described earlier. The categories were ordered, with 'main concept' indicating the strongest relationship to logic and 'not used' the weakest. The algorithm for grouping the multiple-choice questions, which resulted in three disjoint sets, depended on the cumulative ratings from the content analysis procedure. In content analysis, reliability is expressed as the amount of agreement that exists among the ratings given by all of the judges.