A CONTROL CHART FOR ERRORS IN IBM TEST SCORING MACHINES

IN SCORING test papers with the IBM Test Scoring Machine, it is well known that variations in humidity, voltage, and age and condition of an swer sheets can effect the scoring response of the machine. Because of the variability in the scores obtained, some form of checking procedure is fol lowed by mostusersof the scoring machine. The checking usually consists of a complete rescor ing of papers or spot checks. The extensiveness of spot checking depends in part upon local tradi tion and in part upon the kind of precision re quired for the job at hand. C h e c k i ng of scores costs money or time as does any other quality con trol measure. Further, neither local tradition nor any arbitrary "kind" of precision is enough to provide information sufficient to achieve eco nomical scoring checks. Since checking answer sheet scoring is exactly analogous to inspecting industrial production it was decided to investigate one of the statistical quality control devices used in industry. Kimball (1) has proposed a number of sampling plans for analysis of errors in scoring IBM an swer sheets. However, none of the plans in Kim airs paper are presented on the basis of practi cal experience. It is proposed in this paper to present checking procedures based upon regions that seem useful empirically. In a study of the comparability of hand scoring of IBM answer sheets, the agreement between five machine scorers were tabulated. (See top of next page.) Thus, while there is far from per fect agreement among the scorers, there is very good agreement if the criterion of + 1 point is used to define agreement. As a matter of fact, the mean number of agreements + 1 point is 96. 0 and the Standard Deviation is 1. 95 agreements. Since this scoring was done from a random group of papers and the scoring was done care fully, 96 percent agreement (ag r eement between two scores within + 1 point) seems to be an attain able degree of agreement between scorers. This seems an acceptable standard of agreement to use to require as a criterion of acceptability for a set of scores derived from rescoring of IBM answer sheets. In different situations, different stand ards will be necessary. In very short tests where important decisions are to be b a s e d on s c o res within two or three points of each other, obvious ly more rigorous standards must be used. In the situation where there are large samples, long tests, and procedures are somewhat flexible, 96 percent agreement seems adequate. For practi cal considerations, one would not care to use a s c o r ing that agreed less than 90 percent with an other scoring. Actually, 2-1/2 S. D. down from 96 percent agreement is 91 percent agreement. This seems to be an acceptable bound for reject ing scorings. If a group of scorings of papers from a large sample showed agreement between 91 percent and 96 percent between two scorings of the same paper, more papers would be examined before accepting or rejecting the scorings of the larger sample. If we consider each test paper asa random sample from a population of papers, and we re score the paper to see if the original score on the paper falls in the limits of error (+ 1 score point), then we make a yes-no decision whether to accept the score on the paper as correct or not. For a group of such papers we could use the b i n o m i al distribution to test the hypothesis that the popula tion from which the group came possessed the quality, "96 percent of all the papers if rescored would fall in the limit of error (+ 1 score point)." The sequential probability ratio test for the mean of a binomial distribution as presen ted in Wald (2) seems to provide a solution to the prob lem. If we let P0 be the lower bound of the propor tion of defectives in an acceptable group, and Px

Field of study: Education