Classification Accuracy for Tests That Allow Retakes

When tests are used to make classification decisions, estimates of decision consistency and false-positive and false-negative error rates may be more appropriate than reliability as a means of characterizing precision. The decision-theoretic framework that supports this approach is not new, and numerous authors have recommended indices for use in this context. However, one aspect of this framework that has particular significance for licensure and certification testing and yet has received relatively little attention is the impact of retakes. When an examinee receives a passing score on a licensure test, a final classification is typically made. This may be a correct classification or it may be incorrect. (In this paper, when a non-proficient examinee is classified as proficient, it will be referred to as a false-positive error.) When an examinee receives a failing score, the examinee will usually be given the opportunity to repeat the test. An ultimate failing classification will occur only when the examinee has given up or exhausted the allowable retake opportunities. Millman described this effect and noted that this process provides necessary protection for the examinee. The examinee’s proficiency is not measured without error. When a proficient examinee fails a licensure examination due to measurement error, it is appropriate that the examinee should have an additional opportunity to demonstrate proficiency. However, providing additional opportunities for testing not only corrects errors that would penalize proficient examinees but also creates errors that favor non-proficient examinees. These errors may put the public at risk by allowing unqualified candidates to become licensed or certified. This paper presents a theoretical framework describing the factors that influence classification error rates over multiple administrations of a test. It then considers some of the strategies that are available for controlling the false-positive error rate. The main emphasis of the paper is to provide the reader with a realistic sense of the magnitude of the inflation of false-positive errors that may result when retakes are allowed and to provide a framework for considering what, if anything, should be done to control this inflation. A starting point for this discussion is to ask whether allowing retakes has a significant impact on the classification error rate that occurs under typical practice conditions. Consider the example of a test with a reliability of . 92, applied to a normally distributed examinee group where 10% of examinees are non-proficient. If the cut-score is established to fail 10% of examinees (in the population), with a single administration approximately 2% of all examinees might be expected to be misclassified as proficient. (Since 10% of the examinees are non-proficient, this is a false-positive rate of 20%). After two retakes this rate is doubled. A false-positive rate this high based on a single administration would be expected from a test with a reliability of approximately . 69. If the primary purpose of a licensure examination is protection of the public from non-proficient practitioners, it is clear that, in the circumstances described in this example, retakes do significantly impact the classification error rate, and thus the effectiveness of the test.