TEST NORMS AND SAMPLING THEORY*
Authors: Lord Frederic M. 
Published: August 2014

Test norms usually constitute a set of sample statistics (percentiles or percentile ranks) used by the test consumer to make inferences regarding a norms population from which the norms sample was drawn. In such cases, it is important that the selection of the norms sample and the subsequent statistical treatment shall be such as to minimize the inevitable sampling errors in the published norms table.

At least in the case of tests used in our schools and colleges, the problems of optimal sampling methods and optimal statistical procedures are considerably complicated by the fact that whereas the published norms table is most commonly considered as representing a group of students, the sampling unit used in drawing the norms sample is usually the school. Since schools differ markedly among one another in their average score on most tests, the sampling errors in the final norms table will ordinarily be large unless the number of schools in the norms sample is large. The number of students in the norms sample typically has only a weak and indirect relation to the size of the sampling errors in the norms table.

In the most common method of drawing a norms sample, all students in each school selected are tested and included in the norms, provided they are at the proper grade level or levels. This constitutes simple cluster sampling. In such sampling, the mean of the norms sample is usually a biased estimate of the mean of the norms population. An unbiased estimate can be constructed from the sample data (eq. 3), but its sampling error (eq. 4) is likely to be so large in comparison to that of the mean of the norms sample (eqs. 2, 2′, 2″) as to preclude the use of the unbiased estimate in most situations.

Numerical examples for two typical situations show that under simple cluster sampling, from twelve to thirty times as many students must be tested in order to obtain the same accuracy for the norms table that would be obtained under simple random sampling of students. Unfortunately, simple random sampling of students, without regard to their “clustering” in schools, is ordinarily impractical. The best practical procedure to avoid the excessive sampling errors characteristic of cluster sampling is to use two‐stage sampling: a sample of schools is drawn in the usual fashion, and then a random sample of students in each school is selected for testing.

At the lower educational levels, two‐stage sampling may often be feasible only when two or more tests can be normed simultaneously, in which case only a fraction of the students in each classroom need take any one of the test. At the college level, it is usually impossible to arrange for the normative testing of all students at a given grade level in a variety of institutions. In this situation, the common procedure is to test as many students as possible in a relatively small number of colleges. Not only does this increase the sampling errors in the published norms table unnecessarily; it also prevents the sample of students from being a random sample, since any sizable group of students assembled at any given time of day ordinarily excludes those with conflicting scheduled activities. In order to obtain a truly representative norms sample for colleges, it will ordinarily be necessary to test so few students in each college that some time can be found when all students chosen in a given institution can be scheduled for a single testing session.

Three different ways of drawing two‐stage samples are described. In each case, the question of the choice of the sample statistic to be used for estimating the mean score in the norms population is discussed, with particular reference to the bias and sampling error of each estimate. Standard error formulas are derived for three of these estimates (eqs. 13, 18, 20). Some numerical examples are given (Table 1) shoving the economies achieved by using two‐stage sampling instead of simple cluster sampling.

A simple method is given by means of which the same formulas can be used to provide standard errors for each percentile rank in the norms table as well as for the mean of the norms distribution. A numerical example (Table 2) indicates that the advantage of two‐stage sampling over cluster sampling holds for estimating percentile ranks as well as for estimating the arithmetic mean.

Methods of obtaining school‐mean norms from two‐stage‐sampling data are briefly discussed.

It will ordinarily be important to stratify the norms population on certain school characteristics related to test score, such as geographical region and type of support (public, private, or church). Two‐stage sampling may then be carried out within each stratum separately. Sampling errors in the norms can be reduced by stratifying on school size, even when size is unrelated to test score. Since it is difficult or impractical to stratify on more than a few dimensions simultaneously, it will often be preferable to deal with school size by one of the two‐stage sampling methods discussed rather than by stratifying on size.