Probability Sampling with Quotas: An Experiment

A widely used modification of probability sampling techniques involves the use of quotas at the last level of a multistage area probability sample. Here we analyze an experiment comparing this technique, called "probability sampling with quotas" by Sudman, with a full probability design, in two large national surveys. Biases were detected in household size and in men's employment status. Otherwise, results were encouraging for users of carefully conducted PSQ surveys. C. Bruce Stephenson is on the General Social Survey staff at the National Opinion Research Center, University of Chicago, where he is a Ph.D. candidate. Public Opinion Quarterly ? 1979 by The Trustees of Columbia University Published by Elsevier North-Holland, Inc. 0033-362X/79/0043-477/$ 1.75 This content downloaded from 207.46.13.129 on Mon, 27 Jun 2016 05:37:05 UTC All use subject to http://about.jstor.org/terms 478 C. BRUCE STEPHENSON to administer the questionnaire. Because of these tight geographic controls, the attempt to specify the racial and economic composition of the sample by means of quotas could be relaxed. Thus many criticisms to which the earlier quota samples were liable are not relevant to "probability sampling with quotas," particularly those regarding the difficulty of attempting a quota control on economic status, and the great freedom interviewers formerly enjoyed in the choice of a respondent. Probability sampling with quotas, or PSQ for short, has been found quicker and, to a certain extent, cheaper than full probability sampling, but obviously remains open to theoretical attack because of possible bias at the last stage of respondent selection, and more generally because the usual assumptions made in sampling theory do not justify the calculation of sampling error from a sample with nonprobability components. Two lines of approach to this problem have been suggested. The first (Stephan and McCarthy, 1958: Chap. 10), which is really applicable to any type of sample strictly executed, is simply to assume that parameters measured by a well-specified sampling technique will have a sampling distribution, and to estimate the variance of this distribution either through replication or through comparison of carefully designed subsamples. Sudman (1966) went further, and argued that the assumption of a sampling distribution was justified because properly chosen quota categories could be regarded as strata, within which the probability of a person's being interviewed was essentially constant. From this argument he concluded that sampling error could be computed just as in any other clustered probability sample. We shall not have anything further to say about these theoretical questions concerning the legitimacy of assuming sampling distributions for PSQ designs, except for one comment in the conclusion. Such assumptions are better made consciously than unconsciously, but we do not have anything to contribute to their theory, nor do we wish to meddle in a theoretical controversy which still arouses strong feelings. Instead we will concentrate on some empirical evidence regarding the existence or nonexistence of perceptible bias in variables measured with PSQ designs, as compared to a more rigorous full probability design. These comparisons should be of interest to anyone who deals with PSQ samples, regardless of the question of justifying inference from such samples. They are of particular interest since most earlier comparisons of "probability" and "quota" samples were made before the development of PSQ techniques. The data for this analysis are from the 1975 and 1976 General Social Surveys (Davis, 1975, 1976). These surveys are part of a series of national surveys containing a broad variety of demographic, attitudiThis content downloaded from 207.46.13.129 on Mon, 27 Jun 2016 05:37:05 UTC All use subject to http://about.jstor.org/terms PROBABILITY SAMPLING WITH QUOTAS 479 nal, and behavioral items, all repeated either every year or according to a regular schedule. The first three surveys in the series, from 1972 to 1974, were conducted with a PSQ design which is described below in more detail; the 1977 and subsequent surveys with a full probability (FP) design are also described below. In the transition years 1975-76 the sample was split, to allow both methodological comparisons and the splicing of the time series conducted with the two different types of sample. These two surveys have several features useful for the purpose of comparing sample designs. Hundreds of questions were included on the experiment. Many of them were asked both years, thus providing replicated comparisons. Clusters for each subsample were drawn from the NORC national sampling frame in a balanced way, as described below. All interviewing was done at about the same time each year, although the FP subsamples required longer to complete than the PSQ subsamples because of callbacks. Interviewer training and questionnaire layout were identical for the two subsamples, except for matters pertaining to sampling procedure. The FP subsamples included 735 persons in 1975 and 744 in 1976; the PSQ subsamples included 755 persons each year. Response rates for the FP subsamples were 75.6 percent in 1975 and 75.1 percent in 1976 (Smith, 1978). Response rates are not available for the PSQ subsamples, since it is difficult to define and to measure the number of respondents who could have fallen into a PSQ sample but did not.