QUALITY ASSURANCE - ARE WE 

DOING ENOUGH? 

by Brad Pafford 

When you hear the term "quality assurance" what comes to mind? You may think of a 

Japanese automobile, Lee Ioccoca and Chrysler Motors, or perhaps television commercials 

colorfully campaigning the Ford Motor Company's motto, "Quality is Job 1." When you 

think about quality assurance (QA) in terms of NASS surveys, what now comes to mind? 

Does the color go away? Do we get as "pumped up" as what we see in Lee Ioccoca and 

Chrysler Motors? Why not? The American automobile industry found that by ignoring 

quality as the number one priority, Americans stopped buying their cars in favor of Japanese 

products. Similarly, NASS must place quality as the number one priority or the Agency's 

image in the agricultural sector can decline and our data users would diminish, going 

somewhere else for this information! 

How does NASS define "quality assurance"? Is it the control of sampling and nonsampling 

errors by the application of sound statistical techniques? Is it the standardization of 

procedures throughout the Agency? Also, how does NASS define "quality control"? Does it 

refer primarily to enumerator procedures and their evaluation? Whatever the definitions, 

one has only to look at recommendations from Agency and outside-the-Agency review teams 

set up since 1980 (see the Crop Reporting Boards Standards report of 1985, and the 

Statistical Review of Survey Methodology and Estimation of the Statistics Unit of Economic,

Statistics Cooperatives Service, Executive Summary report of 1980). One common theme is 

the need for a more structured and comprehensive QA program. 

What direction do we need to go? To begin, you need to ask yourself what are the 

important parts of a quality survey. If you said that one of these is accurate estimates, you 

would have listed the topic of my discussion from here on. I'd like to begin by focusing on 

one small but very important part of the developing NASS quality assurance program that 

addresses accuracy - the reinterview surveys. 

In surveys, one measure of accuracy is how far away the estimate, such as December 1 total 

hogs and pigs, is from the truth. This difference is called bias. In terms of the December 1 total 

hogs and pigs estimate, bias can be thought of as the difference between the farmer's response 

and the true number of hogs averaged over the sample. The purpose of the reinterview survey 

program is to measure the bias and in turn get a measure of the quality of the Hog and Pig 

estimates. 

Let's see what the reinterview program is about. A sample of survey respondents are personally 

reinterviewed with a reinterview questionnaire using experienced and/or supervisory 

enumerators. The reinterview questionnaire is exactly like the operational survey 

questionnaire, except it is shortened to re-ask only important major survey items such as 

corn and soybean acreage and on-farm storage. Enumerators are instructed to try extremely 
hard to reinterview farm operators. The reinterview is conducted without the enumerator 

or farmer having access to the initial responses, and within a few days of the first interview. 

Immediately after the reinterview questionnaire is completed, the enumerator opens a 

reconciliation form that contains the original responses and, then, any differences in the two 

responses are reconciled. Information is collected on which response was correct, where the 

error occurred, the reason for the discrepancy, and who the respondents were for each 

interview. The final reconciled response is treated as the best answer, or "truth." 

The results from two studies conducted in December 1987 (Ohio, Minnesota, Indiana) and 

March 1988 (Nebraska, Iowa, Pennsylvania) indicate that significant biases exist in reporting 

on-farm grain stocks collected through Computer Assisted Telephone Interviewing (CATI). 

That is, on-farm stocks for corn, soybeans, and all wheat were underestimated in the 

neighborhood of 10 percent to 20 percent. These levels are important because they may apply to 

all telephone collected stocks data, and because NASS is heavily dependent on CATI data 

collection. The under-reporting of stocks has been assumed for some time now because the 

stocks indications were not consistent with outside check data (exports, crushings of soybeans, 

and other supply type data). The value of these bias measures are that they are direct Agency 

measures of survey quality, and can be incorporated into a more structured QA program. 

A similar program was done in the late 1980’s for the Soybean Objective Yield (OY) Survey, 

called a Validation Survey. The difference here is that the measures of truth are actual 

weighings of soybeans as they are taken to the processing stations. Yields for a soybean OY 

field can be calculated from the weight tickets by dividing the recorded bushels by the field 

acreage. These yields are compared with the yield developed from counts of soybean plants, 

number of pods, and weight per pod for the two field plots that are laid out by the enumerator. 

Biases have been measured in the soybean OY survey and the weight ticket yields. This 

measured bias allows for the accuracy portion of the OY quality assurance program to be 

addressed. 

It would be nice to talk about the other QA areas that are being addressed, but space is 

limited. I will stop and say that producing quality surveys is the most important job in 

NASS. Let's do the right things in the future to maintain the long-standing confidence of 

our data users. Borrowing from Ford Motor Co., and emphasizing the phrase "meant to stay 

that way" in terms of working towards a better quality assurance program, I say, "NASS 

statistics are tough and meant to stay that way!"