THE QUALITY OF SURVEY DATA

I . In t roduct ion Qual i ty is an " in" th ing these days. Manufactur ing companies are put t ing qua l i t y assurance systems in place that emphasize "doing i t r igh t the f i r s t t ime. " Claims are made that an increase in qua l i t y goes hand in hand with an increase in p roduc t i v i t y and a decrease in cost. This seems coun te r i n tu i t i ve to many, but the manufacturing companies that are fo l lowing the ideas of Deming, Juran, and Crosby say that th is is t rue. Can we apply some of these same kinds of techniques to improve the qua l i t y of survey data? This leads to an in te res t ing question. What do we mean by qual i ty? How do we measure i t? What means do we use to say that one survey or census is bet ter than another? Quali ty is d i f f i cu l t to def ine, but what is immediately apparent is that qua l i t y is mul t i -d imensional , is perceived in d i f f e ren t ways, and is in t imate ly l inked to the way the end product is to be used. The qua l i t y of survey data is also b u i l t in (or destroyed) at various in te r lock ing steps in the production of the data. Just as one can v isua l ize the assembly-line production of manufactured goods, so can one v isual ize the assembly-l ine production of survey data. We might think of i t in terms of the d iscrete stages of design, co l l ec t ion , processing, and i n te rp re ta t i on . Poor qua l i t y at any of these stages is carr ied through to the f ina l data product. The concept of s t a t i s t i c a l con t ro l , as d iscussed by Shewhart and and others implies that a process is reproducible, that the measurements resu l t ing from a process are the product of an i d e n t i f i a b l e s t a t i s t i c a l universe, and that there is standardizat ion of the process when carr ied out by d i f f e ren t people and in d i f f e ren t places. To be able to detect whether a process is in s t a t i s t i c a l cont ro l , one must be able to i den t i f y cer ta in qua l i t y charac te r is t i cs of that process. I I . Qual i ty Character is t ics One way to arr ive at a set of qua l i t y charact e r i s t i c s is to imagine what we would l i ke to have happen in an ideal s i t ua t i on . I f one d idn ' t need to worry about cost and time res t ra in t s , what are desirable survey charac ter is t ics? Al l things considered, one would opt for the fo l lowing: Frame that covers the complete universe with no ambiguity. Probab i l i t y sampling. Conceptual c l a r i t y . Operational de f i n i t i ons that f i t concepts. Sample design per fec t l y executed. Most knowledgeable respondents always report . Respondents report accurately and completely. Data are recorded accurately. Data coded accurately. Data weighted accurately. Data reported not replaced in processing. Sampling variances computed accurately. Sampling variances small. Ve r i f i ca t i on of interviews shows l i t t l e or no inconsistency. Tabulated resul ts show consistency with other data. Inferences based on data are supported by measures of uncertainty in data. There never has been a survey, nor w i l l there ever be one for which there are no compromises. Rarely do we ever have a universe l i s t i n g for which there are no ambiguit ies; however, in the best s i tua t ions we are confident that the elements not included in the universe l i s t i n g are small in number or in size or both. Most of the surveys on which publ ic pol icy is based are p robab i l i t y samples, but that is not true for some of the indus t r ia l surveys. In these surveys, a cu t -o f f sample is often used in which large indus t r ia l establishments over a cer ta in size are included with cer ta in ty and smaller establishments are omitted completely. In the design stages, careful work is usual ly done on t ry ing to c l a r i f y concepts. The unemployment concept is one which has received much at tent ion and gets reviewed per iod ica l l y by President ia l Commissions. Not only is the concept reviewed but i t s t rans la t ion into operational de f in i t i ons and questions is reviewed. However, for surveys in general th i s is probably an area in which much more work needs to be done, because the t rans la t ion of concepts into de f i n i t i ons and then in to questions is very d i f f i c u l t . Simple pretests often i l l u s t r a t e weaknesses. Sample designs are usual ly not per fec t l y executed because of a var iety of circumstances related to the f a l l i b i l i t y of in terv iewers, the Post Of f ice, or the telephone companies; to the inadequacy of maps; and to other fac tors . The most knowledgeable person for a pa r t i cu la r sample person or establishment is f requent ly not the respondent. To save money, one respondent often reports for an ent i re household. In establishments, the questionnaire may be f i l l e d by people who are new to the company, may not rea l l y understand what is being asked fo r , do not read the detai led i ns t ruc t i ons , and are re luctant to look up records. We tend to think of the questionnaire as presenting the same stimulus to the respondents, whether they are presented with the questionnaire through an intermediary known as an in terv iewer who comes in person or by telephone or whether the questionnaire comes in the mai l . We know that interv iewers are not jus t passive survey instruments. They often determine whether or not a unit responds; they i n t e r pret the quest ionnaire; they represent the survey organizat ion; they record the data. In cases where there are no in terv iewers, the format and appearance of the questionnaire take on an added burden of representing the survey organizat ion and the accompanying ins t ruc t ions for f i l l i n g the form supply the i n te rp re ta t i on . However, i t is a rare survey in which the respondents report f u l l y and accurately. Often they do not understand questions, are unwi l l ing to make the e f f o r t to f ind records to provide answers, forget the answers, or are unwi l l ing to provide them. Interviewers and respondents make errors in recording data. Data often require coding which is carr ied out by c le r i ca l s t a f f working with i ns t ruc t i ons . Items such as occupation, industry , standard