A Framework Relating Questionnaire Design and Evaluation Processes to 

Sources of Measurement Error 1 

James L. Esposito 

Bureau of Labor Statistics, Postal Square Building, Room 4985 

2 Massachusetts Avenue, N.E., Washington, DC 20212 

Esposito_J@bls.gov or Esposito.Jim@bls.gov 

Key Words: Conceptualization, data quality, observation, operationalization, questionnaire evaluation 

1.0 Introduction 

The design, evaluation and administration of survey questionnaires constitute difficult and resource-intensive work. If 

measurement error is to be minimized, survey sponsors and design-team members need to: (1) be aware of the various phases 

of the questionnaire design-and-evaluation process, and (2) understand the various sources of measurement error that could 

undermine data quality. The primary objective of this paper is to describe a two-dimensional framework that interrelates 

these two elements (see Table 1). The first dimension consists of a rudimentary process model that describes in a very 

general and simplistic manner how questionnaires are developed and evaluated (Esposito and Rothgeb, 1997, pp. 543-546, 

561-566; Esposito 2002).2

 The model comprises eight partially recursive and overlapping phases (see section 2.1): four core 

processes (P1: observation, P3: conceptualization, P5: operationalization, and P7: administration) and four corresponding 

evaluation/assessment phases (P2, P4, P6 and P8). The second dimension relates to a descriptive model of measurement 

error that has been articulated by Groves (1987, 1989) and modified superficially by the present author to accomplish specific 

goals (Esposito 2002, pp. 3-6). This model, as modified, comprises five potential sources of error (see section 2.2): (1) 

questionnaire: content specialist; (2) questionnaire: design specialist; (3) interviewer; (4) respondent; and (5) mode. The 

framework is intended more for consideration in the design/redesign and evaluation of interviewer-administered panel 

surveys that have recognized and ongoing societal importance. A second objective of the paper will be to draw attention to 

the early phases of the design-and-evaluation process in an effort to encourage content (and other subject-matter) specialists 

to be more explicit and systematic in their observational and conceptual work and, in so doing, contribute to efforts to 

minimize the magnitude of measurement error that occurs during survey administration. 

2.0 The Framework 

2.1 Questionnaire Design and Evaluation: An Elementary Process Model. As noted, the process model comprises eight 

partially overlapping phases both for initial design and redesign efforts (Table 1; for additional details, see section 2.3): 

 Phase One (P1): Observation [OBS]. Observation constitutes the foundation upon which science and most personal 

knowledge is built, though we sometimes overlook the fact that mental structures formed as a result of past experiences 

invariably influence what we “see” and “hear” in the present. During this initial phase, content specialists (i.e., subjectmatter experts; see section 2.2) focus on observable activity (behavior and events) within various contexts (family; 

community; workplace). While the ideal, at least initially, may be bottom-up processing (i.e., relatively unfiltered 

perception) of domain-specific behaviors and events across a broad range of contexts, it is presumed that observation by 

content specialists involves substantial top-down processing (i.e., experience- or theory-laden perception) across a more 

restricted range of contexts. What survey participants (respondents and interviewers) have observed and know is also 

important, because if there is a significant mismatch between what they and content specialists have observed, the design 

and evaluation of questionnaires is apt to be problematic. The well-informed content specialist not only knows her 

subject matter, she also has a fairly good idea of what her target population knows. Ultimately, and to the extent that 

resources permit, what needs to be established is: (1) what content specialists and potential survey participants “know”, 

(2) how they come to know what they know, (3) how “accurate” or “valid” that knowledge might be, and (4) what 

differences, in terms of experience and knowledge, might exist between the various groups. 

1 The views expressed in this paper are those of the author and do not reflect the policies of the Bureau of Labor Statistics. This paper 

draws heavily on conceptual material from a longer and more empirical paper that was presented at the 2002 International Conference on 

Questionnaire Development, Evaluation and Testing [QDET] Methods (Esposito 2002). 

2

 For more thorough discussions of these topics, the reader has many good choices: Akkerboom and Dehue, 1997; Converse and Presser, 

1986; DeMaio, Mathiowetz, Rothgeb, Beach and Durant, 1993; Forsyth and Lessler, 1991; Fowler, 1995; Goldenberg et al., 2002 (for an 

establishment survey perspective); Oksenberg, Cannell and Kalton, 1991; Platek, 1985; Sudman and Bradburn, 1982; Turner and Martin, 

1984; and Willis, Royston and Bercini, 1991. 
 Phase Two (P2): OBS Assessment. The inclusion of this phase is based on the belief that even subject-matter experts 

can benefit from independent checks of their domain-relevant observations. The concern here has to do with strong 

influence of prior experience and knowledge in framing and potentially distorting observations in the present. Two lines 

of evaluation work might be useful here: The first would assess the observation-based knowledge of subject-matter 

experts, specifically questionnaire content specialists (section 2.2), and the second would assess the range of observationbased knowledge possessed by individuals who share the characteristics of likely survey participants. Ideally, one would 

want to engage a small group of professionals to identify and describe the most relevant behaviors and events that occur 

within the domain of interest. While most social and behavioral scientists (e.g., psychologists, sociologists) have their 

preferred methods for accomplishing this sort of work (e.g., Webb et al., 1966), most also possess theoretical 

perspectives that could conceivably “contaminate” their domain-relevant observations. We are looking for a more 

neutral approach to gathering observational data/information—an approach grounded not in some preferred academic 

theory but rather in what a skilled observer of human behavior and events might see (e.g., Glaser and Strauss 

1967/1999). Ethnographers would appear to be in the best position to perform this sort of observational assessment. 

With their work as the standard, ethnographic researchers could be asked to compare their observations against those of 

content specialists and against those of prospective survey participants—the two lines of research alluded to above— 

noting significant disparities and trying to determine the origins of those disparities. 

 Phase Three (P3): Conceptualization [CPT]. During this phase, the domain of interest (i.e., the relevant “world” under 

investigation) is selectively abstracted and organized into a network of concepts and categories. While the capacity to 

form concept-based categories appears to be a universal human attribute, there is still considerable debate as to how 

these concepts and categories are represented in memory (e.g., Barsalou, 1992; Smith, 1989). Presumably, content 

specialists will differ with respect to which concepts and categories they identify as central and with respect to the 

delineation of causal interrelationships among them. As was the case in phase one (P1), an important consideration here 

is who assumes primary responsibility for observation and conceptualization (e.g., an individual content specialist versus 

an interdisciplinary team) and how these tasks are accomplished (e.g., limited versus comprehensive domain-specific 

observations; discipline-specific versus interdisciplinary theoretical frameworks).3 Another important consideration has 

to do with the degree of correspondence between a sponsor’s or content specialist’s conceptual understanding of the 

target domain and that of prospective survey participants, and how well both sets of understandings reflect what 

“actually exists and takes place” in the target domain. 

 Phase Four (P4): CPT Assessment. The inclusion of this phase is based on the belief that, unlike the perception of 

physical objects, the conceptualization of social reality is much greater extent subject to personal, professional and 

cultural influences. Given this belief, even content specialists can benefit from independent checks of their domainrelevant conceptualizing (e.g., Miller, 2002). In an effort to correct for potential personal (experiential), professional 

(theoretical) and cultural predilections, qualitative research—ethnographic, psychological or domain-specific—should be 

undertaken by skilled and independent professional observers (e.g., cognitive anthropologists and psychologists) to 

evaluate and reconcile the conceptual terms, models and assumptions of subject-matter experts (e.g., Gerber, 1999). 

 Phase Five (P5): Operationalization [OP]. After a decision has been made to gather data by means of a questionnaire, 

content specialists and design specialists assume responsibility for translating survey concepts into questionnaire items 

(see section 2.2, Questionnaire: Design Specialist) and ancillary metadata (e.g., interviewing manuals; classification 

algorithms; see Dippo and Sundgren, 2000). 

 Phase Six (P6): OP Assessment (or Pretesting). During this phase, design specialists—ideally in close collaboration 

with content specialists and field operations staff—assume primary responsibility for developing a plan to formally test 

the draft questionnaire. This testing usually starts with an assessment of how research participants “process” 

questionnaire content cognitively (i.e., comprehension, retrieval, judgment, response/reporting; see Tourangeau, Rips 

and Rasinski, 2000). The influence of other psychological states (e.g., motivational and emotional) on the nature of the 

response process may or may not be considered at this point (e.g., see Cannell, Miller and Oksenberg, 1981). As noted, a 

variety of methods have been proposed/described for use in pretesting questionnaires (see footnote 2). 

 Phase Seven (P7): Administration. After pretesting work is completed, which could involve several P1-P6 iterations, 

and after modifications have been made to the questionnaire and to its pertinent metadata (e.g., interviewer instructions; 

algorithms), the survey instrument is finalized and moved to a production environment. 

3 Hox (1997) provides a scholarly discussion on the topics of conceptualization and operationalization. While the approaches he describes 

for formulating survey questions are appealing in a theoretical sense, my limited experience in this area suggests that the approach used to 

generate questions for large-scale governmental surveys tends to be more empirical and pragmatic. 
 Phase Eight (P8): Quality Assessment. Depending on available resources and the importance of a survey’s data 

products (e.g., unemployment rate, poverty and crime rates; prevalence of cancer and other diseases), the sponsor may 

choose to conduct post-implementation quality-assessment research. Virtually any of the techniques used to pretest a 

draft questionnaire can be used periodically to evaluate whether questionnaire items are adequately capturing and 

measuring the concepts specified by the survey sponsors (for a listing, see Esposito and Rothgeb, 1997, pp. 543-551). 

While social, technological and cultural change complicates all forms of recurring social measurement, the rate of change 

that occurs in various content domains can vary widely. Given a modest rate of change associated with the content of a given 

social survey, one or more redesign efforts can be expected. Design and redesign processes overlap to the extent that content 

and design specialists make use of quality-assessment findings (P8) in their redesign work (RP1 through RP6). 

2.2 Interdependent Sources of Survey Measurement Error. As noted, the framework’s second component involves five 

interdependent sources of measurement error. Groves defines measurement error as “the discrepancy between respondents’ 

attributes and their survey responses” (1987, p. S162) and distinguishes among four sources of measurement error: the 

interviewer, the respondent, the questionnaire, and the mode of data collection (1987, pp. S163-S166; 1989, chapters 8 

through 11). In describing measurement error arising from the questionnaire, we will find it useful to distinguish between the 

contributions of two specialized groups: content specialists (subject-matter experts with program and/or survey development 

responsibilities) and design specialists (survey practitioners/professionals who design and evaluate questionnaires, prepare 

training materials, develop algorithms, et cetera). The rationale for this distinction is rooted in the different roles each group 

assumes in the questionnaire design-and-evaluation process and with the specialized expertise each possesses with regard to 

resolving certain types of issues and problems (theoretical/conceptual versus technical design). From a functional 

perspective, content and design specialists, as an integrated working group, constitute the questionnaire design-andevaluation team; the setting that incorporates the interviewer, the respondent and the collection mode constitutes the 

information/data-collection context. Brief descriptions of the five sources of measurement error are provided below. 

 Questionnaire: Content Specialist. Especially during the observation and conceptualization phases associated with 

questionnaire design, content specialists assume a central role in describing the domain of interest, isolating and defining 

key concepts and categories, and delineating possible relationships among theoretical variables (FCSM, 1988; Hox, 

1997; Turner and Martin, 1984, Chapter 7). Their assumptions and theories, be they explicit or implicit, about how 

domains are structured, about how theoretical relationships change over time, and about why actors behave as they do in 

various situations have a profound impact on questionnaire content and data quality. The more “accurate” their 

observations, concepts, assumptions and theories, the more successful the survey measurement process is likely to be. 

 Questionnaire: Design Specialist. During initial design, questionnaire-design specialists, usually following guidelines 

prescribed by researchers and practitioners (Belson, 1981; Converse and Presser, 1986; Foddy, 1993; Fowler, 1995; 

Sudman and Bradburn, 1982), transform conceptual specifications provided by content specialists into coherent sets of 

questionnaire items and ancillary metadata. Even when conceptual specifications appear reasonably clear and precise, 

this translation/design process can be challenging. 

 Interviewer. With respect to minimizing measurement error, there would appear to be disparate views among 

researchers and practitioners as to the proper role of interviewers in administering surveys (Beatty, 1995; Maynard and 

Schaeffer, 2002). For some, their prescribed role is to administer survey questions in a standardized manner (Fowler and 

Mangione, 1990). For others, their prescribed role is to facilitate the communication of intended “meaning” when 

administering survey questions (Suchman and Jordan, 1990), which may require a more flexible approach to asking 

questions and providing feedback (Conrad and Schober, 2000). Since neither prescribed role can be expected to remove 

interviewers as a potential source of measurement error, survey sponsors need to consider the relative costs and benefits 

associated with efforts to do so. Whatever one’s position on this issue, content and design specialists would be wise to 

resist the temptation to reflexively assign blame to interviewers for questionnaire-administration problems that, on closer 

inspection, might be found to have their locus in early design-and-evaluation work (e.g., P1, P3 and P5). 

 Respondent. In an effort to improve data quality, behavioral scientists: (1) have developed socio-cognitive models of the 

response process (Cannell, Miller and Oksenberg, 1981; Tourangeau, 1984; for a review, see Jobe and Herrmann, 1996); 

(2) have described the types of cognitive errors that can occur at each stage (Tourangeau, Rips and Rasinski, 2000); and 

(3) have devised strategies for identifying questionnaire problems and reducing measurement error (Schwarz and 

Sudman, 1996; Gerber, 1999). Considerable gains appear to have been made in exploiting cognitive strategies to reduce 

error (Sirken et al., 1999; Jobe and Mingay; 1989; cf. O’Muircheartaigh, 1999). Sometimes, however, problems with the 

response process may have a significant motivational component (e.g., content irrelevance, competing demands on the 

respondent’s time). When unmotivated to participate fully in a survey, respondents may engage in satisficing behavior 

(Krosnick, 1991), thus increasing their contribution to the magnitude of measurement error. 
 Mode. The selection of a data-collection mode (or modes, as the case may be) clearly has an impact on estimates of 

measurement error (Tourangeau, Rips and Rasinski, 2000, pp. 289-312; cf. Groves, 1989, pp. 501-552). Oftentimes, the 

choice of mode is dictated by cost considerations and modest increases in measurement error tend to be accepted as part 

of the compromise to reduce survey costs. 

2.3 Additional Details Regarding the Framework (Table 1). Several additional aspects of the framework are worthy of 

note. First, it is presumed that design-and-evaluation work can and often does overlap across phases and that movement 

between certain phases (P1 through P6) is bidirectional and potentially iterative. Second, the phrase “interdependent sources 

of measurement error” has been adopted to reflect the view that measurement error—and accuracy, too—is presumed to be 

the outcome of collaborative/interactive processes involving the various sources of error identified in Table 1. Within a 

given data-collection context, measurement error is presumed to be a byproduct of role- and task-specific activities—Sudman 

and Bradburn’s (1974) terminology (cf. Platek, 1985)—that manifest themselves during the survey administrative phase (P7 

or RP7). Various role- and task-specific activities that are performed inadequately at prior design-and-evaluation phases (P1 

through P6) can be viewed as precursors to measurement error. Third, the actual performance of role- and task-specific 

activities—represented as generically-labeled cell entries (e.g., C12)—is presumed to vary across survey design-andevaluation efforts. Whether a particular cell has an entry or not would depend on whether specific cell-related activities were 

conducted. For example, if content specialists are not involved in pretesting work conducted during the initial questionnaire 

design, then cell C61 would be left blank. Empty cells are problematic in that they represent activity or knowledge gaps that 

are apt to increase the locus and magnitude of measurement error. And lastly, as noted, social, technological and cultural 

change also plays a crucial role in the measurement process. Unless continuously monitored and accounted for by content 

and design specialists, rapid change within a given target domain can have a substantial effect on measurement error. 

3.0 The Importance of Early Design and Evaluation Phases 

Though my experiences in this area are limited to labor force surveys, I believe that measurement error can be substantially 

reduced: (1) if survey sponsors were to provide documentation to the design-and-evaluation team describing work relevant to 

the observation and conceptualization phases of the questionnaire design-and-evaluation process, and (2) if design specialists, 

in close association with subject-matter experts and other behavioral scientists, were provided with the resources needed to 

effectively evaluate that work. Such documentation would substantially enhance the design team’s ability to translate survey 

concepts into relevant and efficient sets of questionnaire items. Without such documentation/metadata, design specialists 

will find themselves ill-prepared to develop and evaluate questionnaire items. 

More explicitly: Prior to attempts to draft questionnaire items at the operationalization stage, and to the extent that such 

research has been adequately funded, the design-and-evaluation team should have carefully reviewed the following: 

 Reports documenting the domain-relevant observations of subject-matter experts, prospective survey participants and 

ethnographic researchers—this addresses the issue of what each group has observed (and presumably “knows”); 

 Reports documenting the domain-relevant conceptualizations of subject-matter experts, prospective survey participants 

and ethnographic researchers—this addresses the issue of how each group has organized what they have observed; and 

 Reports documenting disparities among the various groups as to what each has observed and conceptualized, and an 

assessment as to whether such disparities can be bridged for the purpose of questionnaire-based data collection. 

Our success in minimizing the magnitude of survey measurement error depends greatly on how well these preliminary phases 

of the questionnaire design-and-evaluation process have been conducted and documented. 

References 

Akkerboom, H., and Dehue, F. (1997). “The Dutch Model of Data Collection Development for Official Surveys,” International Journal of 

Public Opinion Research, 9, 126-145. 

Barsalou, L.W. (1992). Cognitive Psychology: An Overview for Cognitive Scientists, Hillsdale, NJ: Erlbaum. 

Beatty, P. (1995). “Understanding the Standardized/Non-Standardized Interviewing Controversy,” J. of Official Statistics, 11, 147-160. 

Belson, W.R. (1981). The Design and Understanding of Survey Questions, Aldershot, England: Gower. 

Cannell, C.F., Miller, P.V. and Oksenberg, L. (1981). “Research on Interviewing Techniques,” in S. Leinhardt (ed.), Sociological 

Methodology, San Francisco: Jossey-Bass, 389-437. 

Conrad, F.G., and Schober, M.F. (2000). “Clarifying Question Meaning in a Household Telephone Survey,” Public Opinion Quarterly, 64, 

1-28. 

Converse, J.M., and Presser, S. (1986). Survey Questions: Handcrafting the Standardized Questionnaire, Newbury Park CA: Sage. 

DeMaio, T., Mathiowetz, N., Rothgeb, J., Beach, M.E., and Durant, S. (1993). Protocol for Pretesting Demographic Surveys at the Census 

Bureau, Washington, DC: U.S. Bureau of the Census. 
Dippo, C., and Sundgren, B. (2000). “The Role of Metadata in Statistics,” Proceedings of the Second International Conference on 

Establishment Surveys, Alexandria, VA: American Statistical Association, 909-918. 

Esposito, J.L., and Rothgeb, J.M. (1997). “Evaluating Survey Data: Making the Transition from Pretesting to Quality Assessment”, in L. 

Lyberg, P. Biemer, M. Collins, E. deLeeuw, C. Dippo, N. Schwarz, and D. Trewin (eds.), Survey Measurement and Process Quality, 

New York: Wiley, 541-571. 

Esposito, J.L. (2002). “Iterative, Multiple-Method Questionnaire Evaluation Research: A Case Study.” Paper presented at International 

Conference on Questionnaire Development, Evaluation and Testing (QDET) Methods, Charlestown, SC. 

Federal Committee on Statistical Methodology [FCSM] (1988). “Measurement of Quality in Establishment Surveys,” Statistical Policy 

Working Paper 15, Washington, DC: Statistical Policy Office, U.S. Office of Management and Budget, 33-42. 

Foddy, W. (1993). Constructing Questions for Interviews and Questionnaires, Cambridge, UK: Cambridge University Press. 

Forsyth, B.H., and Lessler, J.T. (1991). “Cognitive Laboratory Methods: A Taxonomy,” in P.P. Biemer, R.M. Groves, L.E. Lyberg, N.A. 

Mathiowetz, and S. Sudman (eds.), Measurement Errors in Surveys, New York: Wiley, 393-418. 

Fowler, F.J., and Mangione, T.W. (1996). Standardized Survey Interviewing, Thousand Oaks, CA: Sage. 

Fowler. F.J (1995). Improving Survey Questions: Design and Evaluation, Thousand Oaks, CA: Sage. 

Gerber, E. (1999). “The View from Anthropology: Ethnography and the Cognitive Interview,” in M.G. Sirken, D.J. Herrmann, S. 

Schechter, N. Schwarz, J.M. Tanur, and R. Tourangeau (eds.), Cognition and Survey Research, New York: Wiley, 217-234. 

Glaser, B.G., and Strauss, A.L. (1967/1999). The Discovery of Grounded Theory, New York: Aldine de Gruyter. 

Goldenberg, K.L., Anderson, A.E., Willimack, D.K., Freedman, S.R., Rutchik, R.H., Moy, L.M. (2002). “Experiences Implementing 

Establishment Survey Questionnaire Development and Testing at Selected U.S. Government Agencies.” Paper presented at International 

Conference on Questionnaire Development, Evaluation and Testing (QDET) Methods, Charlestown, SC. 

Groves, R.M. (1989). Survey Errors and Survey Costs, New York: Wiley. 

Groves, R.M. (1987). “Research on Survey Data Quality,” Public Opinion Quarterly, 51, S156-S172. 

Hox, J.J. (1997). “From Theoretical Concept to Survey Question,” in L. Lyberg, P. Biemer, M. Collins, E. deLeeuw, C. Dippo, N. 

Schwarz, and D. Trewin (eds.), Survey Measurement and Process Quality, New York: Wiley, 47-69. 

Jobe, J.B., and Herrmann, D.J. (1996). “Implications of Models of Survey Cognition for Memory Theory,” in D.J. Herrmann, M. Johnson, 

C. McEvoy, C. Hertzog, and P. Hertel (eds.), Basic and Applied Memory Research ( Volume 2), Hillsdale, NJ: Erlbaum, 193-205. 

Jobe, J.B., and Mingay, D.J. (1989). “Cognitive Research Improves Questionnaires,” American Journal of Public Health, 79, 1053-1055. 

Krosnick, J.A. (1991). “Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys,” Applied Cognitive 

Psychology, 5, 213-236. 

Maynard, D.W., and Schaeffer, N.C. (2002). “Standardization and Its Discontents,” in D.W. Maynard, H. Houtkoop-Steenstra, N.C. 

Schaeffer, and J. van der Zouwen (eds.), Standardization and Tacit Knowledge, New York: Wiley, 3-45. 

Miller, K. (2002). “Cognitive Analysis of Sexual Identity, Attraction and Behavior Questions.” Cognitive Methods Staff, Working Paper 

Series, No. 32, Hyattsville, MD: National Center for Health Statistics. 

Oksenberg, L., Cannell, C., and Kalton, G. (1991). “New Strategies for Pretesting Questionnaires, J. of Official Statistics, 7, 349-365. 

O’Muircheartaigh, C (1999). “CASM: Successes, Failures, and Potential,” in M.G. Sirken, D.J. Herrmann, S. Schechter, N. Schwarz, J.M. 

Tanur, and R. Tourangeau (eds.), Cognition and Survey Research, New York: Wiley, 39-62. 

Platek, R. (1985). “Some Important Issues in Questionnaire Development,” Journal of Official Statistics, 1, 119-136. 

Sirken, M.G., Herrmann, D.J, Schechter, S., Schwarz, N., Tanur, J.M., and Tourangeau, R. (eds.) (1999). Cognition and Survey Research, 

New York: Wiley. 

Suchman, L., and Jordan, B. (1990). “Interactional Troubles in Face-to-Face Survey Interviews,” JASA, 85, 232-253. 

Sudman, S., and Bradburn, N.M. (1982). Asking Questions: A Practical Guide to Questionnaire Design, San Francisco: Jossey-Bass. 

Schwarz, N., and Sudman, S. (eds.) (1996). Answering Questions: Methodology for Determining Cognitive and Communicative Processes 

in Survey Research, San Francisco: Jossey-Bass. 

Smith, E.E. (1989). “Concepts and Induction,” in M.I. Posner (Ed.), Foundations of Cognitive Science, Cambridge: MIT Press, 501-526. 

Tourangeau, R. (1984). “Cognitive Science and Cognitive Methods,” in T. Jabine, M.L. Straff, J.M. Tanur and R. Tourangeau (eds.), 

Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines, Washington, DC: National Academy Press, 73-100. 

Tourangeau, R., Rips, R.J. and Rasinski, K. (2000). The Psychology of Survey Response, Cambridge, UK: Cambridge University Press. 

Turner, C.F., and Martin, E. (1984). Surveying Subjective Phenomena, Volume 1, New York: Russell Sage Foundation. 

Webb, E.J., Campbell, D.T., Schwartz, R.R., and Sechrest, L. (1966). Unobtrusive Measures: Non-reactive Research in the Social 

Sciences, Chicago: Rand McNally. 

Willis, G.B., Royston, P., and Bercini. D (1991). “The Use of Verbal Report Methods in the Development and Testing of Survey 

Questionnaires,” Applied Cognitive Psychology, 5, 251-267. 
Table 1. A Framework Relating Questionnaire Design-and-Evaluation (D-and-E) Processes to Sources of Measurement Error 

INTERDEPENDENT SOURCES OF MEASUREMENT ERROR (at P7 or RP7) 

Questionnaire D-and-E Team Information/Data Collection Context 

Interviewer Respondent Mode 

(3) (4) (5) 

C13 C14 

C23 C24 

C33 C34 

C43 C44 

C53 C54 C55 

C63 C64 C65 

C73 C74 C75 

C83 C84 C85 

C R13 C R14 

C R23 C R24 

C R33 C R34 

C R43 C R44 

CR53 CR54 CR55 

CR63 CR64 CR65 

CR73 CR74 CR75 

CR83 CR84 CR85 

 

 

 

 

 

 

 

 

INITIAL 

DESIGN 

Content 

Specialist (1) 

Design 

Specialist (2) 

Questionnaire 

Design and 

Evaluation 

Phases 

P1 

P2 

P3 

P4 

P5 

P6 

P7 

P8 

Observation 

Evaluation 

Conceptualization 

Evaluation 

Operationalization 

Evaluation 

Administration 

Evaluation 

C11 

C21 

C31 

C41 

C51 

C61 

C71 

C81 

C12 

C22 

C32 

C42 

C52 

C62 

C72 

C82 

REDESIGN 

Questionnaire 

Redesign and 

Evaluation 

Phases 

RP1 

RP2 

RP3 

RP4 

RP5 

RP6 

RP7 

RP8 

Observation 

Evaluation 

Conceptualization 

Evaluation 

Operationalization 

Evaluation 

Administration 

Evaluation 

CR11 

CR21 

CR31 

CR41 

CR51 

CR61 

CR71 

CR81 

CR12 

CR22 

CR32 

CR42 

CR52 

CR62 

CR72 

CR82