Division

June 1991 Using Structured

Interviewing

Techniques

GAO/PEMD-10.1.5

GAO assists congressional decisionmakers 

deliberative process by furnishing analytic

information on issues and options under

consideration. Many diverse methodologie

needed to develop sound and timely answ

questions that are posed by the Congress. 

GAO evaluators with basic information ab

more commonly used methodologies, GAO

guidance includes documents such as met

transfer papers and technical guidelines.

This methodology transfer paper on using 

interviewing techniques discusses how GA

evaluators should incorporate structured i

techniques when appropriate to performin

It explains when these techniques should b

what steps should be followed. Overall, it 

techniques for designing a structured inter

pretesting, for training interviewers, and fo

conducting the interviews. The original rep

authored by Erwin W. Bedarf in July 1985. 

reissued version, prepared by Kenneth Litk

supersedes the version published in 1985.

Using Structured Interviewing Techniques 

series of papers issued by the Program Ev

and Methodology Division (PEMD). The pu

the series is to provide GAO evaluators wi

various aspects of audit and evaluation me

to illustrate applications, and to indicate w

detailed information is available.

We look forward to receiving comments fr

readers of this paper. They should be addr

Eleanor Chelimsky at 202-275-1854.

Werner Grosshans

Assistant Comptroller General

Office of Policy

Page 1 GAO/PEMD-10.1.5 Structured 
Preface 

Chapter 1 

The Role of

Structured

Interviews in GAO

Evaluations

Chapter 2 

What Is a

Structured

Interview and

When Should It

Be Used?

Chapter 3 

Designing a

Structured

Interview

Identifying Variables and Developing

Questions

Composing Appropriate Questions 

Selecting a Question Format 

Organizing Questions 

Layout Considerations 

Chapter 4 

More on

Interview Design:

Avoiding

Problems

Appropriateness of the Language 

Level of the Language 

Use of Qualifying Language 

Clarity of Language 

Bias Within Questions 

Considerations for Telephone Interviewing

Instruments

Page 2 GAO/PEMD-10.1.5 Structured 
Chapter 5 

Pretesting and

Expert Review

Purpose of Pretest 

Pretest Procedures 

Purpose of Expert Review 

Instrument Redesign 

Chapter 6 

Training

Interviewers

Training Methods 

Interviewer Qualifications 

Chapter 7 

Selecting and

Contacting

Interviewees

Selection of Interviewees 

Contacting Potential Interviewees 

Interview Arrangements 

Protecting the Interviewee 

Chapter 8 

Conducting

Interviews

Developing Rapport and Showing Interest 

Giving the Interviewee a Reason to

Participate

Helping the Interviewee to Be Responsive 

Asking Questions in a Prescribed Order an

Manner

Ensuring Understanding 

Ensuring Nonbias 

Obtaining Sufficient Answers 

Showing Sensitivity to Interviewee Burden 

Chapter 9 

Analyzing the

Data

Nonrespondent Problem 

Data Analysis 

Analysis of Open-Ended Questions 

Page 3 GAO/PEMD-10.1.5 Structured 
Chapter 10 

The Role of

Evaluators and

Specialists on

Each Task

Appendixes Appendix I: Bibliography 

Appendix II: Glossary 

Appendix III: Papers in This Series 

Tables Table 1.1: Evaluation Questions and

Strategies

Table 2.1: Comparison of Data-Collection

Techniques

Table 3.1: Identifying, Developing, and

Selecting Questions

Table 10.1: Functions and Responsibilities 

Evaluators and Specialists

Figures Figure 3.1: Question 1 

Figure 3.2: Questions 2, 3, and 4 

Figure 3.3: Question 5 

Figure 3.5: Question 7 

Figure 3.6: Question 8 

Figure 3.7: Question 9 

Figure 3.8: Question 10 

Figure 3.9: Question 11 

Figure 3.10: Structured Interview Text 

Figure 4.1: Question 1 

Figure 4.2: Question 2 

Figure 4.3: Question 3 

Figure 4.4: Question 4 

Figure 4.5: Question 5 

Figure 7.1: Interviewee Contact Procedure

Example

Page 4 GAO/PEMD-10.1.5 Structured 
Figure 7.2: Example of Telephone Contact

With Potential Interviewee

Figure 7.3: Interviewee Contact Log 

Figure 7.4: Interviewee Contact Log Filled 

Abbreviations

CATI Computer-assisted telephone

interviewing

DCI Data-collection instrument

GAO General Accounting Office

PEMD Program Evaluation and Method

Division

QPL Questionnaire Programming Lan

QUEST A GAO computer program for fo

interviews and questionnaire

Page 5 GAO/PEMD-10.1.5 Structured 
GAO Evaluations

A major responsibility of the General Acco

Office (GAO) is to audit and evaluate the p

activities, and financial operations of feder

departments and agencies and to make

recommendations toward more efficient a

operations.

The broad questions that dictate the objec

GAO evaluation and that suggest the evalu

strategy can be categorized as descriptive, 

or impact (cause-and-effect).1 A descriptiv

evaluation, as the name implies, provides d

information about specific conditions of a 

activity, while a normative evaluation com

observed outcome to an expected level of

performance. An impact (cause- and-effec

evaluation aims to determine whether obs

conditions, events, or outcomes can be att

the operation of the program or activity. A

the type of evaluation questions to be answ

different evaluation strategies are used, as 

table 1.1.

1

We use the term “evaluation” throughout this paper; 

many of the interviewing concepts and procedures ap

GAO audits. The categories of questions are discusse

methodology transfer paper entitled Designing Evalu

bibliography at the end of this paper.

Page 6 GAO/PEMD-10.1.5 Structured 
GO auatos

Table 1.1: Evaluation

Questions and Strategies Type of question Strategy

Descriptive Sample survey

Case study

Available data

Normative Sample survey

Case study

Available data

Impact (cause-and-effect) Field experime

Available data

In a sample survey, data are collected from 

of a population to determine the incidence

distribution, and interrelationship of event

conditions. The case study is an analytic d

of an event, process, institution, or progra

either a single case or multiple cases. The 

experiment compares outcomes associate

program operations with estimates of wha

outcomes would have been in the absence 

program. Available data refers to previous 

data bases previously established and curr

available.

The design of a GAO evaluation encompas

elements:

• the kind of information to be acquired,

• sources of information (for example, types 

respondents),

• methods to be used for sampling sources (

example, random sampling),

• methods of collecting information (for exa

structured interviews and self-administere

questionnaires),

• the timing and frequency of information co

Page 7 GAO/PEMD-10.1.5 Structured 
GO auatos

• the basis for comparing outcomes with an

program (for cause-and-effect questions), 

• the analysis plan.

This paper focuses on the fourth design

element—specifically, structured interview

self-administered questionnaires, structure

interviews are often used when the evalua

strategy calls for a sample survey. Structur

interviews can also be used, however, in fi

experiments where information must be o

from program participants or members of 

comparison group. Similarly, when essenti

same information must be obtained from n

people for a multiple case-study evaluation 

case-study evaluation, it may be beneficial 

structured interviews.

Structured interviews (and other forms of 

data collection, such as the self-administer

questionnaire) are often used in conjunctio

design that employs statistical sampling. T

combination provides data that can be use

projections about the entire population fro

the sample was drawn. We discuss samplin

methodology and generalization in depth i

methodology transfer paper entitled Using 

Sampling.

It should be noted, however, that the steps 

evaluation design process—defining the qu

that dictate the objectives of the study, sel

method of collecting the information, and 

an analysis plan for using the collected inf

answer the questions—are interrelated an

If, for example, a structured interview is u

collect information to answer an evaluatio

the question will determine the contents o

matter of the interview form. Any constrai

Page 8 GAO/PEMD-10.1.5 Structured 
GO auatos

identifying and selecting a sample (for exa

lack of a universe listing of the target popu

make it necessary to refine the original eva

question. Many more examples could be g

demonstrate the iterative nature of this pr

point to remember is that the use of struct

interviewing to collect information is not a

process and cannot be thought of as a sequ

unrelated to or independent of other tasks 

process of answering an evaluation questio

Page 9 GAO/PEMD-10.1.5 Structured 
When Should It Be Used?

For years, GAO evaluators have collected 

through various techniques such as review

and interviewing government and contract

employees, and program participants. Incr

since 1972, we have used what have come 

data-collection instruments (DCIs) on assi

that require the same or uniform informati

numerous cases. A DCI is a document con

questions presented in a systematic, highly 

fashion; its purpose is to enable the evalua

obtain uniform data that can be compared

and, if it is quantitative, subjected to addit

statistical analysis. The form of a DCI vari

according to whether it is to be used in a s

interview, as a self-administered questionn

mailed to individuals or organizations or c

by individuals in a group setting), or as a p

schedule to obtain information from recor

An interview that uses a DCI to gather dat

telephone or face to face, is a structured in

one in which evaluators ask the same ques

numerous individuals or individuals repres

numerous organizations in a precise mann

each interviewee the same set of possible 

In contrast, an unstructured interview con

open-ended questions, which are not aske

structured, precise manner. Different evalu

interpret questions and often offer differen

explanations when respondents ask for cla

Given the need to collect uniform data fro

numerous persons or organizations, when 

evaluator use a structured interview rathe

mail questionnaire or a questionnaire adm

a group setting? There is no hard-and-fast 

discuss some of the advantages and disadv

interviews and questionnaires in the follow

paragraphs. In addition, the characteristic

Page 10 GAO/PEMD-10.1.5 Structured 
eSoud t e Used?

data-collection techniques are systematica

compared in table 2.1.

Table 2.1: Comparison of Data-Collection Techniques

Structured interview Questionn

Extent of advantage

Characteristic or advantage

By

telephone

Face to

face By mail Grou

Methodology

Allows use of probes 3 5 1 

Controls bias of collector 3 2 5 

Can overcome

unexpected events in

data collections 4 5 2 

Facilitates feedback

about instrument or

collection procedures 4 5 2 

Allows oral and visual

inquiry 1 5 2 

Allows oral and visual

response 1 5 2 

Evaluator can control

collection procedures 3 5 1 

Facilitates interchange

with source 4 5 2 

What contents allow

Inclusion of most

relevant variables 3 5 4 

Complex subject matter

to be presented or

derived 3 5 3 

Collection of real-time

data 5 5 4 

Acquisition of historical

data 4 4 4 

Page 11 GAO/PEMD-10.1.5 Structured 
eSoud t e Used?

Structured interview Questionn

Extent of advantage

Characteristic or advantage

By

telephone

Face to

face By mail Grou

Universe or sample

Relevant universe to be

sampled can be

identified 4 5 4 

Facilitates contacting

and getting sample 3 2 4 

Allows use with large

sample 4 3 5 

Allows identity of source

to be known 4 5 3 

Reduces problems from

respondent’s illiteracy 4 5 1 

What time, cost, and

resources minimize

Instrument-development

time 2 3 1 

Instrument-development

cost 3 1 1 

Number of field staff 5 ? 5 

Travel by staff 5 ? 5 

Staff training 2 1 5 

Time required to carry

out activities ? ? 3 

Overall cost 3 1 5 

Results, response, and

quality of data

Maximize rate of return

of data after source is

contacted 4 5 3 

Minimize multiple

contacts of sources 2 2 3 

Page 12 GAO/PEMD-10.1.5 Structured 
eSoud t e Used?

Structured interview Questionn

Extent of advantage

Characteristic or advantage

By

telephone

Face to

face By mail Grou

Minimize follow-up after

initial response 5 5 3 

Increase chance source

will be accurate 4 4 4 

Allow reliability to be

checked 5 5 3 

Allow validity to be

checked 4 4 2 

Facilitate recall of data

by source 4 5 3 

Key:

1 Little or no extent

2 Some extent

3 Moderate extent

4 Great extent

5 Very great extent

? Depends greatly upon study specification

na Not applicable

In the job design phase of an evaluation or 

one-of-a-kind interview during the data co

analysis phase of an evaluation, the less-st

less-guided type of interview may be more 

Face-to-face interviews and telephone inte

generally more successful with responden

reading levels are low in comparison with 

complexity of the questions. In this radio a

television age, some respondent groups un

spoken words and sentences better than w

Page 13 GAO/PEMD-10.1.5 Structured 
eSoud t e Used?

The telephone interview and, even more, t

face-to-face interview enable the interview

establish rapport with the respondents. In

who would ignore mail questionnaires ent

would not answer certain questions on the

persuaded to provide truthful answers in a 

or face-to-face interview. Also, a well-train

interviewer can recognize when a respond

having a problem understanding or interpr

question and can employ the proper techn

assist the interviewee without jeopardizing 

integrity of the interview.

In comparison to the telephone interview, 

face-to-face interview gives the interviewe

opportunity to observe as well as listen. Fo

if it is required or desired that the interview

arrangements be noted, the face-to-face in

would be the choice. Also, more complex 

can be asked in a face-to-face interview th

telephone interview. Respondents can be s

cards with the complete set of possible res

making it easier for them to remember and 

all the choices. In addition, more question

asked. Twenty to 30 minutes is the usual li

telephone interviews, while face-to-face in

can last up to an hour.

Computer-assisted telephone interviewing 

one form of telephone interviewing. In CA

questionnaire or DCI is stored in a comput

questions are displayed on the computer s

during the interview, and the interviewer d

enters the responses into the computer. Te

interview costs generally fall somewhere b

lower mail survey costs and the higher per

interviewing costs. Also, depending on the 

sample, the number of interviewers availab

Page 14 GAO/PEMD-10.1.5 Structured 
eSoud t e Used?

number of questions, and question comple

telephone surveys can be completed quick

In comparison with mail questionnaires, fa

and telephone interviews are much faster 

gathering data. The need to train interview

their time spent traveling and contacting a

interviewing respondents, however, make 

face-to-face interview much more expensi

telephone interviews or mail or group que

Both forms of questionnaire can be longer 

include more complex questions (if the res

group is one that reads well) than is possib

telephone interview.

To administer a questionnaire in a group s

requires that it be practical to assemble th

respondents. Thus, it is normally used in s

which the sample is an entire group or a la

of it, such as an Army company or battalio

many agency employees in one location. G

questionnaires are faster than mail questio

permit some clarification of questions (bu

same extent as interviews). As with mail q

however, the language complexity used in 

questionnaires must be commensurate wit

reading level of the respondents.

In the past, GAO has used structured, face

interviews to study such topics as

• the self-reported experience of Work Incen

participants while in and after leaving the 

• the experience of participants trained by

Opportunities Industrialization Centers, an

• the opinions of Drug Enforcement Agency 

concerning various operating procedures f

periods.

Page 15 GAO/PEMD-10.1.5 Structured 
eSoud t e Used?

We used face-to-face interviews in the first 

because the respondent groups were not o

tend to respond in large numbers to mail

questionnaires, the subject matter was com

relationship to their reading levels, and the 

were too long to be done by telephone. In 

Enforcement Agency evaluation, the face-t

interview was used because time did not p

mail survey, the interview was too long for 

telephone survey, and the agents could no

assembled in a group.

GAO used structured telephone interviews 

such topics as the satisfaction of

• small businesses with management assista

provided by Small Business Administration

contractors and

• individuals having weatherization work do

houses under a federal program.

In both cases, telephone interviews were u

because the number of questions to be ask

small and time precluded a mail questionn

Questionnaires were administered in a gro

as part of GAO studies of

• cadets and midshipmen at the four service 

in regard to attrition and

• employees of the Federal Communication

Commission in regard to a proposed move 

Washington, D.C., area to Pennsylvania.

In both instances, it was practical to assem

respondents. Also, in the latter case, time 

permit a mail questionnaire.

Page 16 GAO/PEMD-10.1.5 Structured 
eSoud t e Used?

In general, GAO uses mail questionnaires m

frequently than group questionnaires, telep

interviews, or face-to-face interviews com

However, an understanding of structured i

techniques is essential for situations in wh

questionnaire cannot be used. Additional d

of structured interviews, questionnaires, a

DCIs, with examples of GAO applications, 

chapter 10.1 of the GAO Project Manual.

1

Questionnaires are discussed in the methodology tra

entitled Developing and Using Questionnaires.

Page 17 GAO/PEMD-10.1.5 Structured 
Designing a structured interview requires 

just writing down a set of questions to be a

this chapter, we first examine the process 

the interview questions are identified, dev

selected; then we describe standard proce

composing and formatting the questions. T

procedures aim to ensure that the data col

reliable and valid and to facilitate trouble-f

and analysis of data, while keeping the bur

interviewee to a minimum.

Reading or even studying this transfer pap

make anyone an expert in writing question

structured interviews. We suggest, therefo

work with measurement specialists from t

methodology, and technical assistance gro

division programming the assignment whe

planning to use a structured interview.

The DCI for structured interviews should b

by a design, methodology, and technical as

group if it involves 10 or more private citiz

firms, or local governments; 5 or more stat

governments; or 25 or more federal agency 

employees.1

 In certain executive agencies, 

designated representatives and established

procedures that must be followed when us

structured interviews and questionnaires.2

1

See GAO’s Project Manual chapter 10.1 on methodo

2

For example, see GAO’s Operations Manual, order 0

“Coordination of General Accounting Work at the De

Defense, Defense Agencies and the Military Departm

Page 18 GAO/PEMD-10.1.5 Structured 
Identifying

Variables and

Developing

Questions

The first step is to formulate the broad, ov

questions to be answered by the evaluation 

Why is the study being done? What do we 

able to say or prove? Are we primarily des

what has taken place in a program? Do we 

compare what has happened with some es

implied standard, a normative-type questio

we want to determine if a program has ma

difference, a cause-and-effect type questio

Examples of such questions are

• Descriptive: “How do graduates of the XYZ 

for the unemployed seek out and find jobs 

community?”

• Normative: “How well does the program m

goals for placing graduates in jobs?”

• Cause-and-effect: “Why do some graduates 

and others do not find jobs?”

The type of question asked will dictate the 

strategy. Also, certain strategies are more 

to answering certain questions.3 However, 

interviews, being simply a method of data 

can be used with several evaluation strate

thus, in a variety of GAO assignments.

After the broad overall questions are devel

must be translated into measurable elemen

form of hypotheses or questions. For the e

mentioned above, to evaluate how particip

jobs would require developing such measu

sources through which participants learne

available jobs, the number of employers co

and the number of job interviews arranged

target population must be identified. The t

population is the source level (individuals

organizations) at which the information is 

3

Formulating overall evaluation questions and selecti

strategies that provide answers is discussed in the m

transfer paper entitled Designing Evaluations.

Page 19 GAO/PEMD-10.1.5 Structured 
gathered. Thus, in the study of how progra

participants found jobs after leaving the pr

target population is the individual particip

program who were trained.4

Next, develop a pool of questions that atte

measure the variables under consideration

questions may include various ways of me

same variable. For example, for age, you m

“How old were you on your last birthday?” 

what day, month, and year were you born?

questions help you determine the individu

the second elicits much more information

which to use. From the pool of questions, 

most useful or appropriate are chosen.

The identification, development, and selec

questions for our example, a study of how 

participants found jobs after leaving a job-

program, are illustrated in table 3.1.

Table 3.1: Identifying, Developing, and Selecting Questions

Task Example

Formulate overall questions How do program participants find jobs after l

XYZ program?

Determine the kind of

information needed

1. Sources through which participant learned 

jobs

2. Number of employers contacted

3. Number of job interviews arranged

4. Number of interviews attended

5. Number of jobs offered

4

Later in the evaluations, data analyses may actually 

higher (more aggregated) level. In the example above

program may be conducted at several locations in a c

cities in a state, and in many states. Thus, several lev

would be possible. The objectives of the evaluation a

sampling plan devised to meet those objectives, how

the level or levels of data analysis.

Page 20 GAO/PEMD-10.1.5 Structured 
Task Example

6. Time (in days) it took to secure a job

7. Time (in days) since participant left progra

data collection

8. Relationship of job obtained to skill. . .

Identify target population Program participants who have left the progra

sample)

Create a question pool 1.1 How did you look for jobs?

1. Look in the newspaper?

2. Ask friends?

3. Go to a state employment office?

4. Go to a private employment office?

5. Look in the telephone book?

6. Drop in on companies?

7. Get information from radio or TV?

1.2 About how many jobs that you were intere

you find out about from

1. The newspaper?

2. A friend?

3. The state employment service?

4. Private employment services?

2.1 How many employers did you contact ab

since you left the program?

2.2 Since you left the program, about how ma

employers did you contact about a job that yo

about from

1. The newspaper?

2. A friend? 3. The state employment service?

3.1 How many . . .

Select questions 1.1 . . .

2.1 . . .

3.1 . . .

Page 21 GAO/PEMD-10.1.5 Structured 
Composing

Appropriate

Questions

When composing interview questions, be s

are appropriate—that is, relevant to the st

directed to the proper persons, and easily 

• Relevance: Questions should be relevant t

being conducted and should have a good p

of yielding data needed for the final report

this would seem obvious, evaluators some

“fishing expeditions” and want to include a

variables that can create an unnecessary b

the interviewee and distract attention from 

purpose of the interview.

• Selection of respondents: Give preliminary

consideration to which people can be expe

answer given questions. A question may be 

a given study, but the choice of persons to 

may be inappropriate.

• Ease of response: Interviews are meant to 

that may otherwise not be documented or

documented, may need some interpretatio

includes opinions and feelings about the st

You should attempt to construct questions 

relatively easy to answer and do not cause 

burden to the interviewee.

Avoid questions that require the interview

perform “audit work” to answer—that is, t

records or other information sources. If us

such questions should be reserved for mai

questionnaires. For telephone interviews, 

questions should be even less complex, be

is less of an opportunity to help the intervi

understand. It is possible to send the quest

beforehand to the person who will be inter

requesting that he or she gather the necess

information in preparation for the intervie

Other questions (or the manner in which t

presented) that cause the interviewee disc

Page 22 GAO/PEMD-10.1.5 Structured 
should be avoided or used with extreme ca

same is true of questions that would tend t

incriminate or show the interviewee in a b

particularly since the interview might term

they were asked. Likewise, avoid personal 

about private matters that do not belong in 

study, as well as questions whose sole pur

embarrass the interviewee (such as testing 

questioning the intelligence of the intervie

seeking information about private habits).

If needed, ask sensitive questions in a mail

questionnaire, where confidentiality or ano

be granted.5 Also avoid questions that cou

unnecessary confrontation, causing the int

and interviewee to take sides and do battle

detracts from the interview task, may caus

can seriously affect the validity of the answ

Also avoid questions that have no answers 

questions that, if you attempt to ask them, 

unusable results. These are not to be confu

course, with questions for which the legiti

answer might be “no basis to judge” or “no 

(presumably, some interviewees will not h

to make a judgment or give an opinion).

Selecting a

Question Format

Considerations in deciding on the format o

question to use include how the question i

or presented, what the interviewee is aske

available response alternatives. Among the 

questions we use are open-ended, fill-in-th

binary-choice, and scaled-response, as dis

below.

5

See our discussion on confidentiality and anonymity 

Page 23 GAO/PEMD-10.1.5 Structured 
Open-Ended

Questions

The open-ended question provides no stru

the answer, allowing the interviewee to dis

he or she wishes, not necessarily what the 

wants to know. By sharpening the question

focus it. For example:

• Broad question: “What happened to you w

were unemployed?”

• Focused question: “How did you manage t

bills while you were unemployed?”

Open-ended questions are easy to write. F

research, they can be used successfully to 

answers that contribute to the formulation 

specific questions and response alternativ

small number of respondents and where a

be qualitative, rather than quantitative, ope

questions may also be useful. If possible, a

open-ended questions with larger numbers 

respondents, whose answers need to be ta

Under such circumstances, content analys

done before attempting to tabulate.6

In CATI questionnaires, the questions shou

designed in accordance with the guideline

established for structured telephone surve

addition, other practices apply. For examp

open-ended questions should be avoided a

possible, primarily because of the time it t

the answer. If the topics addressed in the

questionnaire are at the exploratory stage, 

not recommended. A CATI requires some 

maturity in the understanding of the issues 

investigation. To the extent that open-ende

6

Discussed in the methodology transfer paper entitle

Analysis: A Methodology for Structuring and Analyzi

Material. Also see chapter 9.

7

Discussed in the methodology transfer paper entitle

and Using Questionnaires.

Page 24 GAO/PEMD-10.1.5 Structured 
are included in a CATI, they should be des

easy typing. Such questions take up consid

space in the computer data files. To the ex

possible, they should be moved to the end 

questionnaire and the interviewer should a

record the answers “off-line.” These questi

the potential for interrupting the flow of th

deflating the interview.

A question that actually is closed can be pr

such a way that to the interviewee it appea

open-ended. Do this by preparing a list of p

answers and checking these off during the 

as the interviewee mentions the various al

Do not, however, read the choices to the in

Such questions are more focused and spec

simple, open-ended questions and allow th

possible answers to be narrowed. Question 

3.1 illustrates the technique.

Figure 3.1: Question 1

Fill-In-The-Blank

Questions

This type of question has a simple answer, 

the form of a name, frequency, or amount. 

may prepare a list of alternative answers t

during the interview. Questions 2, 3, and 4 

3.2 illustrate this type of question.

Page 25 GAO/PEMD-10.1.5 Structured 
Figure 3.2: Questions 2, 3, and 4

Binary-Choice

Questions

This is the typical yes-no, true-false type o

good format for obtaining factual informat

generally not opinions or feelings. Since th

interviewee is asked to make a commitme

extreme or another, binary choice is consi

forced choice. Figure 3.3 shows an examp

Figure 3.3: Question 5

Scaled-Response

Questions

In the scaled-response question, you read 

the interviewee a scale—a list of alternativ

responses that increase or decrease in inte

ordered fashion. There are three types: bal

unbalanced, and rating and ranking scales

Page 26 GAO/PEMD-10.1.5 Structured 
Balanced Scales The end points of the balanced scale are u

adjectives or phrases with opposite meani

example, very satisfied and very dissatisfie

name implies, the balanced scale contains 

number of responses on each side of a refe

or neutral response, as shown in question 

3.4.

Figure 3.4: Question 6

This scale expands the binary-choice answ

discussed above, permitting a range of ans

better reflect the way people hold opinion

Unbalanced Scales The unbalanced scale is used when no neg

response is possible. It has a reference poi

a “zero” point or “none”) and the value of t

increases for successive points on the scal

ranges from none to very great.

Figure 3.5: Question 7

Rating and Ranking

Scales

In a rating question, the interviewee is ask

a rating to persons, places, or things accor

specified criteria. The points on the scale c

either numeric or verbal. An example of a 

is shown in figure 3.6.

Page 27 GAO/PEMD-10.1.5 Structured 
Figure 3.6: Question 8

Whether verbal or numerical, a rating scal

that the distance from one point to the nex

same on all parts of the scale.

In a ranking question, the interviewee is as

place items in order according to a specifi

as shown in question 9 in figure 3.7.

Page 28 GAO/PEMD-10.1.5 Structured 
Figure 3.7: Question 9

Ranking questions may have several types 

instructions. You can ask the interviewee t

as in the example, or to select the first (be

last (worst), the top three, or some other

combination.

In contrast to rating, ranking does not imp

distance between points is the same on all 

scale. For example, if Johnson, Green, and 

were ranked 1, 2, and 3, respectively, the in

may not necessarily think that the gap betw

Johnson’s and Green’s performance is the 

gap between Green’s and Smith’s.

When it is necessary to obtain the intervie

opinion as to the distance between items (

example, how much better or worse one e

than others), use a rating question. While a 

question may also produce an ordering, a r

may well give two or more items the same 

you want the interviewee to choose betwe

fewer items but you do not care how much 

or she believes one item is than the others

question is likely to give you what you wan

larger number of items must be ordered, h

will probably be easier for the interviewee

Page 29 GAO/PEMD-10.1.5 Structured 
them than to rank them. It is difficult to ju

order of a large number of items and avoid 

between items, especially in interviews. A 

can be produced by averaging the ratings o

respondents.

Number of Cues The number of cues (alternative responses

scaled-response questions depends on the 

interviewee and type of analysis expected

physical limit, generally, to the number of 

which an interviewee can react, probably a

seven. GAO usually uses five-point scales.

Respondents with a keen interest in the st

expected to handle a greater number of cu

more points on the scale, the better will be 

eventual analysis of the data, since more c

a more sensitive measure and allow the an

greater flexibility in selecting ways to anal

data.

An even number of cues used in a balance

generally eliminates a middle or neutral po

scale and forces the interviewee to commi

positive or negative feeling. The use of an

odd-numbered scale permits a neutral ans

more closely approximates the range of op

feeling that people can have.

When the possible responses do not includ

to judge,” “can’t recall,” or “no opinion,” th

interviewee may feel forced to select an an

is inaccurate. The point is that some peopl

may be unable to answer. If you have good 

believe this is so for members of the respo

group, include in the list of cues read or sh

interviewees the most applicable of the

alternatives—“no basis to judge,” “can’t re

opinion.” If you do not do this, the intervie

guess, make up an answer, or ignore the qu

Page 30 GAO/PEMD-10.1.5 Structured 
Order of Cues The order in which the cues are presented 

used to help offset possible arguments tha

interviewees are biased toward answering 

question in a particular way. Consider a sit

which GAO had preliminary evidence that

participants in a training program were no

counseling. The following question could b

“Job counseling involves someone talking to you abo

apply for a job, how to behave in an interview, etc. T

did you receive job counseling while you were in this 

The choices presented to the interviewee w

• “very great extent,”

• “great extent,”

• “moderate extent,”

• “some extent,”

• “little or no extent.”

In this example, the order of presentation 

choice slightly in favor of the program. Som

interviewees who did not take a strong int

question might select the first choice, indic

they received job counseling to a very grea

This would tend to give us an overall answ

slightly biased toward receiving job couns

When the cues form a scale, only at great e

could we totally eliminate the bias inheren

order in which the alternative responses a

presented.8

 To repeat, the bias is slight. Bu

8

To totally eliminate this type of bias requires that ha

be presented the cues in one order and the other half 

the cues in the opposite order. In our example, half t

would be presented a card on which “very great exte

first (or top) cue and “little or no extent” was the last 

cue. The other half of the sample would be presented 

which “little or no extent” was the first cue and “very 

was the last cue.

Page 31 GAO/PEMD-10.1.5 Structured 
does exist, we should use the logic of bias

question against the hypothesis we are exa

Wording of Cues As indicated in the previous example, the 

in the cues was the “extent” to which som

was performed. When an action or process 

assessed in a question, it is preferable to p

question and the cues in terms of the actio

previous question would generally be reph

“How much job counseling did you receive

could be rephrased as “A very great amoun

counseling,” “A great amount of counselin

moderate amount of counseling,” and so o

Unscaled-Response

Questions

In an unscaled-response question, a list of 

or shown to the interviewee, who is asked 

one from the list or to select all that apply

should consist of mutually exclusive categ

“other” category is usually included as a la

alternative, either to provide for many pos

thought to be rare) answers or if it is thoug

some interviewees will come up with uniq

Question 10 in figure 3.8 is an example of a 

in which only one response is to be given; 

in figure 3.9 is a question in which the inte

may check several responses.

Page 32 GAO/PEMD-10.1.5 Structured 
Figure 3.8: Question 10

Figure 3.9: Question 11

Organizing

Questions

In any DCI, the order in which the question

presented is important. Early questions, w

tone for the collection procedure and can 

responses to later questions, also help you 

know the interviewee and to establish the 

essential to a successful interview.9 For ex

9

Establishing rapport is covered in more detail in cha

Page 33 GAO/PEMD-10.1.5 Structured 
an interview with participants in the XYZ p

the first few questions could review for ac

obtained from agency files such as family

composition, age, and education.

The next questions should also be ones tha

answered with some ease, as you are still d

rapport with the interviewee. Should these 

questions be too difficult or too sensitive f

of relationship developed, the interviewee 

the interview. Remember also that the que

should hold the interviewee’s attention; th

must begin to introduce some “interesting

and the sensitive areas covering the attitud

interviewee.

Present the questions in a logical manner, 

flow of questions in chronological or rever

appropriate. Avoid haphazardly jumping fr

topic to another.

Also, avoid introducing bias in the orderin

questions. For example, to determine wha

interviewee thinks a program’s advantages 

disadvantages are, do not mention the pos

advantages or disadvantages earlier in the 

Generally, the set of questions asked varie

interviewee to interviewee. Many question

only if there is a specific response to a par

question. As a result, several questions ma

skipped. These interrelationships among t

questions constitute the skip pattern of the 

face-to-face interviews and telephone inter

do not use a CATI system, the complexity 

skip pattern should be kept to a minimum

it becomes very difficult for the interviewe

the next question to be asked.

Page 34 GAO/PEMD-10.1.5 Structured 
One of the important advantages of a CAT

questionnaire is that it allows for consider

complexity in the skip pattern, since the b

handled entirely by the computer. Any num

paths can be followed through the questio

Usually, the computer displays the next qu

sequence. Alternatively, conditional skips 

programmed to go from one specific quest

another somewhat later in the questionnai

skips can be based on how the interviewee 

single question or on the responses to seve

questions.

One drawback to a CATI questionnaire is t

multiple-choice questions permitting sever

are not easily handled. It is difficult for an 

to remember all the options when several 

chosen. As a result, multiple-choice questi

allowing the interviewee to check all that a

illustrated in figure 3.9) should be broken 

separate questions, each of which is an alt

response that is “checked” or “not checked

Layout

Considerations

The layout or form of a printed DCI (for no

applications) (see figure 3.10 for an examp

important; it is what you carry into the inte

use as a guide to conducting it. It gives you

on-the-spot instructions for each question 

you to record the answer. Later, the form i

facilitate editing, keypunching, and the sub

computerized analysis.

Page 35 GAO/PEMD-10.1.5 Structured 
Figure 3.10: Structured Interview Text

Page 36 GAO/PEMD-10.1.5 Structured 
Here are some considerations when design

DCI.

Typeface. Generally the text to be read to 

interviewee is set off in a different typefac

instructions that you do not read to the int

In the example presented in figure 3.10, th

read to the interviewee is presented in upp

lowercase, the instructions in upper- and l

italics.

Continuation of questions. Generally, do n

a question in the next column or on the ne

you risk not having the entire question or a

response alternatives presented to the inte

Boxes and lines. Provide open-top boxes f

interviewer to record answers to questions 

require written responses. Place the box o

standard place beside each question to aid 

interviewer and to facilitate editing, data e

subsequent analysis of completed question

Keypunch numbers. These should be place

standard place beside each question to fac

keypunching, when data are to be entered 

computer files.

Skipping questions. If a certain response to 

means that interviewers are to skip the ne

specify this by placing a “GO TO” instructi

the response.

Two computer programs are available to a

design and layout of structured interviews 

questionnaires. They are QUEST (develop

Office of Information Management and

Communications) and QPL (Questionnaire

Programming Language, developed by the 

Page 37 GAO/PEMD-10.1.5 Structured 
methodology, and technical assistance gro

Human Resources Division).

QUEST makes it possible for an evaluator 

questionnaires using laser printers, incorp

typographic and graphic elements (such as 

boxes, arrows, and italic type).10 QUEST

automatically handles many of the layout

considerations mentioned above. Draft an

versions of typeset questionnaires, incorpo

current desktop publishing concepts, can b

generated quickly. The designer develops 

WordPerfect file employing codes that ide

questionnaire elements and control typogr

layout on the page. These codes make it po

correct questionnaires easily, automaticall

renumbering pages, questions, and choice

altering keypunch instructions.

QPL is designed to automate many of the a

involved in gathering and preparing survey 

analysis.11 It was developed primarily to im

CATI questionnaires within GAO; it can als

as a data entry program for other DCIs. In 

the questionnaire is first written in QPL, u

processing program, and then compiled. T

compiled version displays the questions on 

computer screen, one at a time, and then w

interviewer to type a response. The intervi

10The question examples in this paper were prepared 

A manual is available from the Office of Information 

and Communications, Publishing and Communicatio

new technology becomes available, QUEST will be im

make it more powerful and even easier to use. More 

desktop publishing software may also facilitate quest

development.

11See the QPL reference manuals: QPL Reference Ma

2.0 (HRD Technical Reference Manual 1, March 1990

Collection Program (HRD Technical Reference Manu

March 1990); and QPL Data Editing Program (HRD T

Reference Manual 3, March 1990).

Page 38 GAO/PEMD-10.1.5 Structured 
page forward and backward through the

questionnaire to make corrections or revie

The record of the interview is then added t

file. The compiled version can also be con

SAS and SPSS statistical analysis program

process QPL data files. One of the program

QPL system reformats the computer quest

into a written questionnaire, numbering al

questions, drawing open-top boxes for the 

specifying card and column locations for e

in the data file, and writing skip instruction

QUEST, the questionnaire does not incorp

typographic and graphic elements, but QP

easy to review and revise a questionnaire.

Page 39 GAO/PEMD-10.1.5 Structured 
Problems

In this chapter, we suggest further ways to 

good interview questions and to forestall p

with comprehension or bias. As an evaluat

such questions, you need to consider the

appropriateness and level of language used 

interview, the effects of qualifying languag

importance of clarity. We also discuss the 

kinds of bias that can creep into the wordi

interview questions and their effect on the 

the evaluation results.

Appropriateness

of the Language

Whether interviewing language is appropri

inappropriate may relate to what is said, h

said, or when it is said, as discussed below

What is said in the interview is basically di

the written, structured data-collection inst

The DCI is prepared in advance and pretes

interviewers are trained to use it; thus, to s

extent, the appropriateness of the languag

tested. It is the task of the interviewer to tr

faithfully to the interviewee the meaning o

questions. In addition to wording the ques

precisely, you may include supplemental l

the DCI, to be used if the interviewee does 

understand the original wording of a quest

the course of the interview, the interviewe

not understand and different language mu

improvised, such improvisations should be 

considered before the data are analyzed.

How it is said concerns the speech and ma

the interviewer who controls the “presenta

whose delivery of questions may alter thei

meaning. More detailed information on thi

appears in chapter 8.

Page 40 GAO/PEMD-10.1.5 Structured 
obes

When it is said refers to the context of the 

in which each question is placed. Although

designing the DCI, you should be precise a

order in which questions are asked, you m

introduce some variation during the actua

to clarify the questions, review information

postpone potentially sensitive questions. O

interviewee expresses concern or sensitivi

given question, changing the language of a 

question might defuse the concern.

Level of the

Language

When composing interview questions, con

level of the language used. Seek to commu

the level the interviewee understands and 

verbal setting that is conducive to serious

data-gathering yet one in which the intervi

comfortable. In chapter 3, we touched on s

writing approaches to use; here we deal w

questions sound and the atmosphere the la

creates. One problem often encountered is

maintaining a level of language that is neit

nor below the interviewee’s level of under

Speaking over the interviewee’s head inclu

of complex, rare, and foreign words and ex

words of many syllables, abbreviations, ac

and certain jargon. Such language, while it 

appropriate to the interviewer or evaluatio

may not be understood by the interviewee

For example, when interviewing participan

training program, the terms “OJT” and “PS

question may be nothing but alphabet soup 

interviewees; even the words they represe

“on-the-job training” and “public service

employment,” may be over their heads. In 

the actual interview, you would most likel

give further definitions or examples of wh

Page 41 GAO/PEMD-10.1.5 Structured 
obes

meant. When interviewing training program 

however, the use of “OJT” or “PSE” would 

appropriate if the interviewees use the term

Thus, to speak over the interviewee’s head 

communication. Interviewees who are em

their lack of understanding may either not 

guess at the meaning, which can lead to in

answers. Or the interviewee may get the im

that you really do not care about the answ

interest in the interview.

Speaking down to an interviewee is just as 

can oversimplify the language in the DCI t

where the interviewees feel you regard the

ignorant. This approach is demeaning. You 

contacted these individuals because they h

important information to impart. To treat a 

condescendingly—or to let it appear that y

do—negates that importance.

Likewise, take care in using slang, folksy e

and certain jargon. While such language m

develop rapport with the interviewee, the 

of the communication may be lessened.

To avoid error in either direction, pretest b

final wording of the DCI and the interview 

Use of Qualifying

Language

After composing an interview question, yo

it requires an adjective or qualifying phras

a time specified to make the item complet

the interviewee sufficient or complete info

For example, “How many employees do yo

might become “How many full-time-equiva

employees do you have?” and “How many 

you gone to a physician?” might become “H

1

More detailed information on pretesting appears in c

Page 42 GAO/PEMD-10.1.5 Structured 
obes

times have you gone to a physician in the p

months?”

If feedback is possible in the actual intervi

interviewee can ask for further qualificatio

needed. If you have not included the neces

qualifiers in the DCI, however, another int

may qualify in a different way. This could m

resulting data difficult to summarize and a

Also, interviewees, not realizing that qualif

language is absent, may answer the questio

interpret it. Thus, different interviewees w

responding to different questions, based o

interpretations.

Clarity of

Language

The style in which a question is couched c

clarity of communication. We discuss belo

matters as question length, complexity, an

double-barreled questions; double negativ

language; and defining terms.

Length, Complexity,

and Clutter

A question that contains too many ideas or 

may be too complex for the interviewee to

understand, especially if it is presented ora

makes it difficult for the interviewee to rev

of the question. While the interviewee mig

responding to one part of the question, the

interviewer may be interpreting the respon

response to the entire question. You shoul

more than one thought in separate sentenc

the interviewee the proper framework. Fo

“How satisfied or dissatisfied were you wi

amount of time devoted to helping you get 

you were in the XYZ program?” becomes “

about the training experiences you had wh

XYZ program. How satisfied or dissatisfied 

Page 43 GAO/PEMD-10.1.5 Structured 
obes

with the amount of time devoted to helpin

job?”

Likewise, a sentence may contain clutter—

do not clarify the message. Word question

Here are a few tricks to reduce sentence c

• Delete “that” wherever possible—for exam

“Others suggest [that] training can be impr

• Use plain language. For example, for

“aforementioned,” use “previous” or “prev

mentioned.”

• Avoid the passive voice. Substitute pronou

“we,” or “they”) and active verbs; instead o

necessary to obtain,” use “We need.”

Double-Barreled

Questions

A double-barreled question is a classical ex

an unclear question. Consider the followin

get skill training while in the program and 

completing the program?” This question at

determine if there is a relationship betwee

training and getting a job. But if the intervi

answers “yes,” this could mean “yes” to bo

“yes” to the training part only, or “yes” to t

only. Other interviewees, finding the quest

confusing, might not respond. You are pre

questions but the opportunity to record on

answer. Both interviewee and interviewer 

need for only one answer. State the questi

separately.

Double Negatives In phrasing a question, avoid the double ne

which is difficult to answer. For example, 

which of the organizational goals listed be

considered unattainable within the 2-year 

should be reworded to read “Indicate whic

Page 44 GAO/PEMD-10.1.5 Structured 
obes

organizational goals listed below are consi

attainable within the 2-year period.”

Extreme Words Avoid such words as “all,” “none,” “everyth

“never,” and others that represent extreme 

Rarely is a statement using such a word tru

use of extreme words causes interviewees 

the end points of a scale. There are cases w

use of “all” or “none” is appropriate, but th

Where “yes” or “no” answers are expected

can be misleading. For example, if one em

not covered in a question like “Are all of yo

employees covered by medical insurance?

answer is impossible. A better question wo

“About what percent of your employees ar

by medical insurance?”

Alternatively, choices can be provided, as 

1 in figure 4.1.

Figure 4.1: Question 1

Defining Terms Where possible, define key words and con

in questions. For example, when speaking 

“employees,” define and clarify the term. A

talking about part-time, full-time, permane

Page 45 GAO/PEMD-10.1.5 Structured 
obes

temporary, volunteer, white-collar, blue-co

example of how this might be done is

“Consider people who work for your company, are p

your company, work at least 35 hours per week, and 

permanent employees. What percent of these employ

Of course, not all questions need be prece

a definition. As earlier questions are devel

definitions evolve. You may wish to list de

a separate section or on a card to hand to

interviewees for reference.

Bias Within

Questions

A question is biased when it causes intervi

answer in a way that does not reflect their 

positions on an issue. An interviewee may 

be aware of the bias. Problems result when 

interviewees are

• unaware of the bias and influenced to resp

way that is directed by the wording or

• aware of the bias and either deliberately a

way that does not reflect their opinions or

• refuse to answer because the question is b

Bias can appear in the stem (or statement) 

the question or in the response-alternative 

Bias may also result when a question carri

implied answer, choices of answer are une

“loaded” words are used, or a scaled quest

unbalanced. These are discussed below.

Implied-Answer Bias A question’s wording can indicate the soci

acceptable answer. An example is the ques

GAO employees have subscribed to the U.

Bond program. Have you subscribed?” Inte

who are concerned about being different f

norm may answer “yes,” even if they have 

Page 46 GAO/PEMD-10.1.5 Structured 
obes

subscribed. The question could be restated 

you subscribed to the U.S. Savings Bond p

Questions can be worded so as to impel so

to answer in one direction and others in an

both types of interviewee could be unawar

bias in the wording. Such bias usually occu

additional qualifying or identifying informa

added to the question. There is bias in the 

“Which plan is more acceptable to you: the 

designed by Pat Brown, our chief economi

one designed by Chris Green, the consulta

hired?” The interviewee who is not familia

either plan may answer on the basis of wh

plan was generated internally or externally 

organization, although this may have little 

to do with the quality of the plan. A better

presentation would be “Whose plan is mor

acceptable to you: Pat Brown’s or Chris Gr

Bias Resulting From

Unequal Choices

When response alternatives are created, it 

important that they appear to be equal. If u

emphasis is given to one, it may be easier f

interviewee to select that one. Question 2 

illustrates a question with unequal emphas

question 3 in figure 4.3 corrects the unbala

Alternative 3 in question 2 is isolated from 

others because of the words “high-paid,” w

those individuals apart from the others, an

fact that alternative 3 is longer than the oth

Page 47 GAO/PEMD-10.1.5 Structured 
obes

Figure 4.2: Question 2

Figure 4.3: Question 3

Bias From Specific

Words

When used in almost any context, certain w

be considered “loaded,” because they evok

emotional feelings. “American,” “freedom,

and “justice” generally evoke positive feeli

“communist,” “socialist,” “bureaucrat,” and 

holocaust” may evoke negative feelings. Si

difficult to control the emotional connotat

such words, it is usually best to avoid them

Bias From Lack of

Balance

When using a scaled question, avoid bias in 

as well as in the response alternatives. A q

seeks to measure satisfaction with someth

mention both ends of the scale in a balanc

Page 48 GAO/PEMD-10.1.5 Structured 
obes

For example, question 4 in figure 4.4 show

in both the stem and the alternatives, whil

in figure 4.5 shows how this bias is elimina

Figure 4.4: Question 4

Figure 4.5: Question 5

Considerations

for Telephone

Interviewing

Instruments

In general, the same principles described a

to the development of questions and answ

telephone interviews. However, some addi

considerations come into play. The primar

factor is that the cues available in face-to-f

interviews are absent. It is not possible to 

interviewee’s reactions (including confusio

uncertainty, or hostility) and make allowab

2

Proper use of an unbalanced scale was discussed in 

Page 49 GAO/PEMD-10.1.5 Structured 
obes

adjustments in conducting the interview.3 

questions shorter, breaking multiple-choic

into binary questions, and conducting som

face-to-face will overcome some of these d

Another loss in telephone interviewing ari

impersonal nature of the telephone. An int

has a tendency to become flatter in presen

interviewer must counter this tendency by 

continually aware of the enunciation of qu

the QPL system, some words are capitalize

underlined, or put into bold type to help th

interviewer maintain appropriate pitch and 

In summary, designing a structured intervi

not simple. It involves many consideration

choices: the specific questions to be asked

format, language order, and layout. In this 

and chapter 3, we have covered briefly the 

principles that should be followed in maki

choices.4

3

See chapter 8 for details.

4

For more information, consult Bradburn and Sudma

Sudman and Bradburn (1982), as listed in the bibliog

Page 50 GAO/PEMD-10.1.5 Structured 
Pretesting and expert review constitute pe

least appreciated phase in the developmen

structured interview.1 In the desire to mee

for getting the job done, staff may ask “Wh

eliminate the pretest?” or “Do we need out

opinions on the interview form?”

But these are perhaps the most important 

development of the interview, an iterative 

that uses continuing input from evaluators 

technical specialists to derive the final pro

Cannell et al. (1989) indicate, when the eva

little experience with a topic or when the i

has difficulty with a question, substantial w

be necessary to develop questions that wil

desired results. Research has shown that q

formulation may alter results by as much a

50 percent. The pretest and expert review 

give the evaluators feedback as to whether 

stand a chance of doing what they are desi

Following pretesting and expert review, th

redesigned as needed—an iterative proces

occurs after each pretest or group of prete

Purpose of

Pretest

In pretesting, we test the DCI with respond

from the universe of people who will even

considered for the study interviews to pre

well the DCI will work during actual data c

The pretest seeks to determine whether

• the right questions are being asked to obta

needed information,

1

The term “pretest” is not interchangeable with “pilot

usually used in connection with the testing of a struc

interview or questionnaire, while “pilot” implies a tes

of the complete study design at one field location bef

proceeding to implement the design at all selected lo

Page 51 GAO/PEMD-10.1.5 Structured 
• the contents or subject matter of each que

relevant to the respondent and the respon

knowledge to answer the question, and

• the wording and procedures used in condu

interviews are adequate to ensure that vali

reliable results are obtained.

Research (Cannell et al., 1989) has shown 

following to be among the types of problem

arise with survey questions:

• difficulties in asking the question because 

sentence structure, tongue twisters, or wo

difficult to pronounce;

• difficulties in comprehending the question 

difficult vocabulary, complex sentence str

lack of specificity about information or the 

information that is needed;

• lack of common understanding of terms or 

in the question because they are interprete

differently by different respondents or the

interpreted differently from what the inter

designer intends; and

• difficulties in answering the question beca

information is inaccessible or unavailable 

respondent, to the extent that the respond

not want to make the effort to obtain the a

Pretest

Procedures

The number of pretests typically varies de

the size of the survey and the range of con

may affect the survey results. For structur

interviewing of thousands of respondents, 

pretests might be conducted. Sometimes, w

sample is less than 500, a dozen or fewer p

are sufficient, provided they bracket the ra

collection conditions. Discuss the exact nu

the measurement specialist who designed 

a great degree, the pretest procedures for 

Page 52 GAO/PEMD-10.1.5 Structured 
structured interview simulate what would 

during actual data collection. It is importa

many of the procedures involved in condu

structured interview as possible, including 

selection of and contact with the interview

part, pretests should be conducted in the s

to be used in the actual interviews—that is

face-to-face interview pretested in person 

telephone interviews over the telephone. H

telephone and mail surveys should also be 

part in face-to-face interviews. For CATIs, 

generally have fewer than 300 interviews, 

pretests might be sufficient. These pretest

conducted both in person and over the tele

Who Conducts the

Pretest

Two types of staff should represent GAO a

pretest:

• the evaluator working on the job, because 

can best address questions on the content 

and the background of the evaluation, and

• the measurement specialist who designed 

because he or she needs to evaluate the in

process, including how the DCI works, and 

improvements.

The measurement specialist acts as the

interviewer—that is, asks the questions on 

and perhaps the second pretest—while the 

observes. On subsequent pretests, the eval

the questions and the measurement specia

as observer.

Selecting and

Contacting Pretest

Interviewees

Pretest interviewees are drawn (not neces

randomly) from the universe being consid

final study. If the universe is relatively

homogeneous—for example, welfare recip

Page 53 GAO/PEMD-10.1.5 Structured 
pretest subjects need not be exactly balan

various attributes. With a heterogeneous g

as taxpayers or U.S. citizens, however, try 

pretest interviews with high- and low-inco

old and young, the highly educated and les

and women and men. Ideally, the DCI is pr

with several of each of the different kinds 

individuals in a heterogeneous group.

Contact pretest interviewees by telephone 

person to arrange a pretest session. If poss

procedures similar to those proposed for a

collection. Identify yourself, describe wha

agency GAO is and what it does, explain th

the study, and indicate the importance of t

participation. If this is a face-to-face pretes

interviewee to participate by arranging to 

place that is convenient to the interviewee 

distractions. If this is a pretest of a telepho

interview, arrange a time that is convenien

interviewee. (For a more detailed explanat

copies of text to be followed, see chapter 

Conducting the

Pretest

The initial steps of a pretest are the same a

data collection. Give the interviewee any a

background information, even if you have 

this while setting up the interview appoint

an interview is interactive, the interviewee 

probably provide a great deal of feedback 

to answering the questions. Problems with 

procedures often become evident immedia

may be dealt with then, so that the intervie

proceed. Often, if an instruction, word, or 

not understood, the interview cannot cont

Ideally, however, it is desirable to run thro

entire interview without getting sidetracke

Page 54 GAO/PEMD-10.1.5 Structured 
way, you can examine the flow of the inter

estimate the total time needed to complete 

During the pretest, then, your tasks as inte

to

• carry on the normal introduction and ques

an interview without too much interruptio

flow,

• provide explanations or try alternative wo

when the interviewee cannot or will not an

question and note the changes introduced,

• record the answers on the DCI so the reco

procedure and coding technique can be te

• make notes on situations that occur during 

interview that indicate problems with the i

or procedures, and

• conduct a debriefing at the end of the inter

learn what the interviewee thought of the 

but did not express.

With respect to the second item, providing

explanations or alternative wording must b

carefully, since interviewer bias can occur

interview is written as bias-free as possibl

deviating from the prescribed text, you ma

time to rephrase the question adequately a

make a slip in wording that favors or is sla

toward your approach to the situation.

For telephone interviews, it may be easier 

the pretests and they may be more informa

interviewee should be informed that a mea

specialist will be listening for purposes of 

instrument. It may be possible to use a spe

to allow more members of the team to list

notes, and record answers without intrudi

interviewee’s permission, the interview ma

to allow for more detailed examination of 

Page 55 GAO/PEMD-10.1.5 Structured 
With these possibilities, pretesting telepho

interviews may be a lot smoother than pre

face-to-face. However, as mentioned abov

remember to include some face-to-face int

Identifying Problems After a pretest, the evaluator and the meas

specialist review the interview process and 

identify any problems that the interviewer 

asking the questions or the interviewees ap

have in answering questions. If the pretest

problems such as ambiguous interpretatio

difficulties (discussed below), you must re

interview and continue the tests until the p

are resolved, even if this requires unplanne

time. Premature termination of pretests ca

questionable data. Major indicators of prob

include the following:

• slight changes by the interviewer in readin

question;

• major changes by the interviewer in readin

question or not finishing reading the quest

• interrupting the question-reading with an a

• difficulty in answering, as evidenced by th

interviewee’s nervousness, inconsistency i

from question to question, inconsistency b

back and changing answers, or taking too 

answer;

• clarification, explanation, or repetition of 

requested by the interviewee;

• qualified answer provided by the interview

• inadequate answer provided by the intervi

including giving a different answer than on

the instrument or known to the interviewe

• “Don’t know” answer given by the intervie

• refusal to answer by the interviewee.

Page 56 GAO/PEMD-10.1.5 Structured 
The problems fall into two basic categorie

related to instrument design or administra

those concerning the interviewee’s lack of 

or reluctance to answer. The first type can 

controlled by the staff designing the instru

are covered in chapters 3 and 4, while the 

merely recorded as observed behavior.

Research has found (Cannell et al., 1989) t

interviewers are not consistent in identifyi

problems with the questions or providing g

for their revision. Responses can vary by a

50 percent when there are no adequate con

the quality of the questions and procedure

techniques (categorization of respondent b

and use of probe questions) that have been 

are useful particularly when the number o

interviewers is large. The first method sim

tabulating for each question how often eac

the problems mentioned above occurred a

interviews. A small percentage of interview

expected to have some problem for each q

however, for a given question, a high perce

interviews has a specific problem, this sug

question needs revision.

The second method, use of probe question

used by itself or to clarify the nature of the 

identified from the first method. Special pr

questions may be included in the interview 

used at the end to ask interviewees to elab

answer, explain how they interpreted the q

answers, or describe any difficulties. Ther

types of probes:

• general probes ask for more information a

particular questions or the entire interview

Page 57 GAO/PEMD-10.1.5 Structured 
• information retrieval probes ask whether

interviewees had difficulty in recalling the

information necessary to respond to the qu

• comprehension probes ask interviewees h

interpreted particular questions or phrases

Purpose of

Expert Review

Because no instrument is perfect, it is gen

to seek outside commentary on our appro

seek expert review on assignments using s

interviews to help us determine whether

• the questions being asked and the manner 

they are asked are adequate to answer the 

question posed in the evaluation,

• the intended interviewee group will have t

knowledge to answer the questions, and

• the instrument is as well constructed as po

within state-of-the-art confines.

In many instances, officials from the agenc

program is under review serve in this capa

obtaining agency input at this stage, we av

potential problems after data collection, w

and money have already been spent. In oth

staff in other design, methodology, and tec

assistance groups, PEMD staff, or individu

subject-area or evaluation expertise can pr

expert review. In particular, subject-matte

membership associations who provide us 

the respondent universe or sample can pro

review.

Persons providing expert review are not a

interviewees. They do not answer the ques

instead provide a critique.

2

For more information on conducting a pretest debri

methodology transfer paper entitled Developing and 

Questionnaires.

Page 58 GAO/PEMD-10.1.5 Structured 
Instrument

Redesign

The evaluator and the measurement specia

consider the results of the pretest and exp

and make appropriate changes to the DCI. 

are minor, the instrument can be used with

pretests; if extensive, another series of pre

be necessary.

If pretesting can be spread over a longer p

time, more versions of the instrument can 

and a smaller number of interviewees can 

with each version. Changes that are obviou

can be made and the revised version can b

the next pretest. This allows us to use a re

more perfect version on each round of pre

Page 59 GAO/PEMD-10.1.5 Structured 
In most cases, our own evaluators conduc

interviews for GAO studies, but occasiona

employees of other agencies or contractor

Regardless, the interviewers must be train

purpose of the evaluation and the procedu

conducting the interview.

Training Methods GAO uses various ways of training its inter

and helping them maintain their skills thro

data-collection period: a job kickoff confer

interview booklet, role-playing and field pr

supervisory field visits and telephone cont

addition to the items discussed below, inte

training should emphasize the skills descri

chapter 8 for conducting the interviews, w

particular attention to structured interview 

probing techniques, and reinforcements. T

also discussed below.

Kickoff Conference For most projects of any size, a GAO divis

kickoff conference to tell the staff from th

and other divisions the purpose of the eva

make assignments, and to answer question

project is to include structured interviewin

data-collection phase, the conference is us

extended so the interviewers can be given 

instructions on the use of the DCI. Prefera

potential interviewers should attend.

If a region sends only one representative t

kickoff conference, for example, it should 

individual who will be conducting intervie

study. Not all aspects of the training can b

into the interview booklet (discussed in th

section); thus, practice sessions must invo

with the measurement specialist, those wh

Page 60 GAO/PEMD-10.1.5 Structured 
actually conduct interviews and possibly w

others in the region to do so.

The training begins with the evaluator in c

the measurement specialist reviewing the 

the study and how the interview data will f

overall objectives. Then, the data-collectio

procedures are covered in detail, using the 

booklet. The trainers discuss the interview 

question by question, including the need fo

possible rephrasing to be used if a questio

understood by the interviewee, how to rec

answers, and other matters they think cou

The trainees can ask questions, clarify item

typographical errors in the DCI, and sugge

changes from their experience. Even at su

date as the kickoff conference, changes ca

in the DCI to preclude problems being car

actual interviews.

Among the potential problems that the tra

usually make special efforts to address is m

that the interviewers

• know what an adequate answer to each qu

Without this knowledge, they may accept a

inadequate answer. A structured interview 

show the interviewer, by means of the resp

choices, what is adequate and what is inad

this to be learned, the interviewer must un

the DCI.

• ask the questions correctly. The words are 

paper; the interviewers need to be persuad

them in the way they are presented to ensu

standardization of meaning and delivery an

elimination of bias. Even though the instru

pretested, some interviewees will still hav

understanding the language. The interview

know enough about the question that if rew

Page 61 GAO/PEMD-10.1.5 Structured 
for clarity is the only recourse the intervie

rewording will not violate the intent of the 

• do not omit questions they think are answ

other questions. Answers are expected to 

questions, unless instructions call for an it

skipped or the interviewee refuses to answ

(Refusal can be considered an answer.) If 

interviewee gives the answer to a question 

asked, the interviewer should either ask th

anyway or give it as a statement for the int

affirm.

• do not introduce bias in the way they ask t

questions (see the discussion of this in cha

Interview Booklet Where the interview questions are limited 

and not very complex or difficult and the s

members who will conduct the interviews 

develop the DCI, we use the kickoff confer

to inform the interviewers in detail how ea

should be handled.

If, however, a large-scale interview effort i

undertaken, GAO project staff may prepar

that discusses in detail each question in th

booklet is similar to that issued by the Bur

Census to its enumerators.) Typically, GAO

cover not only the interview questions but 

matters such as sampling procedures, con

interviewees, and coding procedures. Thes

discussed below.

Sampling Procedures Where statistical sampling procedures are 

to select interviewees, the booklet shows t

interviewer how to identify the universe an

the sample. The booklet may include a

random-number table, when necessary, an

Page 62 GAO/PEMD-10.1.5 Structured 
both simple random samples and more com

two-stage procedures.

Interviewee-Contact

Procedures

Rules are provided for contacting the pote

interviewee and deciding what to do if the 

refuses or cannot be located. An example 

phone conversation to set up the interview

covered is the log interviewers must keep 

interview contacts to ensure that proper s

maintained. The log makes it possible to a

universe later and to examine possible effe

nonresponse. For CATIs, many of the cont

logging procedures are handled automatic

computer. How this is to be accomplished 

described to the interviewers during traini

Coding Procedures The booklet shows interviewers how to co

various types of question to facilitate editi

keypunching the answers and reviews diff

of questions. This is handled automatically 

Role-Playing

Practice

This is nothing more than two staff membe

turns “playing” interviewer and interviewe

method that should start at the kickoff con

a group session with the measurement spe

observing and critiquing. The role-playing 

continue when the staff members return to 

regions, particularly if regional staff memb

did not attend the conference will also be 

interviews.

Such role-playing gives staff members the 

become familiar with the instrument from 

of the interview. The person playing the in

should challenge the interviewer by giving 

“hard time,” perhaps refusing to answer qu

pretending not to understand them. Somet

serves to show the weaknesses of question

Page 63 GAO/PEMD-10.1.5 Structured 
unclear or lack sufficient response alterna

the evaluator in charge or measurement sp

should be notified so the items can be cha

clarification can be given to all interviewer

For CATIs, the interviewers must also be t

the software requirements. This should be 

the training in the details of a paper versio

DCI. The computer training first focuses o

mechanics of using the computer for a CA

the interviewers how to start the CATI, mo

cursor and step through each screen, mov

forth between questions, and identify parti

situations that may arise.

After the essentials of the DCI and the com

been covered, the interviewers can procee

role-playing, this time using the computer 

office-to-office mock interviews. The evalu

charge or measurement specialist should o

these sessions to identify not only weakne

DCI but also any difficulties in using the co

This role-playing should be practiced for a 

full day.

Field Practice Once evaluators are in the field at the first 

should oversample the number of interview

needed for that site and use some for field

interviews. These interviews are planned a

throw-away cases, identified as such in ad

interview. The data derived from an interv

used in the final analysis, regardless of wh

interview went well or poorly. Interviewin

interviewees who do not count gives interv

chance to get rid of any anxiety and test ou

approach. The interviewees, however, sho

told that this is a practice session. To them

Page 64 GAO/PEMD-10.1.5 Structured 
real thing; they will, therefore, exhibit all t

and concerns of any interviewee.

Obviously, field practice takes some time a

be built into the project schedule. After pr

interviewers should discuss any problems 

and decide where they need to change the

or learn more. Any lasting concerns should 

to the evaluator in charge or the measurem

specialist.

Supervisory Field

Visits

Normally, the evaluator in charge makes fi

during the course of an evaluation. A visit 

data-collection phase, when interviewing h

begun, is valuable, allowing the evaluator i

review the procedures being used to condu

interviews and observe some interviews fi

This quality-assurance checking enables th

in charge to ascertain that interviewers are 

out the standard practices designed into th

structured-interview procedures. If possib

measurement specialist should participate 

the visits.

For CATIs, it may be more difficult to main

supervisory controls. To the extent possib

interview should be recorded on paper as 

made into the computer, so that the accur

computer input can be verified. Large orga

that conduct CATIs frequently provide the 

of a supervisor to monitor calls by intervie

usually at random or when the interviewer

experiences problems. This is not usually 

GAO CATIs. In some instances, it may be u

tape initial interviews, with the interviewe

permission, in order to remove any final pr

associated with the interview administrati

Page 65 GAO/PEMD-10.1.5 Structured 
Supervisory

Telephone Contacts

The evaluator in charge and measurement 

form a team that keeps interviewers inform

changes in procedure and receives comme

the field on progress and problems encoun

These telephone contacts serve as the fina

training interviewers.

Interviewer

Qualifications

Many GAO interviews are highly sensitive, 

data to be obtained can be influenced by s

elements that are in the control of the inte

When GAO uses outside sources to supply

interviewers, it usually retains the right to 

the work of interviewers and, if there is ca

suggest that some be replaced. The same a

GAO evaluators whom the region or divisi

to the project. Staff members who are relu

conduct the necessary interviews or exhib

may not be right for the job and could jeop

data-collection effort.

For CATIs, the skill level and content know

interviewers can be lower than for face-to-

interviews because the questions are gene

simpler and fewer probes need to be used

GAO contracts for CATIs or the use of sho

part-time staff have been quite successful 

alternatives to the use of evaluators.

The qualifications that interviewers exhibi

various training opportunities should be ev

supervisors. If there are any problems that 

corrected through retraining, these intervi

should be replaced.

Page 66 GAO/PEMD-10.1.5 Structured 
This chapter touches briefly on the selecti

interviewees and then discusses in some d

contacting the prospective interviewees, a

the interview, and protecting the interview

informed consent and guarantees of confid

anonymity).

Selection of

Interviewees

For some structured interviews, because t

one person who fits the category of intervi

example, state officials responsible for we

programs), no selection process is needed

More-complex selection procedures that a

required—for example, when the sampling 

for a random sample of program participan

respondent groups—are covered in some d

methodology transfer paper entitled Using 

Sampling. When complex sampling techniq

used and a list of interviewees is generated 

computer, control over the selection and c

interviewees can be automated, as describ

detail below.

Contacting

Potential

Interviewees

Once the potential interviewees have been 

you must contact them, explain what GAO 

and why you need their assistance, and arr

appointment. The interview booklet sets o

be followed in contacting the interviewees

Frequently, when structured interviews ar

interviewees are program participants or b

of federal programs. The universe list is de

a given point in time and a sample is draw

time the sample is contacted for interview

may have passed. This means some of the 

selected for initial telephone contact will h

away, died, or otherwise become inaccess

interviewers. Thus, we oversample and set 

Page 67 GAO/PEMD-10.1.5 Structured 
for replacing individuals who cannot be lo

provisions are illustrated in figure 7.1, whi

rules that

Page 68 GAO/PEMD-10.1.5 Structured 
Figure 7.1: Interviewee Contact Procedures Example

Page 69 GAO/PEMD-10.1.5 Structured 
GAO used to review a nationwide program 

the interviewing of program participants.

When contacting the interviewee by phone

standardized approach. This ensures that y

omit any important information. An examp

an approach is presented in figure 7.2. For 

introductory script can be put onto the com

Naturally, if unexpected events occur, you 

to deviate from this guide. The interview b

contain some samples of unexpected even

provide some guidance on how to deal wit

Maintain a log of all attempted contacts, w

of each interviewee’s name and address, te

number, date and time of the attempted co

the result. This information will be of use l

determining the possible effects of nonres

on the results. Also, it gives the analyst a m

adjusting the universe and plays a role wh

response-weighing is used. An example of 

appears in figure 7.3 and how it looks part

completed is in figure 7.4.

Page 70 GAO/PEMD-10.1.5 Structured 
Figure 7.2: Example of Telephone Contact With Potential Interviewee

Page 71 GAO/PEMD-10.1.5 Structured 
Figure 7.3: Interviewee Contact Log

Page 72 GAO/PEMD-10.1.5 Structured 
Figure 7.4: Interviewee Contact Log Filled in

Page 73 GAO/PEMD-10.1.5 Structured 
For CATIs, particularly those using the QP

data base of respondents can easily be gen

used to provide automated call sheets. Cer

information, such as the time at which an 

was conducted, can be entered automatica

computer. In addition, if information abou

interviewee is already available from other 

can be entered directly into the record bei

generated without the need to ask the inte

(unless some verification is wanted). Final

the interview is completed, selected inform

be transferred to the automated call sheet

the progress in administering the DCI.

A main objective when selecting and conta

interviewees is to avoid bias. By following 

procedures, you can minimize wrong selec

by mistake or because of ease in contactin

Interview

Arrangements

When you interview an individual for a GA

evaluation or audit, the interviewee usuall

GAO a favor. You should, therefore, make 

interview arrangements, including time an

convenient as possible for the interviewee

This may mean conducting the interview a

for you, the interviewer, an inconvenient h

as early morning or late evening. The locat

be a GAO office, an audit site, space provi

agency under review, or some other public 

this is not convenient for the interviewee, 

have to travel to his or her home or place o

employment or to some other such locatio

example, if you must interview farmers, yo

expect them to take time from their work 

travel to a place to meet you; you would n

the farms.

Page 74 GAO/PEMD-10.1.5 Structured 
If the interview contains sensitive question

the interview in certain locations might cr

difficulties. For example, if you are questio

participants of a welfare program about th

and treatment they are receiving, it would 

to conduct the interview in the welfare off

setting might cause interviewees to omit n

comments about the office and its personn

fear that this information would be overhe

affect their benefits.

When interviewing people in their homes, 

encounter frequent interruptions from oth

members, neighbors, and telephone calls. 

and radio programs can also be distracting

Interruptions and distractions also occur w

are interviewed at work. Nevertheless, the

advantages to interviewing people in their 

settings: they generally feel more comforta

have not been inconvenienced by having t

the interview, and they may have records a

sources of information, including other pe

their disposal. Thus, choose the interview 

carefully. On balance, it is more important 

the interview in a setting in which the inte

feels comfortable than to insist on a settin

no distractions.

For CATIs, the same general principles are 

up an interview. You should assume that y

contact with the interviewee will not be a 

time to conduct the interview and ask the 

when would be a good time to set aside a s

amount of time. You should be particularly 

any impatience on the part of the interview

arranging the time. One method is to mail 

to the interviewee asking for a convenient 

Sometimes it is useful to highlight the mai

interview will cover. In addition, if there is 

Page 75 GAO/PEMD-10.1.5 Structured 
information that the interviewee will need 

before the interview, this should be includ

postcard or in a letter.

If the interview is to be taped or if a superv

listening, the interviewee should be so info

the interviewee objects, alternative arrang

should be available. Acceptable alternative

described in the interview guide.

Protecting the

Interviewee

You may encounter interview situations th

the interviewees’ speaking of themselves o

negatively. This could come from asking q

such sensitive issues as personal habits or 

attitudes (for example, political or religiou

reactions to an employer, boss, or employ

To obtain cooperation from interviewees a

the quality of the data and the response ra

need to grant some kind of assurance to th

interviewees that the data collected will no

in a manner that could harm them.

When you first contact interviewees and a

meeting for the interview, usually give the

idea of what types of question you wish to 

seek their cooperation. This is called obtai

informed consent: revealing the contents o

interview in advance of the actual question

giving the interviewee a chance to refuse t

with the interview request. GAO does not u

more defined procedure in which the inter

asked to sign a statement of understanding

advance information is preliminary to GAO

guarantee of protection, which takes the fo

confidentiality or anonymity.

Page 76 GAO/PEMD-10.1.5 Structured 
Confidentiality means that the evaluator c

associate the interviewee’s name with spe

responses but promises not to do so. Evalu

obtain specific written approval before ma

pledges of confidentiality. GAO manageme

approvals and, in certain cases, the congre

requester must also agree to honor the ple

pledge of confidentiality has been given, G

safeguards the information consistent with 

pledge. For details of pledges of confident

to GAO’s General Policy Manual, chapter 7

GAO’s Project Manual, chapter 7.1.

Anonymity assures that GAO staff perform

work on the evaluation will be unaware of 

responses of individual interviewees. Whe

collected through face-to-face interviews c

by GAO interviewers, granting anonymity 

interviewees would be impossible.

Page 77 GAO/PEMD-10.1.5 Structured 
Each participant in the interview—intervie

interviewee—has a role to perform and a s

behaviors that assist in the performance. B

role and behaviors of each one influence t

of the interview, they affect the other parti

interviewer’s role and behaviors can be pr

and acquired through training, while the in

role and behaviors must be observed by th

interviewer, who seeks to modify them as 

to successfully complete the interview.

To oversimplify, the role of the interviewe

the questions, while that of the interviewe

respond with answers. Actually, the interv

perform at least eight major tasks:

• develop rapport with the interviewee and 

interest,

• give the interviewee a reason to participat

• elicit responsiveness from the interviewee

• ask questions in a prescribed order and ma

• ensure understanding,

• ensure nonbias,

• obtain sufficient answers, and

• show sensitivity to the interviewee’s burde

These tasks, which are not isolated but mu

integrated into the interview procedure, ar

below from the viewpoint of the interview

or her responsibilities.

Developing

Rapport and

Showing Interest

Seek to establish a balanced relationship b

interviewee and yourself as an empathetic

individual who is not too different from th

interviewee but who is also an independen

and honest collector of data. Your appeara

mannerisms, body language, and voice wil

the rapport, starting with the contact that 

Page 78 GAO/PEMD-10.1.5 Structured 
interview. Since this is usually done by tel

your voice and verbal mannerisms are extr

important (as they are later in the interview 

Of course, for CATIs, voice and verbal man

are key factors in the success of the interv

Remember that this interview is an artifici

in which nonverbal cues are not available.

Make your verbal and voice cues calm and

unflustered. Speak so the interviewee need 

to hear and understand. Changes in voice i

sighs, or other noises give clues to your fe

moods, as do your facial expressions and b

language. Control these so that the intervie

not pick up impatience, disapproval, or oth

feelings. Ideally, you should not experienc

feelings during the interview, since you are 

to be an impartial, unbiased, and tolerant o

Likewise, you should control expressions 

feelings or agreement with what the interv

saying.

It is important that the interviewer be awa

characteristic nonlinguistic cues such as c

voice, facial expressions, or gestures, sinc

as half of the communication that takes pl

the interview is conveyed by these modes 

expression. Failure to understand these cu

result in miscommunication. More details 

understanding and using these cues are pr

the methodology transfer paper entitled D

and Using Questionnaires.

Your appearance is still another variable th

influences rapport and, therefore, the tone 

interview. Dress to fit both the interview a

interviewee. If the interview is with a state 

official in his office in the capitol, it is app

perhaps mandatory, to wear office-type clo

Page 79 GAO/PEMD-10.1.5 Structured 
and tie for men, and suit or dress for wom

what you would expect the interviewee wo

wearing. Try to live up to the expected sta

the interviewee in this case. Not doing so m

the interview off to a bad start.

If, however, the interview is to take place 

construction site or with young people at a 

youth-recreation site, wear more casual cl

makes sense in that it gives interviewees t

that you understand the nature of their

circumstances. Also, you are not set off as 

totally different from the interviewee.

Giving the

Interviewee a

Reason to

Participate

Generally, interviewees do not benefit dire

the information that they give to GAO. Wh

should they agree to give you their time fo

interview? The reasons are various. Some

interviewees are obliged to cooperate with 

because of their positions and to provide i

on how federal money is being spent. Such

individuals usually understand why they sh

participate and need only be told somethin

evaluation procedures. In other cases, whe

interviewees are not operating some part o

program but are the recipients of funds, su

program beneficiaries and contractors, gre

explanation may be required.

Interviewees who are not aware of the imp

the evaluation and how they can help may 

sincere and well-thought-out answers. You

explanations to them, therefore, are impor

validity of the resulting data.

Page 80 GAO/PEMD-10.1.5 Structured 
Helping the

Interviewee to Be

Responsive

Many people you may contact, especially p

beneficiaries, have never before been inter

during an evaluation or audit. They may ha

interviews and interviews prior to receivin

where they have given name, address, age

children, work experience, and the like. Bu

they have not been asked for their opinion

feelings.

Thus, the individual may need to learn how 

an interviewee. The interviewer should he

process, and while this should not include 

how questions should be answered, it does 

making the interviewee comfortable and c

respondent. For example, you will impart 

that helps the interviewee learn to use an a

format that has been programmed into the 

interview. Where responses form a closed 

interviewee must know how to choose fro

alternatives given.

For CATIs, the development of the questio

sensitive to helping the interviewee. The s

the interview is very sensitive to question 

and ensuring that the interview takes on a

conversational tone. For a CATI, there is a 

tradeoff between formality (with longer, m

difficult questions) and conversation (with 

and easier questions).

During the interview, it may help to reinfo

interviewee with such verbal cues as

• “Un-huh, I see.”

• “Let me get that down.”

• “I see” (repeat answer).

• “I want to make sure I have that right.”

• “It’s useful to get your ideas on this.”

• “Thanks, it’s important to get your opinion 

Page 81 GAO/PEMD-10.1.5 Structured 
• “I see, that’s helpful to know.”

Asking Questions

in a Prescribed

Order and

Manner

The order in which the questions appear in 

structured interview is not accidental. Que

ordered so as to lead the interviewee throu

topics, correctly position sensitive questio

hold the interviewee’s interest. To the grea

possible, you must maintain this order. Th

and phrasing used in the questions also ha

carefully chosen and tested. For the sake o

standardization and understandability, it is 

that these be used as planned. For CATIs, 

simpler, since the order is preprogrammed

The following tips may help:

• ask the questions exactly as they are word

questionnaire,

• ask the questions in the order in which the

presented in the questionnaire,

• ask every question specified in the questio

• read each question slowly (two words per 

• repeat questions that are misunderstood o

misinterpreted,

• do not let the respondent stray from the qu

the interview, and

• keep nonverbal cues as neutral as possible

Remember that for telephone interviews, t

visual contact decreases the ability to mak

interviewee understand.

Ensuring

Understanding

At times, an interviewee will not understan

question, as indicated either by telling the 

so, by not answering, or by providing an an

seems inconsistent or wrong. When this ha

Page 82 GAO/PEMD-10.1.5 Structured 
should use an appropriate probing techniq

the following:

• repeat the question;

• give an expectant pause;

• repeat the respondent’s reply; and

• make neutral questions or comments, such 

“Anything else?” “Any other reason?” “Any 

“How do you mean?” “Could you tell me m

your thinking on that?” “Would you tell me 

have in mind?” “What do you mean?” “Why 

feel that way?” “Which would be closer to 

you feel?”

To maintain the meaning of the questions 

bias them, do this “probing” with care. The

probes should be worked out during the p

Rephrasing the question or adding new qu

should be avoided. If all probes have been 

rephrasing or adding questions is the only 

notes to that effect should be added next t

responses.

Ensuring Nonbias In earlier chapters, we covered bias in the 

question is written or in the selection of in

There can be bias also in the way you pose 

contents of the query, in the introduction o

ideas into a probe, or in your adding certai

emphasis or using certain body language. A

can destroy the neutrality that should char

your presentation. When listening to the in

answer, you can filter out portions of the m

that alter the true response.

Obtaining

Sufficient

Answers

You must learn to judge when an answer i

before going to the next question. If the an

incomplete or vague, you should ensure th

question is understood (as discussed abov

Page 83 GAO/PEMD-10.1.5 Structured 
more out of the interviewee to complete th

At times, the interviewee is allowed to ans

questions in an open-ended fashion, while 

each answer to one of a set of responses o

interview form. You must be sure that the 

has sufficient information to select one of 

answers. Sometimes, you must select two 

responses (not just one) from the set and a

interviewee which one best matches his or 

answer. This should be done, however, on

resort and only after giving the respondent 

time to respond.

On other occasions, an interviewee may no

answer in mind but may need to refer to d

or ask someone else. If this can be done co

and within a short time, encourage the inte

do so.

You can also check the accuracy of the an

by asking for supporting information from 

interviewee. Sometimes the design of the i

has built into it questions to which answer

already been obtained from files or from o

in advance. Use these to check the accura

which the interviewee is answering. Under

of information is often found. As the length 

since a subject event increases, there is a g

tendency for the interviewee either to forg

occurred or to recall it only partially.

Showing

Sensitivity to

Interviewee

Burden

Before conducting an interview, give the in

a general statement of how long it is expec

You are then under some obligation to adh

time limitation.

Frequently, interviewees prolong their ans

adding examples, critical incidents, or oth

Page 84 GAO/PEMD-10.1.5 Structured 
neither you nor the interviewee has a time 

this extension of the interview is acceptab

critical, however, use techniques to speed 

interview so as not to lose valuable answe

end. Besides the length of time taken, the i

can be burdensome because of the amoun

the interviewee needs to go through to pro

information requested. If a relatively unim

question requires a significant amount of t

energy by the interviewee, it may not be w

pursuing.

Page 85 GAO/PEMD-10.1.5 Structured 
The purpose of all the work that has gone 

designing, pretesting, and revising the stru

interview (perhaps many times), obtaining 

review, and finally using it is to obtain data 

properly analyzed, will answer the evaluat

questions. If you have followed the proced

outlined in previous chapters, chances are 

you now have those data. All the hard wor

will have been worth the cost.

You now have uniform data that can be us

answer the evaluation or audit questions. T

questions in your structured interviews we

and precise as the state of the art permits. 

interviewers were carefully trained and ins

to what explanations were to be given whe

interviewees did not understand or had tro

questions. The pretests you conducted and 

review you obtained ensured that the peop

interviewed could give you the data you ne

You need not worry about analyzing narra

responses to a long list of open-ended que

because your interviews contained few, if 

them. Through your preliminary research, 

interviews with program officials and outs

and your pretests, you identified most of th

replies to your questions. Thus, you were a

convert what started out as open-ended qu

closed-format questions with sets of altern

responses that minimized the use of “other

specify” responses.

Transferring the data from the completed 

forms to computer files is comparatively s

have used keypunch boxes. You can have 

keyed directly to disks or magnetic tape an

entered directly into computer files. (Of co

CATIs, the data base will have been genera

Page 86 GAO/PEMD-10.1.5 Structured 
went along.) After verifying the accuracy o

keypunching, you are almost ready to begi

analysis. But before you do this, you must 

if you have a nonrespondent problem.

Nonrespondent

Problem

When you draw a sample of persons from 

to conduct interviews, you intend that all t

be interviewed. Indeed, it is part of the GA

plan that what these selected persons say 

for what the entire universe would have sa

could have been queried. Rarely can the en

be interviewed, however, because of death

to locate people, refusals to be interviewed

on. For telephone interviews, the nonrespo

problem is usually significantly less, since 

frequently have 80- to 90-percent response 

To combat the nonrespondent problem, no

will randomly sample a greater number of 

is statistically required. Nonrespondents c

replaced by randomly drawn substitutes. F

example, if the sampling plan calls for 50 p

be interviewed, you might randomly select 

If the 8th, 20th, 31st, and 49th individuals y

contact have died, you would use cases 51 

as substitutes.

Usually, making a small number of substit

no effect on analysis of the final data. Whe

number of substitutions is made, for exam

20 percent or more, you may have some co

the people you were unable to interview re

unique portion of the universe. For examp

these persons died, they may represent the 

persons in the universe; your data collectio

would not adequately represent the opinio

individuals. If queried, this portion might h

dramatically different answers to all or som

Page 87 GAO/PEMD-10.1.5 Structured 
questions and altered the final results of yo

collection.

There are several ways to assure yourself 

data would not have changed much had th

individuals been contacted: analyzing the r

the nonparticipation, interviewing by telep

subsample on critical questions, comparin

demographic information, and assuming a 

case” answer. These are discussed below.

Analyzing Reasons

for Nonparticipation

When you are trying to contact individuals 

interviews, you have the opportunity to tal

or someone who has information about th

potential interviewee could not be contact

in your log the reason given for not being a

up the interview: for example, death, move

the area, whereabouts unknown, or appare

location but unable to be contacted.

The potential interviewee, when reached, 

to be interviewed, giving such reasons as “

“I don’t give interviews,” “it’s none of your 

or “I don’t understand why you want to tal

never participated in that.”

If you have no data other than this, you ma

some attempt to determine whether the re

for nonparticipation are related to critical 

the interview. For example, if you are rela

services received to the recipients’ state of 

and have missed many interviews because 

potential interviewees’ deaths, this could m

your sample of many interviewees who wo

reported poor well-being. You might then h

place some limitation on the final conclusi

study. There is no statistical test of the exc

that can be used to make this decision.

Page 88 GAO/PEMD-10.1.5 Structured 
Interviewing a

Subsample on

Critical Questions

A second approach to the nonrespondent p

to select a subsample of those not availabl

interview (or the entire group, if it is small 

and conduct a short phone survey of them

some of the critical questions on the instru

course, this does not help if the people cou

located in the first place or were deceased

however, were found but at first refused a

because of time considerations, you may b

collect data on some questions on the pho

answers are then compared to those collec

normal interviewing process, using statisti

procedures to test for significant differenc

Questions on which the two groups differ 

might then be eliminated from the final an

There are some exceptions to this approac

discussed below under “worst-case assum

Comparing

Demographic

Information

Many times, we have a rich data base on a 

of demographic variables for all potential

interviewees. For example, the program fi

information for welfare recipients may con

information on their sex, age, race, educat

status, number of children, and work expe

Thus, if you cannot obtain partial interview

information from a subsample, as discusse

you can compare the demographic variabl

interviewed and those not.

Significant differences on a certain propor

critical demographic variables would cast 

the two groups were essentially the same a

that the absence of these individuals could 

overall results.

Assuming the Worst

Case

Some of the questions you pose will have

binary-choice answers. Your task may be s

Page 89 GAO/PEMD-10.1.5 Structured 
determine whether more people have done 

than have not. Suppose you are to intervie

sample of 100 people, but 20 people could 

contacted and were not replaced by the ne

people on your random list. For the 80 you 

interview, your results show that 47 said “y

said “no.” You are now asked about the 20 

unable to interview. Could they have chan

outcome?

Taking a conservative view, you could attr

their responses to one of the categories. If 

of “yes” votes would allow you to defend a 

finding, then you would want to make the 

that all 20 would have answered “no.” This 

make the final outcome 47 “yes” and 53 “n

these circumstances, the finding would no

support. In other words, a 47-to-33 split of 

with 20 uninterviewed persons is too close 

decision. Had the split been something like 

your case would have stood a chance.

Some of these methods require the collect

additional information. With each method,

assumptions and limitations can influence 

eventual interpretation of the data collecte

the structured interview. Nothing short of 

the interviewees’ answers to the questions 

fully satisfactory.

Data Analysis The edited data now reside in computer fil

have dealt with the nonresponse problem, 

can now begin the data-analysis phase. Th

probably be the most enjoyable part of the 

will begin to see results and imagine how t

will read. Yet this phase will not be easy an

require your full attention.

Page 90 GAO/PEMD-10.1.5 Structured 
At this point, it is useful to address the que

do you mean by data analysis? In GAO, da

carries with it various meanings, ranging f

simple tasks as learning how many membe

surveyed population are 25 years of age or 

how many are under 25 (first-level analysi

investigating causal relationships between 

different achievement levels (if any) of onl

children with only older siblings, children 

younger siblings, and children with both o

younger siblings (third-level analysis).

The analysis to be done will be determined 

degree by the project objectives that have 

established for the structured interview. Y

consider three levels of analysis.

First-Level Analysis In first-level analysis, you concentrate on a

description of the data—for example, how 

responded to each response alternative, bo

absolute numbers and on a percentage bas

example, a question may have asked, “Did 

complete high school?” A description of th

would show how many and what percenta

responded “yes” and how many “no.”

In the language of analysis, this type of des

the data is commonly referred to as freque

tabulations or frequency tables. Although n

analytic activity under this first-level analy

normally the most significant activity.

You may often make a computer run to ob

frequency tabulations during the data-verif

phase, because it will show all values keyp

every question. A review of the run will dis

possible errors in the data base. In the exa

above, “yes” answers may be coded as “1” 

answers as “2.” Any other number showing 

Page 91 GAO/PEMD-10.1.5 Structured 
question would stem from an error on the 

interviewer or the keypuncher.

Second-Level Analysis Second-level analysis begins where the de

the data stopped. In this next level of analy

perhaps the most useful to most GAO effo

analyze the data, one question at a time. C

statistics, such as the mean and median, ca

obtained with the description of the data f

questions where such statistics would be u

appropriate. Remember that if a sample ot

simple random sample has been used, the 

percentages shown on the frequency tabul

must be weighted before you make projec

Therefore, it would be wise to consult a sa

statistician before using the numbers in a b

report.

Having completed the single-question anal

then move to testing the associations betw

of questions in response to hypotheses est

during the design phase. For example, is th

association between a person’s sex and wh

not the person completed college? If the d

that a larger percentage of women comple

than do men, is the difference statistically 

or could it stem from the fact that we stud

sample and not the entire population? Suc

measures as chi-square analysis and corre

analysis are often used to determine how c

can be that apparent associations between 

to two questions do not stem from chance

GAO assignments, second-level analysis is 

the analysis of questionnaire or interview 

Third-Level Analyses Third-level analyses are more complex tha

levels of analysis. They normally take into 

many variables at one time and address m

complex questions. Third-level analyses of

Page 92 GAO/PEMD-10.1.5 Structured 
differences between subgroups of surveye

cases—what factors differentiate students 

federal loans in a timely manner from thos

not—or investigate the influence that a set 

may have on a single variable—what facto

the amount of loans made by the Small Bu

Administration.

Two of the many analytic tools available to

investigate these more complex analytic q

multiple regression analysis and discrimin

analysis.

It is not our intent here to provide a detaile

of the analytic tools available in survey res

do, however, want you to understand that 

and complexity of the analysis phase of a p

vary dramatically, depending primarily upo

objective established for the project. The a

addresses cause-and-effect questions will b

more difficult than the analysis for descrip

normative questions. Regardless of the typ

question being addressed, a large number 

statistical tools are available for the analys

the evaluation. Selecting the most appropr

easy. Once again, we strongly advise that e

and auditors obtain the assistance of a spe

the design, methodology, and technical ass

group, the regional technical assistance gr

PEMD for this phase of the assignment.

Analysis of

Open-Ended

Questions

Answers to open-ended questions may ran

few words to several sentences. Interview

typically give the interviewer some salient 

come quickly to mind but leave out some i

factors. Open-ended questions do not help

interviewees consider an identical range o

After conducting several interviews, interv

Page 93 GAO/PEMD-10.1.5 Structured 
supplement the question by asking the inte

about factors not mentioned, but such sup

questions will not be standard among inter

Thus, the interviewees as a group are not r

to identical questions.

As mentioned briefly in chapter 3, the prop

of open-ended questions requires the use o

complicated, time-consuming process call

analysis.” In brief, you must read and rerea

substantial number of the written respons

with some scheme to categorize the answe

essence, develop a set of alternative respo

develop rules for assigning responses to th

categories. Even with a set of rules, people 

categorize answers differently. Therefore, 

four people must go through each complet

interview and categorize the answers. A m

them must agree if you are to have a reliab

base.

Because content analysis is so time-consu

answers to open-ended questions are often 

unanalyzed. The evaluator or auditor in re

quote from one or a few selected response

open-ended interviews generally do not pr

uniform data that can be compared, summ

further analyzed to answer the evaluation 

questions.

Page 94 GAO/PEMD-10.1.5 Structured 
on Each Task

This paper has discussed the major tasks t

performed to collect data by structured int

Some of these tasks are short and seem re

uncomplicated, such as identifying the tar

population and selecting variables from th

pool. Other tasks appear lengthy and rathe

such as designing the interview form and c

the interview.

All these necessary tasks are carried out in 

cooperative effort by evaluators and speci

Generally, three types of technical experti

involved: measurement, sampling, and dat

The expertise required need not, however, 

three separate individuals. Most specialist

capable of handling more than one functio

depending on the complexity of the job. Ex

functions each staff member might perform 

the evaluation appear in table 10.1. In addi

data-collection tasks 1 to 17, which are dis

this paper, the table includes for complete

tasks that concern data handling, analysis,

interpretation, and reporting.

Table 10.1: Functions and Responsibilities of Evaluators and Specialists

Staff functiona

Task

Evaluator in

charge and

staff

Measurement

specialist

Sampling

specialist Dat

1. Formulate

overall Initi

question

al work Acqui

und re

re job

erstanding

andview

Optional Op

2. Determine

kind of In

information

needed

itial work Re view Optional Op

Page 95 GAO/PEMD-10.1.5 Structured 
oacas

Staff functiona

Task

Evaluator in

charge and

staff

Measurement

specialist

Sampling

specialist Dat

3. Identify the

target

population

Initial work Review Optional Op

4. Create

question pool

Contribute Contribute Optional Dis

ana

and 

5. Select

questions from

the pool

Check for

relevance

Primary role Optional Rev

6. Decide on

final data

collection

method

Check for

job-related

constraints

Outline

advantages and

disadvantages;

recommend

solutions

Design

sampling plan

Rev

7. Plan data

analysis

Participate Participate Participate Par

8. Design

interview form

Assist and

review

Primary role Optional Op

9. Obtain

subject matter

review

Lead Participate — —

10. Conduct

pretest

Participate Lead — —

11. Review of

expert review

and pretest

Participate Lead review Optional Rev

12. Revise

interview form

Review Write Optional Co

13. Kick-off

conference

Give information

on how

instrument

fulfills job needs

Give

instructions on

question

meaning and

instrument use

Give

instructions on

sampling

Op

(de

dat

use

Page 96 GAO/PEMD-10.1.5 Structured 
oacas

Staff functiona

Task

Evaluator in

charge and

staff

Measurement

specialist

Sampling

specialist Dat

14. Train

interviewers to

interview

Assist in training Train using

various

techniques

— —

15. Select

interviewees

Staff uses

sampling plan

Available for

consultation

Available for

consultation

—

16. Contact

interviewees

Staff contact Available for

consultation

Available for

consultation

—

17. Conduct

interviews

Staff conduct Available for

consultation

— —

18. Settle

nonrespondent

problems

Acquire needed

data

Recommend

solutions

Recommend

solutions

Par

19. Edit raw

data

Staff edit Consult — Dire

20. Keypunch — — — Co

arra

21. Edit

keypunch data

Staff edit — — Dire

22. Analyze

data

Consult Consult Consult Ana

23. Interpret

data

Participate Participate Participate Par

24. Draft report Write Write, consult,

review

Write, consult Wri

aA staff member may perform more than one functio

given task in the course of an evaluation.

This table can serve as a guide when reach

agreement about responsibilities when ass

personnel at various times during the deve

and use of the DCI. Staff responsible for a 

function may not be required during some 

Page 97 GAO/PEMD-10.1.5 Structured 
oacas

their presence could prove useful. In other 

large number of staff might be distracting, 

better to work with fewer persons for that 

Page 98 GAO/PEMD-10.1.5 Structured 
Page 99 GAO/PEMD-10.1.5 Structured 
Babbie, E. H. Survey Research Methods. B

Calif.: Wadsworth, 1973.

Bradburn, N. M., and S. Sudman and Assoc

Improving Interview Methods and Questio

Design. San Francisco: Jossey-Bass, 1981.

Cannell, C., et al. New Techniques for Pret

Survey Questions. Ann Arbor, Mich.: Surve

Center, University of Michigan, 1989.

Dawes, R. M. Fundamentals of Attitude Me

New York: John Wiley and Sons, 1972.

Dillman, D. Mail and Telephone Surveys. N

John Wiley and Sons, 1978.

Fowler, F. J. Survey Research Methods. Ne

Park, Calif.: Sage Publications, 1988.

Fowler, F. J., Jr., and T. W. Mangione. Stan

Survey Interviewing: Minimizing Interview

Error. Newbury Park, Calif.: Sage Publicat

Frey, J. H. Survey Research by Telephone, 

Newbury Park, Calif.: Sage Publications, 1

Groves, R. M. Survey Errors and Survey Co

York: John Wiley and Sons, 1989.

Groves, R. M., et al. Telephone Survey Met

New York: John Wiley and Sons, 1988.

Heverson, M. E., L. L. Morris, and C. T. Fit

How to Measure Attitudes. Beverly Hills, C

Publications, 1978.

Page 100 GAO/PEMD-10.1.5 Structured 
Lavrakas, P. J. Telephone Survey Methods

Selection and Supervision. Newbury Park, 

Publications, 1987.

Sudman, S., and N. M. Bradburn. Asking Q

Practical Guide to Questionnaire Design. S

Francisco: Jossey-Bass, 1982.

U.S. General Accounting Office. Content A

Methodology for Structuring and Analyzin

Material, methodology transfer paper 3. W

D.C.: 1982.

U.S. General Accounting Office. Designing

Evaluations, methodology transfer paper 1

Washington, D.C.: 1991.

U.S. General Accounting Office. Developin

Questionnaires, methodology transfer pap

Washington, D.C.: 1986.

U.S. General Accounting Office. Using Stat

Sampling, methodology transfer paper 6. W

D.C.: 1986.

U.S. National Center for Health Statistics. 

Evaluation and Methods Research: A Summ

Studies of Interviewing Methodology. Vita

Statistics, series 2, number 69. Rockville, M

Warwick, D. P., and C. A. Lininger. The Sam

Survey: Theory and Practice. New York: M

1975.

Page 101 GAO/PEMD-10.1.5 Structured 
Bias Words, sentence structure, attitudes, and m

that unfairly influence a respondent’s answ

question. Bias in questionnaire data can st

variety of other factors, including the sequ

questions. Both interviewer and instrumen

exist.

Closed Question A question with more than one possible an

which one or more answers must be selec

Content Analysis A set of procedures for collecting and orga

nonstructured information into a standard

that allows one to make inferences about t

characteristics and meaning of written and 

recorded material.

Cues The alternative responses to questions tha

or decrease in intensity in an ordered fash

interviewee is asked to select one answer 

question.

Data-Collection

Instrument

A highly structured document, sometimes 

DCI, that requires the user or respondent t

provide data in a systematic and highly pre

fashion.

Demographic

Question

A question used in compiling vital backgro

social statistics, such as age, marital status

of household.

Open-Ended

Question

A question that does not have a set of poss

answers from which to make a selection b

the respondent to answer in essay form. O

Page 102 GAO/PEMD-10.1.5 Structured 
questionnaire, the respondent would write 

short answer or fill in a blank. During an in

the respondent would give the interviewer 

unstructured, narrative answer. The interv

would record the response verbatim or sel

features. If a structured interview were us

question might appear to be open-ended to 

interviewee but could be “closed down” by 

interviewer, who would have a set of alter

answers to check.

Probe To examine a subject in an interview in de

several questions.

Qualitative Analysis An analysis that ascertains the nature of th

attributes, behavior, or opinions of the ent

measured. In describing a person, a qualita

analysis might conclude that the person is 

and middle-aged. See also Quantitative An

Quantitative

Analysis

An analysis that ascertains the magnitude, 

size, for example, of the attributes, behavi

opinions of the entity being measured. In d

person, a quantitative analysis might concl

the person is 6 feet 4 inches tall, weights 1

and is 45 years old. See also Qualitative An

Reliability The extent to which a measurement proce

similar results on repeated observations o

condition or event.

Stem The statement portion of a question.

Page 103 GAO/PEMD-10.1.5 Structured 
Structured Interview An interview in which questions to be aske

sequence, and the detailed information to 

are all predetermined; used where maximu

consistency across interviews and intervie

needed.

Target Population The level (item, individual, group, organiza

like) at which data are collected. Data can 

collected at the individual level (for examp

program participant) and analyzed and rep

organizational level (for example, the emp

office). Participants might be asked how m

of counseling they received from the empl

office, whereas the analysis of the data mi

the number of offices that provided given 

counseling.

Validity The extent to which the question being ask

measures the concept that the evaluator w

measure.

Page 104 GAO/PEMD-10.1.5 Structured 
This is a flexible series continually being a

updated. The interested reader should inqu

the possibility of additional papers in the s

The Evaluation Synthesis. Transfer paper 

formerly methods paper I.

Content Analysis: A Methodology for Struc

Analyzing Written Material. Transfer paper 

formerly methodology transfer paper 3.

Designing Evaluations. Transfer paper 10.

formerly methodology transfer paper 4.

Using Structured Interviewing Techniques

paper 10.1.5, formerly methodology transf

Using Statistical Sampling. Transfer paper 

formerly methodology transfer paper 6.

Developing and Using Questionnaires. Tra

10.1.7, formerly methodology transfer pap

Case Study Evaluations. Transfer paper 10

formerly methodology transfer paper 9.

Prospective Evaluation Methods: The Pros

Evaluation Synthesis. Transfer paper 10.1.

methodology transfer paper 10.

(973317) Page 105 GAO/PEMD-10.1.5 Structured 
Ordering Information

The first copy of each GAO report and 

is free. Additional copies are $2 each. 

should be sent to the following addres

accompanied by a check or money orde

out to the Superintendent of Documen

necessary. VISA and MasterCard credi

are accepted, also. Orders for 100 or m

copies to be mailed to a single address 

discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th & G Sts

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling

(202) 512-6000 or by using fax numbe

(301) 258-4066, or TDD (301) 413-000

Each day, GAO issues a list of newly a

reports and testimony. To receive fac

copies of the daily list or any list from 

30 days, please call (202) 512-6000 us

touchtone phone. A recorded menu w

information on how to obtain these lis

For information on how to access GAO 

on the INTERNET, send an e-mail mes

"info" in the body to:

info@www.gao.gov
Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Postage & F

GAO

Permit No