Administrative Conference of the United States

QUALITATIVE RESEARCH METHODS:
A GUIDE FOR ACUS CONSULTANTS

December 10, 2019

Jennifer Nou
University of Chicago

Gregory Huber
Yale University

The views expressed in this document do not necessarily reflect those of the Administrative Conference’s Council, its members,
or its committees. This initial version of the guide was prepared by Gregory Huber and Jennifer Nou: Forst Family Professor of
Political Science, Yale University; Professor of Law, University of Chicago Law School, respectively. The document will be
updated and revised as necessary by the Administrative Conference of the United States (ACUS). Many thanks to Steve Balla,
Karen Bradshaw, Cary Coglianese, David Engstrom, Nicholas Parrillo, Stuart Shapiro, Wendy Wagner, Chris Walker,
University of Chicago IRB staff, and Yale Human Research Protection Program staff for helpful conversations and comments on
the project. Reeve Bull, Shawne McGibbon, Frank Massaro, Gavin Young, and Todd Rubin were also instrumental at ACUS.

TABLE OF CONTENTS
I.

OVERVIEW OF QUALITATIVE INTERVIEWS ............................................... 2
1.
2.

WHAT IS THE PURPOSE OF QUALITATIVE INTERVIEWS? ............................................ 2
WHAT TYPES OF QUALITATIVE INTERVIEWS ARE THERE AND WHAT FACTORS
SHOULD I CONSIDER IN CHOOSING ONE?................................................................... 2

II. PRELIMINARY MATTERS................................................................................... 3
1.

DO MY INTERVIEW QUESTIONS NEED TO BE CLEARED BY THE OFFICE OF
INFORMATION AND REGULATORY AFFAIRS (OIRA) UNDER THE PAPERWORK
REDUCTION ACT (PRA)? ......................................................................................... 3
2. DOES MY STUDY NEED TO BE REVIEWED BY AN INSTITUTIONAL REVIEW BOARD
(IRB) BEFORE I BEGIN RESEARCH?........................................................................... 3
a. WHAT IS THE STANDARD FOR WHETHER RESEARCH MUST BE REVIEWED BY AN
IRB?....................................................................................................................... 4
b. HOW DOES ACUS DEFINE “HUMAN SUBJECTS RESEARCH”? ................................... 4
c. WHAT IF MY EMPLOYING INSTITUTION’S IRB CONSIDERS MY RESEARCH TO BE
“HUMAN SUBJECTS RESEARCH” AND ACUS DOES NOT? ......................................... 4
III. THE INTERVIEW PROCESS ................................................................................ 6
1.
2.
3.
4.
5.
6.
7.
8.
9.

HOW DO I IDENTIFY POTENTIAL INTERVIEWEES AND FROM WHICH AGENCIES? ........ 6
WHAT IS THE ROLE OF THE ACUS STAFF? ............................................................... 7
WHEN IS IT APPROPRIATE TO INTERVIEW INDIVIDUALS EXTERNAL TO
GOVERNMENT? ......................................................................................................... 7
HOW SHOULD I THINK ABOUT HOW MANY INTERVIEWS TO CONDUCT? HOW MANY
INTERVIEWS ARE “ENOUGH”?................................................................................... 8
HOW SHOULD I APPROACH ATTRIBUTION ISSUES? RELATEDLY, WHAT STEPS CAN I
TAKE TO PROTECT THE INTERVIEWEE’S INFORMATION AND IDENTITY? ................... 9
ANY TIPS FOR HOW TO CONTACT POTENTIAL INTERVIEWEES? .................................. 9
SHOULD I CONDUCT THE INTERVIEW BY TELEPHONE OR IN PERSON? WHAT ARE
THE TRADEOFFS? .................................................................................................... 10
ANY TIPS FOR DEVELOPING RAPPORT WITH THE INTERVIEWEE? ............................. 10
SHOULD I RECORD THE INTERVIEW OR ELSE TAKE NOTES?..................................... 11

IV. ANALYZING AND PRESENTING THE RESULTS ......................................... 12
1.
2.
V.

SHOULD I CODE THE INTERVIEWS AND QUANTIFY THE RESULTS AND, IF SO, HOW? 12
HOW SHOULD I PRESENT MY RESULTS? .................................................................. 12
BIBLIOGRAPHY ................................................................................................... 13

1

Qualitative Research Methods: A Guide for ACUS Consultants
The purpose of this document is to provide concise and useable guidance regarding
qualitative interviews with individuals in administrative agencies and those who interact with
them. The guide has been developed in consultation with the Administrative Conference of the
United States (ACUS). It is intended to convey best practices for future ACUS reports, not to
suggest a critique of past work. Consistent with federal plain language guidelines, the guide refers
in the second person to “you,” the ACUS consultant. The bibliography provides a list of references
that directs you to further reading potentially of interest.
I.

Overview of Qualitative Interviews
1. What is the purpose of qualitative interviews?

Qualitative research, like quantitative research, aims to draw descriptive or explanatory
inferences from empirical research. Whereas quantitative research privileges numerical and
statistical methods, qualitative research embraces a number of approaches including interviews
and historical archival work. Qualitative approaches can be especially useful to develop knowledge
about which little is currently known; investigate complicated phenomena; or provide context for
a rich set of dynamics. The most valuable studies often combine both qualitative and quantitative
data. This guide focuses on the former, though it will touch on the latter in the context of coding
interview data.
Qualitative interviews are frequently used by ACUS consultants both (1) for the purposes
of understanding how government operates, i.e., factual knowledge of policies, procedures, etc.
and (2) to gather evaluations and opinions about current operations and alternatives. Given that
many institutional practices are complicated, interviewees may be drawn from multiple positions
within various agencies, from the Executive Office of the President (e.g., Office of Information
Regulatory Affairs), and outside of government.
2. What types of qualitative interviews are there and what factors should I consider in
choosing one?
Interviews can be broadly divided into three categories: unstructured, semi-structured, and
structured (closed). At one end of this spectrum, unstructured interviews are those in which you
have no predetermined set of questions or topics you wish to ask about. They may be most valuable
when the ACUS project is primarily exploratory and preliminary in nature. At the other end,
structured interviews are those in which there is a preset list of questions to be asked; deviations
and extensions from that set of items rarely take place. Structured interviews are the most useful
when you plan to compare and code the responses, since the questions are standardized. The main
drawback, of course, is that structured interviews do not permit more in-depth exploration of
unanticipated topics.
In practice, most ACUS-related interviews are likely to be semi-structured, in which you
have a predetermined list of topics and questions you will ask, with deviations occurring as novel
issues or those that need further exposition arise. Semi-structured interviews allow you to ensure

2

that a desired set of topics is covered while also preserving the flexibility to adjust in light of
learning that takes place during the interview. That said, it is often valuable to pretest your initial
questions. Pretesting involves the systematic administration of the interview to a group of
individuals with characteristics similar to your anticipated interview subjects and under similar
conditions. The goal is to learn if your questions are too vague, off-putting, or otherwise confusing
(e.g., because they combine two or more important issues in a single question). Pretesting also
allows you the opportunity to practice your interview skills.
When you conduct multiple interviews, it is likely that the predetermined set of topics and
questions used will evolve in light of the experience gained in earlier interviews. For these reasons,
you might consider beginning interviews with interviewees who have been at the relevant
institutions or observed them for sufficiently long periods of time to be able to give a bird’s-eye
view of the relevant issues. Those interviewees may also help to shed light on the agency’s
organizational structure, as well as internal and external constituencies that may be appropriate
interview subjects themselves.
II.

Preliminary Matters
1. Do my interview questions need to be cleared by the Office of Information and
Regulatory Affairs (OIRA) under the Paperwork Reduction Act (PRA)?

The PRA is triggered when an agency, within a twelve-month period, collects information
(regardless of form or format), or sponsors the collection of information, from ten or more people
who are not federal employees. “Collection of information” refers to asking identical or structured
questions. If the PRA is triggered, there are steps that ACUS staff must take, including providing
notice in the Federal Register and obtaining clearance from OIRA, before questions can be
administered to the tenth person. If you plan to interview or otherwise administer questions to ten
or more non-federal officials, it is important that you bring this fact to the attention of ACUS staff
members as soon as possible, and they will determine whether any PRA-related steps are
necessary.
Additional Reading:
Office of Information and Regulatory Affairs, Information Collection under the
Paperwork Reduction Act, https://www.whitehouse.gov/sites/whitehouse.gov/files/omb/assets/
inforeg/PRAPrimer_04072010.pdf.
2. Does my study need to be reviewed by an Institutional Review Board (IRB) before I
begin research?
The answer to this question depends on whether your research is determined to be “human
subjects research,” as discussed in 2a. Regardless of whether you are employed by an institution
with an IRB, you should speak with your ACUS project contact to determine whether ACUS
considers your research to be “human subjects research.” If you are employed by an institution
with an IRB, you should also speak with those IRB officials to determine whether they consider
your research to be “human subjects research.” ACUS staff will typically find that research

3

conducted in its name and for its purposes is not “human subjects research”; nonetheless, your
institution’s IRB may still conclude that your research is “human subjects research,” in which case
you would need to abide by its procedures, including a potential IRB review.
a. What is the standard for whether research must be reviewed by an IRB?
The Policy for the Protection of Human Subjects (known as the “Common Rule”), jointly
passed as a regulation by many Federal agencies that fund research, sets the standard for
determining whether research must be reviewed by an IRB. (See, e.g., 45 C.F.R. § 46.101 et seq.,
for the version adopted by the U.S. Department of Health and Human Services). The Common
Rule requires an ethics review by an IRB of “human subjects research” that is “conducted,
supported, or otherwise subject to regulation by any federal department or agency.” Id. Both
federal agencies and institutions conducting federal research (i.e., institutions that are “supported”
by the federal government) will have IRB procedures in place to review covered types of research.
b. How does ACUS define “human subjects research”?
As discussed above, regardless of whether you are employed by an institution with an IRB,
you should speak with your ACUS project contact to determine whether ACUS considers your
research to be “human subjects research.”
ACUS’s position is that, with respect to any IRB obligations the agency itself might
impose, the research it conducts or supports does not typically qualify as “human subjects
research” based on the plain meaning of the term, legislative history and regulatory context (both
of which are largely focused on human health studies), and applicable law. The Common Rule
defines “human subject” in part as “a living individual . . . about whom an investigator . . .
conducting research . . . obtains information . . . or . . . generates identifiable private information.”
45 C.F.R. § 46.102(e)(1) (emphasis added). ACUS’s position is that its projects ordinarily do not
include research “about” any living individual. Rather, consultants conduct research “about” the
administrative state, such as agency practices with respect to adjudication, rulemaking, and
licensing. Interviews are intended to obtain information about these sorts of non-living subjects
rather than about the interviewees themselves. Thus, even if a researcher “obtains information” or
“generates identifiable private information” about the interviewee, because the researcher is not
generally conducting research “about” that living individual, he or she is not conducting “human
subjects research.” However, there may be rare instances in which ACUS does consider your
research to be “human subjects research” so you should speak with ACUS staff at the outset of the
project to determine ACUS’s position and procedures.
c. What if my employing institution’s IRB considers my research to be “human
subjects research” and ACUS does not?
Even if ACUS does not believe your research is “human subjects research,” your
employing institution might. If that is the case, then you must abide by your institution’s
procedures, notwithstanding ACUS’s conclusion.

4

Institutions are permitted to adopt a broader definition of “human subjects research” than
the one laid out in the federal regulations. We cannot, therefore, offer a comprehensive definition
of “human subjects research” as understood by your particular institution’s IRB. At many
institutions, researchers are not permitted to determine on their own whether their research
constitutes “human subjects research.” This determination must generally be made by the IRB. In
any event, you should familiarize yourself with your institution’s IRB policies and procedures,
including the process for determining how key terms might be defined.
Subject to organization-specific variations on this procedure, if the IRB has deemed your
research plans to be “human subjects research” you would then usually apply to your IRB for: (1)
one of several specified exemptions allowing research determined to be “human subjects research”
to skip IRB review; or (2) approval of your research after full review by the IRB. Accordingly, a
first step in obtaining IRB approval is to prepare either an exemption request or apply for IRB
approval. You may also be required to obtain brief training or certification which at many
institutions can be done online. You cannot begin “human subjects research,” including pilot study
interviews, until after your research has been reviewed by the IRB and either deemed exempt or
approved. You will therefore want to build in some lead time before beginning your research to
account for the possibility of this process, perhaps as long as four to six weeks.
Common exemptions that might apply to ACUS-related research considered “human
subjects research” include the following: (1) the researcher does not record personally identifying
information of the interviewee; (2) disclosure of gathered information “would not reasonably place
the subjects at risk of criminal or civil liability or be damaging to the subjects’ financial standing,
employability, educational advancement, or reputation”; and (3) an IRB determines through
limited IRB review that “there are adequate provisions to protect the privacy of subjects and to
maintain the confidentiality of data.” 45 C.F.R. § 46.111(a).
Even if your work is deemed exempt, you may still be required to follow certain procedures
such as obtaining informed consent from the interviewee. If your work is not deemed exempt, the
IRB may still opt to approve it without full review under specific terms. Those terms are most
likely to center on full disclosure of the risks of participating in your study (e.g., an accidental
breach of confidentiality) and evidence that subjects voluntarily choose to consent despite that
risk, possibly with a standardized written or oral consent procedure.
Setting aside questions of formal IRB requirements, a general best practice is to take steps,
wherever reasonable and feasible, to minimize potential risks to subjects. Indeed, regardless of
whether you are bound by IRB requirements, you should consider the broader ethical obligations
that may arise from your research. We discuss some best practices further with regard to interviews
in Question III 5 below.
Additional Reading:
U.S. Department of Health & Human Services, Human Subject Regulations Decision
Charts, https://www.hhs.gov/ohrp/regulations-and-policy/decision-charts/index.html.

5

National Commission for the Protection of Human Subjects of Biomedical and
Behavioral Research, The Belmont Report (Ethical Principles and Guidelines for the Protection
of Human Subjects of Research), https://www.hhs.gov/ohrp/regulations-and-policy/belmontreport/index.html.
III.

The Interview Process
1. How do I identify potential interviewees and from which agencies?

Probably the most difficult task you face conducting qualitative interviews is deciding
whom to talk to and obtaining access to those individuals. In an ideal world without any
constraints, you would obtain all available information from all agencies and individuals affected
by a particular issue to understand what is at stake. Such an approach, however, is often not feasible
for a multitude of reasons, including scarce time, resource constraints, and the need to obtain
cooperation from agencies and interview subjects.
It is thus useful to think of two important research concerns up front when choosing
individuals to interview. The first is about the accuracy and completeness of the information you
obtain in studying any single agency or part of government. The second is whether what you learn
is representative of other agencies and parts of government from which you do not gather data.
Accuracy can be understood as a complete and unbiased description of the object of study.
For example, to ensure that your interviews help you understand what is actually going on, you
will want to make sure you talk to people who both know how things work and can provide a
variety of perspectives. If your area of study affects multiple groups, you will want to talk to those
different groups, and you are also likely to want to talk to people who are both pleased with and
potentially concerned about the current state of affairs. For example, if attempting to understand
agency processes, it may be important to speak to both career staff and political appointees at
various levels. It may also be important to consult staff and appointees in different offices and
bureaus within the agency, particularly if they have relevant subject matter expertise.
In thinking about how a desire for accuracy affects your choice of interview subjects, you
may want to consider the limitations of traditional means of recruiting subjects. People you know,
people with ties to ACUS, or those who are loud critics or supporters of existing policy may not
fully represent the distribution of viewpoints within a given agency or policy domain. One
common method of recruiting subjects is snowball sampling, in which the interviewer relies upon
interview subjects to identify further individuals to contact. ACUS staff and agency contacts are
an important resource for starting this process. You should be conscious, however, of whether this
process of recruiting interview subjects prevents you from learning about opposing viewpoints and
consider alternative ways of identifying interview subjects, such as through secondary sources.
A second, related concept is representativeness — that is, whether what you learn from the
particular part of government you study is likely to be informative of how things work elsewhere
in government. This idea is analogous to the problem that arises in public opinion polls: to what
extent can you extrapolate from the subset of people interviewed to all individuals, including those
not reached by telephone or who refused to participate? There are at least two ways to address

6

concerns about representativeness. One is to try to seek, again in a manner analogous to polling, a
“representative sample” of parts of government affected by a particular policy or topic. In this
regard, it is important to recognize that this concept is distinct from random sampling, which is
usually not obtainable or sensible in this setting. Rather, systematic non-random approaches
attempt to identify interview subjects that can shed light on various aspects of a process or problem.
Chapter 1 of Herbert Kaufman’s The Administrative Behavior of Federal Bureau Chiefs (see
bibliography) discusses this particular issue of agency selection in this context.
If you were writing about rulemaking, for example, you may want to make sure to include
in the study agencies that are and are not subject to review by OIRA, those with relatively smaller
and larger budgets, and those that engage in infrequent and frequent rulemaking. The aim would
be to identify agencies with meaningful variation along a number of dimensions so as to learn
more about how those dimensions may influence their behavior. In doing so, it would be useful to
consult ACUS’s Sourcebook of United States Executive Agencies (Second Edition), which
discusses various forms of structural and personnel-based variation.
The second non-exclusive approach to dealing with representativeness is circumspection:
the researcher can acknowledge the limitations of the data gathering and be explicit that the claims
made on the basis of those agencies and participants are likely limited in light of those not included
in the study.
2. What is the role of the ACUS staff?
As discussed in Question III 1 above, ACUS staff can assist in identifying and reaching
out to agency contacts for interviews. ACUS staff can provide you with a list of ACUS government
members and government-affiliated liaison representatives and their contact information for any
interviews you intend to conduct. In some instances, it may be preferable for ACUS staff to make
the initial contact with the government officials being interviewed. At the outset of the project,
you should discuss with your ACUS project contact the optimal method for conducting outreach.
ACUS staff will include you on emails sent to ACUS members, and you should include
ACUS staff on emails sent to ACUS members. While ACUS staff can assist you in reaching out
to individuals for interviews, ACUS staff typically does not set up or attend interviews due to
resource constraints. However, case-specific exceptions might exist that may warrant ACUS staff
organization of or attendance at interviews. You should discuss more active involvement with
ACUS staff and consider how staff participation might affect potential information disclosure
under the Freedom of Information Act (FOIA) (see Question III 5 below). Once ACUS is aware
of the identity of the interview subject, for instance, that information may be subject to disclosure
under FOIA.
3. When is it appropriate to interview individuals external to government?
Interviews with individuals external to government may be valuable for at least two
reasons. First, individuals may be former government employees with insights into practices from
their periods of government service. Such perspectives can help shed light on past government
practices or changes to those practices. For example, if differences reflect changes in party control

7

or presidential administrations, approaching former government officials serving under political
conditions different from the current one would be particularly valuable. Second, individuals from
the general public may interact with relevant parts of government, such as by participating in
advisory committees or the rulemaking process. As discussed in Question II 1 above, if you plan
to interview ten or more people not acting in their capacity as federal employees, you must bring
this attention to the fact of ACUS staff members as soon as possible so they can determine whether
any PRA procedures apply.
Different ethical and practical concerns arise in talking with individuals outside of
government. On the one hand, former employees are no longer subject to direct retaliation by a
superior, but at the same time, as private citizens, they may no longer enjoy job protections as civil
servants. Their subsequent employment opportunities may thus be affected by comments they
make that could be attributed to them. For this reason, you may need to consider individualized
ways in which to gain their cooperation. Similarly, for practical reasons, it is likely more difficult
to identify and contact relevant interview subjects outside of government.
Some valuable resources on this front may include career websites like LinkedIn as well
as publicly posted resumes if the individual, for example, is currently at a think tank or university.
In addition, past agency directories or the U.S. Policy and Supporting Positions publication —
known as the “Plum Book” — may also be helpful resources to identify former agency officials.
The Plum Book lists more than 9,000 federal civil service positions and the incumbents in the
executive and legislative branches.
4. How should I think about how many interviews to conduct? How many interviews are
“enough”?
From a research perspective, key questions to ask are (1) whether you believe you fully
understand the topic on which you are writing and (2) whether that perspective is robust and
representative of differences in agency environments. In some cases, individuals in a single agency
may have very different perspectives, whereas, in others, differences across agencies may be more
important. See Question III 1 above for a more general discussion.
A related idea that some researchers have found useful in considering sample size is the
idea of theoretical saturation, the point at which no or little new information is being extracted
from the interviews. Factors that may help inform theoretical saturation include: (1) how structured
your interview questions are (the more structured, the sooner saturation is likely to be reached
relative to unstructured questions); (2) the complexity of the topic (the simpler and narrower the
focus, the sooner saturation is likely); and (3) the homogeneity of the interview population (the
more homogenous, the sooner the likely saturation, given the higher probability of shared
perspectives and experiences).
Many prior ACUS consultants report that their decision about how many interviews to
conduct was also necessarily dictated by practical considerations, including the time available to
complete a report and their ability to identify individuals with whom to speak. You should thus
attempt to plan your timelines in advance to pursue methodological best practices as much as
possible.

8

5. How should I approach attribution issues? Relatedly, what steps can I take to protect
the interviewee’s information and identity?
In most cases, interviewees are likely to be willing to speak on the record about the nonsensitive topics raised in ACUS-related projects. However, it is standard interview practice to
provide individuals with the chance to speak without direct attribution. Doing so may increase
their willingness to participate. In such situations, you can state that you will take steps to protect
the subject’s confidentiality and anonymize the interviews before they are sent to ACUS.
Confidentiality is the promise to not disclose a subject’s identity, whereas anonymization is the
removal of any linkage between the data (interview quotes) and the subject’s identity.
Further consideration, however, should be given to whether you are also willing to not
report an interviewee’s agency or agency unit (e.g., sub-agency, office, bureau) to protect a
subject’s anonymity. Reporting the interview data without a sense of which agencies and agency
units were consulted in the report could diminish the usefulness and credibility of the report. Doing
so would also hinder the ability of readers to evaluate independently how representative the data
are. The same tradeoffs exist for state or regional affiliations.
It is also important to be aware that FOIA may require ACUS to release identifying
information it has about interview subjects if such information is provided to ACUS. It is possible
that certain exemptions and privileges may protect this information if there is a FOIA request, but
this is not guaranteed. These observations imply that you cannot promise confidentiality to subjects
(that is, that their identity will not be disclosed) if you provide identifying information to ACUS.
Be aware that an individual’s identity might be inferred from other information he or she has
provided, such as position title or even agency unit if it is sufficiently small.
If confidentiality is important, you will thus want to minimize the chances of accidental
disclosure of identifying information to ACUS. One important way to do this is to avoid sending
ACUS a list of your interviewees. Another way to do this may be to avoid keeping identities in
interview notes (both in the notes themselves as well as in naming files), so that even an accidental
disclosure of those notes would not reveal an interview subject’s identity. Best research practices
include avoiding the collection of unnecessary personally-identifying information and keeping
research notes and other material physically and digitally secure. Those at state institutions should
be especially aware of these best practices, as their research may be subject to state open records
laws as well.
A closely related issue to interviewee identity is the issue of confidentiality regarding thirdparty information and data. On this topic, please consult the relevant provisions of the consultant
contract.
6. Any tips for how to contact potential interviewees?
The two most common ways in which interview subjects are recruited for ACUS projects
are (1) prior personal/professional contacts you might have and (2) ACUS staff-facilitated
introductions with ACUS representatives at different agencies. ACUS has representatives in all or

9

nearly all of the largest federal agencies, and those representatives can help secure contact with
appropriate individuals in smaller agencies.
One practical tip when using email to reach out to individuals is to send a carbon copy
(“cc”) to the ACUS Government Member in the agency or another ACUS contact, such as the staff
attorney on the project. Doing so may help improve response rates, though it can also make it more
difficult to promise anonymity as discussed in Question III 5 above. Another is to secure
cooperation from higher-ranked agency officials before reaching out to subordinates so that those
lower-level officials know their participation is endorsed by agency higher-ups. Even when ACUS
does not have representatives in place, copying an ACUS contact such as the relevant staff attorney
on the correspondence may serve as a useful demonstration of your credibility.
Finally, peer referrals from other interview subjects are often useful both to identify
additional subjects and to garner cooperation, particularly when those referrals are references in
the initial email. See Question III 1 above on “snowball” sampling. As noted in Question III 5
above, one may want to limit communication with ACUS about which particular individuals are
interviewed if the subjects are promised confidentiality.
7. Should I conduct the interview by telephone or in person? What are the tradeoffs?
For logistical reasons, it is often easier to arrange interviews by telephone. However,
telephone interviews make it more difficult to gauge/ensure subject engagement and also limit
non-verbal forms of communication. These constraints may, in turn, render it more difficult to
develop rapport with the subject. For these reasons, research on interview methods often states that
in-person interviews are more likely to yield useful information, a view that seems to comport with
the practice of prior ACUS consultants. Nonetheless, telephone interviews may also be a useful
way to conduct preliminary or background research, to follow up on new topics with prior subjects,
or to reach subjects who could not fit a face-to-face interview into their schedules.
8. Any tips for developing rapport with the interviewee?
Convincing subjects that you are trustworthy and engaged is one of the key elements of
effective interviewing. The individuals you are interviewing are busy. It is thus important to make
clear up front the purpose of the interviews and research topic. Demonstrating that you have done
necessary background research to prepare for the interview and that you are interested and taking
their responses seriously is likely to help in building rapport. So is returning their calls or emails
promptly, answering their pre-interview questions thoroughly, and accommodating their
scheduling needs. In addition, you may want to offer confidentiality or anonymity, subject to the
considerations discussed in Question III 5 above.
You may also want to consider leaving the interview location to the interviewee’s
discretion — hopefully one that is convenient and makes the interviewee the most comfortable,
depending on the nature of the report topic. Possibilities might include the interviewee’s own
office, a conference room within the agency, or a more public meeting place. In addition, you
should consider the tradeoffs involved should you decide to interview small groups of individuals
together. On the one hand, group discussions may identify issues an individual would not have

10

considered on his or her own. They may also be easier to schedule and may save time. On the other
hand, they might foster groupthink. A group setting may also make an individual hesitant to raise
issues, particularly if the group is a mix of both career and political appointees or composed of
individuals at different levels of a managerial hierarchy.
During the interview itself, you should build on your pre-interview rapport by following
commonsense interview etiquette. For example, you should avoid checking your watch/phone,
since doing so may make the interviewee feel rushed or that he or she is not providing helpful
responses. One useful way to think of the interview flow may be to naturally begin with small talk,
then easing into the interview with a brief reiteration of who you are and the research purpose to
signal the beginning of the interview, followed by the questions. You may wish to conclude the
interview by asking whether the subject has anything to add or wishes to clarify anything said
earlier.
9. Should I record the interview or else take notes?
The clear advantage of recording interviews is that it eases your note-taking burden. In
addition, it allows you to more accurately document quotations to highlight in your report.
However, tape recordings identify interview subjects and may diminish the chances that
individuals will speak freely, even when you do not intend to record further identifying
information. (It is important in this regard to review Question III 5 above regarding issues of
confidentiality and accidental disclosure.) When interviews are recorded, best practices are to
transcribe and then erase/delete recordings as soon as feasible, thereby minimizing risks of
accidental disclosure or breaches of confidentiality.
The most sensible alternative is to take careful notes. One option on this front is to take
contemporaneous notes. The main benefit is that you will be able to gain information in real time
and do not have to worry about technical problems with audio recording. One drawback is that it
is very difficult to accurately take notes that fully capture an interview as it is taking place.
Additionally, taking notes may distract from the interview itself, thereby impeding
contemporaneous follow-up inquiries. One common strategy to address this tradeoff is to take
limited notes during an interview (often in outline form or with key quotations) but then to
immediately set time aside after an interview to go back and edit your notes and add additional
material while it is still fresh in your mind. In light of this advice, you may wish to schedule free
time between interviews.
Another option is to write down notes only after the interview is complete. This option may
be most ideal when the topic is sensitive or the circumstances may otherwise render
contemporaneous note taking distracting. The disadvantage, of course, may be your inability to
remember the interview topics and responses in detail. Finally, another option is to hire a dedicated
note taker, freeing you to focus solely on the interview itself. This route, however, obviously
requires more resources, and may, like a tape recorder, compromise trust and rapport. More
generally, several of the readings listed in the bibliography further discuss the tradeoffs among
different ways of documenting interviews.

11

IV.

Analyzing and Presenting the Results
1. Should I code the interviews and quantify the results and, if so, how?

Coding the interview and quantifying the results is most appropriate when you have
decided to use structured interviews, though it is also possible to transcribe and code open-ended
responses (see Question I 2 above). If you seek to characterize the nature of responses during
interviews in a formal quantitative way, you will want to consider what ideas you seek to quantify
and how to reliably measure those ideas. Recording and transcribing the interviews may also be
more valuable in gathering data to ensure greater accuracy and comparability. Beyond these basic
principles, the topic of quantitative interview analysis is a large one that cannot be fully addressed
here. Several of the suggested readings listed in the bibliography address questions of validity and
reliability.
2. How should I present my results?
It is best to describe in a straightforward and direct manner your methodological choices,
your reasons for them, and their respective strengths and limitations. Many of the strongest ACUS
reports have standalone research design or methodology sections. Even if you have promised your
subjects confidentiality, you should also clearly state how you selected interview subjects —
agencies, people within agencies, etc. — so that readers of your report can evaluate for themselves
how representative your results are. See Question III 1 above about issues of generalization and
inference. You should also consider noting in the aggregate how many agencies declined to
participate and, if so, acknowledge that they may have been limited by staff and resource
constraints to avoid imputing motives. It is especially important to keep the potential limitations
of your interview findings in mind as you formulate recommendations that are directed at all
agencies. Accordingly, you may consider narrowing the scope of your recommendations to match
the scope of your report, or else gather sufficient data to support a more broadly aimed
recommendation.
An additional consideration is how to incorporate comments from interview subjects about
your draft report into your final report. Indeed, you may send your draft report or portions of your
draft report to interview subjects for their review and feedback. This is a useful opportunity for
interview subjects to clarify, expand on, and correct issues as necessary. If you would like to
address the additional comments, you should inform ACUS that you are sending the draft report
to interview subjects and set aside sufficient time for the interview subjects to respond and for you
to incorporate any feedback into your report and still ensure that you meet the project deadlines.
If you do offer an interview subject the opportunity to review the draft and provide comments, you
should note to the interviewee that the offer is extended as a courtesy to the interview subject rather
than as an invitation for the interview subject or his or her agency to approve the text of the report.
You should take care that the interview subject understands the scope of the review and submission
of comments. You have the discretion to make appropriate edits to the final report in response to
any comments from interview subjects, and the final report need not specifically identify that any
edits or changes were made in response to an interview subject’s comments.

12

V.

Bibliography

Aberbach, Joel D., and Bert A. Rockman. “Conducting and Coding Elite Interviews.” PS:
Political Science & Politics, vol. 35, no. 4, 2002, pp. 673–676.,
doi:10.1017/S1049096502001142.
Aberbach, Joel D., James D. Chesney, and Bert A. Rockman. 1975. “Exploring Elite
Political Attitudes: Some Methodological Lessons.” Political Methodology 2(1): 1–27.
Berry, Jeffrey M. “Validity and Reliability Issues In Elite Interviewing.” PS: Political
Science & Politics, vol. 35, no. 4, 2002, pp. 679–682., doi:10.1017/S1049096502001166.
Dexter, Lewis Anthony. 2006. Elite and Specialized Interviewing. ECPR Press.
Fenno, Richard F. Home Style: House Members in Their Districts. Boston: Little, Brown,
1978. (See the appendix on participant observation and note taking).
Goldstein, Kenneth. “Getting in the Door: Sampling and Completing Elite Interviews.”
PS: Political Science & Politics, vol. 35, no. 4, 2002, pp. 669–672.,
doi:10.1017/S1049096502001130.
Guest, Greg, Emily E. Namey & Marilyn L. Mitchell, Collecting Qualitative Data: A
Field Manual for Applied Research (2013).
Harvey, William S. “Strategies for conducting elite interviews.” Qualitative Research,
Vol 11, Issue 4, pp. 431 – 441. https://doi.org/10.1177/1468794111404329.
Kaufman, Herbert. 1981. The Administrative Behavior of Federal Bureau Chiefs.
Washington, DC: Brookings Institution Press.
King, Gary, Robert O. Keohane and Sidney Verba. 1994. Designing Social Inquiry:
Scientific Inference in Qualitative Research.
Kingdon, John W. 1973. Congressmen's voting decisions. New York: Harper & Row.
Leech, Beth L. “Asking Questions: Techniques for Semistructured Interviews.” PS:
Political Science & Politics, vol. 35, no. 4, 2002, pp. 665–668.,
doi:10.1017/S1049096502001129.
Peabody, Robert L., et al. “Interviewing Political Elites.” PS: Political Science &
Politics, vol. 23, no. 3, 1990, pp. 451–455., doi:10.2307/419807.
Rubin, Herbert J., and Irene Rubin. 1995. “Chapter 6.” Qualitative interviewing: The Art
of Hearing Data. Beverly Hills, CA: SAGE.

13