Patient-Focused Drug 
Development: Methods to 
Identify What Is 
Important to Patients 
Guidance for Industry, Food and Drug 
Administration Staff, and Other 
Stakeholders 
 
 
 
 
 
 
U.S. Department of Health and Human Services 
Food and Drug Administration 
Center for Drug Evaluation and Research (CDER) 
Center for Biologics Evaluation and Research (CBER) 
February 2022 
Procedural 
 
   
Patient-Focused Drug 
Development: Methods to 
Identify What Is 
Important to Patients 
Guidance for Industry, Food and Drug 
Administration Staff, and Other 
Stakeholders
 
Additional copies are available from: 
Office of Communications, Division of Drug Information 
Center for Drug Evaluation and Research 
Food and Drug Administration 
10001 New Hampshire Ave., Hillandale Bldg., 4th Floor 
Silver Spring, MD 20993-0002 
Phone: 855-543-3784 or 301-796-3400; Fax: 301-431-6353 
Email: druginfo@fda.hhs.gov 
https://www.fda.gov/drugs/guidance-compliance-regulatory-information/guidances-drugs  
and/or 
Office of Communication, Outreach and Development 
Center for Biologics Evaluation and Research 
Food and Drug Administration 
10903 New Hampshire Ave., Bldg. 71, Room 3128 
Silver Spring, MD 20993-0002 
Phone: 800-835-4709 or 240-402-8010 
Email: ocod@fda.hhs.gov 
  https://www.fda.gov/vaccines-blood-biologics/guidance-compliance-regulatory-information-biologics/biologics-guidances 
U.S. Department of Health and Human Services 
Food and Drug Administration 
Center for Drug Evaluation and Research (CDER) 
Center for Biologics Evaluation and Research (CBER) 
February 2022 
Procedural 
  Contains Nonbinding Recommendations 
TABLE OF CONTENTS 
I.  INTRODUCTION .............................................................................................................. 1 
  Overview of the Series of FDA Guidance Documents on Patient-Focused Drug 
Development ................................................................................................................................ 1 
  Purpose and Scope of Guidance 2 ............................................................................................. 2 
II.  METHODS TO IDENTIFY AND UNDERSTAND WHAT IS IMPORTANT TO 
PATIENTS .......................................................................................................................... 3 
  Background Research ................................................................................................................. 3 
  Overview of Methods .................................................................................................................. 3 
III.  QUALITATIVE RESEARCH METHODS ..................................................................... 4 
  Common Qualitative Methods Used To Obtain Patient Input ............................................... 4 
1.  One-on-One Interviews ......................................................................................................... 4 
2.  Focus Groups ........................................................................................................................ 5 
3.  Choosing Between One-on-One Interviews and Focus Groups, In-Person or Remote ........ 6 
  Approaches To Asking the Right Question .............................................................................. 8 
IV.  QUANTITATIVE RESEARCH METHODS .................................................................. 9 
  Choice of Survey Administration Method ................................................................................ 9 
  Considerations for Developing Questions and Response Options for a Survey Instrument
 .................................................................................................................................................... 10 
V.  MIXED METHODS ......................................................................................................... 15 
VI.  MANAGING BARRIERS TO SELF-REPORT ........................................................... 16 
VII.  CONSIDERATIONS FOR USE OF SOCIAL MEDIA ................................................ 18 
REFERENCES ............................................................................................................................ 20 
APPENDIX 1. Other Qualitative Methods ............................................................................... 23 
  Delphi Methods ......................................................................................................................... 23 
  Observational Methods............................................................................................................. 23 
  Facilitated Discussions at Patient Meetings ............................................................................ 23 
  Survey Instruments with Open-Ended Questions .................................................................. 23 
APPENDIX 2. Considerations for Selection of Qualitative Data Collection Methods ......... 24 
APPENDIX 3. Study Materials for Qualitative Studies .......................................................... 27 
APPENDIX 4. Analysis of Qualitative Data ............................................................................. 30 
  Data Preparation ....................................................................................................................... 30 
  Data Analysis ............................................................................................................................. 30 
  Saturation .................................................................................................................................. 32 
  Quality Control ......................................................................................................................... 33 
  Data Reporting .......................................................................................................................... 33 
APPENDIX 5. Screening and Exit Interview Studies/Survey Studies ................................... 34 
i  Contains Nonbinding Recommendations 
TABLE OF TABLES 
Table 1. Potential Strengths and Limitations of Administration Methods ............................. 7 
Table 2. Some Potential Strengths and Limitations of Open- and Closed-Ended Questions
 ............................................................................................................................................. 12 
Table 3. Response Options ......................................................................................................... 12 
Table 4. Potential Strengths and Limitations of Qualitative Data Collection Methods ....... 24 
Table 5. Study Materials ............................................................................................................ 27 
Table 6. Steps Typically Used for Data Analysis in Qualitative Studies................................ 30 
Table 7. Saturation Table Example ........................................................................................... 32 
 
iii  Contains Nonbinding Recommendations 
Patient-Focused Drug Development: Methods to Identify What Is 
Important to Patients 
Guidance for Industry, Food and Drug Administration Staff, and 
Other Stakeholders 
 
 
 
This guidance represents the current thinking of the Food and Drug Administration (FDA or Agency) on 
this topic.  It does not establish any rights for any person and is not binding on FDA or the public. You 
can use an alternative approach if it satisfies the requirements of the applicable statutes and regulations. 
To discuss an alternative approach, contact the FDA staff responsible for this guidance as listed on the 
title page. 
 
 
 
I.  INTRODUCTION 
  Overview of the Series of FDA Guidance Documents on Patient-Focused 
Drug Development 
This guidance (Guidance 2) is the second in a series of four methodological patient-focused drug 
development (PFDD) guidance documents1 that describe how stakeholders (patients, researchers, 
medical product developers, and others) can collect and submit patient experience data2 and 
other relevant information from patients and caregivers to be used for medical product3 
development and regulatory decision-making. The topics that each guidance document addresses 
are described below. 
 
  Methods to collect patient experience data that are accurate and representative of the 
intended patient population (Guidance 1)4 
  Approaches to identifying what is most important to patients with respect to their 
experience as it relates to burden of disease/condition and burden of treatment 
(Guidance 2) 
  Approaches to selecting, modifying, developing, and validating clinical outcome 
assessments to measure outcomes of importance to patients in clinical trials 
(Guidance 3) 
 
1 The four guidance documents correspond to commitments under section I.J.1 associated with the sixth 
authorization of the Prescription Drug User Fee Act (PDUFA VI) under Title I of the FDA Reauthorization Act of 
2017, as well as  requirements under section 3002 of the 21st Century Cures Act (available at 
https://www.fda.gov/downloads/forindustry/userfees/prescriptiondruguserfee/ucm563618.pdf). 
2 21st Century Cures Act: https://www.congress.gov/bill/114th-congress/house-bill/34v.  
3 A drug or biological product. 
4 See the guidance for industry, FDA staff, and other stakeholders Patient-Focused Drug Development: Collecting 
Comprehensive and Representative Input (June 2020). We update guidances periodically. For the most recent 
version of a guidance, check the FDA guidance web page at https://www.fda.gov/regulatory-information/search-fda-
guidance-documents. 
1  Contains Nonbinding Recommendations 
  Methods, standards, and technologies for collecting and analyzing clinical outcome 
assessment (COA) data for regulatory decision-making (Guidance 4), including 
selecting the COA-based endpoint and determining clinically meaningful change in that 
endpoint 
 
Please refer to Guidance 1 and other FDA guidances5 for additional information on patient 
experience data. 
 
In conducting research that involves accessing patient experience data or directly engaging with 
patients, it is important to carefully consider Federal, State, and local laws and institutional 
polices for protecting human subjects and reporting adverse events. For additional information 
about human subjects protection, refer to section IV.A.2 of Guidance 1.  
 
FDA encourages stakeholders to interact early with FDA and obtain feedback from the relevant 
FDA review division when considering collection of patient experience data related to the 
burden of disease and treatment. 6 FDA recommends that stakeholders engage with patients and 
other appropriate subject matter experts (e.g., qualitative researchers, clinical and disease 
experts, survey methodologists, statisticians, psychometricians, patient preference researchers) 
when designing and implementing studies to evaluate the burden of disease and treatment, and 
perspectives on treatment benefits andrisks. 
  Purpose and Scope of Guidance 2 
This guidance will discuss methods for eliciting information from individuals identified in 
Guidance 1. It will discuss best practices in conducting qualitative research and reference-related 
resources7; however, it should not be viewed as providing detailed instructions on how to use 
particular methods or as a substitute for engaging subject matter experts when undertaking the 
work described. 
 
The methods described in this document can be used to elicit what is important to patients, which 
may in turn help inform understanding of disease/condition and clinical trial design. It may also 
help the generation and use of patient experience data, including clinical outcome assessments 
and patient preference information, to inform benefit-risk assessment. 
 
 
5 See FDA guidance for industry Patient Preference InformationVoluntary Submission, Review in Premarket 
Approval Applications, Humanitarian Device Exemption Applications, and De Novo Requests, and Inclusion in 
Decision Summaries and Device Labeling (August 2016), or subsequent guidances in the PFDD series, when 
available.  
6 In addition to the general considerations discussed in this guidance, a study may need to meet specific statutory 
and regulatory standards governing the collection, processing, retention, and submission of data to the FDA to 
support regulatory decisions regarding a marketed or proposed medical products. This guidance focuses on more 
general considerations that apply to many types of studies, and you should consult with the review division and 
applicable guidance regarding any other applicable requirements. 
7 The citation of a scientific reference in this guidance does not constitute FDAs endorsement of approaches or 
methods presented in that reference for any particular study. Study designs are evaluated on a case by case basis 
under applicable legal standards. 
2  Contains Nonbinding Recommendations 
This guidance does not address methods for collecting and analyzing COA data, a topic to be 
covered in later guidance in this series. It also does not address methods for collecting and 
analyzing patient preference information, which is addressed in other FDA guidances.8 
 
The contents of this document do not have the force and effect of law and are not meant to bind 
the public in any way, unless specifically incorporated into a contract. This document is intended 
only to provide clarity to the public regarding existing requirements under the law. FDA 
guidance documents, including this guidance, should be viewed only as recommendations, unless 
specific regulatory or statutory requirements are cited.  The use of the word should in Agency 
guidance means that something is suggested or recommended, but not required. 
II.  METHODS TO IDENTIFY AND UNDERSTAND WHAT IS IMPORTANT TO 
PATIENTS 
  Background Research 
Research to understand what matters most to patients living with a disease or condition to guide 
medical product development should begin with a characterization of the disease or condition 
and currently available therapies. Before conducting studies in patients, which may involve their 
caregivers as well, literature reviews, consultation with relevant subject matter experts, and other 
information sources should be used to develop targeted research questions and select appropriate 
methods to identify what matters most to patients regarding their experience with their disease or 
condition. 
  Overview of Methods 
When planning research, consider whether the study sample reflects the group of individuals 
(patients) of interest (i.e., the target population), including the spectrum of disease or condition 
and patient diversity, and whether the methods used to elicit information from patients are 
appropriate for the research objective and target population.9 
 
Qualitative research methods, quantitative research methods, or mixed-methods research can be 
used to identify what is important to patients. These methods can be used either independently or 
complementarily. When selecting an appropriate research method or set of methods, FDA 
recommends carefully considering the research objectives: 
 
  Qualitative research methods, such as interviews and focus groups, are typically used to 
obtain a deeper understanding of the patient experience by generating in-depth 
 
8 Issues related to patient-reported outcome measures and patient preference information are addressed in the 
following guidances for industry: (1) Patient-Reported Outcome Measures: Use in Medical Product Development to 
Support Labeling Claims (December 2009) and (2) Patient Preference InformationVoluntary Submission, Review 
in Premarket Approval Applications, Humanitarian Device Exemption Applications, and De Novo Requests, and 
Inclusion in Decision Summaries and Device Labeling (August 2016). 
9 For more information on defining research questions and the target population, see the guidance for industry, FDA 
staff, and other stakeholders Patient-Focused Drug Development: Collecting Comprehensive and Representative 
Input (June 2020). 
3  Contains Nonbinding Recommendations 
information about the experiences, perspectives, priorities, preferences, and feelings of 
patients and others, in their own words. 
  Quantitative research methods, such as surveys, are characterized by the collection of 
quantifiable data (e.g., numerical data) and the application of statistical methods to 
summarize the collected patient experience data, to describe, compare, or relate 
measures of patient experience. 
  Mixed methods research, such as a survey instrument with open-ended and fixed-
response questions or interviews combined with administration of a survey instrument 
with fixed-response questions, involves using both qualitative and quantitative 
approaches or methods in a single study or program of inquiry to understand the patient 
experience.  
 
Although this document includes distinct sections for qualitative and quantitative research 
methods, many data collection methods may be used in either approach. For example, interviews 
are commonly used to generate qualitative data, but they also may be used to generate 
quantitative data. Similarly, survey instruments are commonly used to generate quantitative data 
but may also be used to generate qualitative data with open-ended questions. 
III.  QUALITATIVE RESEARCH METHODS 
  Common Qualitative Methods Used To Obtain Patient Input 
One-on-one interviews and focus groups are two common qualitative methods, which will be 
discussed in this section. Other commonly used qualitative methods are summarized in 
Appendix 1.10 There is no single preferred qualitative method for all uses and research questions. 
The method selected should be suitable for the research objective(s) and question(s). FDA will 
review the rationale and appropriateness of the qualitative method chosen. Before selecting 
qualitative data collection methods, consider patient characteristics and potential strengths and 
limitations of the methods discussed in Table_4 in Appendix 2. There also is not a universal 
number of interviews and focus groups to conduct; it is dependent on the research objective. For 
certain methods, it may be guided by the achievement of saturation11 (see Appendix 4).  
1.  One-on-One Interviews 
One-on-one interviews (i.e., one person interviewing another person) involve a discussion on the 
topic of interest between the study participant and a trained interviewer. One-on-one interviews 
offer opportunities to explore topics in-depth at an individual level using probing questions. 
There are three different types of interview methods: structured interviews (interviewer asks a set 
of predefined questions), semi-structured interviews (interviewer asks a set of predefined 
questions and probing questions), unstructured interviews (interviewer asks unplanned or 
spontaneous questions). 
 
10 Many of these methods can be accomplished in person or using technology (e.g., social media, online forums, and 
other web-based methods). 
11 Also referred to as concept saturation, data saturation, or thematic saturation. For further discussion of saturation, 
see e.g., Bernard and Ryan, 2010; Suter, 2012; Creswell, 2013. 
4  Contains Nonbinding Recommendations 
When using one-on-one interviews, it is important to consider the following: 
 
  Patient selection and sample size 
  Selecting sources for participant recruitment (e.g., medical practices/centers, 
academic institutions, research consortia, polling organizations, national panels), 
where applicable, to recruit participants for in-person interviews.  
  Choosing an appropriate sampling method (refer to Guidance 1). 
  Estimating an adequate number of interviews to conduct12, while considering 
number of sites, as well as  geographic and patient representation 
  A greater number may be needed for unstructured interviews, broad or 
complex topics, or diverse populations. 
  Interview and data collection methods 
  Selecting the appropriate interview type and administration method (i.e., in-
person or remote) for the target population; see section III.A.3 for more 
information on administration methods. 
  Designing suitable interview questions and interview guide (focus on concepts of 
importance for context of use and research objectives). 
  Pilot testing interview guide (i.e., administer the interview in a small number of 
participants) to identify and correct any methodological or logistical issues before 
using in the qualitative study. 
  Interview conduct 
  Pilot testing length of interview and appropriateness of length for administration 
method. 
  Selecting and training interviewers to perform interviews (e.g., considering 
expertise in performing interviews and other factors based on the characteristics 
of the disease or condition and target population under study). 
  The emotional burden for the respondent (potential for heightened 
emotions, including anxieties and discomfort among patients and 
caregivers), as well as the emotional burden for the interviewer (potential 
for emotional distress associated with hearing about difficult patient and 
caregiver experiences), may affect responses. 
2.  Focus Groups 
Focus groups involve a conversation with a group of participants led by a moderator. The 
moderator can explore issues at the individual level, as well as encourage discussions among 
participants at the group level. This approach elicits a range of experiences. 
 
There are different types of focus groups13; however, the most common type is the single focus 
group, an interactive discussion of a topic by a set of participants and a moderator(s) as one 
group in one place. Generally, a moderator uses a semi-structured discussion guide to direct the 
group conversation. 
 
 
12 Note the sample size estimated before the study may not match the final sample size. 
13 For additional discussion of focus groups, see Guest et al., 2013; Greenbaum, 2000. 
5  Contains Nonbinding Recommendations 
The considerations for focus groups are similar to one-on-one interviews (e.g., choosing an 
appropriate administration method and sampling method, designing suitable questions and 
discussion guide, pilot testing the discussion guide and length of meeting). Other important 
considerations for focus groups include: 
 
  Selecting a trained moderator with the appropriate skills 
  Choosing the number of focus groups to conduct, which may vary based on factors 
such as: 
  Complexity of the topic(s) (e.g., all versus some impacts of a disease or 
condition on multiple dimensions of a patients quality of life, dependent upon 
therapeutic area and research question(s)) 
  Sensitivity of the topic(s) (e.g., separation of genders for sensitive topics such as 
sexual function) 
  Diversity of the participant sample 
  Number of subgroups planned (e.g., different age groups, disease or condition 
severity groups, gender) 
  Determining the sample size for each focus group: generally, the goal is to keep the 
group small enough to enable the elicitation of in-depth responses from each participant 
but large enough to get a wide variety of perspectives across different severity levels 
and demographic representation within the target disease or condition 
  Although there is no set number recommended for a focus group sample, 
sample sizes between 5 and 10 participants are common.14  
  A group may become fragmented (e.g., multiple, simultaneous conversations 
occur) when there are too many participants, decreasing the likelihood of 
engagement and responses from each individual. 
3.  Choosing Between One-on-One Interviews and Focus Groups, In-Person or Remote 
Both one-on-one interviews and focus groups may be useful to capture patient experience data. 
For example, focus groups could be used in preliminary research to explore a broad topic 
followed by one-on-one interviews to obtain more detailed information on the topic of interest.  
 
Each interview method has its own strengths and limitations. One-on-one interviews can be used 
to obtain more detailed (in-depth) individual experiences; address sensitive topics; and explore 
diseases or conditions with many different symptoms that vary from patient to patient (i.e., 
heterogenous disease/condition presentation).  Focus groups allow for participant interaction and 
can capture a range of perspectives within a population in a shorter time frame. 
 
One-on-one interviews and focus groups can be conducted in-person or remotely (e.g., use of 
computers or telephones). 
 
FDA does not have a single recommended administration method for conducting interviews and 
focus groups; however, the method should be appropriate for the selected target population, 
study characteristics, and study objective(s). FDA will review the rationale and appropriateness 
of the administration method chosen for the intended use. 
 
14 For additional discussion on size of focus groups, see Hennink and Levy, 2014. 
6  Contains Nonbinding Recommendations 
 
Each administration method has strengths and potential limitations. A growing body of literature 
suggests that there are no marked differences between interview administration methods 
regarding the accuracy of the data collected.15 
 
Table 1 lists some potential strengths and limitations for in-person and remote administration 
methods. Some strengths and limitations are not entirely distinct for the methods. 
Table 1. Potential Strengths and Limitations of Administration Methods 
Administration 
Method  Strengths  Limitations 
In-person    Allows for collection of both    Cost can be prohibitive (e.g., travel 
verbal and nonverbal responses  costs, facility/room space rental 
to help inform data  fees). 
interpretation.    It may be a hardship for patients 
  A variety of written and  and/or caregivers to attend in-
brainstorming exercises (e.g.,  person (e.g., health status, 
ranking exercises for concept  logistical or economic constraints). 
of interest) can be incorporated 
into the interview/discussion 
guide to help elicit information. 
Telephone (audio only)    Participation is not limited to a    A rapport may be difficult to 
geographic location  study  establish between the 
sampling can be nationwide or  interviewer/moderator and 
worldwide (i.e., ability to  participant. 
obtain geographically and    Participants may not have a private 
socially diverse populations).  space to feel comfortable 
  Participants can be involved in  participating. 
the comfort of their homes or    Disruptions (e.g., background 
location of their choice.  noise and presence of family 
  Useful for participants who are  members) can interfere with sound 
unable to travel, have poor  quality and cause distractions. 
health status, and/or have 
limited mobility or disability. 
  No travel costs or facility/room 
space rental fees. 
Online/virtual video    Potential for participants to see    Participants need to have access to 
conferences  each other if using a web cam.  the appropriate technology (e.g., a 
  Allows opportunity to see  computer, webcam, internet 
facial expressions.  service). 
  Also, see strengths for in-   Technical problems (e.g., wireless 
person and telephone  internet signal problems, 
administration.  audio/video issues). 
  Potential security problems with 
personal data collection via online. 
  Potential costs for virtual 
conferencing platform(s). 
 
15 Block and Erskine, 2012; Cachia and Millward, 2011; Shapka et al., 2016; Vogl, 2013. 
7  Contains Nonbinding Recommendations 
Administration 
Method  Strengths  Limitations 
  See also the limitations for 
telephone administration. 
  Approaches To Asking the Right Question 
Regardless of method, the way questions are framed is critical to collecting unbiased patient 
input. Although spontaneous responses are ideal, there are situations in which participants may 
need to be prompted. 
 
Prompts (i.e., open-ended questions to stimulate and provoke participation in the discussion) are 
used to help the interviewer/moderator gain more information, particularly if the participant does 
not initially provide detailed responses. Prompt questions should avoid leading the participant. 
Leading questions (i.e., questions that include or imply the desired answer to the question in the 
phrasing of the question) are problematic because they may result in biased or false/misleading 
answers (results). They may also lead to a missed opportunity to hear an unexpected insight. 
 
Approaches to avoid leading questions include (but are not limited to): 
 
  Do not suggest an answer. 
  Do not assume you know what the participant is thinking or feeling. 
  Do not ask questions that cast judgment on a participants belief, choice, or perspective, 
or imply that you prefer the participant to respond in one way versus another. 
 
The boxed text that follows offers some examples of probing questions or prompts that are 
leading or otherwise problematic and offers potential solutions. 
EXAMPLES 
Example 1 
Research Objective: 
Determine what aspects of peripheral artery disease patients would like to see improved with 
treatment. 
Leading Probing Question: 
Wouldnt you consider it most important to improve your walking distance, for example, how far 
you walk around the track when you exercise? 
Problem: This question guides the respondent to provide an answer that is more favorable or 
preferred by the researcher. Additionally, the for example clause may not include relevant 
examples for the research participant. 
Potential Solution: Consider rephrasing as: 
Tell me how peripheral artery disease impacts you. What would you like to see improve with 
treatment? 
8  Contains Nonbinding Recommendations 
EXAMPLES 
Example 2 
Research Objective: 
Determine what factors to consider when deciding whether to treat your condition with medication. 
Probing Question That May Be Perceived as Casting Judgment: 
Could you tell me why you are not treating your condition with medication? 
Problem: This question implies that the interviewer is potentially casting judgment on the 
participants beliefs or choices. 
Potential Solution: Consider rephrasing as: 
What did you consider when deciding whether to treat your condition with medication? 
IV.  QUANTITATIVE RESEARCH METHODS 
Survey research methods are commonly used to collect quantitative data from patients and 
relevant stakeholders. Refer to Guidance 1 for considerations for data management and data 
analysis (including data analysis plans). In designing a survey instrument, it is important to 
decide how to administer the survey instrument and how to design and test the instructions, 
questions, and response options. 
  Choice of Survey Administration Method 
Survey instruments can be self-administered or interviewer-administered. Self-administered 
surveys can be paper-based, telephone-based (e.g., interactive voice response system), or 
electronic-based (e.g., computers, tablets, smartphone). Interviewer-administered surveys can be 
conducted in-person or remotely. 
 
Choice of administration method may be driven by a variety of factors. For example, an 
interviewer-administered survey instrument, a survey instrument using an interactive voice 
response system, or a survey instrument easily completed using common assistive technology on 
a computer may be useful in patients with visual impairment. 
 
Self-administration allows participants to respond at their own pace and at their convenience. 
Compared to interviewer-administered surveys, self-administration typically is less costly and 
removes the potential for interviewer bias. Computer-assisted and administered survey 
instruments can navigate skip patterns, help minimize missing data, and allow real-time data 
collection and analysis compared with other methods. 
 
Administering survey instruments in a clinical trial at screening and/or exit visits (e.g., occurs at 
or after the end-of-treatment visit or last clinic visit) may add greater depth to understanding the 
burden of disease or condition, treatment, and trial participation, as well as provide more detail 
on patients perspectives on treatment benefits and harms, which may help inform drug 
development (e.g., future clinical trial design) and COA development (see Appendix 5).16 
 
16 For a discussion of additional steps to consider when administering survey instruments in noninterventional 
studies, see Cooper et al., 2006. 
9  Contains Nonbinding Recommendations 
  Considerations for Developing Questions and Response Options for a Survey 
Instrument 
Survey instrument instructions and questions should be: 17 
 
  Well-aligned with the research objective(s) and designed to answer the research 
questions 
  Specific to the targeted concept for the research question (e.g., disease/condition 
symptoms and impacts, current treatment, past treatments, treatment side effects) with 
minimal redundancies across questions 
  Well-understood by participants to enhance consistency of response, including having 
been: 
  Assessed for appropriateness for the ability of the population (e.g., reading and 
writing ability (literacy); ability to understand health information (health 
literacy); and ability to understand and use numbers (numeracy)_) 
  Developed to include natural, familiar language (e.g., minimal use of clinical 
terminology) as used by patients when discussing the concept of interest 
  Assessed for translatability of questions if used in multinational and 
multicultural studies; ultimately, the survey instrument should be translated and 
culturally adapted for all languages and cultures where it will be administered18 
  Assessed for applicability of the content (although sometimes a not applicable 
response option is also needed for a question) 
  Tested through interviews of respondents in the target population to make sure 
they interpret the survey instrument instructions, questions, and responses as 
intended and can respond accordingly (including that question stems and 
response options are appropriate and meaningful); alternatively, tested through 
the survey instrument (e.g., inclusion of survey questions to measure 
comprehension of attributes, questions, and concepts in a format that can be 
analyzed so that comprehension is part of the main analysis) 
  Formatted in a simple manner to maximize the ease of use for respondents and 
interviewers 
  Assessed for potential response bias (e.g., the tendency of respondents to answer 
questions in a manner they perceive may be viewed favorably by others) 
  Tested to ensure electronic or web-based technology is functional and/or compatible to 
administer the survey instrument, if applicable 
  Scripted to ensure standardization, if administered by an interviewer 
 
FDA discourages the use of: 
 
  Incomplete questions (e.g., Age? Reason last saw doctor?) 
  Leading questions 
 
17 Questions for survey instruments can be generated from multiple sources, e.g., standard textbooks (e.g., Streiner et 
al., 2015). 
18 Additional considerations regarding translation and culturable adaptation of survey instruments can be found in 
Survey Research Center, 2016; Wild et al., 2005. 
10  Contains Nonbinding Recommendations 
  Unclear or confusing questions (i.e.,  poorly worded questions) 
  Double-barreled or multi-barreled questions (i.e., a question that asks about two or 
more concepts at once) 
  Double negatives (i.e., a sentence that includes two negatives) 
 
When developing questions, consider the respondent effort to complete the survey instrument 
because this effort can contribute to respondent burden, which can result in lower-quality data 
and nonresponse. Minimize respondent burden by including a simple survey design and layout 
(including simple questions), nonrepetitive questions, pretesting of questions, and time testing 
(estimate time to complete survey). 
 
EXAMPLES 
Double-Barreled Question: 
How embarrassed and self-conscious have you been because of your condition? 
The question above is asking about two different concepts in a single question (i.e., presents 
two questions in a single question). 
Combining these concepts into one question makes it unclear what is being measured. Once 
respondents answer the question, it likely will be impossible to know which concept the 
respondents were thinking about when they answered the question (unless it was an 
interviewer-administered question and further probing was done). 
One way to address this issue is to split the question into two or more parts. For example, 
1.  How embarrassed have you been because of your condition? 
2.  How self-conscious have you been because of your condition? 
Double-Negative Question: 
Do you agree or disagree with the following statement? I do not have symptoms that are not 
of concern. 
The question above uses two negatives in one question, which may cause misinterpretation 
from respondents. 
One approach to address this issue is to avoid using any negative statements to the extent 
possible. For example: 
Do you agree or disagree with the following statement? I have symptoms that are of 
concern. 
If the double negative is unavoidable, the question should be pretested to ensure that 
respondents have a clear understanding of the question. In some cases, the negative word may 
need to be underlined to catch the participants attention. 
It is also important to note that negative questions (single- and double-negative questions) are 
difficult to use with administration methods where there is no visually accessible (i.e., visible) 
question (e.g., telephone or interactive voice response systems). Negative questions may also 
lead to challenges when translating questions. Instruments should be developed so that they 
have the potential to be implemented across all methods of data collection. 
11  Contains Nonbinding Recommendations 
There are two types of questions that can be used in survey instruments:19 
 
  Closed-ended questions (questions with fixed set of response options) 
  Open-ended questions (questions without a fixed set of response options, e.g., free text)  
 
Table 2 lists examples of closed- and open-ended questions, as well as some strengths and 
potential limitations of using different question types.  
Table 2. Some Potential Strengths and Limitations of Open- and Closed-Ended Questions 
Question Type  Examples  Strengths  Limitations 
Closed-ended  Which of the following    Respondent typically    May not provide 
questions  health conditions do you  can reliably answer the  respondent with a 
currently have?  question when response  comprehensive list of 
  Asthma  options are given  response options 
  Acne    Researcher typically    Response options may 
  High blood pressure  can reliably interpret  not be applicable to the 
  Glaucoma  answers  respondent 
  May be easy  and quick 
for respondents to 
record answers 
Open-ended  What health conditions    May obtain answers    May be challenging to 
questions  do you have?  that were unplanned  analyze 
  Provides opportunity 
for respondents to 
answer questions in 
their own words 
 
Table 3 examines some different types of response options; note that this is not an exhaustive 
list. 
Table 3. Response Options 
Response Option  Examples  Limitations 
Checklist  Please check to indicate if you ever had    Checklists may not cover all the 
any of the following conditions (please  possible responses or may not be 
select all that apply):  applicable to all respondents. 
  Diabetes  Adding an Other response 
  Kidney disease  option with an associated free text 
  Stroke  box at the end of the checklist 
  High blood pressure  may minimize this limitation. 
  Asthma   
  Heart attack 
 
19 These types of questions also can be asked in interviews and focus group discussions. 
12  Contains Nonbinding Recommendations 
Response Option  Examples  Limitations 
Dichotomous  Have you ever been diagnosed with    May force respondents to choose 
(two-response  glaucoma?  between a narrow set of response 
options)    Yes  options, resulting in a response 
  No  that does not completely capture 
I have been diagnosed with glaucoma.  t heir experiences/feelings 
  True 
  False 
Rankings  Rank the importance of the following    Ranking scales can be a difficult 
characteristics of a treatment for lung  task for respondents, particularly 
cancer (fill in your rank order in the spaces  if there are several response 
provided using the numbers 1 through 5,  options (e.g., >5) and/or if 
with 1 indicating most important and 5  respondents have poor numeracy 
indicating least important).  skills 
  Ranking scales do not capture 
  Treatment relieves symptoms 
why something is important or 
  Treatment has few side effects 
unimportant to respondents 
  Treatment will increase survival 
  Ranking scales address questions 
  Treatment can be taken as a pill 
in relation to each other rather 
  Treatment can be taken monthly 
than individually, which can be 
difficult for respondents 
  It may not be possible to measure 
how much distance there is 
between levels of importance for 
each rating. 
 
13  Contains Nonbinding Recommendations 
Response Option  Examples  Limitations 
Rating scales  Numeric    Decreased ability to accurately 
Rate your pain at its worst in the last 24  use rating scales at extremes of 
hours.  age. 
  0 (no pain)    Although distances between 
  1  response categories appear 
  2  equidistant, the observed 
  3  distances or change may vary. 
  4  For example: 
  5  o  The difference between 
  6  rating pain 2 versus 3 
  7  may not be the same 
difference in pain when 
  8 
comparing 8 versus 9 
  9 
ratings on a numeric rating 
  10 (worst imaginable pain) 
scale 
Verbal    The difference between 
o
Rate your pain at its worst in the last  rating pain mild versus 
24 hours.  moderate may not be the 
  None  same difference in pain 
  Mild  when comparing 
  Moderate  moderate versus 
  Severe  severe ratings on a 
verbal rating scale 
How often have you had pain during the 
past week? 
  Not at all 
  A little 
  Quite a bit 
  All the time 
Visual analog  How severe has your abdominal pain been    Respondents may be unable to 
scale (VAS)  today? (place a mark (I) on the line below)  estimate distances on the VAS 
line precisely 
  Cannot be administered 
verbally 
  Higher rates of missing data 
(Dworkin et al., 2005; Hawker 
et al., 2011) 
  Inconsistencies with the length 
of the VAS line 
  Marks may not be clear 
The ordering of questions in a survey instrument also may be important. Priming occurs when 
information presented in an earlier question causes respondents to adjust their responses to 
subsequent questions. 
Ways to avoid priming include the ordering and spacing of questions. The order of questions 
generally should flow from general to more specific to avoid order bias.20  Appropriate spacing 
 
20 For additional details on order bias, refer to Browne and Keeley, 1998. 
14  Contains Nonbinding Recommendations 
of questions (e.g., separate topics on different pages or electronic screens) can also minimize 
response bias. 
 
The use of screening questions can result in priming; however, in some instances, a screening 
question may be useful to ensure that the survey instrument is appropriate and relevant to the 
respondent. 
EXAMPLE 
Scenario: A survey instrument has been designed to assess the burden of using a colostomy bag (stoma 
bag). 
A screening question would be useful to avoid including survey respondents in which this subject 
matter may not be of relevance. 
Screening question: Do you currently use a stoma bag? Yes/No 
V.  MIXED METHODS 
Studies using mixed methods research can use a sequential or concurrent design to explore or 
confirm a concept of interest.21 
 
EXAMPLES 
Example 1 
Mixed methods study based on a qualitatively driven sequential design 
A qualitative study uses focus groups to gain broad insight into patients living with diabetes. Following 
the focus groups, one-on-one interviews are conducted in a different set of individuals to obtain a 
deeper level of understanding of the experiences and feelings of patients with diabetes. The information 
from this qualitative research (i.e., focus groups and one-on-one interviews) is subsequently used to 
develop a diabetes-specific survey instrument to better understand the prevalence of the generated 
concepts in a larger patient population. 
Example 2 
Mixed methods study based on a quantitatively driven sequential design 
A study explores depression and anxiety in patients with acute coronary syndrome (ACS) by 
administering a survey instrument to patients with ACS (quantitative component). Analyzing the 
survey data, researchers find an association between depression and anxiety for female patients with 
ACS. In the second phase of the study, the researchers conduct follow-up qualitative interviews with a 
sample of the patients, targeting depressed male and female patients with ACS, to explore why the 
relationship appeared to be present only for female patients (qualitative component). 
Example 3 
Mixed methods study based on a concurrent design 
A study examines the patient experience with living with amyotrophic lateral sclerosis through one-on-
one interviews with patients and caregivers (qualitative component). Within the study, there is 
concurrent collection of symptom checklist data (quantitative component). The data from both study 
components are analyzed separately before being compared. 
 
21 For additional details on how to design and operationalize mixed method measurement studies, refer to Johnson 
and Christensen, 2014; Johnson and Christensen, 2017; and Teddlie and Tashakkori, 2009. 
15  Contains Nonbinding Recommendations 
FDA encourages researchers to consider the goals and objectives of using a mixed methods 
research approach and how the results from both qualitative and quantitative research 
components are intended to be used together.22 
 
Reasons to use a mixed methods research design may include: 
 
  Harmonizing and confirming results from different methods (triangulation) 
  Supplementing and clarifying results from one method with results from another 
method (complementarity) 
  Using results from one method to inform the design of another method 
  Discovering inconsistencies, contradictions, and new perspectives, and reframing of 
questions or results from one method with questions or results from the other method 
(initiation) 
  Expanding the scope (range) of a research question by using different methods for 
different components of the research question (expansion) 
 
Questions researchers should ask to determine which mixed methods research approach to use to 
address their research question(s): 
 
  Will qualitative or quantitative methods be more predominant in the study, or will both 
be given equal status (i.e., equal weight or equal priority) in the study? 
  Should qualitative and quantitative components be carried out concurrently or 
sequentially? 
 
With a mixed methods research approach, there is a possibility that the findings from the 
qualitative and quantitative components may appear to conflict with each other.23 FDA 
encourages researchers to consider approaches that increase the understanding and interpretation 
of potentially conflicting findings. The selected approach should be guided by both the context 
and the research question(s). 
VI.  MANAGING BARRIERS TO SELF-REPORT  
Typical techniques and/or methods to obtain self-report data may not be appropriate for all target 
populations and may need to be tailored to the population of interest. Consider the following 
approaches to obtain self-report data across broad populations: 
 
  Participants with different capabilities (e.g., physical, sensory, intellectual, and/or 
communication)  
  Ensure usability of materials by participants with varying abilities or aids 
(e.g., low vision, tremor, using mobility aides). 
  Use written materials with large or adjustable font and that are screen 
reader accessible for respondents who are visually impaired. 
 
22 Johnson and Christensen, 2014. 
23 For additional information on how to address conflicting findings from mixed methods research, see Slonim-Nevo 
and Nevo, 2009. 
16  Contains Nonbinding Recommendations 
  Offer materials that can be read with a screen reader, or in an interview 
format, for those who cannot read. 
  Use instruction modification, infrared eye-trackers, and computerized 
tasks for people who are minimally verbal or nonverbal. 
 
  Evaluation of the capabilities of people with intellectual disabilities through 
appropriate procedures or measurements to ensure they can validly provide their 
direct insights 
  Consider certain populations may have a limited attention span (e.g., young 
children).  
  Plan to conduct interviews in shorter segments. 
 
  Pediatric populations 
  Ask young children to participate in drawing activities to help elicit concepts. 
  Use props.24 
 
  Populations with caregivers 
  For respondents who have a caregiver, the caregiver generally should not be 
present during a patient interview (e.g., they may be asked to sit outside the 
room). In cases where it is important for the caregiver to be present with the 
patient (e.g., for patient comfort), the caregiver could sit behind the 
respondent to minimize influencing the interview (either verbally or 
nonverbally).25 In either case, the research protocol should include a clear 
plan to document the presence and/or assistance of a caregiver and how data 
from the patient and caregiver will be collected and reported in these 
instances, so that the source of the information is clear for analysis and 
reporting. 
 
For patients who are unable to self-report, eliciting what behaviors caregivers observe in the 
patients (including things the patients tell them) can help to avoid proxy reporting (i.e., reporting 
from the caregiver as if they were the patient).26 Proxy reporting can lead to inappropriate 
inferences and may not be reflective of what a patient may be truly thinking or feeling. 
 
Sometimes the barrier to self-report is language, dialect, or culture. Consider the following: 
 
  Questions used in multicultural qualitative and quantitative studies should be culturally 
sensitive and use language that is adapted to the culture of interest to ensure appropriate 
responses.  
  It is important to understand and consider how cultural differences may affect 
participants responses.  
 
24 For additional information on concept elicitation in children, see Matza et al., 2013. 
25 For additional information on considerations for including caregivers in qualitative interviews, see Matza et al., 
2013. 
26 For more information on factors to consider when deciding the feasibility of self-report, see FDA guidance for 
industry Patient-Focused Drug Development: Collecting Comprehensive and Representative Input (June 2018). 
17  Contains Nonbinding Recommendations 
  Poorly translated survey instruments can prevent researchers from collecting data 
comparable to that of surveys in the source (original) language.27 Ideally, translatability 
assessment should be performed early during development of a survey instrument to 
address the needs of different nationalities, regions, and cultures.28 
  In both qualitative and quantitative studies, translation and cultural adaptation 
procedures for multinational, multiregional, and/or multicultural survey studies should 
be used to keep the meaning of questions similar.  
  In survey instruments, it is generally helpful to keep the format of the questions similar 
across translations, considering the differences among languages, and to retain the 
properties of the instrument, such as range of response options and scoring. 
VII.  CONSIDERATIONS FOR USE OF SOCIAL MEDIA 
Social media may be an approach to collect qualitative and/or quantitative data: 
 
  Data can be collected qualitatively through passive observation of social media 
discussions or information; observations can occur retrospectively or prospectively. 
  Data can be collected prospectively by administering a survey instrument in a social 
media setting. Best practices for designing and implementing studies using survey 
instruments and technology also are applicable to the use of social media to conduct a 
survey.  
 
Consider the following when using social media data: 
 
  Choose an appropriate research design.  
  Mixed methods sequential research designs can further strengthen the depth of 
knowledge gained from social media data. 
  Carefully select the source(s) of the social media with the research question in mind, 
because findings across social platforms may be distinctly different (e.g., certain 
platforms may have strong advocacy/support community presence; others may 
predominantly capture industry/academic perspectives surrounding certain issues).  
  Different social media communities appeal to different segments of the 
population, and the degree of a communitys user anonymity may affect what 
users are willing to discuss. Research using social media data should examine a 
variety of social media networks and communities to obtain data that can be most 
generalized to the population of interest to the extent possible (i.e., 
generalizability).29 A discussion of the strengths and limitations of using social 
media data in qualitative, quantitative, and mixed methods research can be found 
in Guidance 1. 
  Use appropriate methods to collect and analyze data. 
 
27 Survey Research Center, 2016; Wild et al., 2005. 
28 Further information on translation and culturable adaptation of survey instruments can be found in Acquadro et 
al., 2018; Survey Research Center, 2016; Wild et al., 2005. 
29 It is important to consider ethical standards (e.g., disclosure, consent, data ownership) for the collection and 
analysis of social media data, and any applicable federal, state, and institutional rules. For a discussion on ethical 
considerations, refer to Gleibs, 2014. 
18  Contains Nonbinding Recommendations 
  Data collection methods should address potential limitations (e.g., lacking 
mechanisms to verify patient characteristics, such as identity or diagnosis) and 
how these limitations can affect data integrity and interpretation. 
  Assess data quality. 
  Verify content and sources to avoid false information (e.g., bots).  
  Protect privacy. 
  Although verified patient community data may be used, in some cases, it may be 
appropriate to allow users to remain anonymous or post under a username (e.g., 
blogs and forums), particularly when topics are of a sensitive nature.   
19  Contains Nonbinding Recommendations 
REFERENCES 
Please note that the citation of a scientific reference in this guidance does not constitute FDAs 
endorsement of approaches or methods presented in that reference for any particular study. 
Study designs are evaluated on a case by case basis under applicable legal standards. 
 
Acquadro C, Patrick DL, Eremenco S, et al., Emerging good practices for Translatability 
Assessment (TA) of Patient-Reported Outcome (PRO) measures. Journal of Patient-
Reported Outcomes, vol. 2(8), pp. 1-11, 2018. 
Bernard HR, Ryan GW, Analyzing Qualitative Data: Systematic Approaches. Los Angeles, CA: 
SAGE, 2010. 
Block ES, Erskine L, Interviewing by Telephone: Specific Considerations, Opportunities, and 
Challenges. International Journal of Qualitative Methods, vol. 4, pp. 428-445, 2012. 
Browne MN, Keeley SM, Asking the Right Questions: A guide to Critical Thinking (5th Ed.). 
Upper Saddle River, NJ: Prentice Hall, 1998.  
Cachia M, Millward L, The Telephone Medium and Semi-Structured Interviews: A 
Complementary Fit. Qualitative Research in Organizations and Management: An 
International Journal, vol. 6(3), pp. 265-277, 2011. 
Cooper CJ, Cooper SP, Del Junco DJ, et al., Web-Based Data Collection: Detailed Methods of a 
Questionnaire and Data Gathering Tool. Epidemiologic Perspectives & Innovations, vol. 
3(1), p. 1, 2006. 
Creswell JW, Qualitative Inquiry & Research Design: Choosing Among Five Approaches. 
Thousand Oaks, CA: SAGE Publications, Inc., 2013. 
Dworkin RH, Turk DC, Farrar JT, et al., Core Outcome Measures for Chronic Pain Clinical 
Trials: IMMPACT Recommendations. Pain, vol. 113, pp. 9-19, 2005. 
FDA, Guidance for industry, Patient Preference Information Voluntary Submission, Review in 
Premarket Approval Applications, Humanitarian Device Exemption Applications, and De 
Novo Requests, and Inclusion in Decision Summaries and Device Labeling, August 2016. 
FDA, Guidance for industry, Patient-Reported Outcome Measures: Use in Medical Product 
Development to Support Labeling Claims, December 2009. 
Gibbs GR, Analysing Qualitative Data. Second edition. London, England: SAGE Publications 
Ltd., 2018. 
Gleibs IH, Turning Virtual Public Spaces Into Laboratories: Thoughts on Conducting Online 
Field Studies Using Social Network Sites. Analyses of Social Issues and Public Policy, 
vol. 14, pp. 352-370, 2014. 
Greenbaum TL, Moderating Focus Groups: A Practical Guide for Group Facilitation. Thousand 
Oaks, CA: SAGE Publications, Inc., 2000. 
Guest G, Namey E, Mitchell M, Collecting Qualitative Data. London: SAGE Publications, Ltd., 
pp. 172-222, 2013. 
20  Contains Nonbinding Recommendations 
Hawker GA, Mian S, Kendzerska T, et al., Measures of Adult Pain: Visual Analog Scale for 
Pain (VAS Pain), Numeric Rating Scale for Pain (NRS Pain), McGill Pain Questionnaire 
(MPQ), Short-Form McGill Pain Questionnaire (SF-MPQ), Chronic Pain Grade Scale 
(CPGS), Short Form-36 Bodily Pain Scale (SF-36 BPS), and Measure of Intermittent and 
Constant Osteoarthritis Pain (ICOAP). Arthritis Care & Research, vol. 63, Supplement 11, 
pp. S240-252, 2011. 
Hennink M, Leavy P, Focus Group Discussions. Cary: Oxford University Press, 2014. 
Johnson, RB, Christensen, L, Educational Research. Thousand Oaks, CA: SAGE Publications, 
Inc., 2014. 
Johnson, RB, Christensen, L, Educational Research: Quantitative, Qualitative, and Mixed 
Approaches. Thousand Oaks, CA: SAGE Publications, Inc., 2017. 
Keeney S, McKenna H, Hasson F, The Delphi Technique in Nursing and Health Research. John 
Wiley & Sons, 2010. 
Matza LS, Patrick DL, Riley AW, et al., Pediatric Patient-Reported Outcome Instruments for 
Research to Support Medical Product Labeling: Report of the ISPOR PRO Good Research 
Practices for the Assessment of Children and Adolescents Task Force. Value in Health, 
vol. 16(4), pp. 461-479, 2013. 
Shapka JD, Domene JF, Khan S, et al., Online Versus In-Person Interviews with Adolescents: 
An Exploration of Data Equivalence. Computers in Human Behavior, vol. 58, pp. 361-367, 
2016. 
Slonim-Nevo V, Nevo I, Conflicting Findings in Mixed Methods Research: An Illustration 
From an Israeli Study on Immigration. Journal of Mixed Methods Research, vol. 3(2), 
pp. 109-128, 2009. 
Streiner DL, Norman, GR, Cairney J, Health Measurement Scales: A Practical Guide to Their 
Development and Use (5th ed.). New York, NY: Oxford University Press, 2015. 
Survey Research Center, Guidelines for Best Practice in Cross-Cultural Surveys. Ann Arbor, 
MI: Survey Research Center, Institute for Social Research, University of Michigan at Ann 
Arbor, 2016. Retrieved July 03, 2018 (available at http://www.ccsg.isr.umich.edu/). 
Suter WN. Introduction to Educational Research: A Critical Thinking Approach. Thousand 
Oaks, CA: SAGE Publications, Inc., 2012. 
Tashakkori A, Creswell JW, Editorial: The new era of mixed methods. Journal of Mixed 
Methods Research, vol. 1(1), pp. 3-7, 2007. 
Teddlie C, Tashakkori A, Foundations of Mixed Methods Research: Integrating Quantitative 
and Qualitative Approaches in the Social and Behavioral Sciences. Thousand Oaks, CA: 
SAGE Publications, Inc., 2009. 
Tong A, Sainsbury P, Craig J, Consolidated Criteria for Reporting Qualitative Research 
(COREQ): A 32-item Checklist for Interviews and Focus Groups. International Journal for 
Quality Health Care, vol. 19(6), pp. 349-357, 2007. 
Vogl S, Telephone Versus Face-To-Face Interviews: Mode Effect on Semi-Structured 
Interviews With Children. Sociological Methodology, vol. 43(1), pp. 133-177, 2013. 
21  Contains Nonbinding Recommendations 
Wild D, Grove A, Martin M, et al., Principles of Good Practice for the Translation and Cultural 
Adaptation Process for Patient-Reported Outcomes (PRO) Measures: Report of the ISPOR 
Task Force for Translation and Cultural Adaptation. Value Health, vol. 8(2), pp. 94-104, 
2005. 
22  Contains Nonbinding Recommendations 
APPENDIX 1. Other Qualitative Methods 
In addition to one-on-one interviews and focus groups (refer to section IIIA), there are other 
qualitative methods that can be used to elicit what is important to patients, some of which are 
described in the following sections. 
  Delphi Methods 
The Delphi Panel technique is a multistage survey process with the intent to achieve consensus 
among experts, which may include patients, on a topic or issue.30 
  Observational Methods 
Observational research methods can involve observations of patients by the researcher in a 
naturalistic setting (e.g., home or school); a research facility; or virtual environment (e.g., online 
communities, social media) to generate data related to symptoms or daily life functioning. These 
methods often involve assessment of events, patient attitudes, and behaviors over a period of 
time. Observational methods might be used to help understand experiences described through 
other methods. Examples of scenarios where these methods could be useful include (but are not 
limited to): 
 
  In-person observations of children with attention deficit hyperactivity disorder in a 
classroom setting 
  Room surveillance live or through use of video recordings to capture behavior while 
sleeping 
  Room surveillance for observation of aggressive behaviors or confusion in patients with 
advanced Alzheimers disease 
  Social media listening (e.g., observing interactions among social media users in an 
online community) to understand how patients with a disease or condition describe 
their experience with treatment 
  Facilitated Discussions at Patient Meetings 
Facilitated discussions in well-organized public meetings that include patients, caregivers and 
patient representatives can generate useful public input and patient perspectives in specific 
disease/condition areas or topics. FDA has organized and led such meetings under its PFDD 
initiative. FDA also welcomes patient organizations to identify and organize patient-focused 
collaborations to generate public input on other disease/condition areas, using the process 
established by FDA-led PFDD meetings as a model.31 
  Survey Instruments with Open-Ended Questions 
See section IV. 
 
30 There are many different Delphi methods described in the literature that can generate consensus data (Keeney et 
al., 2010). 
31 See https://www.fda.gov/drugs/developmentapprovalprocess/ucm579400.htm. 
23  Contains Nonbinding Recommendations 
APPENDIX 2. Considerations for Selection of Qualitative Data Collection Methods 
Table 4. Potential Strengths and Limitations of Qualitative Data Collection Methods 
Data Collection 
Method  Strengths  Limitations 
One-on-one    Can gain in-depth and broad    Time-intensive (e.g., length of time 
interviews  information on the topic of  it takes to conduct several patient 
interest, including nuanced data  interviews)  
about an individuals experience    Participants may be uncomfortable 
and perspective   providing complete or truthful 
  Can gain an understanding of  information on sensitive topics to 
how a respondent interprets a  interviewers in person 
question that might be included    Studies can be expensive (i.e., staff 
in a questionnaire  time for conducting multiple 
  Flexible format  can tailor  individual interviews and data 
interviews to generate  analysis) 
appropriately detailed 
information based on research 
needs (e.g., through use of 
probing questions) 
  Greater scheduling flexibility 
compared with focus groups 
  Privacy and confidentiality 
some people may be reluctant to 
share certain things in a group 
setting 
  Can be conducted in-person at a 
study site or at a persons home 
or via telephone or video 
conference 
Focus groups    Can gain in-depth information on    May not be efficient in covering 
the topic of interest as a whole  maximum depth on an issue for 
  Flexible format (see above one- each individual 
on-one interviews)    Participants may become distracted 
  Efficiency  elicit feedback from  by other participants in the group 
multiple participants at one time    Participants may experience peer 
  Participants can react to and  pressure within the group 
build on each others ideas    Single individuals might dominate 
  Relatively inexpensive  the conversation, preventing 
multiple perspectives from being 
shared 
  Group setting may inhibit some 
individuals from providing sensitive 
information  
  Participants may be uncomfortable 
with speaking in a group setting 
and/or public setting 
  Less flexibility in scheduling for a 
group of people, which can lead to 
recruitment challenges 
24  Contains Nonbinding Recommendations 
Data Collection 
Method  Strengths  Limitations 
Delphi panels    May provide a method for    Lack of universal guidelines for 
reaching consensus among  process 
appropriate experts and    Definitions of expert opinion are 
stakeholders on important issues  variable 
and topics    No clear standards for acceptable 
  Anonymous32 process, when  level of consensus, which can make 
appropriate, reduces the role of  the Delphi process lengthy 
ego and interpersonal issues in    Size of expert panel should be 
reaching consensus  considered because it is difficult to 
  Information can be collected  achieve consensus among a larger 
remotely (e.g., via email or file- group 
sharing software) and at the    Implications for lack of anonymity 
convenience of the participant  in the case of modified Delphi 
panel methods 
  Potentially high respondent burden 
if using a lengthy Delphi survey 
instrument 
  Can be time-consuming and costly 
(e.g., high key opinion leader 
remuneration costs) 
Observations of    Opportunity to observe the    May be time-consuming and 
patient behavior or  participant's experience  logistically cumbersome to execute 
events (e.g.,    May be low burden for  if conducted in natural settings 
ethnography studies,  participants because the  (e.g., study environments may vary 
which may occur in  observation does not require  across locations) 
real-world settings  active participation    Patient and others privacy needs to 
and in real-time;    Potential to observe episodic  be addressed given patients will be 
social media  behavior and signs in a real- observed in their daily lives 
listening; videos)  world context    Some concepts and experiences are 
  Observations do not rely on  not observable 
patient or caregiver report    Can be expensive (e.g., equipment 
if recording behaviors, staff time 
for observing in real-time) 
  Participant behavior may be 
affected by observer presence 
  When conducted in naturalistic 
settings, variability among settings 
may affect the reliability and 
generalizability of the results 
 
32 Responses are anonymous only to group members; researchers are aware of respondent identities. Similar to the 
methods used in reporting aggregated data for interviews and focus groups, responses will be reported using a 
unique identifier assigned to each expert.  
25  Contains Nonbinding Recommendations 
Data Collection 
Method  Strengths  Limitations 
Facilitated    Gain in-depth information on the    Input is limited to patients who can 
discussions in  topic of interest   attend the meeting, which may 
organized patient    Efficiency  elicit feedback from  affect the reliability and 
conferences/meetings  multiple participants at one time  generalizability of the results 
  Can include real-time public    Although panelists speak to the 
polling exercises  moderator, participants do not 
interact with each other in the same 
way that focus group participants 
do 
  Representativeness and clinical 
confirmation of diagnosis may be 
difficult to determine 
  See the potential limitations of 
focus groups 
 
26  Contains Nonbinding Recommendations 
1  APPENDIX 3. Study Materials for Qualitative Studies 
2  Table 5 discusses considerations of special relevance to designing and implementing study materials for qualitative studies of patient 
3  experience. 
4  Table 5. Study Materials33 
Study 
Components  Considerations 
Material 
Research    Details on how the research will be    Outline clear research objectives and questions; identify potential sources of 
protocol  conducted  bias 
  Evidence to support the conduct of the study    Specify details on target population, including demographics, clinical 
(e.g., unmet need)  characteristics (e.g., phenotype, genotype, disease/condition severity), and 
  Description of all research-related activities  other pertinent characteristics (e.g., geographic representation) 
and study activities that patients will    Use screening material to ensure the target population of interest is 
undergo  recruited 
  Specify how data will be prepared for analysis (e.g., transcription, audio-
/video-recorded, internet data, metadata, archives) 
  Include information regarding projected clinical site enrollment 
characteristics (e.g., geographic location, referral/academic centers versus 
community centers) to help further characterize the study sample 
  See Guidance 1 for details regarding considerations for study sampling and 
representativeness 
  Identify the number and duration of discussion sessions you plan to 
conduct; this should be dependent on: 
  Number of objectives and research questions 
  Demographic and cultural/linguistic diversity of the target population 
(e.g., by age, sex, gender, race, educational level), including number of 
subgroups (e.g., disease/condition severity levels, phenotypes, 
informants (just patients or patients and their caregivers)) 
 
33 For the documents discussed in the table, there may also be other generally applicable regulations, guidance(s), standards and/or requirements. 
27  Contains Nonbinding Recommendations 
Study 
Components  Considerations 
Material 
Interview/    Interviewer/moderator instructions    Use terms participants can understand and avoid technical terms when 
discussion    Study instructions  possible (e.g., choose to use the term shortness of breath rather than 
guide    Warm-up questions (e.g., questions to  dyspnea) 
establish connection with participant)    Avoid asking leading questions that guide participants to respond with a 
  Core topic-related questions  preferred answer 
  Wrap-up questions    Avoid asking questions that imply you are casting judgment on a 
  Discussion conclusion  participants beliefs or choices 
  Use open-ended questions rather than closed-ended questions, where 
appropriate, to elicit spontaneous information from participants 
  Probing questions may be appropriate based on the direction of the 
conversation (e.g., if patient misinterprets the question) 
  Frame questions within the context of a participants experiences; avoid 
questions about abstract or theoretical concepts to the extent possible 
  Consider eliciting specific data by framing questions using targeted 
approaches such as: 
  Diary questions (patients asked to describe a typical day) 
  Critical incidents (patient reports worst/best experience) 
  Free listing (e.g., patients list all symptoms, impacts, treatments) 
  Ranking (e.g., patients rank importance of symptom, treatment benefit) 
Training    Detailed coverage of the research protocol    Train staff using standardized training materials (e.g., training documents,  
materials  contents  slides) 
  Consent/assent forms    Provide refresher training 
  Mock discussion session (staff can    Training on use of technology, where applicable 
evaluate flow of discussion)    The consent/assent form should contain the appropriate information (i.e., 
elements of informed consent) 
  Train investigators/staff to present elements of informed consent/assent to 
prospective study participants and document the corresponding 
consent/assent 
Glossary    Definitions of terminology    Clearly define key terminology within the text and ensure consistent 
terminology is used throughout study document(s) 
Coding    Codes (category or concept descriptions)    Outline clear instructions for categorization, including code definitions, 
dictionary    Coding structure  instructions, and considerations 
28  Contains Nonbinding Recommendations 
Study 
Components  Considerations 
Material 
  Memos (ideas or thoughts on how codes    Derive initial codes from prior knowledge (e.g., natural history, conceptual 
are derived))  model, disease model, discussion guide structure) 
  Develop a hierarchical organization of main codes and detailed codes 
Data    Data to be collected    Should be established when planning a research study (i.e., before data 
analysis    Analytic methods, including coding  collection begins) 
plan  software    Determine sample size needed for the study 
  Identification of coders/analysts (including    Specify method for achieving and documenting saturation 
credentials)    Identify and specify appropriate analytic methods for data type 
  Plans for resolving discrepancies among    Identify analyses assumptions  
coders and other quality assurance 
  Consider approach most appropriate to present data (e.g., tables, figures) 
measures (e.g., double-coding exercises) 
  Description of coding stages (e.g., initial 
coding, interim checks  including plans 
for coding dictionary refinement) 
  Plans for data visualization 
  Table/figure shells 
5 
29  Contains Nonbinding Recommendations 
6  APPENDIX 4. Analysis of Qualitative Data 
7    Data Preparation 
8  Qualitative data can be voluminous, so it is important to have a standardized method to analyze 
9  and interpret the volume of data in a practical and consistent way. Qualitative data should be 
10  prepared before analysis. Preparation can include aggregation or transcription of data from 
11  different sources, including: 
12   
13    In-person interviews or focus groups 
14    Recordings (e.g., audio, video, online) 
15    Internet (e.g., social media, chat room dialogues) 
16    Metadata (e.g., date of interview, name of interviewer, demographic details of 
17  respondent, source of field notes, initial ideas of analysis) 
18    Data Analysis 
19  Table 6 provides considerations for analyzing qualitative data. Note the steps for data analysis in 
20  qualitative studies may be iterative and are not necessarily sequential. 
21  Table 6. Steps Typically Used for Data Analysis in Qualitative Studies 
Steps  Description 
Compiling and    Arrange notes from research and other data collection in a useful and 
organizing data  standardized order (electronic storage and computer programs) 
Describing and    Break down compiled data into smaller pieces 
classifying data    Reorganize pieces into different groupings/sequences (e.g., codes) 
Interpreting data    Use the grouped/sequenced data to identify the larger meaning of the data 
  Connect concepts from the data to other evidence (e.g., relevant literature, 
expert opinion) 
  Evaluate whether no new and important concepts have appeared (i.e., 
saturation) if applicable to the research objective 
Representing and    Package data in a way that can be easily understood (e.g., text, tables, figures) 
visualizing data 
22  Transcripts should be analyzed using methods appropriate for categorization and aggregation of 
23  study results. There are different approaches to describe and classify qualitative data, some that 
24  may involve coding and some that may not. FDA generally recommends that qualitative data are 
25  coded for regulatory submissions. 
26   
27  If a coding approach is selected for analysis, considerations commonly include but are not 
28  limited to the following: 
29   
30    Select the appropriate coding approach for the data of interest 
31    Determine the appropriate level of detail for what is to be coded (e.g., line-by-line 
32  coding or select segments of text) 
33    Decide what data is relevant enough to be coded 
34    Move methodically to a slightly higher conceptual level initially when coding data 
30  Contains Nonbinding Recommendations 
35    Carefully consider the grammatical form of the coded words to best summarize the 
36  basic topic of a passage of qualitative data (e.g., actions versus processes versus nouns) 
37   
38  When you have literature, expert input, and appropriate knowledge, a preliminary coding 
39  dictionary34 can be developed. In many instances, codes are driven by the data; the coding 
40  dictionary will then evolve as new concepts are identified and emerge from the data. See the 
41  examples below. 
42   
EXAMPLES 
Example 1 
Coding line-by-line (applying codes to each line of qualitative data) 
01 INTERVIEWER 
02 How do you feel when you take your 
03 medicine? 
04 PATIENT 
Fatigue  05 I feel extremely tired after taking my medicine. I 
Time-sensitive medication  06 am not sure if it is related to the time of day that I 
Interference with daily activities  07 take it or not. Regardless, I cannot complete chores  
Limits physical functioning  08 around the house or take long walks. 
Rash  09 I also have noticed a rash along my upper arm, 
Itchy  10 which has caused a lot of itching. 
Example 2 
Using data to generate themes and codes 
01 PATIENT   
02 Because I was in extreme pain, my doctor wanted to  Pain 
03 re-evaluate some of my meds. The doctor told me I   Requesting regimen evaluation 
04 would have to stay in the hospital for monitoring. I  Hospitalization 
05 was afraid that this was would not be covered under  Medical access 
06 my insurance. I ended up calling my family to see if  Family support 
07 they could visit me. 
43   
44  If a coding approach is not selected for analysis, methods that are commonly used include but are 
45  not limited to the following: 
46   
47    Arrange notes (notes about original data) in a thematical manner 
48    Ensure your notes precisely cite the original data (or precisely locate the places in the 
49  database) 
50    Implement a procedural check (take notes and crosswalk them backwards into the 
51  original database) 
52   
53  It is important to note that regardless of analytic method, you should maintain a methodical 
54  analytical procedure to avoid nonsystematic and inconsistent judgments. 
 
34 A coding dictionary is a guide with predetermined concept categories and descriptions (related to the research 
objectives and questions) that is developed before data collection and analysis. 
31  Contains Nonbinding Recommendations 
55    Saturation 
56  When conducting iterative rounds of interviews, saturation is the point when no new relevant or 
57  important information emerges and collecting additional data will not likely add to the 
58  understanding of how patients perceive the concept of interest.  Saturation is sometimes used in 
59  qualitative research as a principle for discontinuing data collection and/or analysis.  However, 
60  the use of this principle will be dependent on the research objective. Although there are no set 
61  criteria or methodology for how saturation may be evaluated, the steps indicated in the example 
62  below is one approach. 
63   
EXAMPLE 
Concepts reported in the first 25 percent of planned interviews with patients are compared to the next 25 
percent of planned interviews after they are conducted. Both sets of interviews (50 percent of the originally 
planned number) are compared with the next 25 percent of the planned interviews, and subsequently all these 
interviews (75 percent) are compared to the next 25 percent of interviews.  If saturation has not been reached 
by the initial planned sample size, or more information is needed in certain patient groups in order to fully 
assess saturation, more interviews may be needed. The goal of the saturation process is to compare the amount 
of new information that is observed in the first interview set to the second interview set and so forth. 
Interviews are typically conducted until saturation is met, and no new important concepts are emerging from 
the last set of interviews with a diverse group of patients.   
64  Table 7 shows a saturation grid example summarizing focus group data. In this example, the 
65  researchers identified two symptoms (i.e., Symptom A and Symptom B) based on literature 
66  review and subject matter expert input. In addition, the researchers identified additional 
67  symptoms based on the transcripts (i.e., Other Emergent Symptoms). Before assuming that 
68  saturation may have been met, it is important to review focus group participant demographics to 
69  assess representativeness (i.e., do the focus group participants represent the intended target 
70  population as much as possible?). 
71  Table 7. Saturation Table Example 
Concept  Group 1  Group 2  Group 3  Group 4 
Symptom A  51  4  3  3 
Symptom B  41  3  5  3 
Symptom C  31  4  4  3 
Symptom D    21  1  2 
Symptom E    31  2   
Symptom F  41  3  5  4 
Symptom G      11   
72  N=6 patients per focus group; 24 total patients 
73  1 Highlighted cells indicate the first time a concept is mentioned, and numbers within the cells indicate the number of patients in that focus group 
74  endorsing the concept. 
32  Contains Nonbinding Recommendations 
75    Quality Control 
76  To improve quality control of data, consider the following: 
77   
78    Thoroughly read all qualitative transcripts and reread the transcripts 
79    Ensure codes are applied consistently to all data and apply quality assurance checks 
80  throughout the coding and analysis process 
81    Have multiple coders (e.g., each coder recodes qualitative data more than once to 
82  ensure reliability) 
83    Examine agreement among multiple coders to avoid inconsistent coding to the 
84  extent possible (e.g., a subset of data is coded by multiple coders using the same 
85  coding framework to examine the extent to which the data is coded in the same 
86  way) 
87    Document how data are collected and analyzed in a transparent manner (i.e., audit 
88  trail, memo to file) 
89    Analyze and summarize concepts emerging from the interviews in the order that data 
90  are collected (i.e., as interviews are conducted) and display in a table or grid 
91    Data Reporting 
92  Regarding reporting findings from qualitative studies, qualitative data should be presented in a 
93  clear manner. It is generally helpful to include participant statements, in the participants own 
94  words, to represent the qualitative data. Stakeholders should use their best judgment on how best 
95  to present and/or report the data.35 
 
35 For more information on how to report qualitative data, see Tong et al., 2007; Gibbs, 2018, EQUATOR Network 
website and database (https://www.equator-network.org/reporting-guidelines-study-design/qualitative-research/). 
33  Contains Nonbinding Recommendations 
96  APPENDIX 5. Screening and Exit Interview Studies/Survey Studies 
97  Screening/exit interviews and survey instruments may be implemented within the context of a 
98  clinical study. They can be helpful in obtaining patient feedback regarding various topics, such 
99  as the following: 
100   
101    Reported changes in symptoms or functioning (worsening/improvement/no change) 
102  experienced by patients throughout a trial; changes may be related to benefits, 
103  tolerability, and/or unintended effects 
104    Participant treatment expectations 
105    Anticipated and unanticipated symptoms and side effects 
106    Viability of proposed dosing regimen 
107    Patients experience with clinical trial participation, for example: 
108    In blinded studies, whether they thought they could tell they were on the 
109  experimental treatment (or not) and why they thought they were on that 
110  treatment 
111    Thoughts regarding study procedures and study participation 
112    Experience with methods of data collection (e.g., user experience with 
113  electronic data entry) 
114    Benefit-risk perspective(s) from the patient/caregiver 
115   
116  Screening/exit interviews and survey instruments can also be conducted with caregivers to obtain 
117  similar feedback on some of the above topics, if they are observable.  Caregivers may have 
118  unique perspectives important for medical product development programs. 
119   
120  The following are examples of potential strengths associated with conducting screening/exit 
121  interviews and survey instruments: 
122   
123    Contribute cumulative evidence on demographics, medical history, and aspects of the 
124  patient experience 
125    Inform initial development or refinement of a clinical outcome assessment in early 
126  medical product development through cognitive interviews as part of a mixed-method 
127  approach 
128    Add greater depth to data in rare diseases (or possibly other diseases or conditions with 
129  not much patient input) where stand-alone qualitative studies are less feasible 
130    Obtain participant input on meaningful outcomes and meaningful change by eliciting 
131  patient definitions of symptom improvement, stability, or worsening (this topic will be 
132  discussed further in other guidances) 
133    Provide information on trial integrity (e.g., whether blinding to treatment assignment 
134  was maintained) 
135   
136  Potential limitations of screening/exit interviews and survey instruments include: 
137   
138    Extra burden on site staff (e.g., additional operational procedures beyond the clinical 
139  study) 
140    Extra burden for patients/caregivers, on top of standard clinical study protocol 
34  Contains Nonbinding Recommendations 
141    For interviews, issues might arise regarding interview scheduling, administration time 
142  and confidentiality (e.g., certain sites/countries cannot share participant contact details 
143  with third-party vendors who might be conducting the interviews) 
144   
145  If screening/exit interviews are implemented, FDA generally recommends a trained, neutral 
146  third-party interviewer to conduct the interviews. Interviews should generally be conducted 
147  before (i.e., screening interviews) or after (i.e., exit interviews) patients complete the main 
148  portion of the clinical study to avoid any potential compromise of study integrity. 
35