Survey Methodologies 
to Assess REMS Goals 
That Relate to 
Knowledge 
Guidance for Industry 
 
 
 
DRAFT GUIDANCE
 
 
 
For questions regarding this draft document, contact (CDER) Office of Surveillance and 
Epidemiology, Division of Risk Management, Brian Gordon at 301-796-3960 or Doris Auth at 
301-796-0487, or (CBER) Office of Communications, Outreach and Development at 800-835-
4709 or 240-402-8010. 
 
 
 
 
 
 
 
 
 
 
U.S. Department of Health and Human Services 
Food and Drug Administration 
Center for Drug Evaluation and Research (CDER) 
Center for Biologics Evaluation and Research (CBER) 
 
January 2019 
Procedural
   
Survey Methodologies 
to Assess REMS Goals 
That Relate to 
Knowledge 
Guidance for Industry 
 
 
 
Additional copies are available from: 
 
Office of Communications, Division of Drug Information  
Center for Drug Evaluation and Research 
Food and Drug Administration 
10903 New Hampshire Ave., Bldg. 51, Room 2201 
Silver Spring, MD 20993-0002  
Phone: 301-796-3400; Fax: 301-847-8714; Email: druginfo@fda.hhs.gov  
http://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/default.htm 
 
and/or 
 
Office of Communication, Outreach and Development  
Center for Biologics Evaluation and Research 
 Food and Drug Administration 
10903 New Hampshire Ave., Bldg. 71, Room 3128 
Silver Spring, MD 20993-0002 
Phone: 800-835-4709 or 240-402-8010; Email: ocod@fda.hhs.gov  
http://www.fda.gov/BiologicsBloodVaccines/GuidanceComplianceRegulatoryInformation/Guidances/default.htm    
 
 
U.S. Department of Health and Human Services 
Food and Drug Administration 
Center for Drug Evaluation and Research (CDER) 
Center for Biologics Evaluation and Research (CBER)  
 
January 2019 
Procedural
  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
TABLE OF CONTENTS 
 
 
I.  INTRODUCTION............................................................................................................. 1 
II.  BACKGROUND ............................................................................................................... 1 
III.  REMS ASSESSMENT KNOWLEDGE SURVEYS: DESIGN, CONDUCT, AND 
DATA ANALYSIS ............................................................................................................ 3 
A.  Endpoints of Interest ..................................................................................................................... 4 
B.  Sampling Considerations............................................................................................................... 6 
1. Stratified Random Samples .............................................................................................................. 6 
2. Stratified Random Samples With Oversampling .............................................................................. 7 
3. Cluster Samples (With or Without Stratification) ............................................................................ 7 
C.  Sample Size Considerations .......................................................................................................... 8 
D.  Participant Recruitment .............................................................................................................. 10 
E.  Survey Conduct ............................................................................................................................ 11 
1. Eligibility Criteria .......................................................................................................................... 11 
2. Survey Administration .................................................................................................................... 11 
F.  Survey Questionnaire Development ........................................................................................... 13 
1. Questionnaire Design Considerations for Patient Surveys ........................................................... 14 
2. Questionnaire Design Considerations for Healthcare Provider Surveys ...................................... 16 
IV.  STATISTICAL CONSIDERATIONS WHEN ANALYZING AND PRESENTING 
RESULTS ........................................................................................................................ 17 
V.  SUMMARY OF IMPORTANT POINTS ..................................................................... 19 
A.  Assessment Survey Protocol ........................................................................................................ 19 
B.  Survey Results and Interpretation ............................................................................................. 21 
REFERENCES ............................................................................................................................ 24 
  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
Survey Methodologies to Assess REMS Goals That Relate to 
1 
Knowledge 
2 
Guidance for Industry1 
3 
4   
5   
6  This draft guidance, when finalized, will represent the current thinking of the Food and Drug 
7  Administration (FDA or Agency) on this topic.  It does not establish any rights for any person and is not 
8  binding on FDA or the public.  You can use an alternative approach if the approach satisfies the 
9  requirements of the applicable statutes and regulations.  To discuss an alternative approach, contact the 
10  FDA staff responsible for this guidance as listed on the title page.  
11   
12   
13   
14  I.  INTRODUCTION  
15   
16  This guidance provides recommendations to industry on conducting risk evaluation and 
17  mitigation strategies (REMS) assessment surveys, used to evaluate respondent knowledge of 
18  REMS-related information.  This guidance discusses general principles and recommendations 
19  related to conducting REMS assessment knowledge surveys, including study design, survey 
20  instrument development, survey data collection and processing, and data analysis.   
21   
22  In general, FDAs guidance documents do not establish legally enforceable responsibilities.  
23  Instead, guidances describe the Agencys current thinking on a topic and should be viewed only 
24  as recommendations, unless specific regulatory or statutory requirements are cited.  The use of 
25  the word should in Agency guidances means that something is suggested or recommended, but 
26  not required.  
27   
28  II.  BACKGROUND 
29   
30  Section 505-1 of the Federal Food, Drug, and Cosmetic Act (FD&C Act) (21 U.S.C. 355-1), as 
31  added by the Food and Drug Administration Amendments Act of 2007 (FDAAA) and later 
32  amended by the Food and Drug Administration Safety and Innovation Act (FDASIA), authorizes 
                                                 
1 This guidance has been prepared by the Center for Drug Evaluation and Research (CDER) in cooperation with the 
Center for Biologics Evaluation and Research (CBER) at the Food and Drug Administration.  
 
  1  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
33  FDA to require REMS for certain drugs2 if FDA determines that a REMS is necessary to ensure 
34  the drugs benefits outweigh its risks.3,4,5   
35   
36  REMS may include a Medication Guide, a patient package insert, and/or a communication plan.6   
37  FDA also may require certain elements to assure safe use (ETASU) as part of a REMS for a 
38  drug.7   
39   
40  Every proposed REMS for a new drug application (NDA) or biologics license application (BLA) 
41  must have a timetable for submission of REMS assessments,8 that: 
42   
43    Includes assessments submitted to the FDA by the dates that are 1) 18 months, 2) 3 years 
44  after the strategy is initially approved, and 3) in the 7th year after the strategy is so 
45  approved; and 
46    Is at a  frequency specified in the strategy, and  can be increased or reduced in frequency 
47  under certain circumstances and eliminated under certain circumstances.   
48   
49  With limited exceptions, REMS assessments are also required when submitting a supplemental 
50  application for a new indication for use, when required by the strategy, and whenever FDA 
51  determines that an assessment is needed to evaluate whether the strategy should be modified to 
52  ensure the benefits of the drug outweigh the risks or to minimize the burden on the healthcare 
53  delivery system of complying with the strategy.9  In addition to the required assessments, an 
54  applicant may voluntarily submit an assessment of an approved REMS at any time.10 
55   
56  Section 505-1(g)(3) of the FD&C Act specifies that a REMS assessment shall include, with 
57  respect to each goal in the strategy, an assessment of the extent to which the approved strategy, 
58  including the elements, is meeting the goal or whether the goal or elements should be modified.  
59  The FD&C Act does not specifically describe how an applicant should conduct this assessment.  
60   
                                                 
2 For the purposes of this document, drug refers to a prescription drug or biological product for which there is a 
pending or approved application, including a new drug application (NDA), abbreviated new drug application 
(ANDA), or a biologics license application (BLA). 
 
3 Public Law 110-85, September 27, 2007, available at https://www.gpo.gov/fdsys/pkg/PLAW-
110publ85/html/PLAW-110publ85.htm. 
4 Public Law 112-144, July 9, 2012, available at http://www.gpo.gov/fdsys/pkg/PLAW-112publ144/pdf/PLAW-
112publ144.pdf. 
5 See Section 505-1(a) of the FD&C Act. 
6 Sections 505-1(e)(2)-(3) of the FD&C Act. 
7 See Section 505-1(f)(1) of the FD&C Act.   
8 See Section 505-1(c)-(d) of the FD&C Act. 
9 See Section 505-1(g)(2) of the FD&C Act. 
10 See Section 505-1(g)(1) of the FD&C Act. 
 
  2  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
61  FDA reviews REMS Assessment Reports11 to evaluate whether the REMS is meeting its goals. 
62  The Assessment Report is the document applicants submit that contains information generated 
63  from the analysis of the metrics outlined in the REMS Assessment Plan, which specifies how the 
64  applicant intends to assess the performance of the REMS program in meeting its risk mitigation 
65  goals and objectives12.   
66   
67  Many REMS include a goal related to knowledge,13 such as to inform or educate patients and 
68  healthcare providers (i.e., those who prescribe, dispense, or administer drugs) about the serious 
69  risks associated with and safe use of a drug.  When such knowledge goals are part of a REMS, 
70  the REMS assessment plan generally includes, as appropriate, a survey to evaluate patients and 
71  healthcare providers understanding14, 15 of the serious risks associated with, and safe use of, the 
72  drug.   
73   
74  The majority of applicants use surveys to evaluate patients and healthcare providers 
75  understanding of the serious risks associated with, and safe use of, their drugs to assess REMS 
76  knowledge goals.  Though surveys are not the only means to assess these types of goals, this 
77  guidance describes best practices for the design, conduct, and data analysis of the results of 
78  REMS assessment knowledge surveys.  It incorporates input obtained from the June 7, 2012, 
79  public workshop entitled REMS Assessments:  Social Science Methodologies to Assess Goals 
80  Related to Knowledge,16 and the comments regarding the public workshop and accompanying 
81  FDA issue paper, submitted to the docket opened in association with the workshop.   
82   
83  III.  REMS ASSESSMENT KNOWLEDGE SURVEYS: DESIGN, CONDUCT, AND 
84  DATA ANALYSIS  
85   
86  When designing and conducting a REMS assessment knowledge survey, it is important to: 
87   
88    state the objective of the survey and how it relates to assessing the REMS goal(s) 
89   
                                                 
11 For purposes of this guidance, the REMS Assessment Report is the document applicants submit that contains 
information generated from the analysis of the metrics outlined in the REMS Assessment Plan. 
 
12 For more information on the assessment of REMS see the draft guidance for industry REMS Assessment: 
Planning and Reporting, available at  
https://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/default.htm 
 
13 For purposes of this guidance, knowledge is defined as the fact or condition of knowing something with 
familiarity gained through experience or association; the fact or condition of being aware of something; the range of 
ones information or understanding. 
14 For purposes of this guidance, understanding means to know and comprehend the nature or meaning of; the power 
of comprehending; the power to make experience intelligible by applying concepts and categories. 
15 Some applicants have also included survey questions to assess the behavior changes and burden associated with 
the REMS program requirements in their REMS assessment knowledge surveys. 
16 The transcript from the June 7, 2012, public workshop is available at 
https://web.archive.org/web/20170211122735/http://www.fda.gov/downloads/Drugs/NewsEvents/UCM310935.pdf  
  3  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
90    identify the target population for the survey (e.g., patients, parents/caregivers, prescribers, 
91  pharmacists, other healthcare providers) 
92   
93    identify the REMS key messages based on the REMS goals and objectives and other 
94  relevant REMS materials.  These messages should be grouped into appropriate REMS 
95  key message domains for assessment, each of which might cover several related REMS 
96  key messages and can be assessed with multiple questions or items in the survey. For 
97  example, the REMS key messages for Drug X are (a) risk A, (b) risk B, and (c) the need 
98  for patient counseling.  The proposed protocol might group these three REMS key 
99  messages into two domains, Risk A and Risk B, with the relevant patient counseling 
100  messages for risk A and risk B grouped into the domains of Risk A and Risk B, 
101  respectively. 
102   
103    specify and provide a rationale for the survey design used to meet the survey objectives 
104  (e.g., participant selection and recruitment methods, survey administration modalities), 
105  and for the statistical analysis plan (including any stratification by important factors, 
106  sample size calculations, and response rate goals [i.e., the proportion of invited subjects 
107  who actually completed the survey]). 
108   
109    construct a survey instrument and test for reliability and validity with regard to survey 
110  purposes 
111   
112    prespecify which questions and items will be used to assess each REMS key message 
113  domain 
114   
115    minimize factors that might contribute to a biased survey (e.g., unrepresentative 
116  sampling, faulty recruitment strategies, leading questions that bias the responses in a 
117  particular direction, missing data) 
118   
119    develop strategies to minimize burden on respondents and maximize participation 
120   
121  A.  Endpoints of Interest 
122   
123  The endpoints of interest are measurements of knowledge of the REMS key messages in a target 
124  population of interest (e.g., patients, prescribers, dispensers, or others, as appropriate).  At the 
125  subject level, the knowledge endpoint is a binary outcome indicating whether the subject knows 
126  or does not know the key message.  In a target population, the endpoint is the knowledge rate of 
127  a REMS key message.  The knowledge rate is the proportion of subjects who know the key 
128  message out of all subjects; it is also the chance that a given subject in the target population 
129  knows the key message. 
130   
131  Assessing the knowledge of everyone in the target population through a census is not feasible in 
132  most cases. Therefore, we recommend in section III.B using a probability random sample 
133  selected from an appropriate sampling frame to estimate the target population knowledge rate. 
134  To assess a subjects knowledge of REMS key messages, we recommend in section III.F using a 
135  well-designed survey instrument with at least one item for each REMS key message. When 
  4  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
136  multiple questions or items are within a REMS key message domain, then a subjects knowledge 
137  of that REMS key message is a composite score based on the answers to those items. The study 
138  protocol should clearly specify how the endpoint for each REMS key message domain is 
139  calculated from the survey items.   
140   
141  It is important to identify the endpoints pertaining to the REMS key messages for the target 
142  population when developing the survey methodology and instruments.  Consider, for example, a 
143  drug that has a REMS which includes mandatory prescriber training and a goal of informing 
144  prescribers about certain serious risks and safe use associated with the drug. The three REMS 
145  key messages conveyed in the prescriber educational materials are (1) the serious risk, (2) the 
146  dosing and administration considerations related to mitigating that risk, and (3) patient 
147  monitoring requirements.  Thus, the endpoints of interest would be the following knowledge 
148  rates: 
149   
150  1.  the proportion of prescribers who know about the serious risk  
151   
152  2.  the proportion of prescribers who know the dosing and administration considerations 
153   
154  3.   the proportion of prescribers who know the patient monitoring requirements 
155    
156  4.  the proportions of prescribers who know only one, two, or all of the REMS key messages 
157   
158  Analyzing answers to questions about the three REMS key messages from a probability sample 
159  of prescribers provides estimates and confidence intervals of these endpoints.  
160   
161  A prescriber survey instrument could include questions to assess other endpoints of interest (e.g., 
162  the prescribers knowledge of appropriate patient selection criteria, appropriate counseling 
163  recommendations, and instructions to provide to patients).  A patient survey instrument could 
164  include questions to assess a patients knowledge (e.g., of the serious risks, safe use, counseling 
165  instructions, monitoring requirements, and which symptoms would necessitate contacting the 
166  prescriber).  
167   
168  Before conducting a REMS assessment knowledge survey, it is important to propose and provide 
169  justification for a performance threshold in the study protocol.  This threshold is the minimum 
170  knowledge rate that, if achieved, indicates the REMS met the goal of communicating the REMS 
171  key messages (i.e., if the surveyed samples knowledge rates of the REMS key messages meet or 
172  exceed the threshold, then communicating this REMS key message would be considered 
173  successful).  In addition, if the knowledge rates are below the prespecified threshold, it is 
174  important to propose steps to achieve the desired knowledge rates (e.g., by enhancing REMS 
175  educational materials or outreach activities).   
176   
177  The performance threshold for each survey should be determined on a case-by-case basis.   The 
178  choice of the prespecified threshold should be informed by the public health implications of a 
179  lack of knowledge of a REMS key message, which in turn depends on the indication, the severity 
180  and prevalence of the safety concerns, and the target population.  Although there is no standard 
  5  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
181  knowledge performance threshold that is generally accepted for all REMS programs, in most 
182  cases it should be 80 percent or higher for each REMS key message domain.  
183   
184  B.  Sampling Considerations 
185   
186  Since it is rarely possible to survey the entire target population of those who take a drug, or those 
187  who prescribe or dispense a drug, sampling of the target population is more feasible.  REMS 
188  assessment knowledge surveys should be based on the principle of statistical inference, which 
189  generalizes findings from a sample to a target population.17  The generalizability of the survey 
190  results to the target population is based on how representative the sample is of the target 
191  population.  
192     
193  An initial step in designing the survey is identifying a sampling frame of the target population. 
194  That is, a list of those in the target population from which the sample can be selected.  Examples 
195  of sampling frames for a REMS target population include individuals in a REMS-mandated 
196  patient registry, patient enrollment database, or prescriber or pharmacy enrollment database. 
197  When a REMS does not include mandatory databases, possible sampling frames of patients and 
198  healthcare providers could be chain pharmacies, voluntary patient registries, or other sources.  It 
199  is important to provide the rationale for choosing the selected sampling frame that takes into 
200  consideration the representativeness of the sampling frame relative to the target population. 
201  Evaluating representativeness of a survey sample of the target population includes identifying the 
202  demographic and clinical characteristics of the sample and comparing them to known 
203  characteristics in the population.  
204   
205  To sample from the selected frame, we recommend using a probability random sample.  
206  Probability random sampling assumes every member of the sampling frame has a known and 
207  non-zero chance of selection into the sample and ensures the sample is representative of the 
208  frame with respect to all characteristics, known or unknown, measured or unmeasured.  This 
209  ensures the sample is not subject to selection bias.  Probability random sampling design does not 
210  protect against nonresponse bias; however, when a bias exists because of nonresponse, 
211  probability random sampling gives the ability to quantify the magnitude of this bias to assess its 
212  impact on the main findings.18  
213   
214  A simple random sample (SRS) is a probability random sample in which each member in the 
215  sampling frame has the same probability of selection into the sample.  Stratified random samples, 
216  stratified random samples with oversampling, and cluster samples (with or without stratification) 
217  are more elaborate probability random samples, discussed in the remainder of this section. 
218  Although selecting an SRS from a frame may be easier than more elaborate sampling designs, a 
219  more elaborate design may have advantages over an SRS in ease of conduct. 
220   
221  1.  Stratified Random Samples 
222   
                                                 
17 Altman 1997, p. 490. 
18 Levy and Lemeshow 2008, p. 18-19 and p. 35-36. 
 
  6  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
223  In a stratified random sample, the population is first divided into homogeneous strata (e.g., 
224  gender, ethnicity, or socioeconomic status), then a simple random sample is taken from each 
225  stratum.19,20  This sampling design is useful when the expected variation in knowledge between 
226  strata is larger than the expected variation within each stratum.  In that case, we expect the 
227  precision of the estimated overall population knowledge rate to be higher in a stratified design 
228  than in an SRS of the same size. Therefore, it is important to account for the stratified sampling 
229  design in the calculation of the overall knowledge estimates and their standard errors.21 
230   
231  2.  Stratified Random Samples with Oversampling 
232   
233  Although the distribution of individuals with a particular characteristic (e.g., sex, race) in the 
234  SRS sample should be approximately proportional to the distribution in the target population, 
235  occasionally it may be of interest to have disproportionate samples with respect to certain 
236  characteristics to ensure adequate sample sizes upon which knowledge estimates are based.  
237  Oversampling some subgroups of interest in a stratified random sample design leads to more 
238  precise knowledge estimates in those subgroups. Examples of oversampling for subgroups of 
239  interest include: 
240    
241    [Serious Risk] is associated with Drug X.  People taking Drug X who live in a certain 
242  part of the country are more susceptible to [Serious Risk].  All patients who take Drug X 
243  are eligible for the survey assessment.  Oversampling would be for the patients who live 
244  in that certain part of the country because that population is more susceptible to the risk. 
245   
246    [Serious Risk] is associated with Drug Y.  [Serious Risk] is more prevalent in people who 
247  are of a certain age category.  All patients who take Drug Y are eligible for the survey 
248  assessment.  Oversampling would be for the patients who are within that certain age 
249  category because that population is more susceptible to the risk. 
250   
251    [Serious Risk] is associated with Drug Z.  [Serious Risk] can happen to females of 
252  childbearing potential (females who can become pregnant).  All healthcare providers who 
253  prescribe Drug Z are eligible for the survey assessment.  Oversampling would be for the 
254  healthcare providers who treat females of childbearing potential with Drug Z, because 
255  that population of healthcare providers treats the population that is susceptible to the risk. 
256   
257  3.  Cluster Samples (With or Without Stratification) 
258   
259  A cluster sample consists of all or a random sample of subjects from a random sample of 
260  clusters.  Example of a cluster is a clinic or a hospital.  A cluster survey typically requires a 
261  larger sample of subjects from a few clusters of entities, and requires a more complex sampling 
262  design and analysis than an SRS. However, conducting a survey using a cluster design may be 
                                                 
19 Aday and Cornelius 2006, p. 131-132.  
20 Dillman et al. 2014, p. 76. 
21 Levy and Lemeshow, p. 121-142. 
 
  7  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
263  more feasible than conducting the survey with an SRS design.22  For example, a two-stage 
264  cluster survey design of patients can consist of selecting clinics at random in the first stage then 
265  selecting patients at random within sampled clinics in the second stage.  In the example, 
266  surveying more patients from a few clinics may be logistically easier than surveying a SRS 
267  sample which may have fewer patients but from many more clinics scattered across the country. 
268  However, results from cluster surveys are more complex to analyze than results from an SRS 
269  because in a cluster design, cluster size and correlations of knowledge among subjects in the 
270  same cluster should be taken into account in determining precision of estimates of knowledge.  It 
271  is important to account for the cluster sampling in calculating precision of estimates.   
272   
273  If an applicant demonstrates that using a probability random sample is not feasible, then the 
274  applicant should discuss using other types of surveys to assess knowledge with the Agency. 
275   
276  C.  Sample Size Considerations 
277   
278  In this section, sample size refers to the number of survey participants in a population of interest. 
279  The number of subjects invited to participate in a survey should be larger than the target sample 
280  size to account for nonresponse.  
281   
282  For a given target population, we recommend that the sample be large enough to precisely 
283  estimate the knowledge rate in that population.  Sample size calculation considers two important 
284  elements: sampling design (see section III.B, above) and precision of the estimated knowledge 
285  rate.  This guidance discusses three sampling designs:  SRS, cluster sample, and stratified 
286  random sample.   
287   
288  Precision is a measure of sampling uncertainty.  In this guidance, precision is the absolute 
289  difference of the sample estimate and the lower bound of the two-sided 95 percent confidence 
290  interval of this estimate.  We recommend it be controlled at 5 percentage points or less in the 
291  sample size calculation.  This 5 percentage points represents an absolute value and is not relative 
292  to the knowledge rate. 
293   
294  Other elements that may enter in the sample size calculation are the target populations size and 
295  assumed knowledge rate: 
296   
297    Target population size refers to the number of individuals in the target population.  The 
298  target population depends on the REMS goals and is usually one of three groups: all 
299  subjects who take the drug, all subjects who prescribe the drug, or all subjects who 
300  dispense the drug.   
301   
302    Assumed knowledge rate used for the sample size calculation is the best guess or 
303  expected knowledge rate in the target population based on relevant published literature, 
304  similar surveys, or pilot studies.  For a more conservative estimate of the sample size 
305  while keeping everything else the same, one might choose to use a 50 percent knowledge 
306  rate, which would yield the largest sample size estimate. 
                                                 
22 Levy and Lemeshow, p. 223-230.  
  8  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
307   
308  Determining the appropriate sample size is easier for an SRS survey than for a cluster sample or 
309  stratified random sample.  Table 1 presents the estimated sample sizes with a target precision of 
310  4 percent for different assumed knowledge rates and target population sizes.  It also shows how 
311  the appropriate sample size for an SRS survey with 4 percent target precision increases as the 
312  assumed knowledge rate approaches 50 percent, and the appropriate sample size for an SRS 
313  survey decreases from large target population sizes to small target population sizes. For example, 
314  given a target precision of 4 percent, an assumed knowledge rate of 50 percent, and a large target 
315  population size, the appropriate sample size is 601. 
316   
317  Table 1: Estimated Sample Size for a Simple Random Sample with a Target Precision of 
318  4% (Half-Width of 95% Confidence Interval) for Different Assumed Knowledge Rates and 
319  Different Target Population Sizes 
Sample Size When Assumed Knowledge Rate Is -- 
Target Population Size 
90%  80%  70%  60%  50% 
50,000  217  385  505  577  601 
5,000  208  357  459  517  536 
500  151  218  252  268  273 
320   
321  For a clustered design, a stratified random sample, or other complex designs, the sample size to 
322  achieve a certain precision differs from the sample size in a SRS by the design effect factor.23  
323  The design effect of a survey design is the ratio of the variance of estimates from that design, 
324  divided by the variance of estimates from an SRS of the same size. For example, when the 
325  design effect is 2, the sample size should be twice as large as an SRS survey to achieve the same 
326  target precision. Conversely, when the design effect is 0.5, the sample size can be half the size of 
327  an SRS survey to achieve the same target precision. 
328   
329  The appropriate sample size to achieve a certain precision is higher in a cluster design than in an 
330  SRS by the design effect factor because the variance of estimates in the cluster design is higher 
331  than in an SRS (i.e., design effect factor > 1).24  The design effect in a cluster design increases 
332  with the number of subjects in each cluster and with intra-cluster correlation coefficients (ICC)25.   
333   
334  The appropriate sample size to achieve a certain precision is generally lower in a stratified design 
335  than in an SRS by the design effect factor because the variance of estimates in a stratified design 
336  is lower than for an SRS (i.e., design effect factor  1).26  The variance in a stratified design 
337  takes into account within stratum variability and allocation proportions or weights for the strata.  
                                                 
23 Aday and Cornelius, p. 175-178. 
24 Levy and Lemeshow, p. 223-230. 
25 The formula for design effect using ICC and cluster size (M)  is 1+(ICC)*(M -1). The definition of ICC is the 
ratio of between cluster variance to the sum of between cluster variance and within cluster variance.  ICC is a 
measure of how related two measurements are within each cluster compared to between clusters.  ICC ranges from 
zero to one, with zero indicating no correlation of survey responses within a cluster and one indicating all survey 
responses within a cluster are identical. 
26 Levy and Lemeshow, p.121-142. 
  9  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
338   
339  When there are multiple REMS key messages, presenting a 95 percent confidence interval for 
340  each REMS key risk message independently can give a false sense of precision, as the overall 
341  confidence level for all REMS key messages is much less than 95 percent due to multiplicity.  
342  Therefore, we recommend having as few REMS key message domains as possible.  Any REMS 
343  key message domains should be constructed in a meaningful way.  
344    
345  D.  Participant Recruitment 
346   
347  Recruiting participants for REMS assessment knowledge surveys can be challenging.  There is 
348  no single best way to recruit participants for these surveys because of the variation in how drugs 
349  are prescribed and dispensed and in patient populations.  In general, we recommend a  
350  multimodal approach be used to recruit participants (e.g., sending survey invitations via U.S. 
351  mail, email, and phone).27   
352   
353  A description of the recruitment sources and justification for selecting the recruitment sources 
354  should be provided in the survey protocol.  Combining several sources of potential survey 
355  participants may ensure a broader cross section of the patient or healthcare provider population 
356  of interest in the sampling frame, provide a robust approach to lower nonresponse, meet the 
357  target sample size, and lower potential nonresponse bias.  Survey participants should be made 
358  aware that taking part in the survey is voluntary.    
359   
360  1.  Methods to Minimize Non-Response 
361   
362  Using multiple recruitment sources might still result in a low response rate unless additional 
363  recruitment methods are employed.  Follow-up contact to nonrespondents using a different mode 
364  than what was used to make initial contact can increase response rates during recruiting.28  Using 
365  social validation is especially useful in nonrespondent follow-up.  For example, informing 
366  nonrespondents that others have completed the survey can encourage them to respond to the 
367  survey.29   
368   
369  Finally, cluster samples and multi-stage survey sampling can make reaching participants easier 
370  and increase response rates. 
371   
372  Using multimodal approaches and other study designs can not only reduce nonresponse rates, it 
373  can also reduce nonresponse bias, which is possible even in probability surveys because 
374  knowledge of key messages of subjects who chose to respond to a survey may be different from 
375  knowledge of those subjects who chose not to respond to a survey. Thus, high nonresponse 
376  threatens the validity of a survey. By increasing response rate, the survey and estimated 
377  knowledge rate are more generalizable to the target population. 
378     
                                                 
27 Dillman et al., p. 400 - 403. 
28 Dillman et al., p. 417 - 418. 
29 Dillman et al., p. 30. 
  10  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
379  E.  Survey Conduct 
380   
381  1.  Eligibility Criteria 
382   
383  It is important to consider appropriate inclusion and exclusion criteria for the target population 
384  participating in REMS assessment knowledge surveys and provide an appropriate explanation in 
385  the protocol for the criteria selected.  The disposition of survey participants at various stages of 
386  inclusion or exclusion should be provided in the REMS assessment report.  
387   
388  In general, eligible patient participants should: 
389   
390    be age 18 or older (or have a parent/caregiver respond for a patient who takes the drug 
391  and is under 18 years of age) 
392   
393    be currently taking the drug with the approved REMS, or have taken the drug within a 
394  reasonable timeframe30   
395   
396    not be currently participating in a clinical trial involving the drug 
397   
398    not have participated in a previous REMS assessment survey for the same drug 
399   
400    Not be currently employed by the applicant, FDA, or the third party conducting the 
401  REMS knowledge assessment survey, or have other conflicts of interest that might affect 
402  their answers 
403   
404  In general, eligible prescriber and pharmacist participants should: 
405   
406    be certified and/or enrolled to prescribe or dispense the drug31 or have prescribed or 
407  dispensed the drug within a reasonable timeframe 
408   
409    not have participated in a previous REMS assessment survey for the same drug 
410   
411    not be currently employed by the applicant, FDA, or the third party conducting the 
412  REMS knowledge assessment survey, or have other conflicts of interest that might affect 
413  their answers. 
414   
415  2.  Survey Administration 
416   
417  It is important to inform potential survey participants that their participation is voluntary, they 
418  can cease their participation at any time in the course of taking the survey, and they can contact 
                                                 
30 Determining eligibility based on how recently a patient has used a REMS drug might present a trade-off between 
recall and the ability to recruit adequate numbers of patients.  You might wish to shorten this interval or even limit 
participation to current users if it will not interfere with recruitment.  In any case, the criteria should be specified. 
31 Applicable if there is a REMS requirement for prescribers or pharmacists to be enrolled or certified to be able to 
prescribe or dispense the drug. 
 
  11  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
419  the study sponsor with questions or concerns about their participation.  Potential participants 
420  should be informed that their answers will not affect their ability to prescribe, dispense, or 
421  receive the drug.  
422   
423  The purpose of the REMS knowledge assessment survey is to evaluate the target populations 
424  knowledge retention about the serious risks and safe use of the drug, not to evaluate their ability 
425  to read and comprehend the educational materials at the time of the survey. Therefore, 
426  respondents should not be offered an opportunity to read or see the Medication Guide, 
427  prescribing information, or any other REMS-related educational materials as part of the survey 
428  process. To remind the survey respondents what these REMS-related educational materials are, 
429  respondents may be provided a text description of the Medication Guide and illegible rendition 
430  of these materials for relevant survey questions. 
431   
432  Two general methods should be considered to administer the survey and collect survey data, 
433  each with its own advantages and disadvantages: self-administration (mail or Internet survey) 
434  and use of a trained interviewer (e.g., individual or group interviews, in-person or telephone 
435  interviews) to ask the questions. Survey administration modality should be decided based on the 
436  study question and target population, as well as the following additional factors:  
437   
438    potential response rate  
439   
440    noncoverage and nonresponse bias32  
441   
442    complexity of concepts and survey questions  
443   
444    length of questionnaire and time needed to complete the survey 
445   
446    special characteristics of the population of interest 
447   
448  Ideally, survey participants should be offered multiple survey administration modalities for 
449  completing REMS surveys (e.g., mail, telephone, Internet, in person).  Offering multiple survey 
450  modalities may: 
451    
452    reduce coverage error when a single mode cannot adequately cover the population of 
453  interest 
454    
455    improve response rate, with the recognition that some people prefer certain modes and 
456  may not respond to others 
457    
458    reduce nonresponse bias by obtaining responses from people who may be difficult to 
459  reach via certain modes  
460    
                                                 
32 Aday and Cornelius 2006, p. 112-113. 
  12  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
461    reduce measurement error by offering modes to ask/answer questions in non-
462  embarrassing ways or allowing adequate time for survey participants to read and answer 
463  questions at their own pace 
464   
465    improve timeliness of survey completion by reaching the target sample size quickly33 
466   
467  F.  Survey Questionnaire Development 
468   
469  The survey questionnaire should be designed to accurately capture respondents understanding of 
470  the REMS key messages and optimize the validity of the information collected.  Wording, 
471  question structure, and question sequences can significantly affect the validity and 
472  interpretability of the data collected.  Experts in questionnaire design should be involved when 
473  designing the questionnaire.  Applicants should ensure that the proposed survey instruments 
474  reliability and validity are tested and the results are acceptable.  
475   
476  The following recommendations merit particular consideration when designing the survey 
477  questionnaire:  
478   
479    Use audience-appropriate vocabulary and pretested questions. 
480   
481    Use direct, specific, and unambiguous questions. 
482   
483    Use a variety of question types, such as open-ended, close-ended, multiple-choice, and 
484  true/false. 
485   
486    Design questions to ask about knowledge, not about opinions.  Appropriate responses to 
487  true/false questions are not agree or disagree. 
488   
489    Frame questions in a positive way.   
490   
491  Examples of positively worded questions (recommended format): 
492   
493  o  You should tell your healthcare provider about [Symptom] before taking Drug X. 
494  (True/False/I dont know) 
495  o  [Serious Risk] is associated with the use of Drug X. (True/False/I dont know) 
496   
497  Examples of negatively worded questions: 
498   
499  o  You should not tell your healthcare provider about [Symptom] before taking Drug X. 
500  (True/False/I dont know) 
501  o  Treatment with Drug X does not cause [Serious Risk]. (True/False/I dont know) 
502   
503    Include questions to test whether respondents can successfully apply knowledge they 
504  learned from the REMS education materials. 
                                                 
33 Dillman et al., p. 400 - 403. 
  13  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
505   
506    Ensure questions are not biased or leading. 
507   
508    Ensure each question has an I dont know option.  When listing response categories for 
509  multiple-choice questions, I dont know provides respondents permission to admit they 
510  do not know and thus avoid guessing.  It is also important that dichotomous questions
511  for example, yes/no questions about potential drug side effectsoffer an I dont know 
512  option. 
513   
514    Advise respondents to select I dont know rather than guess an answer.   
515   
516    Develop lists of answer categories that include all reasonable possible answers.34 
517   
518    Multiple-choice responses should be mutually exclusive and independent. 
519   
520    Include foils or decoy answers (i.e., plausible incorrect answers) as possible answer 
521  options. 
522   
523    Randomize the order of multiple-choice responses for each question.35 
524   
525    Randomize correct responses between true and false and include an equal (or near 
526  equal) number of answers that are true as are false. 
527   
528    Include an instruction to select all that apply, when appropriate. 
529    
530    Do not include any information that educates or influences a respondents ability to 
531  answer subsequent questions. 
532   
533    Do not allow respondents the opportunity or ability to skip ahead or go back to previous 
534  questions in the survey. 
535   
536    Offer education for incorrect and I dont know responses at the end of the survey, not 
537  after each question. 
538   
539    Specify answer key for each question in the survey.  
540   
541  1.  Questionnaire Design Considerations for Patient Surveys 
542   
543  Questionnaires for assessing patient knowledge should include questions about the specific risks 
544  or safety information conveyed in the patient-directed REMS materials (e.g., how to recognize 
                                                 
34 Dillman et al., p. 135 - 136. 
35 Dillman et al., p. 146-148. 
  14  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
545  and what to do if they experience symptoms of an adverse event).  The following list includes 
546  some recommended considerations when designing a patient questionnaire: 
547     
548    Derive the REMS risk-specific questions from information specified in the REMS goals 
549  and the REMS patient education materials, including the Medication Guide.  As 
550  applicable, this includes all the risks or adverse events conveyed (real, potential, or 
551  theoretical), signs and symptoms of the adverse events, and what to do if symptoms of an 
552  adverse event occur.   
553   
554    Use text from the patient-directed REMS materials verbatim for questions and answers. 
555   
556    Ensure questions and answers are worded at the 6th- to 8th-grade reading level. 
557   
558    When applicable, to help assess the distribution of the Medication Guide, include 
559  questions about receipt of the Medication Guide in the patient survey. 
560   
561    The knowledge questions are the highest priority in the REMS assessment knowledge 
562  survey.  When the survey includes questions other than those related to knowledge (e.g., 
563  questions about patient receipt of the Medication Guide), organize the questions so the 
564  risk-specific questions are listed first, followed by the other questions.  Demographic 
565  information should be collected last or as part of any screening questions. 
566   
567    Before questions about receipt of the Medication Guide, include text that describes a 
568  Medication Guide and, if possible and depending on how the survey is administered, a 
569  link to or a picture showing an illegible version of the Medication Guide. 
570   
571  Example:  
572   
573  Now we are going to ask you some questions about the Medication Guide you 
574  may have received with Drug X.  The Medication Guide is a paper handout that 
575  contains important information about the risks associated with use of Drug X and 
576  how to use Drug X safely.  Medication Guides always include the title Medication 
577  Guide, followed by the word Drug X and its pronunciation.  The Medication 
578  Guide usually has sections titled What is the most important information I should 
579  know about Drug X?, What is Drug X?, and Who should not take Drug X?. 
580   
581    When applicable, use the following (or similar) questions to assess receipt and use of the 
582  Medication Guide: 
583   
584  o  Who gave you the Medication Guide for Drug X? (Select all that apply.) 
585  a)  My doctor or someone in my doctors office. 
586  b)  My pharmacist or someone at the pharmacy. 
587  c)  Someone else (please explain): ________________________. 
588  d)  I dont know. 
589  e)  I did not get a Medication Guide for Drug X. (If you checked this, skip to 
590  question XXX) 
  15  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
591   
592  o  When did you receive the Medication Guide for Drug X? (Select all that apply.) 
593  a)  With my first prescription only. 
594  b)  Not with my first prescription, but with some refills. 
595  c)  Not with my first prescription, but with every refill. 
596  d)  With my first prescription and every refill. 
597  e)  I dont know. 
598   
599  o  Did someone explain the Medication Guide for Drug X to you?  
600  a)  Yes, my doctor or someone in my doctors office.  
601  b)  Yes, my pharmacist or someone at the pharmacy. 
602  c)  Yes, someone else (please explain): _____________________. 
603  d)  No. 
604  e)  I dont know. 
605   
606  2.  Questionnaire Design Considerations for Healthcare Provider Surveys 
607   
608  The healthcare provider survey should include a section with questions about the specific risks 
609  and safety information conveyed in the REMS goals and REMS educational materials.  The 
610  following list includes some recommended considerations when designing a healthcare provider 
611  questionnaire: 
612   
613    Questions should cover all aspects of healthcare provider activities required under the 
614  REMS (e.g., knowledge of risks related to the REMS; proper patient selection criteria; 
615  proper patient counseling points; proper monitoring of patients; other REMS-compliance 
616  requirements; and receipt of REMS educational materials). 
617   
618    Questions should solicit answers based on information contained in the Prescribing 
619  Information and REMS materials, not on the respondents experience or opinion. 
620   
621  Example: 
622   
623  According to the Prescribing Information, Drug X can cause which of the following 
624  serious side effects?  (Select all that apply.) 
625   
626    Use the following (or similar) questions to assess receipt and use of the educational 
627  materials, as applicable.  Include a question about each educational piece required as part 
628  of the REMS. 
629   
630  Example: 
631   
632  Before today, which of the following educational materials were you aware of, did you 
633  receive, and did you read/view with regard to Drug X? (Select all that apply.) 
634   
  16  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
Educational Material  Aware  Received  Read/Viewed 
Full Prescribing Information   Yes   Yes   Yes 
 No   No    No  
 Dont know   Dont know 
Medication Guide   Yes   Yes   Yes 
 No   No    No  
 
 Dont know   Dont know 
Dear Healthcare Provider Letter   Yes   Yes   Yes 
 No   No    No  
 
 Dont know   Dont know 
Drug X Educational Brochure   Yes   Yes   Yes 
 No   No    No  
 Dont know   Dont know 
Drug X Educational Video   Yes   Yes   Yes 
 No   No    No  
 
 Dont know   Dont know 
Something else (please specify):   Yes   Yes   Yes 
 No   No    No  
_________________________ 
 Dont know   Dont know 
635   
636    Use the following (or similar) question to assess which sources healthcare providers use 
637  to learn about the risks associated with the drug. 
638   
639  Example: 
640   
641  o  From what sources have you learned about the risks associated with the use of Drug 
642  X? (Select all that apply.) 
643  a)  Sales representative 
644  b)  Drug X REMS.com 
645  c)  Drug X.com 
646  d)  Full Prescribing Information 
647  e)  Drug X REMS Letter to Healthcare Providers Letter 
648  f)  Drug X REMS Letter to Professional Societies 
649  g)  Drug X REMS Factsheet for Healthcare Providers 
650  h)  Drug and prescribing information databases 
651  i)  Other (please specify): _____________ 
652   
653  IV.  STATISTICAL CONSIDERATIONS WHEN ANALYZING AND PRESENTING 
654  RESULTS 
655   
656  When the survey is completed, analyses and presentations of results should follow the 
657  prespecified statistical analysis plan described in the survey protocol.  The REMS assessment 
658  report should provide data management information, characteristics of study subjects, and 
659  assessment knowledge survey results. 
660   
661  The REMS assessment report should include study subjects characteristics using a range of 
662  descriptive statistics, including means, standard deviations, medians, minimum and maximum 
663  values (for continuous variables), interquartile range, and frequency distributions (for nominal or 
664  ordinal variables).  To demonstrate that the sample is representative of the target population, 
665  applicants should compare the characteristics of the sample to the known characteristics of the 
  17  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
666  sampling frame and target population and discuss the reasons for any discrepancy and its 
667  potential influence on the validity of the inference from the sample.  Applicants should also 
668  compare known information about nonresponders to that of responders and discuss the potential 
669  impact of nonresponder bias on the study results. 
670   
671  In addition, the REMS assessment report should include knowledge rate estimates and their 95 
672  percent confidence interval for each question and each REMS key message domain.  For REMS 
673  key message domains measured by multiple questions and answers, prespecified and 
674  appropriately and meaningfully constructed composite scores can be used to present the findings 
675  regarding each domain.  
676   
677  For complex survey designs, such as stratified random sample or cluster sample surveys, the 
678  knowledge rate of a key message in the overall population should be estimated from a weighted 
679  average of knowledge rates in different clusters or strata, where the weights depend on the 
680  sampling design and any nonresponse adjustments that may be important.  Both the knowledge 
681  rate estimates and the associated precision estimates should reflect the probability sampling 
682  design used for the study.  Because the appropriate analysis methods for the point estimates and 
683  their precision depend on the design, they should be prespecified.  Post hoc stratification and 
684  weighting cannot salvage a poorly designed survey or poorly collected data.    
685   
686  The REMS assessment report should also include analyses of data from all survey respondents 
687  during the reporting period.  When the number of completed survey questionnaires received is 
688  higher than expected, all data from completed surveys should be included in the analysis and 
689  report; the report should not sample or select from the survey respondents.  
690   
691  When a returned or completed questionnaire contains unanswered items, unanswered questions 
692  should be considered as missing, not wrong.  Results reporting should follow methods to handle 
693  missing values prespecified in the statistical analysis plan.  
694   
695  For each survey question, the results presented should include at least the total number of 
696  respondents that selected each answer choice, the number that did not select any answer, and the 
697  corresponding percentages of total respondents.   
698   
699  Example: 
700   
701  True or False:  According to the Prescribing Information, [Serious Risk] is associated 
702  with the use of Drug X?   
703   
704    Out of the 500 respondents who were asked this question, 450 responded  
705  o  200 (40% of 500) selected True 
706  o  125 (25% of 500) selected False 
707  o  125 (25% of 500) selected I dont know 
708  o  50 (10%) did not select any answer 
709   
710  Finally, the REMS assessment report should include subgroup analyses when important 
711  subgroups are prespecified in the statistical analysis plan.  Examples of possible subgroups of 
  18  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
712  interest are those patients who read or did not read the Medication Guide, those patients who 
713  have most recently been prescribed a drug, or those who have been taking it for some time.  
714  Additional examples are subgroups based on socioeconomic or ethnic background or high-risk 
715  medical conditions.  FDA supports judicious reporting of subgroup analyses.  However, 
716  conclusions based on the whole survey sample are the primary interest in a REMS knowledge 
717  assessment survey.  Basing conclusions for the target population on a subset of the data is 
718  considered an inappropriate practice.36  Presentation of a partial analysis as the main finding can 
719  distort the conclusion from the survey.  
720   
721  V.  SUMMARY OF IMPORTANT POINTS  
722   
723  To assist in developing survey protocol and presenting REMS assessment knowledge survey 
724  methodology and results, the following detailed information should be included in the 
725  submissions of proposed survey protocol37 and REMS assessment report, as applicable:  
726   
727  A.  Assessment Survey Protocol  
728   
729    background information on the regulatory history of the REMS, epidemiological data of 
730  the risks that the REMS is intended to address, and epidemiological data on the target 
731  populations for REMS assessment knowledge surveys (e.g., the number of patients and 
732  prescribers) 
733   
734    inclusion and exclusion criteria for potential survey participants and reasons for the 
735  selected criteria 
736   
737    main characteristics of each target population group, including a description of the 
738  population for each of the following target groups, as appropriate: 
739   
740  o  those who are prescribed the drug (as much as possible in terms of age, race, 
741  ethnicity, education, gender, geography, indication for use) 
742  o  those who prescribe the drug (as much as possible in terms of geography, 
743  indication for which they prescribe the drug, medical specialty, gender) 
744  o  those who dispense the drug 
745   
746    rationale for choosing a particular sampling frame and a description of the sampling 
747  frame and its relationship to the target population 
748   
749    for each REMS key message domain, specify which questions are used to evaluate it, 
750  how each question is scored, and how the endpoint for each key message is derived from 
751  these questions 
752   
                                                 
36 Altman, p. 486. 
37 See the draft guidance for industry REMS Assessment: Planning and Reporting, available at [insert website], in 
which procedures for REMS assessment submission are outlined 
 
  19  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
753    prespecified thresholds for success and their justification for each REMS key message 
754  domain 
755   
756    rationale for choosing a particular survey design, such as cluster or stratified random 
757  sample 
758   
759    description of impact of design on overall population knowledge rate estimate and 
760  precision 
761   
762    derivation of the appropriate sample size of responders based on the sampling frame, the 
763  survey design, the anticipated non-response rate, and the target precision of the 
764  knowledge rate 
765   
766    derivation of the number of people to be contacted to achieve the targeted number of 
767  participants 
768   
769    strategy used to select sources of potential survey participants 
770   
771    recruitment plan, including the modes that will be used 
772   
773    plan to provide any incentive to participants 
774   
775    recruitment materials 
776   
777    rationale for the planned survey administration modalities 
778   
779    strategies to minimize nonresponse 
780   
781    description of the steps used to develop the survey instrument  
782   
783    methods used to pretest the survey questionnaire and its findings 
784   
785    survey instruments, including any introductory text and screening questions 
786   
787    techniques used to train surveyors (if applicable). 
788   
789    statistical analysis plan 
790   
791    methods used to control for potential limitations or bias 
792   
793    methods to handle missing values in responses to the items of the questionnaire 
794   
795    system for coding, entering, and verifying study data 
796   
  20  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
797    planned procedures for coding, categorizing, and analyzing verbatim responses to open-
798  ended questions 
799   
800    answer key for all close-ended questions 
801   
802    process for offering education for incorrect responses 
803   
804  B.  Survey Results and Interpretation 
805   
806    methodology used, including a description of any deviations from the proposed protocol 
807   
808    nature of the recruitment effort and the response rate, defined as the proportion of invited 
809  subjects who took the survey 
810   
811    relevant demographic characteristics of respondents compared to nonrespondents based 
812  on available information 
813   
814    relevant demographic characteristics of survey respondents and whether they completed 
815  the survey 
816   
817    reasons why respondents failed to complete the survey, when known 
818   
819    representativeness of the survey sample of the target population 
820   
821    the knowledge rates, with 95 percent confidence intervals, for each domain of REMS key 
822  messages and each question related to each domain of REMS key messages 
823   
824    potential biases, the suspected magnitude and direction of these biases, and their potential 
825  impact on the interpretation of the survey findings 
826   
827    a conclusion about the extent to which the REMS goals related to knowledge are met, 
828  how that determination was made, and proposed changes to the REMS, when applicable. 
829   
  21  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
830  REFERENCES 
831   
832  Aday, Lu Ann, and Llewellyn J.  Cornelieus.  Designing and Conducting Health Surveys: A 
833  Comprehensive Guide (Third Edition), San Francisco, CA: Wiley, 2006. 
834   
835  Altman, Douglas G.  Practical Statistics for Medical Research, New York: Chapman & Hall, 
836  1997. 
837   
838  Dillman, Don A., Jolene D. Smyth, and Leah Melani Christian.  Internet, Phone, Mail, and 
839  Mixed-mode Surveys: The Tailored Design Method (Fourth Edition), Hoboken, New Jersey: 
840  John Wiley & Sons, Inc., 2014. 
841   
842  Levy, Paul S., and Stanley Lemeshow. Sampling of Populations: Methods and Applications 
843  (Fourth Edition), New York: John Wiley & Sons, Inc., 2008. 
  24