Department of Health and Human Services 
OFFICE OF 
 
INSPECTOR GENERAL 
 
 
 
 
 
 
FDA L  C D
  ACKS  OMPREHENSIVE  ATA 
 
T D W R
O  ETERMINE  HETHER  ISK 
 
 
E M
VALUATION AND  ITIGATION 
 
STRAT EGIES IMPROVE DRUG SAFETY  
 
 
Daniel R. Levinson  
Inspector General 
 
February 2013 
OEI-04-11-00510   
   
EXECUTIVE SUMMARY:  FDA LACKS COMPREHENSIVE DATA TO 
DETERMINE WHETHER REMS IMPROVE DRUG SAFETY, OEI-04-11-00510 
 
WHY WE DID THIS STUDY  
 
The Food and Drug Administration (FDA) requires manufacturers to submit structured 
plans, known as Risk Evaluation and Mitigation Strategies (REMS), for drugs associated 
with known or potential risks that may outweigh the drugs benefits.  If FDA does not 
properly monitor REMS performance, it cannot ensure that the public is provided 
maximum protection from a drugs known or potential risks.  However, FDA does not 
have the authority to require, but may request, drug manufacturers (i.e., sponsors) to 
submit specific information regarding REMS effectiveness.   
 
HOW WE DID THIS STUDY 
 
We reviewed approved REMS since program inception in 2008 through 2011 and 
conducted structured interviews with FDA officials about FDAs efforts to evaluate 
REMS components.  We also reviewed 49 sponsors REMS assessments and FDAs 
reviews of these assessments to determine the extent to which sponsors assessments 
were complete, were submitted to FDA within required timeframes, and indicated that 
REMS were meeting their goals.  We also determined whether FDA evaluated the 
elements to assure safe use (ETASU) of one drug in each year of the program, as 
required by Federal law. 
 
WHAT WE FOUND 
 
FDA approved 199 REMS between 2008 and 2011, 99 of which were still required in 
2012.  Nearly half of sponsor assessments for the 49 REMS we reviewed did not include 
all information requested in FDA assessment plans, and 10 were not submitted to FDA 
within required timeframes.  FDA determined that 7 of the 49 REMS we reviewed met 
all of their goals.  However, FDA has not identified reliable methods to assess the 
effectiveness of REMS.  Finally, FDAs assessment review times exceeded its goal of 
60 days for all but one sponsor assessment, which reduces sponsors time to make 
suggested changes before submitting subsequent assessments.   
    
WHAT WE RECOMMEND 
 
Our findings raise concerns about the overall effectiveness of the REMS program.  To 
address these concerns, we made seven recommendations regarding FDAs evaluation 
and assessment of REMS and its review of sponsors REMS assessments.  FDA 
concurred with six of our recommendations.  For the remaining recommendation, to seek 
legislative authority to make FDA assessment plans enforceable, FDA did not state 
whether it concurred or did not concur.  However, FDA agreed that this recommendation 
should be considered if another opportunity arises to pursue legislative changes to the 
statutory provisions that describe the requirements for REMS assessments.   
 
   
   
TABLE OF CONTENTS 
 
Objectives .................................................................................................... 1 
Background .................................................................................................. 1 
Methodology .............................................................................................. 10 
Findings ..................................................................................................... 14 
FDA approved 199 REMS between 2008 and 2011, 99 of which 
were still required in 2012 ............................................................. 14 
Nearly half of the 49 sponsor assessments we reviewed did not 
include all information requested in FDA assessment plans, and 
10 were not submitted to FDA within required timeframes .......... 15 
FDA determined that 7 of the 49 REMS we reviewed met all of 
their goals ....................................................................................... 16 
FDA has not identified reliable methods to evaluate the 
effectiveness of REMS .................................................................. 18 
FDAs assessment review times exceeded its 60-day goal for        
all but one sponsor assessment ...................................................... 20 
Conclusion and Recommendations ............................................................ 22 
Agency Comments and Office of Inspector General Response .... 25 
Appendixes ................................................................................................ 27 
A:  Assessment Review Memorandums Provided by FDA ........... 27 
B:  Agency Comments ................................................................... 29 
Acknowledgments ..................................................................................... 34 
 
   
   
OBJECTIVES 
1.  To describe the Risk Evaluation and Mitigation Strategies (REMS) the 
Food and Drug Administration (FDA) approved for drugs between 
program inception in 2008 and 2011. 
2.  To determine the extent to which sponsors REMS assessments 
indicate that REMS are complete, are meeting their goals, and are 
submitted to FDA within required timeframes. 
3.  To determine the extent to which FDA has evaluated the effectiveness 
of REMS. 
4.  To determine the extent to which FDA reviews sponsors REMS 
assessments within its goal of 60 days. 
BACKGROUND  
The Federal Food, Drug, and Cosmetic Act (FDCA), as amended by the 
Food and Drug Administration Amendments Act of 2007 and the Food and 
Drug Administration Safety and Innovation Act of 2012, authorizes FDA 
to require REMS for certain drugs and biological products (hereinafter 
referred to as drugs) to assure that their benefits outweigh their risks.1, 2  
REMS are structured plans to manage specific risks of drugs that are 
effective but associated with known or potential risks (e.g., death, injury) 
that, without REMS, may outweigh benefits.  When FDA requires a 
REMS, the drug manufacturer (i.e., sponsor) must develop, implement, 
and assess it.3  FDA reviews and approves each REMS. 
 
1 Biological products include vaccines, blood and blood products, gene therapies, and 
allergenic extracts.  Therapeutic biological products generally fall under the definition of 
drugs, which include substances intended for the diagnosis, cure, mitigation, treatment, or 
prevention of disease and which meet certain other requirements.  FDA, Drugs@FDA 
Glossary of Terms.  Accessed at 
http://www.fda.gov/Drugs/informationondrugs/ucm079436.htm#B on June 15, 2012.    
2 The Food and Drug Administration Amendments Act (P.L. 110-85, Sept. 27, 2007) added  
 505-1 to the FDCA.  The Food and Drug Administration Safety and Innovation Act 
(FDASIA) (P.L. 112-144, July 9, 2012) further amended FDCA  505-1. 
3 Under FDCA  505-1, the responsible person submitting a covered application to market a 
drug or the holder of such approved application is required to submit, implement, and assess 
REMS.  In this report, we refer to the responsible person as the sponsor.  Generally, a drug 
manufacturer submits and holds a drugs application and is the drugs sponsor.  FDA, Drug 
Development and Review Definitions.  Accessed at 
http://www.fda.gov/Drugs/DevelopmentApprovalProcess/HowDrugsareDevelopedandAp
proved/ApprovalApplications/InvestigationalNewDrugINDApplication/ucm176522.htm              
on April 24, 2012. 
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  1 
   
   
FDA has the authority to, among other things, deem a drug misbranded or 
impose civil monetary penalties on sponsors that violate approved 
REMS.4, 5   
Structure of REMS 
Each REMS includes one or more overall goals to achieve a particular 
safety-related health outcome or understanding by patients and/or health 
care providers of a drugs risks.6  Examples of REMS goals include 
preventing fetal exposure to a drug and informing prescribers, patients, 
and pharmacists of the serious risks and safe-use conditions for a drug. 
REMS include one or more strategies to provide additional safety in the 
usage of certain drugs.  For example, REMS may require sponsors to 
create additional safety information for patients or to ensure that health 
care providers that prescribe the drug are specially certified.7 
For brand-name drugs, each REMS must include a timetable for sponsors 
to submit REMS assessments.8, 9  The standard timetable requires sponsors 
to submit assessments of REMS effectiveness by 18 months, 3 years, and 
7 years after approval of the REMS.10  FDA may require sponsors to 
submit assessments at different intervals specified in the REMS or 
eliminate the timetable after 3 years if it determines that a drugs risks 
 
4 A drug may be deemed to be misbranded if it is subject to a REMS and the sponsor fails to 
comply with a requirement of the REMS.  FDCA  502(y).  Misbranded drugs may not be 
introduced into, delivered, or received into interstate commerce.  FDCA  301 (a) and (c).  
5 Sponsors that violate REMS requirements may be subject to civil monetary penalties of up 
to $250,000 per violation.  Civil monetary penalties are not to exceed $1 million in a single 
proceeding.  Civil monetary penalties may increase to $10 million for continued violations.  
FDCA  303(f)(4)(A).  Additionally, a sponsor may not introduce or deliver for introduction 
into interstate commerce an approved drug if a REMS is required for the drug and the sponsor 
fails to maintain compliance with the requirements of the approved REMS or with other 
requirements of  505-1 of FDCA.  FDCA  505(p).    
6 FDA, Draft Guidance for Industry:  Format and Content of Proposed REMS, REMS 
Assessments, and Proposed REMS Modifications, p. 9.  Accessed at 
http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guid
ances/UCM184128.pdf on June 23, 2011.  This draft guidance will represent FDAs current 
thinking on this topic when finalized, but will not bind FDA or the public. 
7
 FDCA  505-1(c)(2), 505-1(e)(2), and 505-1(f)(3)(A). 
8 FDCA  505-1(c)(1) and (d).  The timetable for assessments specifies when sponsors will 
submit the assessment to FDA, not when the assessment will be performed.   
9 REMS for drugs subject to abbreviated new drug applications (i.e., generics) do not require a 
timetable for assessments.  FDCA  505-1(i)(1)(AB).    
10
 FDA often requires more frequent assessments (e.g., at 6-month intervals or annually) for 
drugs associated with the most serious risks.   
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  2 
   
   
have been adequately identified and assessed and are being adequately 
managed.11   
FDA may require one or more of the following components in addition to 
the timetable for assessments:  medication guide or patient package insert, 
a communication plan, or elements to assure safe use (ETASU).12  FDA 
determines which of these additional components to require according to a 
drugs risks.   
Medication Guides and Patient Package Inserts.  Medication guides are 
paper handouts that include FDA-approved information about the safe and 
effective use of a drug.  Medication guides are not unique to REMS; FDA 
oversees their content and format under separate regulations and may 
require them without requiring a REMS.13  Some REMS may require 
patient package inserts, which also contain drug safety information, in 
addition to medication guides.14, 15 
Communication Plans.  Communication plans describe how sponsors will 
inform health care providers about a drugs risks and/or the components of 
a REMS.  These plans may describe how a sponsor will send letters to 
health care providers and disseminate information through professional 
societies about a drugs risks.16   
   
 
11 FDCA  505-1(d)(4)(A) and (C).  When FDA eliminates a REMS timetable for 
assessments, it may continue to require the REMS but the sponsor is no longer required to 
submit assessments.   
12 FDCA  505-1(e)(13) and 505-1(f)(13). 
13 If FDA requires a medication guide not as a component of a REMS, sponsors are not 
required to assess them.  See 21 CFR 208. 
14 Patient package inserts are required for some prescription drugs, including estrogens and 
oral contraceptive products and are considered to be part of a drugs labeling.  Manufacturers 
of other drugs may voluntarily provide patient package inserts.  Required patient package 
inserts must be dispensed to patients with the drug.  21 CFR 310.501, 21 CFR 310.515.  See 
also, FDA, Guidance:  Drug Safety Information  FDAs Communication to the Public, p. 10.  
Accessed at 
http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guid
ances/ucm072281.pdf on December 19, 2012. 
15 FDA does not expect to require both patient package inserts and medication guides 
frequently.  FDA may allow sponsors to include an existing patient package insert in a REMS 
instead of a medication guide or require sponsors to convert a patient package insert into a 
medication guide, if it meets medication guide requirements.  FDA, Guidance for Industry:  
Format and Content of Proposed REMS, REMS Assessments, and Proposed REMS 
Modifications, p. 10.  Accessed at 
http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guid
ances/UCM184128.pdf on June 23, 2011. 
16 FDCA  505-1(e)(3). 
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  3 
   
   
ETASUs.  FDA may require ETASUs for drugs that are associated with a 
serious adverse drug experience if, without the ETASUs, FDA would not 
approve the drug or would withdraw the drugs approval.17, 18  If drugs 
were initially approved without ETASUs, FDA must determine that other 
REMS components are not sufficient to mitigate risks before it requires 
ETASUs.19   
FDA can require one or more of the following ETASUs: 
  health care providers who prescribe the drug have certain training, 
experience, or certification;  
  pharmacies, practitioners, and/or other health care settings that 
dispense the drug are specially certified; 
  the drug is dispensed only in certain health care settings (e.g., 
hospitals, appropriately equipped physicians offices); 
  the drug is dispensed only to patients with evidence or 
documentation (e.g., laboratory test results, signed 
acknowledgement of risks) that they can safely use the drug; 
  each patient using the drug is subject to monitoring (e.g., patients 
must periodically contact the prescriber or undergo laboratory 
tests); and 
  each patient using the drug is enrolled in a registry.20 
Each ETASU must (1) correspond to the specific serious risk listed in the 
labeling of a drug; (2) considering such risk, not be unduly burdensome on 
patient access to a drug, particularly considering patients with serious or 
life-threatening conditions and patients who have difficulty accessing 
health care; and (3) to the extent practicable, minimize the burden on the 
health care delivery system.21 
 
17 A serious adverse drug experience is an adverse event associated with the use of a drug that 
results in death, the immediate risk of death, inpatient hospitalization or prolonging existing 
hospitalization, a persistent or significant incapacity or substantial disruption of the ability to 
conduct normal life functions, a congenital anomaly or birth defect, or a medical or surgical 
intervention to prevent these outcomes.  FDCA  505-1(b)(4). 
18 FDCA  505-1(f)(1)(A). 
19 FDCA  505-1(f)(1)(B). 
20 FDCA  505-1(f)(3)(AF).  
21 FDCA  505-1(f)(2)(A), (C), and (D).  Specifically, to minimize the burden on the health 
care system, ETASUs should conform with the ETASUs of drugs with similar serious risks 
and be designed to be compatible with established distribution, procurement, and dispensing 
systems for drugs.  FDCA  505-1(f)(2)(D).  Additionally, each ETASU must be posted 
publicly by the Secretary within 30 days with an explanation of how it will mitigate the safety 
risk.  FDCA  505-1(f)(2)(B).    
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  4 
   
   
When FDA requires certain ETASUs, the REMS may also include an 
implementation system.22  Sponsors may use implementation systems to 
take reasonable steps to monitor, evaluate, and improve the 
implementation of ETASUs.23  Examples of implementation systems 
include requiring sponsors to conduct periodic audits of health care 
settings that use the drug or to maintain a database of all certified entities 
to ensure that certification requirements are met.24 
REMS components are not exclusive.  For example, REMS that require 
ETASUs generally also require a communication plan and/or a medication 
guide.  FDA refers to three types of REMS according to the most intensive 
(i.e., primary) required component:  medication guide, communication 
plan, and ETASU REMS.  See Table 1 for examples of drugs that require 
the various REMS components. 
   
 
22 ETASUs that may include an implementation system are:  requiring special certification for 
pharmacies, practitioners, or other health care providers that dispense the drug; dispensing the 
drug in certain settings; and dispensing the drug only to patients with certain evidence or 
documentation.  FDCA  505-1(f)(4). 
23 FDCA  505-1(f)(4). 
24 FDA, Guidance for Industry:  Format and Content of Proposed REMS, REMS Assessments, 
and Proposed REMS Modifications, pp. 1415.  Accessed at 
http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guid
ances/UCM184128.pdf on June 23, 2011. 
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  5 
   
   
Table 1:  Examples of Drugs Requiring REMS 
Approved  Primary REMS 
Drug  Risks  Required REMS Components* 
Indication  Component 
Neuropsychiatric 
symptoms, including 
changes in behavior,  Timetable for assessments,  Medication 
Chantix  Smoking cessation 
agitation, depressed  medication guide  guide 
mood, and suicidal 
thoughts and actions  
Treatment of adult 
patients with 
Unintended exposure 
complicated skin and  Timetable for assessments, 
Vibativ  skin structure  of pregnant women  medication guide, communication  Communication 
resulting in  plan 
infections caused by  plan 
teratogenicity 
susceptible Gram-
positive bacteria 
Management of 
moderate to severe 
pain when a 
OxyContin  Potential for abuse, 
continuous, around- Timetable for assessments, 
controlled- misuse, overdose, and  ETASU 
the-clock, opioid  medication guide, ETASUs 
release tablets  addiction 
analgesic is needed 
for an extended 
period 
Timetable for assessments; 
Management of type  medication guide; communication 
Avandia  Myocardial infarction  ETASU 
II diabetes  plan; ETASUs, including an 
implementation system 
Source:  Office of Inspector General (OIG) analysis of approved REMS, 2011. 
* The required components of these REMS may change over time.  This table includes REMS components required as of  
December 31, 2011.  
Criteria for Requiring REMS 
FDA may require a REMS before approving an application to market a 
drug (i.e., preapproval REMS) or after a drug is on the market (i.e., 
postapproval REMS).25  FDA must consider the following factors to 
determine whether a preapproval REMS is necessary: 
  the estimated size of the population likely to use the drug, 
  the seriousness of the disease or condition that the drug treats, 
  the expected benefit of the drug, 
  the duration of treatment, 
   
 
25 FDCA  505-1(a). 
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  6 
   
   
  the seriousness of any known or potential adverse events related 
to the drug and the background incidence (i.e., frequency) of 
these events in the population likely to use the drug, and 
  whether the drug is a new molecular entity.26, 27  
FDA may require a postapproval REMS if it becomes aware of new safety 
information after a drug is on the market and determines that a REMS is 
necessary to ensure that the benefits of the drug outweigh the risks.28, 29  
For example, FDA required a postapproval REMS for the drug Wellbutrin 
in response to reports of suicide-related events in patients using the drug 
for smoking cessation.30   
REMS Approval Process 
When FDA requires a REMS, it notifies the drugs sponsor in a REMS 
notification letter.  The REMS notification letter instructs the sponsor to 
submit a proposed REMS containing the components FDA determines are 
necessary.  FDA reviews the proposed REMS and approves it after an 
interactive process with the sponsor.31  After FDA approves a proposed 
REMS, it notifies the drugs sponsor in a REMS approval letter.  
FDA works with sponsors to develop an FDA assessment plan for each 
REMS, which is included in the REMS approval letter.  Assessment plans 
describe the information that FDA requests sponsors to collect to complete 
a valid assessment of whether a REMS is meeting its goals.32  For 
example, FDA may request that a sponsors assessment include the 
 
26 A new molecular entity is an active ingredient that has never been marketed in the United 
States in any form.  FDA, Drugs@FDA Glossary of Terms.  Accessed at 
http://www.fda.gov/Drugs/informationondrugs/ucm079436.htm#N on September 2, 2011.   
27 FDCA  505-1(a)(1)(AF). 
28 FDCA  505-1(a)(2)(A).  New safety information is derived from scientific data (e.g., 
clinical trials, adverse event reports) about serious or unexpected risks associated with a drug 
that FDA has become aware of since the drug was approved, since the REMS was required, or 
since the last assessment of the approved REMS.  These scientific data may be based on a new 
analysis of existing information.  FDCA  505-1(b)(3).   
29 FDA also considers the factors listed in FDCA  505-1(a)(1)(AF) when requiring a 
postapproval REMS. 
30 FDA, NDA 018644/S-041, NDA 023858/S-048 Approval Letter, February 26, 2010.  
Accessed at 
http://www.accessdata.fda.gov/drugsatfda_docs/appletter/2010/018644s041,020358s048l
tr.pdf on November 1, 2011. 
31 FDA, CDER 21st Century Review Process Desk Reference Guide, p. 53.  Accessed at 
http://www.fda.gov/downloads/AboutFDA/CentersOffices/CDER/ManualofPoliciesProc
edures/UCM218757.pdf on May 3, 2012.    
32 FDA, Guidance for Industry:  Format and Content of Proposed REMS, REMS Assessments, 
and Proposed REMS Modifications, p. 18.  Accessed at 
http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guid
ances/UCM184128.pdf on June 23, 2011.   
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  7 
   
   
number of patients enrolled in a user registry or an evaluation of patients 
understanding of a drugs risks. 
Sponsors Assessments of REMS 
Sponsors of brand-name drugs must submit their assessments of REMS 
for FDAs review according to the timetable in the approved REMS.33  
FDCA requires FDA to review sponsors assessments but does not require 
FDA to perform its own assessments of each REMS.34  FDA may require 
sponsors to submit an additional assessment if one is needed to evaluate 
whether the REMS should be modified.35  Sponsors may also submit 
voluntary assessments of REMS at any time.36  Sponsors assessments 
must include information required by FDCA and should also include 
information requested in FDA assessment plans.   
Requirements for Sponsors Assessments.  At the time of our review, 
assessments of REMS with ETASUs were required to include an 
evaluation of the extent to which each ETASU is meeting the goal stated 
in the REMS in addition to complying with the approved timetable.  
Under current amendments to FDCA, all REMS assessments must 
include, with respect to each goal of the REMS, an assessment of the 
extent to which REMS are meeting the goals or of whether the goals or 
REMS should be modified.37  FDCA does not require sponsors to include 
additional information about the effectiveness of REMS in assessments.  
FDA may take enforcement actions (e.g., deem a drug misbranded, impose 
civil monetary penalties) if sponsors fail to meet the assessment 
requirements outlined in FDCA. 38  However, FDA does not have authority 
to take enforcement actions if sponsors assessments do not include 
information requested in FDA assessment plans.   
FDAs Reviews of Sponsors Assessments.  Multidisciplinary teams made 
up of staff in FDAs Division of Risk Management within the Office of 
Surveillance and Epidemiology, the respective drug review division(s) in 
the Office of New Drugs, and the Office of Compliance review sponsors 
REMS assessments.  Staff in the Division of Risk Management complete 
review memorandums that summarize the teams findings.  The teams 
 
33 FDCA  505-1(d).  Sponsors of generic drugs are not required to submit REMS 
assessments according to a timetable. 
34 FDCA  505-1(h)(1).  FDAs requirement to assess the ETASUs of at least one drug each 
year is distinct from its responsibility to review sponsors assessments.     
35 FDCA  505-1(g)(2)(C). 
36 FDCA  505-1(g)(1).   
37
 FDCA  505-1(g)(3).  Because this was not required at the time of our review, we 
determined only whether assessments of REMS with ETASUs included assessments of the 
extent to which each ETASU was meeting the stated goal in the REMS. 
38 FDCA  502(y) and 303(f)(4)(A).   
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  8 
   
   
review assessments for completeness and determine whether REMS are 
meeting their goals or whether modifications to the REMS or FDA 
assessment plans are required.  A Division of Risk Management review 
memorandum documents whether a sponsors assessment (1) is complete, 
(2) indicates the REMS is meeting its goals or needs modifications, and 
(3) identifies any deficiencies that may affect a REMSs ability to mitigate 
risks.39 
The Office of Compliance also completes review memorandums, which 
document whether sponsors submit complete assessments according to the 
approved timetable.  However, the Office of Compliance defers the 
analysis of assessment data and of whether REMS are meeting their goals 
to the Division of Risk Management.40 
FDCA requires FDA to promptly review sponsors assessments.41  
Although FDA does not define promptly, FDA officials indicated that 
their goal is to complete all assessment reviews within 60 days of 
submission if it does not discuss REMS modifications with sponsors.42  
Figure 1 illustrates the relationship of the REMS, FDA assessment plan, 
sponsors assessment, and FDA review memorandum. 
Figure 1:  Relationship of REMS Documents 
REMS  FDA Assessment  Sponsor's  FDA Review 
Plan  Assessment  Memorandum 
Enforceable  Unenforceable  Document  Document 
FDA document  FDA document  containing  containing 
outlining REMS  outlining how  sponsor's  FDA's review of 
components  sponsors  assessments of  sponsor's 
should assess  REMS  assessment 
REMS'  effectiveness 
effectiveness 
 
Source:  OIG analysis of REMS documents, 2012. 
FDAs Evaluation of REMS 
Federal law (i.e., FDCA) requires FDA to evaluate the ETASUs of at least 
one REMS each year to determine whether the ETASUs (1) assure safe 
 
39 Federal regulations and FDA policy do not define a complete assessment.  Complete is 
defined later for the purposes of this study. 
40 Based on OIG review of Office of Compliance review memorandums. 
41 FDCA  505-1(h)(1). 
42Since FDA set this goal, some FDCA provisions regarding the review of sponsors 
assessments have changed because of FDASIA.  Before FDASIA was enacted, FDA opened a 
90-day discussion period to work with sponsors to determine whether REMS modifications 
were necessary and to discuss possible modifications.  FDASIA removed the provision 
regarding the 90-day discussion period, but FDA may still discuss REMS modifications with 
sponsors and is still required to promptly review sponsors assessments. 
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  9 
   
   
use of the drug; (2) are unduly burdensome on patient access to the drug; 
and (3) to the extent practicable, minimize the burden on the health care 
delivery system.43   
In 2008, FDA issued a 5-year plan (20082012) to enhance and modernize 
its drug safety activities.  In the 5-year plan, FDA states that it will 
develop a plan to identify, develop, validate, and assess the effectiveness 
of REMS components.44   
METHODOLOGY 
Scope 
We reviewed and described the REMS that FDA approved between 
program inception in 2008 and 2011.  Additionally, we reviewed sponsors 
most recent assessments to determine whether they were complete.  We 
reviewed FDAs review memorandums for each of the sponsor 
assessments to determine whether FDA notified sponsors about 
incomplete assessments, concluded that REMS were meeting their goals, 
and completed its reviews within 60 days.  We also determined the extent 
to which FDA has evaluated REMS with ETASUs, as required by Federal 
law.     
Our review included 199 REMS.  We reviewed sponsors assessments and 
FDA review memorandums for the 49 REMS that required assessments 
according to their approved timetables as of December 31, 2011.45  We 
also reviewed the second most recent assessment (i.e., prior assessment) 
and FDA review memorandums for the 14 REMS that required multiple 
assessments during our timeframe.46  See Appendix A for a list of REMS 
and associated review memorandums provided by FDA. 
We limited our review of sponsors assessments to those required by the 
timetable in each REMS.  We did not include REMS for generic drugs in 
our review because sponsors of these drugs are not required to submit 
 
43 FDCA  505-1(f)(5)(B).  FDA has been required to complete this evaluation for one drug 
with an ETASU REMS each year since 2008. 
44 FDA, PDUFA Drug Safety 5-year Plan 20082012, December 2008, p. 8.  Accessed at 
http://www.fda.gov/downloads/ForIndustry/UserFees/PrescriptionDrugUserFee/UCM119
244.pdf on September 23, 2011.  The 5-year plan is distinct from FDAs required evaluation 
and is not specific to ETASUs. 
45 Fifty-seven REMS required at least one assessment by December 31, 2011.  We did not 
include eight REMS in this analysis because FDA did not provide review memorandums for 
them.  This does not necessarily indicate that FDA failed to review these assessments.  These 
assessments may have been under FDA review during our data collection period. 
46 Twenty REMS required multiple assessments by December 31, 2011.  We did not include 
six REMS in this analysis because FDA did not provide prior review memorandums for four 
REMS and did not provide the most recent review memorandums for two.  
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  10 
   
   
REMS assessments according to a timetable.47  We did not review 
sponsors assessments for which FDA did not provide review 
memorandums.  Additionally, we did not independently determine the 
extent to which REMS are meeting their goals and relied on FDA 
statements in review memorandums.   
Data Collection and Analysis 
We reviewed relevant statutes, policies, and guidance documents related to 
REMS.  We conducted structured interviews with and requested 
documentation from FDA officials.  Finally, we analyzed REMS, FDA 
assessment plans, sponsors assessments, and FDA review memorandums.   
Describing REMS Approved Between 2008 and 2011.  We used FDAs 
online list of REMS to calculate the number of REMS that FDA approved 
between March 25, 2008, when FDAs authority to require REMS began, 
and December 31, 2011, when we collected our data.  We described the 
components of these REMS and determined how many REMS were no 
longer required as of December 31, 2011.     
Determining Completeness and Timeliness of Sponsors Assessments.  We 
analyzed the 49 most recent sponsor assessments to determine whether 
sponsors included information requested in FDA assessment plans and 
required by Federal law.  We based our determination on an independent 
review of sponsor assessments and FDA assessment plans.  We did not 
rely on statements in FDA review memorandums to determine whether 
assessments were complete.  However, if FDA review memorandums 
indicated that FDA no longer requested sponsors to submit certain pieces 
of information, we did not consider the information to be missing.  To 
determine whether sponsors submitted their assessments according to the 
approved timetable, we compared assessment submission dates to the 
timetable in each approved REMS.  We also requested documentation of 
any enforcement actions FDA has taken in response to sponsor 
assessments that did not comply with Federal requirements. 
Determining Whether REMS Were Meeting Goals.  We analyzed FDA 
review memorandums associated with the 49 most recent sponsor 
assessments to determine whether FDA concluded that REMS were 
 
47
 Our review includes the drug isotretinoin, which was approved as an acne treatment and 
marketed as Accutane.  FDA required a single, shared-system REMS for Accutane and its 
generic equivalents but Accutanes approval was withdrawn at the request of its sponsor.  
Several of its generic equivalents remain on the market and are under the originally approved 
single, shared-system REMS.   Sponsors of these generic drugs have submitted assessments in 
accordance with the timetable included the Accutane REMS.   
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  11 
   
   
meeting their goals.48  We considered determinations of whether REMS 
were meeting their goals to be in four categories:  (1) the REMS was 
meeting its goals, (2) the REMS was not meeting its goals, (3) FDA 
reviewers could not determine whether the REMS was meeting its goals, 
and (4) FDA reviewers did not determine whether the REMS was meeting 
its goals.  
We did not independently determine whether REMS were meeting their 
goals; instead, we relied on statements from FDA reviewers in review 
memorandums.  For example, a review memorandum for a REMS that 
FDA determined was not meeting its goals states that ... we find that the 
goals are not being met.  A review memorandum for a REMS for which 
FDA could not determine was meeting its goals states that ... due to a 
lack of valid data, we cannot comment on whether the REMS is meeting 
its educational goals.  We considered REMS to be in the fourth category 
if review memorandums did not contain a statement from the reviewer 
regarding whether the REMS was meeting all of its goals.   
For the 14 REMS that required at least one prior assessment, we analyzed 
the prior review memorandums to determine whether reviewers stated that 
the assessments identified deficiencies that may affect the REMS ability 
to mitigate risks.49  If reviewers noted deficiencies in prior assessments, 
we determined whether the reviewers noted the same deficiencies in the 
most recent sponsor assessments.  We also compared FDAs 
determinations about whether REMS were meeting their goals in review 
memorandums for the prior assessments to determinations in the most 
recent review memorandums. 
Determining the Extent of FDAs Evaluation of REMS.  We conducted 
structured interviews with FDA officials regarding (1) the evaluation of 
ETASUs, as required by FDCA; and (2) FDAs plan to identify, develop, 
validate, and assess the effectiveness of REMS components, as described 
in its 5-year plan (20082012) to enhance and modernize drug safety 
activities.     
Determining FDAs Assessment Review Times.  We determined FDAs 
review times for sponsor assessments for which FDA did not discuss 
REMS modifications with sponsors.  For these 29 sponsor assessments, 
we used the submission dates on the assessments and the dates on the 
 
48 Reviewers determinations regarding whether REMS are meeting each goal are found 
in the discussion section of review memorandums.  Only Division of Risk Management 
review memorandums include this determination; however, the memorandums reflect the 
reviews of a multidisciplinary team. 
49 Reviewers note deficiencies in the discussion and recommendations sections of review 
memorandums. 
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  12 
   
   
associated FDA review memorandums to determine whether FDA 
completed reviews within its goal of 60 days.50 
Limitations 
We did not review FDA communication with sponsors regarding their 
assessment reviews.  We based reports of actions that FDA took after 
assessment reviews (e.g., discussing REMS modifications with sponsors) 
on the recommendations stated in FDAs Division of Risk Management 
review memorandums, not on FDA communication with sponsors.   
Standards 
This study was conducted in accordance with the Quality Standards for 
Inspection and Evaluation issued by the Council of the Inspectors General 
on Integrity and Efficiency. 
   
 
50 For Division of Risk Management review memorandums, we used the dates of review as 
the review completion dates.  For Office of Compliance review memorandums, with one 
exception, we used the electronic signature dates for the review completion dates. 
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  13 
   
   
FINDINGS 
FDA approved 199 REMS between 2008 and 2011,      
99 of which were still required in 2012  
FDA requires REMS to manage specific risks of drugs that are effective 
but associated with known or potential risks (e.g., death, injury) that, 
without REMS, may outweigh the drugs benefits.  Of the 199 approved 
REMS, 119 required medication guides only and 48 required 
communication plans.  For drugs with the most serious risks, FDA 
approved 32 REMS with ETASUs.51  These drugs would not be approved 
or their approval would be withdrawn without the ETASUs.52  Of the 199 
approved REMS, FDA approved 74 preapproval REMS in response to 
information in drug applications.  The remaining 125 postapproval REMS 
were required in response to new safety information discovered after the 
drugs were already on the market or for drugs with ETASUs already in 
effect before REMS program initiation.53   
As of December 31, 2011, FDA no longer required 100 of the 199 
approved REMS.  Ninety-two of the REMS that are no longer required 
included medication guides only, seven required communication plans, 
and one required ETASUs.  Medication guides existed prior to REMS and 
may be required for drugs without REMS.  Following program inception, 
FDA required REMS for all drugs that required medication guides.  
However, in November 2011, FDA published a guidance document 
describing when medication guides would be required without also 
requiring REMS.  It also provided sponsors of REMS requiring only 
medication guides with information about how to submit proposals to 
eliminate the REMS.54   
   
 
51 REMS components are not exclusive.  The 48 REMS that required communication plans 
may have also required medication guides, and the 32 REMS that required ETASUs may have 
also required communication plans and/or medication guides.  We categorized the REMS 
according to their primary component as of December 31, 2011. 
52 Of the 32 REMS with ETASUs, 28 also required implementation systems.   
53
 FDA worked with sponsors to implement ETASUs for certain drugs prior to the REMS 
program.  After REMS program initiation, FDA required sponsors of these drugs to submit 
REMS proposals in accordance with  909(b)(3) of the Food and Drug Administration 
Amendments Act (P.L. 110-85, Sept. 27, 2007). 
54 FDA, Guidance for Industry:  Medication Guides  Distribution Requirements and 
Inclusion in REMS, pp. 78, November 2011.  Accessed at 
http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guid
ances/UCM244570.pdf on August 2, 2012.  This guidance states that it represents FDAs 
current thinking on this topic but does not bind FDA or the public.  
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  14 
   
   
According to the approved timetable in each REMS, 57 REMS required 
sponsors to submit at least 1 assessment by December 31, 2011.55  FDA 
provided review memorandums for 49 of these REMS.56  Of these 
49 REMS, 11 required medication guides only, 19 required 
communication plans, and 19 required ETASUs.57   
Nearly half of the 49 sponsor assessments we 
reviewed did not include all information requested in 
FDA assessment plans, and 10 were not submitted to 
FDA within required timeframes 
FDA assessment plans request sponsors to submit specific information to 
enable FDA to determine whether REMS are meeting their goals.  FDA 
reviewed 49 sponsor assessments as of December 31, 2011.  Almost half 
(23 of 49) of the sponsor assessments did not include all of the 
information requested in FDA assessment plans.  See Table 2 for the 
number of assessments that lacked information requested in assessment 
plans by primary REMS component.  For example, one sponsor did not 
include the number of pharmacies that were deauthorized to dispense the 
drug because of noncompliance with the REMS, as requested in the 
assessment plan.  The same sponsor also did not include the amount of the 
drug shipped to health care providers compared to actual patient orders, as 
requested.  In review memorandums for the 23 assessments that did not 
include information requested in assessment plans, reviewers 
recommended that FDA notify the sponsors of 6 that their assessments 
were incomplete.  FDA review memorandums for the remaining 17 
assessments did not indicate that FDA no longer requested sponsors to 
submit the missing information.  
   
 
55 This number does not include REMS that were no longer required as of December 31, 
2011, or REMS for generic drugs, which are not subject to a timetable for assessments. 
56 For the remaining eight assessments, FDA had not completed assessment reviews as of 
December 31, 2011.  As of December 31, 2011, FDAs assessment reviews did not meet its 
goal of 60 days for three of these eight assessments.  
57 Of the 19 REMS with ETASUs, 12 required implementation systems. 
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  15 
   
   
Table 2:  Sponsors Assessments Lacking Information Requested in 
Assessment Plans 
Number of Assessments 
Lacking Information 
Primary REMS Component 
Requested in Assessment 
Plans 
ETASU  10 
Communication plan  8 
Medication guide  5 
     Total  23 
                                   Source:  OIG analysis of sponsor assessments. 
Sponsors and health care providers have expressed concerns about the 
challenges associated with collecting data on the compliance of third 
parties (e.g., patients, pharmacies, drug distributors).  Specifically, 
concerns about patient confidentiality and the lack of a standardized 
format for sponsor assessments may contribute to the lack of data in some 
assessments.58     
FDA does not have authority to take enforcement actions against sponsors 
that do not include requested information in their REMS assessments.   
Some items requested in FDA assessment plans are also required by 
Federal law.  An assessment of the extent to which the ETASUs met the 
goal stated in the approved REMS was required by Federal law, but was 
missing from seven assessments.59   
Further, 10 of 49 of sponsor assessments were not submitted to FDA by 
the dates specified in the approved timetables, as required by FDCA.  
These assessments were submitted between 3 and 70 days after the dates 
specified in the timetables, with a median of 17 days.  FDA has the 
authority to take enforcement actions against sponsors that do not meet 
Federal requirements for REMS assessments.  However, since program 
inception in 2008, FDA has not done so.   
FDA determined that 7 of the 49 REMS we reviewed 
met all of their goals 
Using limited information in sponsor assessments, FDA determined that 7 
of the 49 REMS we reviewed were meeting all of their goals and that 21 
 
58 FDA, Transcript of REMS Public Meeting, July 28, 2010, pp. 208 and 284.  Accessed at 
http://www.fda.gov/downloads/Drugs/NewsEvents/UCM224950.pdf on July 3, 2012. 
59 Assessments of the extent to which the ETASUs met the goals stated in the approved 
REMS were required only for the 19 REMS with ETASUs.  FDASIA changed this 
requirement to no longer apply specifically to ETASUs.  According to FDCA  505-1(g)(3), 
all sponsor assessments must now include, with respect to each goal of the REMS, 
assessments of the extent to which the REMS are meeting the goals or of whether the 
goals or REMS should be modified.   
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  16 
   
   
were not.  FDA review memorandums indicated that FDA could not 
determine whether 17 REMS were meeting all of their goals.  FDA review 
memorandums did not contain statements regarding whether the remaining 
four REMS were meeting all of their goals.60   
As shown in Table 3, FDA determined that 1 of 19 REMS with ETASUs, 
which are required for the riskiest drugs, was meeting all of its goals.  In 
review memorandums, FDA reviewers stated that eight REMS with 
ETASUs were not meeting all goals and that they could not determine 
whether eight others were meeting all goals.  FDA did not determine 
whether two REMS with ETASUs were meeting their goals.   
                     Table 3:  Determinations of Whether REMS Were Meeting Goals  
Primary REMS  Meeting All  Not Meeting  Unable To  Did Not 
Total 
Component  Goals   All Goals  Determine  Determine 
ETASU  1  8  8  2  19 
Communication plan  2  10  5  2  19 
Medication guide  4  3  4  0  11 
Total  7  21  17  4  49 
               Source:  OIG analysis of FDA review memorandums, 2012. 
FDA most often determined that REMS were not meeting their 
goals because of deficiencies in patient and prescriber 
awareness of drug risks 
When FDA determined that REMS were not meeting all of their goals, it 
most often identified deficiencies related to patient awareness of risks (14 
of 21 assessments) and/or prescriber awareness of risks (12 of 21 
assessments).  For example, because a survey showed that patients had a 
low understanding of key risk messages, FDA determined that one REMS 
was not meeting its goal to inform patients of a drugs risks.  FDA 
discussed REMS modifications with sponsors for 16 of the 21 REMS it 
determined were not meeting all of their goals.   
FDA did not discuss REMS modifications with sponsors of the remaining 
five REMS it determined were not meeting their goals.61     
 
60 If an FDA review memorandum did not contain a statement from the reviewer regarding 
whether the REMS was meeting its goals, we concluded that FDA did not make that 
determination. 
61 In some cases, FDA may have suggested that sponsors modify REMS assessments or 
education materials rather than modifying the REMS.  FDA indicated that these suggestions 
were not consistently included in review memorandums. 
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  17 
   
   
Nearly all deficiencies identified by prior FDA assessment 
reviews were still present in the sponsors most recent 
assessments 
FDA identified 22 deficiencies that may affect a REMS ability to mitigate 
risks.62  In 11 prior assessments we reviewed, 19 of 22 deficiencies 
identified by FDAs prior assessment reviews were identified again in 
reviews of the most recent assessments.  Reviewers identified deficiencies 
that included low patient exposure to medication guides, low 
understanding of risk messages among patients and/or prescribers, and 
poor compliance with the REMS among health care providers. 
FDA could not determine whether REMS were meeting their 
goals most often because of incomplete information in 
assessments 
For 8 of 17 assessments, FDA could not determine whether REMS were 
meeting their goals because of a lack of information included in the 
sponsors assessments.  For example, FDA could not determine whether 
one REMS was meeting its goal to inform patients of the drugs risks 
because the sponsor did not include an assessment of patients 
understanding of risks in the assessment.  FDA could not determine 
whether an additional three REMS were meeting their goals because 
assessments were required too early in the implementation of the REMS to 
draw conclusions.63 
FDA could not determine whether five REMS were meeting their goals 
because of concerns about the quality of the surveys used in the sponsor 
assessments.64  FDAs concerns generally related to small sample sizes or 
the use of surveys that did not meet FDAs standards.  However, FDA does 
not specify in its assessment plans the sample size that would enable it to 
determine whether a REMS is meeting its goals. 
FDA has not identified reliable methods to evaluate the 
effectiveness of REMS 
FDA has not developed a plan to identify, develop, validate, and assess the 
effectiveness of REMS components, as stated in its 5-year plan        
(20082012) to enhance and modernize drug safety activities.  
 
62 A sponsors second most recent assessment is considered the prior assessment.  FDA 
provided review memorandums for 14 prior assessments and identified deficiencies in 11. 
63
 Two of these assessments were required 6 months after REMS approval.  The third 
assessment was required 12 months after approval, but the drugs introduction to the market 
was delayed, which delayed data collection for the assessment. 
64
 Two of these five REMS assessments also lacked information.  FDA reviewers did not 
explain why they were unable to determine whether REMS were meeting their goals for the 
remaining sponsors assessments. 
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  18 
   
   
Additionally, FDA has not met the Federal requirement to evaluate the 
ETASUs of one REMS each year.  Further, FDA has identified limitations 
in methods used to evaluate the effectiveness of REMS.  
FDA has evaluated the ETASUs of one REMS since program 
inception in 2008 
According to FDCA, FDA must evaluate the ETASUs for at least one drug 
each year to determine whether they (1) assure safe use of the drug; (2) are 
unduly burdensome on patient access to the drug; and (3) to the extent 
practicable, minimize the burden on the health care delivery system.  
Between 2008 and 2011, FDA approved 32 REMS with ETASUs to 
address drugs with the most serious risks.  As of December 31, 2011, FDA 
had completed an evaluation for one drug with an ETASU. 
On December 1, 2011, FDAs Drug Safety and Risk Management 
Advisory Committee met to evaluate the ETASUs of isotretinoin.65  At the 
meeting, FDA and stakeholders discussed the impact of the isotretinoin 
REMS on patients and the health care delivery system.  FDA and industry 
representatives noted several challenges associated with completing the 
required evaluation, including the lack of baseline (pre-REMS) utilization 
data and the inability to account for changes in utilization because of 
factors not related to the REMS.   
FDA stated that it had not formally evaluated a REMS with ETASUs in 
the first 3 years of the REMS program because it was focused on 
developing REMS and implementing the program.  Additionally, FDA 
stated that REMS must be implemented for a sufficient period of time 
before they can be properly evaluated.66   
Because FDA has completed this evaluation for just 1 of 32 drugs with 
ETASUs, it has limited data to demonstrate that the remaining REMS with 
ETASUs effectively ensure safe use of drugs or meet statutory 
requirements to minimize burdens on patients and the health care system.   
 
65 Isotretinoin was originally approved as an acne treatment and marketed as Accutane.  
Accutane is no longer on the market, but its generic equivalents remain on the market under a 
single, shared-system REMS for isotretinoin.  The ETASUs require that prescribers and 
pharmacies be specially certified, patients have evidence of safe-use conditions, and sponsors 
maintain pregnancy registries.  Minutes of the Advisory Committee meeting are available to 
the public.  Accessed at 
http://www.fda.gov/AdvisoryCommittees/CommitteesMeetingMaterials/Drugs/DrugSafet
yandRiskManagementAdvisoryCommittee/ucm250295.htm on July 19, 2012.  
66 Prior to the use of REMS, FDA used Risk Minimization Action Plans (RiskMAPs) to 
manage risks of certain drugs.  The ETASUs for isotretinoin were fully implemented in 2006 
under a RiskMAP.  Claudia Karwoski, Opening Remarks for Drug Safety and Risk 
Management and Ophthalmic Advisory Committee Meeting, December 1, 2011.  Accessed at 
http://www.fda.gov/downloads/AdvisoryCommittees/CommitteesMeetingMaterials/Drug
s/DermatologicandOphthalmicDrugsAdvisoryCommittee/UCM285047.pdf  on September 
20, 2012.   
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  19 
   
   
FDA has identified limitations with using survey data to assess 
REMS 
FDA often requests that sponsors include results of surveys regarding 
patients and/or health care providers knowledge of risks in REMS 
assessments.67  FDA made such requests in each of the 49 sponsor 
assessments we reviewed.  Of these 49 sponsor assessments, 40 included 
survey data in response to this request and 9 did not.   
However, in a White Paper issued in June 2012, FDA identified several 
limitations in some sponsors surveys.  These included sample sizes too 
small to draw conclusions, survey populations that do not reflect the 
demographics of the target population, bias caused by convenience 
samples, and the lack of objective standards to measure knowledge of 
risks.68  FDA states that because of these limitations, surveys may not 
always be the best method for assessing the effectiveness of REMS 
educational components.69   
FDAs assessment review times exceeded its 60-day 
goal for all but one sponsor assessment 
FDCA requires FDA to promptly review sponsor assessments.70  Although 
FDA does not define promptly, FDA officials indicated that their goal is 
to complete all assessment reviews within 60 days of a sponsors 
assessment submission if FDA does not discuss REMS modifications with 
sponsors.  FDA did not discuss REMS modifications with sponsors for 
29 of the 49 sponsor assessments we reviewed.71  FDAs assessment 
review times exceeded 60 days for 28 of these 29 sponsor assessments.  
FDAs average assessment review time was 73 days and the median 
review time was 69 days.72  Reviews of three sponsor assessments were 
completed at least 120 days after the sponsor submitted them.73  If FDA 
 
67 FDA, Risk Evaluation and Mitigation Strategy Assessments:  Social Science Methodologies 
to Assess Goals Related to Knowledge, pp. 12.  Accessed at 
http://www.fda.gov/downloads/Drugs/NewsEvents/UCM301966.pdf on September 13, 
2012. 
68
 Convenience samples are selected on the basis of what is most feasible for the researcher 
and are therefore not always representative of the larger population. 
69 Ibid, pp. 56.   
70 FDCA  505-1(h)(1). 
71 FDAs 60-day goal for reviewing sponsors assessments does not apply to the 
remaining 20 assessments for which it discussed REMS modifications with sponsors.  
72 The average and median review times do not include reviews that were not complete as 
of December 31, 2011. 
73 The FDA review memorandum for one of these REMS indicated that it is included in 
the opioid class REMS, which was under development at the time of the review and may 
have contributed to delays in review completion.  The FDA review memorandums for the 
remaining two REMS do not indicate a reason for delays. 
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  20 
   
   
does not review sponsor assessments in a timely manner, sponsors may 
have limited time to implement suggested changes to a REMS before 
submitting the next assessment.   
   
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  21 
   
   
CONCLUSION AND RECOMMENDATIONS 
FDA requires REMS for drugs associated with known or potential risks 
that, without REMS, may outweigh the drugs benefits.  REMS are 
enforceable, structured plans to manage specific risks associated with 
drugs.  However, FDA does not have the authority to take enforcement 
actions against sponsors that do not include all information requested in 
FDA assessment plans.  If FDA does not have comprehensive data to 
monitor the performance of REMS, it cannot ensure that the public is 
provided maximum protection from a drugs known or potential risks. 
Between program inception in 2008 and 2011, FDA approved 199 REMS, 
99 of which were still required in 2012.  However, FDA does not have 
comprehensive data from sponsor assessments to determine whether 
REMS are meeting their goals because many sponsor assessments are 
incomplete or include data that do not meet FDAs standards.  
Additionally, using the limited information in sponsor assessments, FDA 
determined that 7 of 49 REMS were meeting all of their goals and that 
21 were not, raising questions about the effectiveness of REMS.   
Further, FDA has completed the federally required evaluation for 1 of 
32 drugs with ETASUs since program inception.  Therefore, FDA has 
limited data to demonstrate that the remaining REMS with ETASUs 
effectively ensure safe use of drugs or meet statutory requirements to 
minimize burdens on patients and the health care system.  Additionally, 
FDA has not identified reliable methods for evaluating REMS.   
Finally, FDAs assessment review times exceeded 60 days for all but one 
sponsor assessment, which may limit sponsors ability to implement 
suggested changes to the REMS before submitting the next assessment.  
In conclusion, our findings raise concerns about the overall effectiveness 
of the REMS program.  To address these concerns, we recommend that 
FDA: 
Develop and implement a plan to identify, develop, validate, 
and assess REMS components 
This plan should fulfill FDAs commitment in its 5-year plan and outline 
ways that FDA will assess the effectiveness of REMS beyond reviewing 
sponsors assessments.  
FDA should also identify and implement reliable methods to assess the 
effectiveness of REMS.  FDA should decrease its reliance on survey data 
in sponsors assessments and work with sponsors and health care providers 
to develop more accurate evaluation methods. 
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  22 
   
   
Additionally, FDA should continue to hold discussions with stakeholders, 
similar to the public meetings held in 2010 and 2012, about the issues and 
challenges associated with assessing the effectiveness of REMS 
components.74 
Identify REMS that are not meeting their goals and take 
appropriate actions to protect the public health 
FDA should consistently discuss potential REMS modifications with 
sponsors when it determines that a REMS is not meeting its goals.  FDA 
should work with sponsors to determine the most appropriate 
modifications to address the REMS deficiencies.  If a REMS undergoes 
multiple modifications and continues not to meet its goals, FDA should 
consider removing the drug from the market. 
If FDA cannot determine whether a REMS is meeting its goals, it should 
work with sponsors to obtain any additional information that it needs to 
make this determination.  FDA should not wait until it reviews the next 
sponsor assessment to determine whether a REMS is meeting its goals.   
Evaluate the ETASUs of one REMS each year as required by 
Federal law 
FDA should determine whether the ETASUs (1) assure safe use of the 
drug; (2) are unduly burdensome on patient access to the drug; and (3) to 
the extent practicable, minimize the burden on the health care delivery 
system.  FDA fulfilled this requirement for the first time in 2011 and 
should continue to conduct formal evaluations of at least one REMS with 
ETASUs each year.  
Through these evaluations, FDA can determine how effective various 
ETASUs are, whether ETASUs cause barriers to patient access, and which 
ETASUs are the most burdensome for health care providers.  Evaluations 
would also inform FDAs decisions about the most effective ETASUs for 
mitigating specific risks.  FDA should use this information to change or 
eliminate any ETASUs that are both burdensome and ineffective in 
assuring safe use of a drug. 
Identify incomplete sponsor assessments and work with 
sponsors to obtain missing information  
FDA should consistently notify sponsors that their assessments are 
incomplete and request that sponsors provide the missing information as 
soon as possible.  FDA should also work with sponsors to obtain the 
missing information. 
 
74 75 Fed. Reg. 34453 (June 17, 2010) and 77 Fed. Reg. 26292 (May 3, 2012). 
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  23 
   
   
Clarify expectations for sponsors assessments in FDA 
assessment plans 
FDA should make assessment plans as specific as possible to ensure that 
data included in sponsors assessments will enable FDA to determine 
whether REMS are meeting their goals.  For example, FDA should specify 
sample sizes needed to complete valid surveys.  FDA should use 
assessment plans to establish quality standards for data to be submitted in 
sponsors assessments.  If FDA cannot determine whether a REMS is 
meeting its goals because of the quality of data in a sponsors assessment, 
it should notify the sponsor of this concern and, if necessary, clarify data 
standards for future assessments.    
Seek legislative authority to enforce FDA assessment plans  
FDA currently does not have authority to take enforcement actions when 
sponsors do not submit all information requested in assessment plans.  
FDA should work with the appropriate stakeholders to seek legislative 
authority to enforce assessment plans.  This would allow FDA to take 
regulatory actions when a sponsors assessment did not include all items 
requested in the FDA assessment plan.  Further, enforcement authority 
should encourage sponsors to submit the amount and quality of 
information FDA needs to determine whether REMS effectively mitigate 
risks. 
Ensure that assessment reviews are timely 
FDCA requires FDA to promptly review sponsor assessments.  FDA 
officials indicated that their goal is to complete all assessment reviews 
within 60 days of sponsors assessment submissions if FDA does not 
discuss REMS modifications with sponsors.  Since FDA set this goal, 
some FDCA provisions regarding the review of sponsors assessments 
have changed because of FDASIA.  If FDA chooses to change this goal of 
60 days, it should define promptly as it pertains to reviewing sponsor 
assessments and complete assessment reviews within this timeframe.  
FDA should identify ways to complete assessment reviews within its goal 
of 60 days or the newly established timeframe.  This would allow time for 
informed discussions with sponsors about REMS modifications, if 
necessary.  This would also enable sponsors to make timely changes to 
REMS and/or REMS assessments in response to FDAs review.  To 
maximize review resources, FDA could prioritize assessment reviews for 
REMS with ETASUs as these REMS are required for drugs with the most 
serious risks and generally require more frequent assessments. 
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  24 
   
   
AGENCY COMMENTS AND OFFICE OF INSPECTOR GENERAL 
RESPONSE 
In its comments on the draft report, FDA concurred with six of our 
recommendations and, for the remaining recommendation, agreed that 
seeking legislative change should be considered if another opportunity 
arises to do so.  FDA agreed with the need to improve future REMS 
assessments but stated that there would always be challenges in measuring 
the impact of REMS.  FDA provided historical context to describe the 
evolving nature of pharmaceutical risk management.  FDA also provided 
information about the REMS Integration Initiative which was created in 
2011. 
With regard to our first recommendation, FDA noted that the REMS 
Integration Initiative will include contributions from key stakeholders on 
the development, implementation, and effectiveness of REMS tools. 
With regard to our second recommendation, FDA stated that it has worked 
and will continue to work with sponsors to determine the best response 
when it determines that a REMS is not meeting its goals.  Additionally, 
FDA stated that it plans to develop draft guidance concerning how to write 
REMS goals and the metrics for determining whether these goals have 
been met.  
With regard to our third recommendation, FDA noted that it faced 
challenges, including insufficient experience in evaluating REMS and 
insufficient assessment data available, in meeting the Federal requirement 
to evaluate the ETASUs of at least one drug each year.  FDA also noted 
that while it did not formally meet this requirement until 2011, it discussed 
certain REMS with ETASUs at various advisory committees prior to 2011. 
With regard to our fourth recommendation, FDA stated that it has worked 
and will continue to work with sponsors to obtain information missing 
from assessments if that information is necessary to determine whether a 
REMS is meeting its goals.  If FDA determines that the information is not 
necessary to determine whether a REMS is meeting its goals, it will 
reevaluate the need for the information. 
With regard to our fifth recommendation, FDA stated that it is committed 
to holding public workshops to gather stakeholder input.  FDA plans to 
issue draft guidance on methodologies for assessing whether REMS are 
meeting are their goals and the impact of REMS on patient access and 
burden on the health care system.  Additionally, FDA noted that it has 
reviewed and will continue to review proposed assessment protocols if 
sponsors submit them prior to initiating the REMS assessment. 
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  25 
   
   
FDA did not explicitly concur with our sixth recommendation but agreed 
that it should be considered if another opportunity arises to pursue 
legislative changes to the statutory provisions that describe REMS 
assessment requirements. 
Finally, with regard to our seventh recommendation, FDA stated that it is 
examining the internal assessment review process for ways to improve the 
timeliness of FDA assessment reviews.   
We support FDAs efforts to address these issues and encourage it to 
continue making progress in these areas.  For the full text of FDAs 
comments, see Appendix B.  We made minor changes to the report based 
on FDAs technical comments. 
   
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  26 
   
   
APPENDIX A 
 
Assessment Review Memorandums Provided by FDA 
Division of Risk  Office of 
Management  Compliance  Prior Assessment 
Review  Review  and Review 
Name of Drug  Memorandum  Memorandum  Memorandum(s)* 
ETASU 
Abstral sublingual tablets  X  X    
Erythropoiesis stimulating 
agents  X  X  X 
Butrans transdermal system  X       
Caprelsa tablets  X  X    
Entereg capsules  X  X  X 
Exalgo extended-release 
tablets  X  X  X 
Isotretinoin   X  X    
Letairis tablets  X  X    
Lumizyme   X     X 
Nplate for subcutaneous 
injection  X  X  X 
Onsolis buccal soluble film  X       
Oxycontin 
controlled-release tablets  X     X 
Promacta tablets  X  X  X 
Revlimid capsules  X  X  X 
Sabril tablets and oral 
solution  X     X 
Soliris injection  X  X  X 
Suboxone sublingual film  X  X  X 
Thalomid capsules  X       
Tracleer tablets  X  X    
Communication Plan 
Actemra injection  X  X    
Ampyra extended-release 
tablets  X       
Botox and Botox Cosmetic 
injection  X  X    
Dulera inhalation aerosol  X  X    
Dysport injection  X  X    
Effient tablets  X  X    
continued on next page 
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  27 
   
   
Appendix A, continued 
Assessment Review Memorandums Provided by FDA 
Division of Risk  Office of 
Management  Compliance  Prior 
Review  Review  Assessment and 
Name of Drug  Memorandum  Memorandum  Memorandum(s) 
Communication Plan 
Embeda extended-release 
tablets  X  X  X 
Forteo injection  X  X    
Kalbitor injection  X  X    
Krystexxa injection  X  X    
Multaq tablets  X  X  X 
Myobloc injection  X  X    
Samsca tablets  X    
Stelara injection  X  X     
Tasigna capsules  X       
Vibativ injection  X       
Xenazine tablets  X  X  X 
Xiaflex injection  X  X    
Zortress tablets  X  X    
Medication Guide 
Actoplus Met tablets  X       
Actoplus Met XR 
extended-release tablets  X       
Actos tablets  X       
Duetact tablets  X       
Androgel 1.62% gel  X  X    
Chantix tablets  X  X    
Nucynta immediate release 
tablets  X       
Testim gel  X  X    
Venlafaxine hydrochloride 
extended-release tablets  X  X    
Vivitrol extended-release 
injectable suspension  X       
Zyban sustained-release 
tablets, Wellbutrin, 
Wellbutrin 
sustained-release tablets, 
and Wellbutrin 
extended-release tablets  X  X    
   Total  49  33  14 
*Some REMS had not required multiple assessments as of December 31, 2011.  A blank in this column does not necessarily             
 
indicate that a sponsor did not complete a required assessment.  
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  28 
  APPENDIX B 
Agency Comments 
..,.,-
.... 
(  t.E# 
m<:PARTMJ:NT OF HEAI.T!IA...lii.D HU\iAN SERV,lCES 
....~~ 
Food and Drug Administration 
Silver Spring , MD 20993 
DATE: 	 December 10 , 2012 
TO: 	 Daniel R. Levinson 
Inspector General 
FROM: 	 Peter Lurie, MD, MPH 
Acting Associate Commissioner for Policy and Planning 
SUBJECT: 	 Food and Drug Administration's Response to Office of Inspector General 
Draft Report: FDA Lacks Conrprellensive Data to Determ i11e Whether REMS 
Improve Drug Safety (OEI-04-11-00510) 
FDA is providing the attached response to the Office of Inspector General's draft report. 
FDA appreciates the opportunity to comment on this draft report. 
Peter Lurie 
Acting Associate Commissioner for Policy and Planning 
Attachment 
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  29   
   
 
 
 
 
 
   
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  30 
   
   
   
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  31 
   
   
 
 
 
 
 
 
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  32 
   
   
 
 
 
 
 
 
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  33 
   
   
ACKNOWLEDGMENTS 
This report was prepared under the direction of Dwayne Grant, Regional 
Inspector General for Evaluation and Inspections in the Atlanta regional 
office, and Jaime Durley, Deputy Regional Inspector General.   
Sarah Langford served as the lead analyst for this study.  Central office 
staff who provided support include Clarence Arnold, Debra Roush, and 
Talisha Searcy. 
 
 
 
FDA Lacks Comprehensive Data To Determine Whether REMS Improve Drug Safety (OEI-04-11-00510)  34 
   
  
Office of Inspector General 
http://oig.hhs.gov 
 
The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, as 
amended, is to protect the integrity of the Department of Health and Human Services 
(HHS) programs, as well as the health and welfare of beneficiaries served by those 
programs.  This statutory mission is carried out through a nationwide network of audits, 
investigations, and inspections conducted by the following operating components: 
Office of Audit Services 
The Office of Audit Services (OAS) provides auditing services for HHS, either by conducting 
audits with its own audit resources or by overseeing audit work done by others.  Audits 
examine the performance of HHS programs and/or its grantees and contractors in carrying 
out their respective responsibilities and are intended to provide independent assessments of 
HHS programs and operations.  These assessments help reduce waste, abuse, and 
mismanagement and promote economy and efficiency throughout HHS. 
Office of Evaluation and Inspections 
The Office of Evaluation and Inspections (OEI) conducts national evaluations to provide 
HHS, Congress, and the public with timely, useful, and reliable information on significant 
issues.  These evaluations focus on preventing fraud, waste, or abuse and promoting 
economy, efficiency, and effectiveness of departmental programs.  To promote impact, OEI 
reports also present practical recommendations for improving program operations.  
Office of Investigations 
The Office of Investigations (OI) conducts criminal, civil, and administrative investigations 
of fraud and misconduct related to HHS programs, operations, and beneficiaries.  With 
investigators working in all 50 States and the District of Columbia, OI utilizes its resources 
by actively coordinating with the Department of Justice and other Federal, State, and local 
law enforcement authorities.  The investigative efforts of OI often lead to criminal 
convictions, administrative sanctions, and/or civil monetary penalties. 
Office of Counsel to the Inspector General 
The Office of Counsel to the Inspector General (OCIG) provides general legal services to 
OIG, rendering advice and opinions on HHS programs and operations and providing all 
legal support for OIGs internal operations.  OCIG represents OIG in all civil and 
administrative fraud and abuse cases involving HHS programs, including False Claims Act, 
program exclusion, and civil monetary penalty cases.  In connection with these cases, OCIG 
also negotiates and monitors corporate integrity agreements.  OCIG renders advisory 
opinions, issues compliance program guidance, publishes fraud alerts, and provides other 
guidance to the health care industry concerning the anti-kickback statute and other OIG 
enforcement authorities.