FRAMEWORK FOR FDA’S

REAL-WORLD

EVIDENCE

PROGRAM

www.fda.gov

December 2018
Framework for FDA’s 

Real-World Evidence Program

December 2018

Framework for FDA’s Real-World Evidence Program
1

Table of Contents

Introduction .................................................................................................................................................3

Definitions of Real-World Data and Real-World Evidence ........................................................... 4

Scope of RWE Program Under 21st Century Cures Act ..................................................................7

Current Use of RWD for Evidence Generation ................................................................................8

Generating Evidence Regarding Safety and Effectiveness ............................................................ 8

Supporting FDA’s Regulatory Decisions of Effectiveness ............................................................. 9

Trial Designs Using RWD to Generate Evidence ......................................................................... 11

Framework for Evaluating RWD/RWE for Use in Regulatory Decisions...............................13

Using Trials or Studies with RWD/RWE for Effectiveness Decisions..........................................13

Assessing Fitness of RWD for Use in Regulatory Decisions .........................................................14

Potential for Study Designs Using RWD to Support Effectiveness ..............................................19

Regulatory Considerations for Study Designs Using RWD ......................................................... 22

Data Standards — Appropriate Data Standards for Integration and Submission to FDA.......... 24

Stakeholder Engagement ...................................................................................................................26

Conclusion.................................................................................................................................................27

Glossary ......................................................................................................................................................28

References..................................................................................................................................................30

Appendix ...................................................................................................................................................34
Framework for FDA’s Real-World Evidence Program

2
3

Introduction

1 Public Law 114-255.

2

 For purposes this framework, unless otherwise noted, the term drug refers both to a drug approved under section 505(c) or (j) of the 

Federal Food, Drug, and Cosmetic Act (FD&C Act) and to biological products licensed under section 351 of the Public Health Service Act 

(PHS Act) 42 USC 262. 

3

 In this document, when we refer to the use of RWE to support a regulatory decision, we mean that the evidence provides support for 

or helps provide support for the regulatory decision.

4

 FDA issued the guidance Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices on August 31, 2017.

Introduction

The 21st Century Cures Act (Cures Act),1 signed into law on December 

13, 2016, is designed to accelerate medical product development and 

bring new innovations and advances faster and more efficiently to the 

patients who need them. Among other provisions, the Cures Act added 

section 505F to the Federal Food, Drug, and Cosmetic Act (FD&C Act). 

Pursuant to this section, the Food and Drug Administration (FDA) 

has created a framework for evaluating the potential use of real-world 

evidence (RWE) to help support the approval of a new indication for 

a drug2 already approved under section 505(c) of the FD&C Act or 

to help support or satisfy drug postapproval study requirements.3 In 

addition to drug and biological products approved under section 505(c), 

this framework is also intended for application to biological products 

licensed under the Public Health Service Act. The framework does not 

cover medical devices.4

FDA’s RWE Program will be multifaceted. It will involve demonstration 

projects, stakeholder engagement, internal processes to bring senior 

leadership input into the evaluation of RWE and promote shared 

learning and consistency in applying the framework, and guidance 

documents to assist developers interested in using real-world data 

(RWD) to develop RWE to support Agency regulatory decisions. This 

document outlines the framework FDA plans to use to implement its 

RWE Program.
Framework for FDA’s Real-World Evidence Program

4

5

 The definition of RWE provided by section 3022 of the Cures Act was subsequently revised by a technical amendment in Section 901 

of the FDA Reauthorization Act of 2017 (Public law 115-52).

Definitions of Real-World Data 

and Real-World Evidence 

Section 505F(b) of the FD&C Act defines RWE as “data regarding the 

usage, or the potential benefits or risks, of a drug derived from sources 

other than traditional clinical trials” (21 U.S.C. 355g(b)).5 In developing 

its RWE program, FDA believes it is helpful to distinguish between the 

sources of RWD and the evidence derived from that data. Evaluating 

RWE in the context of regulatory decision-making depends not only 

on the evaluation of the methodologies used to generate the evidence 

but also on the reliability and relevance of the underlying RWD; these 

constructs may raise different types of considerations. For the purposes 

of this framework, FDA defines RWD and RWE as follows: 

Real-World Data (RWD) are data relating to patient health status 

and/or the delivery of health care routinely collected from a variety 

of sources.

Real-World Evidence (RWE) is the clinical evidence about the usage 

and potential benefits or risks of a medical product derived from 

analysis of RWD.
5

Introduction

6

 Randomized and observational study designs are discussed further in this framework. For a discussion about large simple trials, see 

Peto et al. 1995 and Eapen et al. 2014. For a discussion about pragmatic clinical trials, see Ford and Norrie 2016.

7

 In this document and for purposes of the RWE program, we also refer to a clinical trial as a type of clinical study. This should not be 

read to suggest that FDA considers clinical trials to be studies under section 505(o), which authorizes FDA to require postapproval clinical 

trials and studies under specific conditions.

Examples of RWD include data derived from electronic health records 

(EHRs); medical claims and billing data; data from product and disease 

registries; patient-generated data, including from in-home-use settings; 

and data gathered from other sources that can inform on health status, 

such as mobile devices. RWD sources (e.g., registries, collections of 

EHRs, administrative and medical claims databases) can be used for 

data collection and, in certain cases, to develop analysis infrastructure to 

support many types of study designs to develop RWE, including, but not 

limited to, randomized trials (e.g., large simple trials, pragmatic clinical 

trials) and observational studies (prospective or retrospective).6

Clinical Trials and Observational Studies 

Covered by the RWE Program

It is important to distinguish among the trial designs and studies that 

will be covered by the RWE Program. Under FDA’s RWE Program, 

evidence from traditional clinical trials will not be considered RWE. 

However, various hybrid or pragmatic trial designs and observational 

studies could generate RWE. FDA’s RWE Program will cover clinical 

trials that generate RWE in some capacity (i.e., sources other than 

traditional clinical trials) and observational studies. 

Clinical Trials. For the purposes of this framework, a clinical trial is 

defined as a research study in which one or more human subjects are 

prospectively assigned to one or more interventions (which may include 

placebo or other control) to evaluate the effects of those interventions on 

health-related biomedical or behavioral outcomes.7

Although there is no single definition of a traditional clinical trial — and 

indeed trials vary considerably in design and conduct — for the purposes 

of this framework, FDA generally considers a traditional clinical trial 

to be one that is usually supported by a research infrastructure that is 

largely separate from routine clinical practice and is designed to control 

variability and maximize data quality. A traditional clinical trial is more 

likely to have restrictive eligibility criteria that are designed to ensure 

that the participants have the disease of interest or have characteristics 

in which detection of a drug effect (if one is in fact present) is more 
Framework for FDA’s Real-World Evidence Program 6

likely than it would be if the population were less precisely defined. 

Traditional clinical trials are usually randomized, double-blind trials in 

which both the investigator and patient are unaware of which treatment 

is being administered. These trials typically use separate procedures 

and/or identified research personnel, or both, to collect specified data 

using standardized procedures and detailed case report forms (CRF) 

that are separate from routine medical records, although some data 

(e.g., locally obtained lab data) may be transcribed from those records 

into the CRF. Personnel follow specified protocol directives to conduct 

scheduled monitoring and encourage precise adherence to study 

procedures. 

Some clinical trials may use hybrid design. For example, certain 

elements of a clinical trial could rely on the collection and analysis 

of RWD extracted from medical claims, EHRs, or laboratory and 

pharmacy databases. Researchers could collect other data specifically 

for the trial, such as results of exercise stress tests or radiographic 

analyses, using methods typical of a traditional clinical trial. A hybrid 

trial could use RWD for one clinical outcome (e.g., hospitalization, 

death), while other elements were more traditional (e.g., specified 

entry criteria, monitoring and collection of additional study endpoints 

by dedicated study personnel). FDA will consider these hybrid trial 

designs to have the potential to generate RWE. Clinical trial designs 

can also include some elements that more closely resemble routine 

clinical practice, which are sometimes described as “pragmatic” 

elements. These pragmatic clinical trials often rely on RWD and have 

the potential to generate RWE. 

Observational Studies. For purposes of this framework, 

observational studies are non-interventional clinical study designs 

that are not considered clinical trials. FDA considers a retrospective 

observational study to be one in which the study identifies the 

population and determines the exposure/treatment from historical 

data (i.e., data generated prior to the initiation of the study). The 

variables and outcomes of interest are determined at the time the study 

is designed. In a prospective observational study, the population of 

interest is identified at the start of the study, and exposure/treatment 

and outcome data are collected from that point forward. The start 

of the study is defined as the time at which the research protocol for 

the specific study question is initiated. Observational clinical studies 

might be another way to generate RWE that is relevant to effectiveness 

determinations. Therefore, the RWE Program will also consider the 

evaluation of observational clinical studies using RWD to support 

product effectiveness determinations.
7

Introduction

Scope of RWE Program Under 21st Century Cures Act 

Under the Cures Act, FDA’s RWE Program must evaluate the potential 

use of RWD to generate RWE of product effectiveness to help support 

approval of new indications for drugs approved under FD&C Act Section 

505(c) or to help to support or satisfy postapproval study requirements. 

FDA’s RWE Program will also apply to biological products licensed 

under section 351 of the Public Health Service Act. 

RWD can also be used to improve the efficiency of clinical trials, even if 

not used to generate RWE regarding product effectiveness. For example, 

RWD can help with: 

• Generating hypotheses for testing in randomized controlled trials 

• Identifying drug development tools (including biomarker 

identification)

• Assessing trial feasibility by examining the impact of planned 

inclusion/exclusion criteria in the relevant population, both within 

a geographical area or at a particular trial site

• Informing prior probability distributions in Bayesian statistical 

models 

• Identifying prognostic indicators or patient baseline characteristics 

for enrichment or stratification

• Assembling geographically distributed research cohorts (e.g., in 

drug development for rare diseases or targeted therapeutics)

Because the use of RWD to improve efficiencies of drug development 

programs that rely primarily on traditional clinical trials is already well 

established and generally encouraged by FDA, that approach will not be 

the focus of this framework. The framework covers the use of RWE in 

the other areas described in this document.

“As the breadth and 

reliability of RWE 

increases, so do the 

opportunities for FDA 

to make use of this 

information.”

Scott Gottlieb, FDA Commissioner

National Academies of Science, 

Engineering, and Medicine, 

Examining the Impact of RWE on 

Medical Product Development, 

September 19, 2017
Framework for FDA’s Real-World Evidence Program

8

8

 A list of collaborating institutions is available at https://www.sentinelinitiative.org/collaborators.

Current Use of RWD for 

Evidence Generation 

Generating Evidence Regarding 

Safety and Effectiveness 

FDA has a long history of using RWE to monitor and evaluate the safety 

of drug products after they are approved (postmarket). FDA’s primary 

source for executing pharmacoepidemiologic queries and studies is 

electronic health data (medical claims and pharmacy dispensing data) 

in the Sentinel System, which as of August 2018 has data on more 

than 100 million individuals within a network of 18 data partners and 

collaborating institutions.8 Multiple data sources are often queried to 

conduct an analysis of how drugs and biologics are used and to evaluate 

safety issues. FDA’s Center for Drug Evaluation and Research (CDER) 

and Center for Biologics Evaluation and Research (CBER) perform 

safety monitoring studies through pharmacoepidemiologic research 

projects under the Sentinel Initiative (Ball et al. 2016). 

Based on the identification of a safety concern, FDA may begin 

planning for a study using the Sentinel System even before a drug 

is approved. FDA first designs a query that can be executed by the 

Sentinel System data partners to provide information on the safety 

question. In addition, FDA designs studies to examine safety questions 

identified after a drug is approved. For example, FDA used the Sentinel 

System to evaluate the risk of stroke after using antipsychotics (Taylor 

2017), the risk of seizures after using ranolazine (Eworuke 2017), and 

the risk of venous thromboembolism after an extended or continuous 

cycle of oral contraceptives (Meony 2017).

CDER and CBER also perform pharmacoepidemiologic studies in 

collaboration with other Federal partners including the Centers 

for Medicare & Medicaid Services (CMS) and the Veterans Health 

Administration. In addition, CDER uses the Clinical Practice Research 

Datalink, which captures United Kingdom longitudinal patient-level 

EHR data. CDER uses RWD from the Centers for Disease Control and 

Prevention National Electronic Injury Surveillance System-Cooperative 

Adverse Drug Event Surveillance project, an active surveillance system 

operating in approximately 60 hospital emergency departments across 

the United States to specifically evaluate drug abuse, misuse, and the 

potential for self-harm.
9

Current Use of RWD for Evidence Generation

In addition, CBER has created focused surveillance efforts within the 

Sentinel System for vaccine safety, using the Post-Licensure Market 

Rapid Immunization Safety Monitoring (PRISM) system, and for blood 

components and blood derived products with the Blood Surveillance 

Continuous Active Surveillance Network (BloodSCAN). In September 

2017, CBER launched the Sentinel Biologics Effectiveness and Safety 

(BEST) system, an expansion of the Sentinel Initiative and other 

evidence generation capabilities that adds a number of new data 

sources, including EHR and medical claims data, new analytic tools, 

and new collaborating institutions.

However, the use of RWE to support effectiveness determinations is 

much more limited. For example, CBER has used CMS data to evaluate 

the comparative effectiveness of preventative vaccines, including 

standard-dose and high-dose (Izurieta et al. 2015); egg-based, cellbased, and adjuvanted influenza vaccines (FDA et al. 2018); and the 

effectiveness of a herpes zoster vaccine (Izurieta et al. 2017). 

Supporting FDA’s Regulatory 

Decisions of Effectiveness 

In limited instances, FDA has accepted RWE to support drug product 

approvals, primarily in the setting of oncology and rare diseases. 

When approval is based on a single-arm interventional trial — often 

when using a parallel assignment control arm is unethical or not 

feasible and usually when the effect size is expected to be large, based 

on preliminary data — the supportive RWE has consisted of data on 

historical response rates drawn from chart reviews, expanded access, 

and other practice settings.

Blincyto (blinatumomab) was initially approved under accelerated 

approval for the treatment of Philadelphia chromosome-negative 

relapsed or refractory B-cell precursor acute lymphoblastic leukemia, 

based on evidence of complete remission (CR) and duration of CR from 

a single-arm trial, the response rate of which was compared to historical 

data from 694 comparable patients extracted from over 2,000 patient 

records from European Union and U.S. clinical study and treatment 

sites (Przepiorka et al. 2015). Further study in a randomized controlled 

trial was required by FDA to verify the clinical benefit (Tower study 

NCT02013167).
Framework for FDA’s Real-World Evidence Program

10

 Trials Generating RWE

Randomized trial using RWD to assess dose 

response: ADAPTABLE (Aspirin Dosing: A PatientCentric Trial Assessing Benefits and Long-Term 

Effectiveness; start date April 2016) (NCT02697916). 

This pragmatic clinical trial compares two 

commonly used aspirin doses, 81 mg and 325 mg, 

by randomizing 20,000 patients with a history of 

myocardial infarction or known atherosclerotic 

cardiovascular heart disease to one of the two doses. 

The trial uses electronic algorithms to identify 

potential participants from the National PatientCentered Clinical Research Network (PCORnet) 

health system partners. The trial is integrated 

into routine clinical care with minimal inclusion/

exclusion criteria and no treatment protocol 

requirement beyond the assignment to one of 

the two doses of aspirin. The trial is using EHRs 

and claims data (through PCORnet) to capture 

primary endpoints such as death, hospitalization 

for non-fatal myocardial infarction or non-fatal 

stroke, and secondary endpoints such as coronary 

revascularization procedures, hospitalization 

for serious bleeding, and other patient-reported 

outcomes (Hernandez et al. 2015). 

Randomized trial using an established 

registry: VALIDATE-SWEDEHEART (The 

Bivalirudin versus Heparin in ST-Segment and 

Non-ST-Segment Elevation Myocardial Infarction 

in Patients on Modern Antiplatelet Therapy in 

the Swedish Web System for Enhancement and 

Development of Evidence-Based Care in Heart 

Disease Evaluated According to Recommended 

Therapies Registry Trial; December 2014—May 2017) 

(NCT02311231). This trial was a registry-based, 

multicenter trial in which patients were randomized 

to bivalirudin or heparin during percutaneous 

coronary intervention. The composite endpoint was 

myocardial infarction, all-cause mortality, and major 

bleeding at 6 months. A national population-based 

Swedish registry platform was used for continuous 

enrollment (all patients at participating centers 

were evaluated for enrollment), randomization, data 

collection (including baseline demographic data, 

procedural data, and clinical outcomes), and follow 

up (Fröbert et al. 2013). 
11

Current Use of RWD for Evidence Generation

Trial Designs Using RWD to Generate Evidence 

Randomized Controlled Trials Integrated into 

Health Care Systems

The integration of clinical trials into health care systems and capturing 

outcomes from clinical practice is not new. For example, in the late 

1980s, researchers in Italy (Maggiono et al. 1990) conducted a 

randomized clinical trial at the site of routine clinical care by 

randomizing 11,712 patients with acute myocardial infarction to 

receive either standard care or standard care and 1.5 million units 

of streptokinase intravenously. A stated objective of the trial was to 

eliminate the divide between clinical research and clinical practice 

(Rovelli et al. 1987). Nearly all cardiac care units in Italy participated in 

the trial, and outcomes included in-hospital mortality and mortality at 

6 and 12 months. 

As previously noted, clinical trials can be integrated into the health care 

system and can include some pragmatic elements. Trial integration 

should facilitate collection of outcomes and serious adverse events 

using RWD. The box Trials Generating RWE provides examples 

of trials using RWD to generate evidence about the potential benefits 

or risks of a drug. These trials share certain characteristics, including 

use of outcomes when there may be less diagnostic variability, which 

therefore may be well captured in RWD sources. In addition, the lack 

of blinding is less likely to influence outcomes such as myocardial 

infarction or stroke.

Observational Studies Using RWD to Generate RWE

Observational studies have been used to support regulatory safety 

decisions; however, treatment assignment based upon physician 

judgment, rather than random assignment, creates a challenge for 

establishing causal inference that must be addressed to support the 

acceptability of observational studies for effectiveness decisions. 

Randomization is used to prevent bias in allocation of the intervention 

by creating study groups balanced for risk factors for the targeted 

outcome and has been considered a critical element in establishing a 

causal relationship between medication use and health care outcomes.

Although observational studies may provide credible evidence, there 

is a stronger scientific justification for deriving evidence of a drug 

effect from randomized controlled trials as compared to observational 

studies. Despite literature citing examples where observational and 
Framework for FDA’s Real-World Evidence Program

12

randomized trials have reached similar conclusions about treatment 

effect (Benson and Hartz 2000; Anglemyer et al. 2014), there are also 

examples when effects identified in observational studies could not be 

reproduced in randomized trials or when the effect sizes differed in 

direction or magnitude (e.g., Cooper et al. 2014; Hemkens et al. 2016; 

Guadino et al. 2018).

FDA is aware of recent efforts to use rigorous design and statistical 

methods to replicate randomized trial results with observational studies 

and to derive general rules that might increase the chance of obtaining 

valid results using RWD in observational study designs (Franklin and 

Schneeweiss 2017). As previously noted, although some examples show 

concordance between randomized trials and observational studies, 

other examples show divergence. As recognized by the authors of a 

recent observational study that was designed to replicate the findings of 

a regulatory drug trial (Fralick et al. 2017), retrospective reviews of the 

literature comparing randomized clinical trial results to observational 

studies “provide single summarizations of the differences between 

these two approaches but provide few insights on the validity of 

individual real-world data analyses.” The authors further suggested 

that “to establish a meaningful baseline, the FDA will need many sets of 

randomized clinical trials with prospectively designed, nonrandomized 

analyses to match the populations included in randomized clinical 

trials across a range of clinical questions, each investigated with a set of 

designs and methods following rigorous epidemiologic principles.” 

As part of its RWE Program, FDA will evaluate the potential role of 

observational studies in contributing to evidence of drug product 

effectiveness. Efforts to replicate the results of randomized controlled 

trials using more rigorously designed observational studies may 

provide insight into the opportunities and limitations of using these 

designs in regulatory decisions.
13

Framework for Evaluating RWD/RWE for Use in Regulatory Decisions

Framework for Evaluating RWD/

RWE for Use in Regulatory Decisions

Using Trials or Studies with RWD/RWE for 

Effectiveness Decisions

As previously noted, there is considerable interest in using RWD to 

generate RWE to support regulatory decisions about the effectiveness 

of drug products. FDA has used RWD primarily in its evaluation of 

safety and only in limited circumstances to inform decisions about 

effectiveness. FDA’s RWE Program will therefore focus on exploring the 

potential of RWD/RWE to support regulatory decisions about product 

effectiveness. Specifically, FDA’s RWE Program will evaluate the 

potential use of RWE to support changes to labeling about drug product 

effectiveness, including adding or modifying an indication, such as a 

change in dose, dose regimen, or route of administration; adding a new 

population; or adding comparative effectiveness or safety information. 

The framework will include consideration of the following:

1. Whether the RWD are fit for use

2. Whether the trial or study design used to generate RWE can 

provide adequate scientific evidence to answer or help answer the 

regulatory question 

3. Whether the study conduct meets FDA regulatory requirements 

(e.g., for study monitoring and data collection) 

FDA intends to use this three-part approach to evaluate individual 

supplemental applications, as appropriate, and more generally to guide 

FDA’s RWE Program. The RWE Program will involve the establishment 

of demonstration projects, engagement with stakeholders, the use of 

internal processes that bring senior leadership input into the evaluation 

of RWE and promote shared learning and consistency in applying 

the framework, and the development of guidance documents to assist 

sponsors interested in using RWE to support their work.

Additionally, to efficiently process RWD and submit it for evaluation 

to FDA, appropriate data standards are necessary. A data standard is 

a set of rules about how a particular type of data should be structured, 

defined, formatted, or exchanged between computer systems. Data 

standards make submissions predictable and consistent and have a form 

“FDA will work with its 

stakeholders to understand 

how RWE can best be used 

to increase the efficiency of 

clinical research and answer 

questions that may not have 

been answered in the trials 

that led to the drug approval, 

for example how a drug works 

in populations that weren’t 

studied prior to approval.”

Janet Woodcock, M.D., Director, CDER
Framework for FDA’s Real-World Evidence Program

14

that an information technology system or a scientific tool can use. To 

work with RWD across multiple sources, data may need to be put into 

a common format, sometimes referred to as a common data model 

(CDM), with common representation (terminologies, vocabularies, 

coding schemes). FDA recognizes the importance of developing data 

standards to maximize the utility of RWD and is working on identifying 

relevant standards and methodologies for collection and analysis of 

RWD. FDA has already been active in developing data standards for 

regulatory use and will continue to expand its work in this area. FDA 

will consider data standards along with the other critical aspects of the 

RWE Program. 

Assessing Fitness of RWD for 

Use in Regulatory Decisions

Assessing Data Reliability (Data Accrual and 

Data Assurance) and Relevance

The strength of RWE submitted in support of a regulatory decision 

depends on the clinical study methodology and the reliability (data 

accrual and data quality control (data assurance)) and relevance of the 

underlying data. In general, FDA does not endorse one type of RWD 

over another. Data should be selected based on their suitability to 

address specific regulatory questions. For the purposes of evaluating 

drug safety, for example, FDA has already outlined its perspective 

on using RWD available in electronic health care data systems for 

safety studies in its guidance for industry and staff Best Practices for 

Conducting and Reporting Pharmacoepidemiologic Safety Studies 

Using Electronic Healthcare Data (Pharmacoepidemiologic Guidance). 

The Pharmacoepidemiologic Guidance includes recommendations for 

evaluating the data sources used in pharmacoepidemiologic safety studies. 

FDA has gained considerable experience assessing electronic health 

care data (e.g., EHRs, medical claims data, registries) through 

experience with the Sentinel System and other data systems. To assess 

the RWD used to generate RWE in the Sentinel System, FDA considers 

data reliability and relevance. Reliability includes data accrual and 

data quality control (data assurance). For medical claims data, this 

would include the assessment that the International Classification of 

Diseases (ICD) codes (e.g., ICD-ver.10-Clinical modification (CM)), 

which describe the medical diagnosis for which a claim is submitted, 

are present and in the appropriate format for use with the Sentinel 

System CDM and analysis tools. For EHRs, since laboratory test results 

obtained during routine care are not uniformly coded or documented in 
15

Framework for Evaluating RWD/RWE for Use in Regulatory Decisions

a standardized manner between organizations or within organizations 

over time, the reliability assessment includes checking the laboratory 

data for completeness, consistency, and trends over time, including the 

use of reporting standards such as the Logical Observation Identifiers, 

Names, and Codes (LOINC) system. 

While the reliability assessments consider whether the codes or 

combinations of codes adequately represent the underlying medical 

concepts they are intended to represent, the relevance assessment 

considers whether the data are fit for purpose and include an 

assessment of whether the data capture relevant data on exposure, 

outcomes, and covariates. In the context of the Sentinel System, the 

relevance assessment also considers whether the CDM contains the 

critical data elements and whether available analytic tools are sufficient 

to address each question of interest.

FDA intends to adapt this approach to assess sources of RWD used 

to generate RWE of drug product effectiveness, recognizing that the 

specific elements to consider will likely differ by RWD type and the 

type of research for which the data are intended. For example, U.S. 
Framework for FDA’s Real-World Evidence Program

16

government agencies, health care insurers, and researchers have 

substantial experience checking data quality, validating the data, and 

using medical claims data for public health and research purposes 

(mostly in observational studies), and there is a good understanding of 

the strengths and limitations of using medical claims data as RWD. On 

the other hand, EHR data, which should provide more detailed data on 

the patient beyond information contained within medical claims data 

(i.e., diagnosis codes and prescriptions or other procedures), do not 

usually show standardized data in structured fields that can be readily 

extracted and compared across systems. In addition, certain covariates 

(e.g., obesity, smoking, alcohol use) and outcomes (e.g., mortality, 

symptomatic changes) may not be consistently captured in EHRs or 

medical claims data. It will be important to examine data relevance 

to determine whether the full range of outcomes that could be used 

for study endpoints (e.g., disease exacerbations, hospitalization for 

specialized conditions, other kinds of disease outcomes) are captured.

Data from other countries can be another valuable source of RWD, but 

their fitness for use in FDA regulatory decision-making could be limited 

by important differences in health care systems. Using data from 

other countries might require analyses that consider the differences in 

medical practice, health care delivery, and data reliability and relevance 

compared to the United States. FDA’s Pharmacoepidemiologic 

Guidance provides considerations for using international electronic 

health care data in safety studies. FDA’s RWE Program will explore 
17

Framework for Evaluating RWD/RWE for Use in Regulatory Decisions

those considerations when using international electronic health care 

data to generate RWE about drug product effectiveness for regulatory 

decision-making. 

Patient registries are another source of RWD that could be used to 

generate RWE. A patient registry is an organized system that uses 

observational study methods to collect uniform data (clinical and 

other) to evaluate specified outcomes for a population defined by 

a particular disease, condition, or exposure and that serves one or 

more predetermined scientific, clinical, or policy purposes (Gliklich 

et al. 2014). Registries are generally defined either by diagnosis of a 

disease (disease registry) or usage of a drug, device, or other treatment 

(exposure registry) (CTTI 2017). Their fitness for use in generating 

RWE requires sufficient processes, such as those to gather follow-up 

information when needed, to ensure data quality, and to minimize 

missing or incomplete data. 

Although different RWD sources will have different strengths and 

limitations, the selection of appropriate RWD sources should be based 

on the regulatory question of interest and should be collected and 

maintained in a way that provides an appropriate level of reliability.

Program Item. Building on the Pharmacoepidemiologic Guidance, 

FDA plans to issue guidance on how to assess the reliability and 

relevance of RWD from medical claims and EHRs used to generate 

RWE regarding drug product effectiveness. FDA will also examine how 

to assess the reliability and relevance of registry data and international 

electronic health care data.

Addressing Gaps in RWD Sources 

EHRs and medical claims data may not capture all data elements 

needed to answer the question of interest. We expect that these sources 

will generally record major events like hospitalization, but other changes 

in medical status (e.g., worsening of depression or anxiety, increased 

joint pain, changes in severity of a dermatologic condition, worsening 

asthma) may not be reliably and consistently documented in the EHR, 

if at all. Even when captured, the way the data elements are captured 

in the EHR may limit their accessibility. For example, a patient’s 

symptoms may be recorded in unstructured data in the physician’s note 

without the use of standardized language or a standard symptom scale. 

Data on changes in the severity of chronic conditions over time are not 

likely readily available in medical claims data.
Framework for FDA’s Real-World Evidence Program

18

9

 See FDA guidance Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims.

More effective development and use of RWE will require that 

information within EHRs be more accessible and connected, and tools 

must be developed to facilitate searching such records. As described 

previously, both EHRs and medical claims data may not capture patient 

experience, unless the patient’s experience is recorded as a clinical 

event by health care providers (e.g., hospitalization or referral to 

another provider). Patient reported outcomes,

9

 changes in responses 

to medication, or non-serious adverse events might not be reported to 

caregivers or at health care visits. Even if the data are captured in the 

EHR, different health care systems may use EHRs in different formats, 

making it difficult to collect the same data across the varying records. 

Moreover, because health care systems often are not interoperable, it is 

difficult to integrate their data systems. This makes it especially difficult 

to capture data as patients move between providers or seek care in 

other facilities. Medical claims data may be able to follow an individual 

over time and across sites of care, but may be limited if individuals 

change insurers. Medical claims data also often lack the clinical 

granularity needed to answer certain questions. 

FDA will explore strategies for filling gaps in data that may be difficult 

to obtain from currently used EHRs and medical claims data, including 

exploring the use of mobile technologies, electronic patient reported 

outcome tools, wearables, and biosensors.

Another important challenge is the difficulty in connecting or 

integrating various data sources contributing information about an 

individual patient. Unlike in some countries, patients in the United 

States do not have a single identification number that is used across 

private health care systems, similar to the social security number that 

are used by the U.S. government in the Medicare coverage system. 

Therefore, methods will need to be developed to address duplication 

of patient information in different data sources and to enable linking 

data about a single patient across data sources while protecting 

patient privacy. In health care settings in which a significant number 

of individuals move between health care plans and insurers, it may 

be difficult to capture outcomes that occur over years rather than 

months. FDA’s RWE Program provides an opportunity to address these 

challenges and improve the reliability and relevance of RWD sources.

Program Item. FDA will review and, where applicable, publish 

guidance on potential gaps in RWD sources and strategies to 

address them.
19

Framework for Evaluating RWD/RWE for Use in Regulatory Decisions

Potential for Study Designs 

Using RWD to Support Effectiveness 

Randomized Designs Using RWD

RWD can be used in a variety of ways and be included in different study 

designs. As previously noted, FDA sees promise in the opportunities 

created by pragmatic clinical trials, including broader inclusion/

exclusion criteria and streamlined data collection. 

Depending on the nature of the disease, the patient population, 

interventions, and other factors relevant to the trial, it may be possible 

to design hybrid trials that include both traditional and pragmatic 

clinical trial elements. FDA, therefore, will evaluate the strengths and 

limitations of pragmatic approaches using RWD, considering some 

important factors:

• What types of interventions and therapeutic areas might be wellsuited to routine clinical care settings? 

• What is the quality of data that can be captured in these settings?

• How many patients can be accessed (particularly when outcomes 

are rare)?

• What are the variations inherent in clinical practice?

When blinding of treatment is infeasible, FDA will seek to identify 

situations when bias resulting from lack of blinding can potentially 

be controlled using outcomes that are less likely to be influenced 

by knowledge of treatment assignment, such as clinically objective 

outcomes (e.g., stroke, tumor size). However, even when using 

objective assessments, approaches to ensure consistency in outcome 

ascertainment and reporting will be important. Finally, FDA will explore 

the feasibility of different randomization approaches in pragmatic 

clinical trials, including cluster randomization by institution or practice.

Program Item. FDA’s RWE Program will develop guidance on 

considerations for designing clinical trials that include pragmatic 

design elements and that generate evidence of effectiveness for 

regulatory decisions. FDA will explore pragmatic approaches to each 

stage of a clinical trial, including recruitment and enrollment of 

patients, strategies for facilitating interventions, and approaches to 

assessing outcomes.
Framework for FDA’s Real-World Evidence Program

20

10 See FDA guidance E 10 Choice of Control Group and Related Issues in Clinical Trials. 

11 Recent recommendations on good study practices from the International Society for Pharmacoeconomics and Outcomes Research 

(ISPOR) and the International Society for Pharmacoepidemiology (ISPE) characterize hypothesis-generating studies as exploratory studies. 

These are studies that do not hypothesize the presence of a specific treatment effect or its magnitude, but primarily serve as a first step 

to learn about possible treatment effects. In contrast, Hypothesis Evaluating Treatment Effectiveness (HETE) studies are defined as studies 

that evaluate the presence or absence of a prespecified effect or its magnitude and are designed to test a specific hypothesis in a specific 

population (Berget et al. 2017). HETE studies, as defined by ISPOR/ISPE, are the primary focus of the Pharmacoepidemiologic Guidance.

Non-randomized, Single Arm Trials with External RWD Control

External controls (e.g., historical controls) are a possible type of control 

arm in an adequate and well-controlled study.10 Typically, the external 

control arm uses data from past traditional clinical trials, but in some 

cases, RWD have been used as the basis for external controls. Using 

external controls has limitations, including difficulties in reliably 

selecting a comparable population because of potential changes in 

medical practice, lack of standardized diagnostic criteria or equivalent 

outcome measures, and variability in follow-up procedures. Collection 

of RWD on patients currently receiving other treatments, together 

with statistical methods, such as propensity scoring, could improve the 

quality of the external control data that are used when randomization 

may not be feasible or ethical, provided there is adequate detail to 

capture relevant covariates. 

Program Item. Guidance on the use of RWD to generate external 

control arms is also being considered.

Observational Studies

Pharmacoepidemiology studies are observational studies that examine 

how drugs are used and their effects in populations. Broadly speaking, 

pharmacoepidemiology focuses on selecting the appropriate data, 

design, and analysis to obtain a valid and unbiased answer to the 

question of interest using RWD. 

FDA has substantial experience evaluating and providing guidance 

on pharmacoepidemiology studies that use RWD in the context of 

safety. As previously stated, for the purposes of evaluating drug safety, 

FDA has already outlined its perspective on using RWD available in 

electronic health care data systems (medical claims and EHRs) for 

safety studies in the Pharmacoepidemiologic Guidance. This Guidance 

is primarily focused on studies designed to test prespecified hypotheses, 

as opposed to hypothesis-generating studies, and emphasizes that 

investigators should submit protocols to FDA before study initiation 

and final reports upon completion.11 FDA anticipates applying many of 
21

Framework for Evaluating RWD/RWE for Use in Regulatory Decisions

12 For more information about registration and display, see https://clinicaltrials.gov/.

the principles outlined in that guidance when evaluating observational 

studies to support changes in labeling that are within the scope of this 

framework (i.e., those that require evidence of effectiveness). 

In considering whether data gathered through observational 

study designs are appropriate to generate RWE for the purpose of 

supporting effectiveness determinations, FDA intends to evaluate 

multiple questions of interest that could affect the ability to draw a 

reliable causal inference, including, for example, the role of existing 

evidence (e.g., the natural history of the disease) and how the inclusion 

of a more diverse population can result in a heterogeneity of treatment 

effects making it difficult to detect smaller effect sizes.

In the context of retrospective observational studies using RWD, FDA 

will focus on critical questions such as the following: 

1. What are the characteristics of the data (e.g., contain data on a 

relevant endpoint, consistency in documentation, lack of missing 

data) that improve the chance of a valid result?

2. What are the characteristics of the study design and analysis that 

improve the chance of a valid result?

a. Can an active comparator improve the chance of a valid result?

b. Given potential unmeasured confounders in non-randomized 

RWD studies, as well as potential measurement variability in 

RWD, is there a role for non-inferiority designs?

3. What sensitivity analyses and statistical diagnostics should be prespecified for observational studies using RWD to generate RWE 

for effectiveness?

In addition to study design and data considerations, transparency 

about study design and analysis before execution is critical for ensuring 

confidence in the results. ClinicalTrials.gov was established to promote 

transparency by requiring many clinical trials to be registered publicly 

on this website and to post certain summary trial results after the trial 

is complete.12 Currently, there is no similar reporting requirement 

for observational studies, although parties can voluntarily register 

observational studies on ClinicalTrials.gov.
Framework for FDA’s Real-World Evidence Program

22

The potential lack of up-front transparency, especially in retrospective 

observational study design and conduct, coupled with the fact that 

retrospective analyses in electronic datasets can be conducted multiple 

times relatively inexpensively with varying study design elements, makes 

it possible to conduct numerous retrospective studies until the desired 

result is obtained and then submit only favorable results as if they 

were the result of a single study with a prespecified protocol. FDA will 

consider policies to prevent such practices, including recommendations 

from experts and other stakeholders. For example, FDA will consider 

the recently published task force recommendations from the 

International Society for Pharmacoeconomics and Outcomes Research 

(ISPOR) and the International Society for Pharmacoepidemiology 

(ISPE) on good procedural practices for treatment effectiveness studies, 

including transparency and reproducibility (Berger et al. 2017; Wang et 

al. 2017). FDA will also explore whether differences in retrospective and 

prospective observational designs require different approaches. 

Program Item. Adapting and building on the Pharmacoepidemiologic 

Guidance, FDA plans to issue guidance about observational study 

designs using RWD, including whether and how these studies might 

provide RWE to support product effectiveness in regulatory decisionmaking. FDA will also consider reporting requirements for such studies 

used to support effectiveness determinations.

Regulatory Considerations for 

Study Designs Using RWD

The use of EHRs, mobile health technology, and other electronic 

data-capture technology, together with new trial and study designs 

using RWD, has the potential to streamline and improve the efficiency 

of clinical studies. Nevertheless, these advances may also raise new 

questions about the applicability of certain regulatory requirements, 

including requirements for informed consent and appropriate oversight 

and monitoring. To facilitate the electronic capture of data from 

health care systems for research purposes while maintaining adequate 

documentation for FDA to validate the source and reliability of the 

data, FDA has already provided guidance on the use of electronic 

source data, electronic signatures, and EHRs. FDA has also issued the 

guidance Use of Electronic Informed Consent Questions and Answers

in December 2016 on using electronic informed consent. Additional 

guidance may be needed to address different study designs using RWD 

to generate RWE for effectiveness determinations.
23

Framework for Evaluating RWD/RWE for Use in Regulatory Decisions

Use of Electronic Source Data

There are several FDA guidance documents that address the use 

of electronic data in clinical investigations. In September 2013, 

FDA published the guidance Electronic Source Data in Clinical 

Investigations, which promotes capturing source data in electronic 

format and provides recommendations on the capture, review, 

and retention of electronic source data in FDA-regulated clinical 

investigations. In July 2018, FDA published the final guidance Use of 

Electronic Health Records in Clinical Investigations. This guidance 

provides recommendations on ensuring the integrity of EHR data that 

are collected and used as electronic source data in clinical investigations 

of medical products. 

FDA’s Part 11 regulations (21 CFR part 11) focus on the quality, 

authenticity, and reliability of electronic records from their point of 

creation to their modification, maintenance, archiving, retrieval, or 

transmission. In June 2017, FDA published a draft guidance Use of 

Electronic Records and Electronic Signatures in Clinical Investigations 

Under 21 CFR Part 11 - Questions and Answers. This guidance discusses 

procedures to help ensure that electronic records and electronic 

signatures are trustworthy and reliable and proposes a risk-based 

approach when deciding to validate electronic systems, implement audit 

trails for electronic records, and archive records that are pertinent to 

clinical investigations.
Framework for FDA’s Real-World Evidence Program

24

Program Item. FDA will consider whether these guidance documents 

adequately address concerns relevant to different study designs using 

RWD to generate RWE for drug product effectiveness determinations or 

whether additional guidance on use of electronic source data is needed.

Regulatory Considerations for 

Clinical Studies Generating RWE

FDA will also examine how FDA’s regulatory requirements are applied 

to data from randomized clinical trials that are integrated into the 

health care system and observational studies when they are intended 

to generate RWE for regulatory decision-making. For example, we will 

examine the use of risk-based13 and central monitoring for clinical trials 

that are integrated into the health care system.

In addition, there are multiple guidance documents that address FDA 

inspections, recordkeeping, and record retention requirements for 

regulated entities within the clinical trial enterprise. FDA will consider 

whether currently available guidance adequately addresses inspectionrelated concerns relevant to different sources of RWD used to generate 

RWE regarding the safety and effectiveness of drug products (e.g., 

informed consent, system access, records that must be available and 

viewable for review upon request at an FDA inspection).

Other regulatory considerations outside of FDA’s authority may affect 

the acquisition and use of RWD, including the Health Insurance 

Portability and Accountability Act (HIPAA), but these will not be 

addressed within the FDA framework.

Program Item. FDA plans to finalize the guidance Use of Electronic 

Records and Electronic Signatures in Clinical Investigations Under 

21 CFR part 11 - Questions and Answers and consider its applicability 

to different study designs. FDA will also issue additional guidance as 

needed on regulatory considerations raised by different study designs 

using RWD to generate RWE that is submitted to support drug product 

effectiveness.

13 FDA issued the guidance Oversight of Clinical Investigations—A Risk-Based Approach to Monitoring in August 2013.
25

Framework for Evaluating RWD/RWE for Use in Regulatory Decisions

Data Standards — Appropriate Data Standards for 

Integration and Submission to FDA

Along with other activities under FDA’s RWE Program, FDA will assess 

the data standards and implementation strategies required to use 

RWD/RWE at FDA, identify any gaps between those requirements and 

existing FDA systems, and recommend a path forward to ensure that 

RWD/RWE solutions are an integral part of the full drug development 

and regulatory life cycle at FDA. Activities for this work include:

• Identify data standards and implementation considerations that 

apply to proposed uses of RWD/RWE at FDA

• Review existing RWD/RWE-driven work, both internally and with 

external stakeholders, to identify gaps that need to be addressed 

◦ This evaluation could include projects such as development and 

use of relevant data standards, implementation strategies for 

applications and databases to connect with RWD sources, and 

strategies for coping with variations in data quality. 

• Collaborate with internal and external stakeholders to adapt or 

develop standards and implementation strategies for RWD/RWEdriven solutions at FDA 

• Integrate RWD/RWE-driven solutions with existing FDA systems 

◦ This activity could include assessment of topics such as Health 

IT strategies for RWD receipt and processing for potential use 

at FDA, impact on reviewer workload, and tools and training 

needed for FDA reviewers.
Framework for FDA’s Real-World Evidence Program

26

14 This meeting satisfies a PDUFA VI commitment to gather input on topics related to the use of RWE for regulatory decision-making.

Stakeholder Engagement 

Stakeholder engagement has been, and will continue to be, an 

important part of FDA’s RWE Program. In addition to FDA’s 

stakeholder engagement initiatives outlined in this section, the 

Appendix details research demonstration projects with stakeholders to 

facilitate the use of RWE in regulatory decision-making. 

Internal Engagement. In December 2017, FDA launched an internal 

website and outreach effort to engage FDA staff in activities supporting 

FDA’s program to evaluate the use of RWE for regulatory decisions. At 

the center of this effort was the launch of the RWE Subcommittee of 

CDER’s Medical Policy and Program Review Council, which includes 

leadership across CDER and CBER. The RWE Subcommittee will serve 

as a cross-cutting forum for RWE issues and will focus on CDER’s 

evaluation of RWE and guide policy development. Staff from the 

Office of New Drugs may consult with the RWE Subcommittee when 

evaluating the use of RWD and RWE to inform regulatory decisions 

about product effectiveness. The RWE Subcommittee will provide 

advisory recommendations as needed on whether underlying data, 

methods, and other study design elements are appropriate to provide 

support for a regulatory decision. The Subcommittee will also provide 

review divisions with additional resources to evaluate the use of RWE, 

consider how stakeholders propose to use RWE, and identify areas in 

which policy development will facilitate consistent practices.

External Engagement. In September 2017, through its cooperative 

agreement with the Duke Margolis Center for Health Policy, FDA 

convened a public meeting that explored the use of RWE for regulatory 

decisions.14 Representatives from industry, academia, patient advocacy 

groups, and other stakeholders discussed, among other things, 

opportunities and challenges associated with applying RWD and RWE 

“The efforts of the FDA should provide insights regarding potential uses of RWE for 

regulatory decisions, but are just one aspect of a larger challenge. If RWD and RWE 

are to be effectively leveraged for public health purposes, there will need to be 

shared learning and collaboration across clinicians, patients, health care systems, 

pharmaceutical companies, and regulators.”

Corrigan-Curay, J., Sacks, L., and Woodcock, J. 

JAMA Viewpoint, September 4, 2018
27

Conclusion

15 CTTI was established by FDA and Duke University as a public-private partnership in 2007. It comprises more than 80-member 

organizations and several individual patient/caregiver representatives working to identify and promote practices that will increase the 

quality and efficiency of clinical trials.

to demonstrate product effectiveness, including the data acquisition, 

study design, and analytic methods necessary to establish 

causal inference. The workshop informed development of FDA’s 

RWE framework.

FDA is also engaged in a project by the Clinical Trial Transformation 

Initiative (CTTI)15 to evaluate the use of RWD in randomized trials to 

generate RWE about medical products.

In addition, through the National Academy of Sciences, Engineering, 

and Medicine, FDA supported a three-series workshop titled 

“Examining the Impact of Real World Evidence on Medical Product 

Development.” FDA staff engaged in the planning of each workshop, 

and FDA leadership participated in these workshops.

• September 2017 — Workshop 1: Incentives

• March 2018 — Workshop 2: Practical Approaches

• July 2018 — Workshop 3: Application

FDA will continue to engage stakeholders through public meetings and 

other forums as part of its RWE Program.

Conclusion

FDA will continue its efforts to evaluate and explore ways and methods 

to optimize the utility of RWE. FDA has developed this Framework to 

guide its RWE Program. FDA already has taken the initiative to pilot 

projects that further the understanding of potential uses of RWD and 

RWE and will continue these efforts. Throughout this process, FDA is 

making stakeholder engagement a key aspect of its RWE Program. 
Framework for FDA’s Real-World Evidence Program

28

Glossary

Case Report Form (CRF): a specialized document in clinical 

research that is protocol driven and is used to collect the essential 

information for a clinical trial on each participant. All data on each 

patient participating in a clinical trial are documented in the CRF. CRFs 

can be in paper or electronic form.

Clinical Trial: a research study in which one or more human subjects 

are prospectively assigned to one or more interventions (which may 

include placebo or other control) to evaluate the effects of those 

interventions on health-related biomedical or behavioral outcomes. 

Clinical trials are interventional clinical studies. 

ClinicalTrials.gov: a data bank established to promote transparency 

by requiring many clinical trials to be registered publicly on the 

ClinicalTrials.gov website and to post certain summary trial results 

after the trial is complete.

Data Standard: a set of rules about how a particular type of data 

should be structured, defined, formatted, or exchanged between 

computer systems. Data standards make submissions predictable, 

consistent, and have a form that an information technology system or a 

scientific tool can use. 

Medical Claims Data: the compilation of information from medical 

claims that health care providers submit to insurers to receive payment 

for treatments and other interventions. Medical claims data use 

standardized medical codes, such as the World Health Organization’s 

International Classification of Diseases Coding (ICD-CM), to identify 

diagnoses and treatments.

Observational Study: a non-interventional clinical study design that 

is not considered a clinical trial. 

Observational Study, Prospective: a study in which the population 

of interest is identified at the start of the study, and exposure/treatment 

and outcome data are collected from that point forward. The start of the 

study is defined as the time the research protocol for the specific study 

question was initiated.
29

Glossary

Observational Study, Retrospective: a study that identifies the 

population and determines the exposure/treatment from historical data 

(i.e., data generated before the initiation of the study). The variables and 

outcomes of interest are determined at the time the study is designed. 

Patient Registry: an organized system that uses observational study 

methods to collect uniform data (clinical and other) to evaluate specified 

outcomes for a population defined by a particular disease, condition, or 

exposure and that serves one or more predetermined scientific, clinical, 

or policy purpose. Registries are generally defined either by diagnosis of 

a disease (disease registry) or usage of a drug, device, or other treatment 

(exposure registry). 

Patient Reported Outcome (PRO): a measurement based on a 

report that comes directly from the patient (i.e., study subject) about 

the status of the patient’s health condition without amendment or 

interpretation of the patient’s response by a clinician or anyone else. A 

PRO can be measured by self-report or by interview, provided that the 

interviewer records only the patient’s response. 

Real-World Data (RWD): data relating to patient health status and/

or the delivery of health care routinely collected from a variety of sources.

Real-World Evidence (RWE): clinical evidence regarding the 

usage and potential benefits or risks of a medical product derived from 

analysis of RWD.

Sentinel Initiative: a long-term effort to create a national electronic 

system for monitoring FDA-regulated medical products. 

Sentinel System: a system being developed and implemented 

in stages, to expand FDA’s existing postmarket safety surveillance 

capabilities by enabling FDA to actively gather information about the 

safety of regulated medical products once they reach the market.
Framework for FDA’s Real-World Evidence Program

30

References

Anglemyer A., Horvath H.T., and Bero, L. (2014). Healthcare Outcomes Assessed with Observational Study Designs 

Compared with Those Assessed in Randomized Trials (Review), Cochrane Database of Systematic Reviews, Issue 

4, Art No MR000034. doi:10.1002/14651858.MR000034.pub2.

Ball, R., Robb, M., Anderson, S.A., and Dal Pan, G. (2016). The FDA’s Sentinel Initiative—A Comprehensive Approach 

to Medical Product Surveillance, Clinical Pharmacology & Therapeutics, 99(3):265-268. doi:10.1002/cpt.320. 

Benson, K. and Hartz, A.J. (2000). A Comparison of Observational Studies and Randomized, Controlled Trials, New 

England Journal of Medicine, 342:1878-1886. doi:10.1056/NEJM200006223422506.

Berger, M.J, Sox, H., Willke, R.J., Brixner, D.L., Hans-Georg, E., Goettsch, W., Madigan, D., Makady, A., Schneeweiss, 

S., Tarricone, R., Wang, S.V., Watkins, J., and Mullins, C.D. (2017). Good Practices for Real-World Data Studies of 

Treatment and/or Comparative Effectiveness: Recommendations from the Joint ISPOR-ISPE Special Task Force 

on Real-World Evidence in Health Care Decision Making, Pharmacoepidemiology and Drug Safety, 26(9):1033-

1039. doi:10.1002/pds.4297.

Clinical Trial Transformation Initiative (CTTI) (2017). CTTI Recommendations: Registry Trials. Retrieved from 

https://www.ctti-clinicaltrials.org/files/recommendations/registrytrials-recs.pdf. 

Cooper, C.J., Murphy, T.P., Cutlip, D.E., Jamerson, K., Henrich, W., Reid, D.M., Cohen, D.J., Matsumoto, A.H., 

Steffes, M., Jaff, M.R., Prince, M.R., Lewis, E.F., Tuttle, K.R., Shapiro, J.I., Rundback, J.H., Massaro, J.M., 

D’Agostino, R.B., and Dworkin, L.D. (2014). Stenting and Medical Therapy for Atherosclerotic Renal-Artery 

Stenosis, New England Journal of Medicine, 370(1):13-22. doi:10.1056/NEJMoa1310753.

Eapen, Z.J., Lauer, M., and Temple, R.J. (2014). The Imperative of Overcoming Barriers to the Conduct of Large, 

Simple Trials. Journal of the American Medical Association, 311(14): 1397-1398. doi:10.1001/jama.2014.1030.

Eworuke, E. (2017). Integrating Sentinel into Routine Regulatory Drug Review: A Snapshot of the First Year Risk of 

Seizures Associated with Ranolazine [Power Point Presentation]. Retrieved from https://www.sentinelinitiative.

org/sites/default/files/communications/publications-presentations/Sentinel-ICPE-2017-Symposium-Snapshotof-the-First-Year_Ranexa-Seizures.pdf.

Food and Drug Administration, Center for Medicare Services, and Acumen Team. (2018). Centers for Disease Control 

and Prevention, Advisory Committee on Immunization Practices Meeting: Relative Effectiveness of Cell-cultured 

versus Egg-based Influenza Vaccines, 2017-18 [Power Point Presentation]. Retrieved from https://www.cdc.gov/

vaccines/acip/meetings/downloads/slides-2018-06/flu-03-Lu-508.pdf. 

Ford, I. and Norrie, J. (2016). Pragmatic Trials. New England Journal of Medicine, 375:454-463. doi:10.1056/

NEJMra1510059.

Fralick, M., Kesselheim, A.S., Avorn, J., and Schneeweiss, S. (2018). Use of Health Care Databases to Support 

Supplemental Indications of Approved Medications, JAMA Internal Medicine, 178(1): 55-63. doi:10.1001/

jamainternmed.2017.3919.
31

References

Franklin, J.M., and Schneeweiss, S. (2017). When and How Can Real World Data Analyses Substitute for Randomized 

Controlled Trials, Clinical Pharmacology & Therapeutics, 102(6):924-933. doi:10.1002/cpt.857.

Fröbert, O., Lagerqvist, B., Olivecrona, G., Omerovic, E., Gudnason, T., Maeng, M., Aasa, M., Angerås, O., Calais, 

F., Danielewicz, M., Erlinge, D., Hellsten, L., Jensen, U., Johansson, A.C., Kåregren, A., Nilsson, J., Robertson, 

L., Sandhall, L., Sjögren, I., Östlund, O., Harnek, J., and James, S.K. (2013). Thrombus Aspiration during STSegment Elevation Myocardial Infarction, New England Journal of Medicine, 369:1587-1597. doi:10.1056/

NEJMoa1308789.

Guadino, M., Di Franco, A., Rahouma, M., Tam, D.Y., Iannaccone, M., Deb, S., D’Ascenzo, F., Abouarab, A.A., Girardi, 

L.N., Taggart, D.P., and Fremes, S.E. (2018). Unmeasured Confounders in Observational Studies Comparing 

Bilateral Versus Single Internal Thoracic Artery for Coronary Artery Bypass Grafting: A Meta-Analysis, Journal of 

the American Heart Association, 7:e008010. doi.org/10.1161/JAHA.117.008010.

Gliklich, R.E., Dreyer, N.A., and Leavy, M.B., editors. (2014). Registries for Evaluating Patient Outcomes: A User’s 

Guide [Internet]. 3rd edition. Rockville (MD): Agency for Healthcare Research and Quality (US); 2014 Apr. 1, 

Patient Registries. Retrieved from https://www.ncbi.nlm.nih.gov/books/NBK208643.

Hemkens, L.G., Contopoulos-Ioannidis, D.G., and Ioannidis, J.P. (2016). Agreement of Treatment Effects for 

Mortality from Routinely Collected Data and Subsequent Randomized Trials: Meta-Epidemiological Survey, BMJ, 

352:i493. doi:10.1136/bmj.i493.

Hernandez, A.F., Fleurence, R.L., and Rothman, R.L. (2015). The ADAPTABLE Trial and PCORnet: Shining Light on a 

New Research Paradigm, Annals of Internal Medicine, 163(8):635-636. doi:10.7326/M15-1460. 

Izurieta, H.S., Thadani, N., Shay, D.K., Lu, Y., Maurer, A., Foppa, I.M., Franks, R., Pratt, D., Forshee, R.A., MaCurdy, 

T., Worrall, C., Howery, A.E., and Kelman, J. (2015). Comparative Effectiveness of High-dose versus Standarddose Influenza Vaccines in US Residents Aged 65 Years and Older from 2012 to 2012 Using Medicare Data: a 

Retrospective Cohort, Lancet Infect Dis, 15(3):293-300. doi:10.1016/S1473-3099(14)71087-4.

Izurieta, H.S., Wernecke, M., Kelman, J., Wong, S., Forshee, R., Pratt, D., Lu, Y., Sun, Q., Jankosky, C., Krause, P., 

Worrall, C., MaCurdy, T., Harpaz, R. (2017). Effectiveness and Duration of Protection Provided by the Liveattenuated Herpes Zoster Vaccine in the Medicare Population Ages 65 Years and Older, Clinical Infectious 

Diseases, 64(6):785-793. doi.org/10.1093/cid/ciw854.

Khozin, S., Abernethy, A.P., Nussbaum, N.C., Zhi, J., Curtis, M.D., Tucker, M., Lee, S.E., Light, D.E., Gossai, A., Sorg, 

R.A., Torres, A.Z., Patel, P., Blumenthal, G.M., and Pazdur, R. (2018). Characteristics of Real-World Metastatic 

Non-small Cell Lung Cancer Patients Treated with Nivolumab and Pembrolizumab During the Year Following 

Approval, Oncologist, 23(3): 328-336. doi: 10.1634/theoncologist.2017-0353.

Maggiono, A.P., Franzosi, M.G., Fresco, C., Turazza, F., and Tognoni, G. (1990). GISSI Trials in Acute Myocardial 

Infarction, CHEST Journal, 97(4), Supplement: 146S-150S. doi:10.1378/chest.97.4_Supplement.146S.
Framework for FDA’s Real-World Evidence Program

32

Mishra, P., Florian, J., Peter, J., Vainorius, M., Fried, M.W., Nelson, D.R., and Bimkrant, D. (2017). Public-Private 

Partnership: Targeting Real-World Data for Hepatitis C Direct-Acting Antivirals, Gastroenterology, 153(3):626-

631. doi:org/10.1053/j/gastro.2017.07.025. 

Moeny, D. (2017). Integrating Sentinel into Routine Regulatory Drug Review: A Snapshot of the First Year Venous 

Thromboembolism after Continuous or Extended Cycle Oral Contraceptive Use [Power Point Presentation]. 

Retrieved from https://www.sentinelinitiative.org/sites/default/files/communications/publicationspresentations/Sentinel-ICPE-2017-Symposium-Snapshot-of-the-First-Year_Lybrel.pdf.

O’Brien, E. and Curtis, L. (2017). Leveraging Electronic Health data in a Multinational Clinical Trial: Early Learnings 

from the HARMONY-OUTCOMES EHR Ancillary Study [Power Point Presentation]. Retrieved from https://www.

nihcollaboratory.org/Pages/GR-Slides-07-14-17.pdf. 

Przepiorka, D., Ko, C., Deisseroth, A., Yancey, C., Candau-Chacon, R., Chiu, H., Gehrke, B.J., Gomez-Broughton, 

C., Kane, R.C., Kirshner, S., Mehrotra, N., Ricks, T.K., Schmiel, D., Song, P., Zhao, P., Shou, Q., Farrell, 

A.T., and Pazdur, R. (2015). FDA Approval: Blinatumomab, Clinical Cancer Research, 21(18): 4035-4039. 

doi:10.1158/1078-0432.CCR-15-0612. 

Peto, R., Collins, R., and Gray, R. (1995). Large Scale Randomized Evidence: Large Simple Trials and Overview of 

Trials, Journal of Clinical Epidemiology, 48(1): 23-40. doi:10.1016/0895-4356(94)001.

Rovelli, F., De Vita, C., Feruglio, G.A., Lotto, A., Selvini, A., and Tognoni, G. (1987). GISSI Trial: Early Results and 

Late Follow-Up. Gruppo Italiano per la Sperimentazione della Streptochinasi nell’Infarto Miocardico, Journal of 

the American College of Cardiology, 10(5), Supplement 2: 33B-39B. doi:10.1016/S0735-1097(87)80426-6. 

Taylor, L.G. (2017). Integrating Sentinel into Routine Regulatory Drug Review: A Snapshot of the First Year 

Antipsychotics and Stroke Risk [Power Point Presentation]. Retrieved from https://www.sentinelinitiative.org/

sites/default/files/communications/publications-presentations/Sentinel-ICPE-2017-Symposium-Snapshot-ofthe-First-Year_Antipsychotic_stroke.pdf.

Wang, S.V., Schneeweiss, S., Berger, M.L., Brown, J., de Vries, F., Douglas, I., Gagne, J.J., Gini, R., Klungel, O., 

Mullins, C.D., Nguyen, M.D., Rassen, J.A., Smeeth, L., and Sturkenboom, M. (2017). Reporting to Improve 

Reproducibility and Facilitate Validity Assessment in Healthcare Database Studies V1.0, Pharmacoepidemiology 

and Drug Safety, 26(9):1018-1032. doi:10.1002/pds.4295.
33

References

FDA Guidances Noted in this Document 

In August 2017, FDA issued the guidance Use of Real-World Evidence to Support Regulatory Decision-Making for 

Medical Devices. This guidance clarifies how the FDA evaluates real-world data to determine whether they are 

sufficient for generating the types of real-world evidence that can be used in FDA regulatory decision-making for 

medical devices. 

In May 2013, FDA issued the guidance Best Practices for Conducting and Reporting Pharmacoepidemiologic 

Safety Studies Using Electronic Healthcare Data (Pharmacoepidemiologic Guidance). This guidance includes 

recommendations for evaluating the data sources used in pharmacoepidemiologic safety studies. 

In December 2009, FDA issued the guidance Patient-Reported Outcome Measures: Use in Medical Product 

Development to Support Labeling Claims. This guidance describes how the FDA reviews and evaluates existing, 

modified, or newly created PRO instruments used to support claims in approved medical product labeling.

In May 2001, FDA issued the guidance E 10 Choice of Control Group and Related Issues in Clinical Trials. This 

guidance is intended to assist applicants in choosing a control group for clinical trials intended to demonstrate the 

efficacy of a treatment.

In December 2016, FDA issued the guidance Use of Electronic Informed Consent Questions and Answers. This 

guidance provides recommendations on the use of electronic systems and processes that employ multiple 

electronic media to obtain informed consent for both HHS-regulated human subject research and FDA-regulated 

clinical investigations of medical products.

In September 2013, FDA issued the guidance Electronic Source Data in Clinical Investigations. This guidance 

promotes capturing source data in electronic format and provides recommendations on the capture, review, and 

retention of electronic source data in FDA-regulated clinical investigations.

In July 2018, FDA issued the guidance Use of Electronic Health Records in Clinical Investigations. This guidance 

provides recommendations on ensuring the integrity of EHR data that are collected and used as electronic source 

data in clinical investigations of medical products. 

In June 2017, FDA issued the draft guidance Use of Electronic Records and Electronic Signatures in Clinical 

Investigations Under 21 CFR Part 11 - Questions and Answers. This guidance discusses procedures to help ensure 

that electronic records and electronic signatures are trustworthy and reliable and proposes a risk-based approach 

when deciding to validate electronic systems, implement audit trails for electronic records, and archive records 

that are pertinent to clinical investigations.

In August 2013, FDA issued the guidance Oversight of Clinical Investigations- A Risk-Based Approach to Monitoring. 

This guidance assists sponsors of clinical investigations in developing risk-based monitoring strategies and plans 

for investigational studies of medical products.
Framework for FDA’s Real-World Evidence Program

34

Appendix 

DEMONSTRATION PROJECTS

The following are demonstration projects that FDA is funding to inform 

the Agency about the use of RWD and RWE.

Current Demonstration Projects to Inform Assessment of 

Fitness for Use in Regulatory Decisions

Standardization and Querying of Data Quality Metrics and 

Characteristics for Electronic Health Data: One of the challenges in 

using data from EHRs is that no standards exist for describing the 

quality and completeness of electronic health data. Understanding 

the characteristics of a data source is critical for investigators in 

their determination about whether the data are fit for a specific use. 

Effective use of the growing number of data sources and distributed 

networks will require adoption of a uniform approach to describing 

the quality characteristics of electronic health data, as well as the data 

capture characteristics at the institutional, provider, and health plan 

levels and data domain level. FDA is supporting a project to develop, 

test, and implement a standards-based approach to describing data 

characteristics and quality and presenting relevant metrics. 

Source Data Capture from EHRs: Using Standardized Clinical 

Research Data: Another challenge with using EHRs is that they were 

developed for clinical care, not clinical research. Despite overlap in the 

information needed for both, the way information is captured may not 

always be optimized for research use. FDA is collaborating with the 

University of California in San Francisco’s (UCSF) Carol Franc Buck 

Breast Care Center on developing the source data capture from EHRs 

(One Source) program. 

This project will demonstrate a single-point, data capture approach 

from the EHR to an electronic data capture system (EDC) using open, 

consensus-based standards. This could allow data collected in the 

EHR to be used as part of an FDA-regulated clinical trial, eliminating 

the need for duplicate entry, and potentially saving time, money, and 

eliminating an opportunity for errors. This project will provide a cloudbased HIPAA and 21 CFR Part 11-compliant tool for researchers of 

patient-centered outcomes to seamlessly integrate EHR and 

EDC systems.
35

Appendix

HARMONY-OUTCOMES Ancillary Study (NCT02465515): FDA is 

supporting research to examine the feasibility of using EHRs in clinical 

research. Through a collaboration between the Duke Clinical Research 

Institute and GlaxoSmithKline (GSK), the HARMONY-OUTCOMES 

Ancillary Study will examine how EHRs at selected trial sites in GSK’s 

cardiovascular outcome trial for albiglutide could have been used 

to identify eligible participants, populate baseline characteristics of 

participants into an electronic CRF, and identify clinical endpoints 

(O’Brien and Curtis 2017). 

INFORMED Collaborations in Oncology: FDA is collaborating on the 

use of RWD to generate RWE. For example, as part of the Information 

Exchange and Data Transformation (INFORMED), the Oncology Center 

for Excellence is collaborating with Flatiron Health to examine how RWD 

can be used to gain insights regarding the safety and effectiveness of new 

cancer therapies (Khozin et al. 2018). In addition, in June 2017, FDA’s 

partnership with CancerLinQ, the American Society of Clinical Oncology’s 

big data initiative, was announced. FDA and CancerLinQ will be using 

real-world, aggregate, de-identified patient care data from oncology 

practices to look at a variety of issues related to the appropriate use of 

newly approved therapies. The initial focus will be on immunotherapy 

agents approved for melanoma. By working with these data to explore 

questions around the use of new oncologic agents, FDA will better 

understand how to evaluate the relevance and quality of these data. 

FDA-Catalyst Program: The FDA-Catalyst is an initiative that leverages 

the Sentinel infrastructure and other capabilities of the Sentinel System 

while supplementing it with data from interventions or interactions 

with health plan members, providers, or both. FDA-Catalyst provides a 

platform to answer a wider range of questions than can be addressed by 

the Sentinel System data alone. 

FDA MyStudies — Mobile Application: As part of the FDA-Catalyst 

program, the FDA MyStudies mobile device application was developed 

to support informed consent as well as secure data collection from 

patients at multiple study sites or data partners in real time. Specified 

administrators can download patient responses from a patient data 

storage environment that is compliant with the HIPAA and Federal 

Information Security Management Act and can be linked to existing 

electronic health data. The code for MyStudies is open source so 

software developers can improve upon its capabilities. MyStudies has 

an associated web-based study design portal that allows for the creation, 

distribution, and modification of questionnaires so it can be configured 

for different therapeutic areas and health outcomes, which reduces 

software development hurdles for non-FDA users. As an initial proof of 
Framework for FDA’s Real-World Evidence Program

36

concept for drug safety and effectiveness research, pregnant women used 

the app to provide information on exposures such as prescription and 

OTC drug use and health care outcomes during the last quarter of 2017. 

Current Demonstration Projects to Inform Assessment of 

Study Designs Using RWD to Support Effectiveness

Implementation of a Randomized Controlled Trial to Improve 

Treatment with Oral Anticoagulants in Patients with Atrial Fibrillation 

(IMPACT-AFib) (NCT02082548): Through FDA-Catalyst, FDA is 

currently supporting a randomized trial using data from the Sentinel 

System to test whether a patient and provider educational intervention 

can increase anticoagulant use for individuals who, per the data within 

the Sentinel System, have atrial fibrillation and are at increased risk of 

stroke. Not only is this a critical public health question, it is also a proof 

of concept trial for conducting interventional effectiveness trials using 

the Sentinel infrastructure. 

Hepatitis C Therapeutic Registry and Research Network (HCVTARGET): HCV-TARGET is an international research consortium 

created to inform the ongoing transformation of hepatitis C treatment 

and research. HCV-TARGET includes 112 academic and community 

sites in 31 states, Puerto Rico, Canada, and Europe as well as 

partnerships with multiple industry sponsors, FDA, and the patient 

advocacy community. In 2011, HCV-TARGET established a national 

registry to observe patients undergoing hepatitis C treatment and 

coordinate real-world monitoring on a national scale for new therapies 

as they enter the market. Since inception in 2011, HCV-TARGET has 

enrolled more than 10,000 patients treated with HCV direct-acting 

antiviral-based regimens approved by FDA (Mishra et al. 2017). 

Effectiveness Research with Real World Data to Support 

FDA’s Regulatory Decision-Making: In May 2018 FDA launched 

a partnership the Brigham and Women’s Hospital/Harvard Medical 

School that aims to identify and reproduce a representative set 

of approximately 30 phase 3/4 randomized control trials with 

observational RWD analyses to delineate the circumstances under 

which observational studies using existing electronic administrative 

health care data can replicate the results of clinical trials. The 

investigators will develop and make a scalable analytics platform 

available to FDA staff for evaluation of RWD utility. 
37

Appendix

Current Demonstration Projects on 

Data Standards and Use of RWD

Common Data Model (CDM) Harmonization Project: Various 

initiatives have created networks of RWD to capture data across 

health care delivery systems. Participating organizations within these 

networks map their databases to a CDM that provides consistent 

format and content. This project seeks to develop a meta-CDM that 

enables a researcher to make a single query that is usable across four 

large RWD networks: FDA’s Sentinel System, the Observational Health 

Data Sciences and Informatics (OHDSI), PCORnet, and the Accrual of 

Patients to Clinical Trials Network (ACT)/ Informatics for Integrating 

Biology & the Bedside (i2b2). This project is aimed to achieve a 

sustainable data network infrastructure, promote interoperability and 

foster the creation of a Learning Health System. An overview of the 

project is accessible on the OHDSI website.
U.S. Food and Drug Administration

10903 New Hampshire Ave. 

Silver Spring, MD 20993

www.fda.gov