FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 1 

FDA Webinar: Use of Real-World Evidence to Support Regulatory DecisionMaking for Medical Devices

Moderator: Irene Aihie

October 10, 2017 

1:00 pm ET 

Coordinator: Welcome and thank you for standing by. At this participants, will be on a 

listen only mode until the question and answer portion. If at that time you 

would like to ask a question, press star one. Today’s conference is also being 

recorded. If you have any objections, please disconnect at this time. And 

now I would like to turn the call over to you host today to Miss Irene Aihie. 

Ma’am you may begin. 

Irene Aihie: Hello and welcome to today’s FDA webinar. I’m Irene Aihie of CDRH’s 

Office of Communication and Education. 

On August 31, 2017, the FDA issued the final guidance documents used as 

real work evidence to support regulatory decision making for medical devices. 

A guidance plan by policy agencies determines whether real world data 

making sufficient for use in regulatory decisions. To adopt changes to 

evidentiary standards we use to make those decisions. 

It’s also clarifies how you plan to evaluate real world data to determine 

whether it may be sufficiently relevant and reliable for various regulatory 

decisions and when an investigation of the extension IDE may be needed to 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 2 

correct and use real world data for purposes of determining the safety and 

effectiveness of a device. 

The purpose of this webinar is to help clarify the agency’s recommendations 

to make it a constant to the guidance documents. Today Josh Chetta a 

biomedical engineer in the Office of Device Evaluation and Alex Hu a 

medical device epidemiologist from the Division of Epidemiology in the 

Office of Surveillance and Biometrics here at CDRH will present an overview 

of the final guidance. 

Following the presentation, we will open the lines for your questions related 

to the information provided during the presentation. Additionally, there are 

other interest subject matter experts to assist us with the Q&A portion of our 

webinar. Now I give you Josh. 

Josh Chetta: All right. Thank you for that introduction. So briefly our agenda for the talk. 

First, we’ll go over background. Cover the differences between the draft and 

the final version of the guidance document, discuss the regulatory framework 

in which this information fits and then we’ll cover some highlights of the final 

guidance including examples and issues of data quality. 

But first we want to highlight the publication of this document is only part of 

a larger initiative to leverage non-traditional data sources for regulatory 

purposes. Increasing access to and use of real world evidence is a part of 

CDRH’s strategic priorities and recent legislation includes sections addressing 

real world evidence as well.

The definitions from the guidance document are as follows, real world data is 

defined as data relating to patient health status and/or the delivery of health 

care routinely collected from a variety of sources. And real world evidence is 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 3 

defined as clinical evidence regarding the usage and potential benefits or risks 

of a medical product derived from analysis of real world data. Taken together 

a functional definition for real world evidence is an analysis of information 

collected from routine clinical care. And these definitions in the guidance 

make clear that for information to be considered evidence, care must be taken 

at multiple stages - during the data collection process in the approach to the 

analysis and in any use to evaluate how the resulting information may or may 

not prove to be adequate to support the regulatory decision at hand. And this 

is no different from a standard clinical investigation. 

The structure of the guidance covers the scope, background, general considerations and then it 

goes into when an investigational device exemption may be needed and 

covers data quality and finally goes on to examples. We’ll largely stick to this 

structure for our presentation today. 

The guidance discusses how the FDA will evaluate whether real world 

evidence is of sufficient quality to inform a regulatory decision for a particular 

medical device, but the document is not all inclusive and we want to make 

clear that it does not address the use of non-clinical data, adverse event 

reports, secondary use of previously collected clinical trial data or systematic 

literature of these.

In addition, it does not endorse a specific study design, study conduct or 

analytical methodology that could be used for generating or interpreting real 

world data. 

As background we would like to take a minute to consider where real world 

evidence comes from and why we’re interested in it.

First we want to point out the vast majority of medical devices don’t require 

clinical data to be provided to the agency before being marketed in the United 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 4 

States. However, for devices where clinical data may be necessary to support 

a marketing application, for example novel or high-risk devices, the traditional 

pathway to collect clinical data is relatively linear. 

The sponsor conducts a clinical investigation, sometimes under an IDE, and 

the information collected in that study is then evaluated in a marketing 

application that is subsequently submitted to the agency. Additional post 

market information such as confirmatory evidence may be collected in a postapproval study or safety issues maybe evaluating post markets surveillance

activities.

These clinical studies tend to be well controlled trials conducted in a specific 

patient population with clinical endpoints designed to support very specific 

labeling claims and indications for use. These results may not be 

generalizable beyond specific patient population under study and new trials 

may be needed for label changes and device modifications.

We know that separate from medical research, the routine use of medical 

devices in the United States health care system generates a huge amount of 

data found in sources like electronic health records, laboratory test results and 

claims databases.

The agency recognizes how important these data sources are because they 

include information covering the experience of physicians and patients on the 

actual use of devices in practice and this may be different from what we see in 

research studies. This constellation of information sources are used to inform 

clinical decision making, develop new hypotheses for testing and drive device

innovation. 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 5 

This innovation may result in new clinical studies and in some cases high 

quality evidence generated from real world device use may be submitted 

directly in a marketing application. So rather than a linear pathway in which 

multiple one-off trials must be repeatedly conducted, the question is whether 

we can take advantage of this information being generated to evaluate device 

use dynamically as population and use change through the total product life 

cycle.

The guidance document makes clear that we will consider any valid scientific 

evidence regardless of its origin without preference, but for some regulatory 

questions, a traditional clinical study may be the most effective mechanism to 

collect the necessary information. There are benefits and shortcomings of 

traditional clinical studies, as well as information from real world device use 

and this document is intended to communicate our expectations to facilitate 

the use of these non-traditional data sources.

The draft guidance document was released last summer and many commenters

provided feedback to the agency. It was important that these be addressed and 

incorporated into the final version of the document and we’ll go over a few of 

the more common ones now. 

The public comments that we received can be broadly summarized into four 

categories. We were asked whether this guidance applies only to certain 

classes or types of devices. We were also asked to provide a deterministic 

score sheet for when real world evidence is applicable to a specific regulatory 

question. There were concerns about the potential for the use of real world 

evidence to lower the evidentiary standard that we use to make regulatory 

decisions and we were also asked to clarify how the use of real world 

evidence impacts the IDE process. 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 6 

First the final guidance makes clear that real world evidence may be provided 

as support for any regulatory decision related to a device and this includes 

diagnostics, software, class 1 and class 2 devices, as well as for pre-market 

and post-market decisions. 

Second, it was not possible to develop a scoring tool or pass/fail criteria for 

the use of real world evidence. However, we added additional detail on what 

aspects the agency will consider when evaluating real world data and real 

world evidence. 

Regarding the question of data quality, the guidance is explicit that the 

evidentiary standard for all regulatory decision remains unchanged. Using 

real world evidence does not lower the bar and we also expanded sections of 

the document that discuss data quality and applicability to a question and 

these will be discussed in more detail later in this presentation.

Finally, the guidance includes an expanded section on when the collection of 

data from real world use of device might require an IDE. We’ll also discuss 

this is more detail later.

Now moving from the background information to the information contained in 

the guidance document, we want focus our attention now on what many of the 

public comments asked about mainly data quality. In this section we’ll cover 

how the agency intends to evaluate the quality of information collected from 

routine care. 

A reminder that the framework within which we make our evaluations is laid 

out in our governing statues and regulations. 21CFR makes it clear that a 

sponsor may submit any form of evidence to the agency but that only valid 

scientific evidence will be used to support a regulatory decision. This 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 7 

regulation goes to list the range of potentially acceptable sources and this 

includes information from well-controlled investigations all the way down to 

reports of significant human experience. Finally, this regulation lists data 

sources which are insufficient to be considered for use in evaluating the 

regulatory decision. 

And we’ve highlighted these sections to emphasize that in CDRH we’ve 

always accepted and reviewed clinical data from a variety of sources and part 

of that review has always focused on the quality of the data. This document 

does not change that rather it formalizes the process we’ve been using and 

lays out our expectations. 

Simply put, information provided to us needs to be fit for purpose. It must be 

complete, consistent, accurate, and needs to contain all critical data elements 

to evaluate a medical device and any associated claims. Within any benefit 

risk decision, the information must be relevant and reliable such that an 

evaluation safety and effectiveness can be undertaken. 

What this means is that certain decisions will require a higher-level data than 

others. For instance, the first of the kind class 3 device may require more 

information than post-market surveillance for a lower risk device. The 

guidance includes significant detail on how data relevance and reliability will 

be assessed and we’ll spend some time walking through those ideas but we 

want to point out that quality needs to be built in into the entire evidence 

generating process: from primary collection through compilation analysis and 

use. 

So relevance is the idea that the data adequately addresses the applicable 

regulatory question or requirement in part or in whole. In other words, is the 

information relevant to the question being asked? Broadly the data should 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 8 

include appropriate variables collected with sufficient detail to capture the 

device exposure and outcome of interest. For example, the endpoints must be 

clinically relevant and be well-defined in a consistent and meaningful way. 

The patient population should be appropriate and representative of any 

potential labeled use and finally the data must be amenable to sound clinical 

and statistical analysis and the results must be interpretable using informed 

clinical and scientific judgement. If the data are considered relevant to the 

specific regulatory question, then we’ll evaluate whether we can have 

confidence in the underlying data and data analysis used. 

The data refers to (unintelligible) which play into overall reliability. Data 

accrual and data assurance. We’ll first discuss data accrual which refers to 

how the data were collected. We’ll consider multiple aspects of the collection 

process in our evaluation. For example, we’ll look for pre-specification of 

standardized data elements, the use of the common definitional framework 

and data dictionary, and the timeframe for collection. 

In addition, we’ll consider the data sources and technical method used to 

capture the data. The patient selection process should be appropriate and 

should limit bias in the study population and the appropriate patient 

protections must be in place to protect the rights, welfare, safety and privacy 

of the patients. 

The other aspect of data reliability is data assurance and quality control. And 

this refers to ensuring adequate people and processes are in place to minimize 

bias and errors and to ensure data integrity. One consideration is the process 

used to populate data elements. Were they abstracted manually or was an 

algorithm used to automatically populate them? Additionally, there needs to 

be documentation to, documentation of and adherence to source verification 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 9 

procedures. The data should be complete and consistent for the specified 

analysis. The data should be consistent and poolable across site and over time 

and there should be ongoing training programs to ensure personnel are 

knowledgeable of the data collection, handling, and transmission procedures. 

These lists of example considerations are not intended to be exhausted by any 

means. We encourage you to look carefully through the guidance document 

and contact us with any questions through pre-submission process. 

Additionally, although they tend to be focused on registries we encourage you 

to consult the published resources from other stakeholders that include best 

practices and recommendations for addressing issues related to data quality. 

The guidance document lists a number of resources for reference. 

We'll now move on to a discussion of when collection of data from routine 

clinical care might require an IDE. In general, whether the collection of real 

world data requires an IDE depends on the intended use of that information. 

The FDA does not regulate how health care practitioners use legally marketed 

devices or make health care decisions within the context of the legitimate 

patient practitioner relationship.

Health care practitioners regularly use medical devices off label in the course 

of normal medical care and we consider these types of interactions to be an 

important source of real world data. These data may be adequate to support 

regulatory decisions if found to be of sufficient quality. 

It's important to point out that the FDA's regulations are only one part of the 

legal framework governing protection of patients and the patient's protected 

health information in the U.S. It was not within the scope of this guidance to 

address the entirety of human subjects protection questions related to real 

world data collection. Rather the guidance document focuses on when an IDE 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 10

might be needed for activities related to the collection of real world data or 

real world evidence.

21CFR812 gives the agency authority to grant an investigational device 

exemption allowing clinical investigations of medical devices to determine 

safety and effectiveness. Whether the collection of real world data for a 

legally marketed device requires an IDE depends on the particular situation 

and use. If the device is being used in the normal course of medical practice, 

an IDE would likely not be required. 

However, if data are being gathered to determine the safety and effectiveness 

of the device and the process for gathering the data would include treatment 

decisions is likely not within the normal course of medical practice and an 

IDE may be required. Given the complexity and nuance of these issues, we 

strongly encourage sponsors, investigators and IRBs to contact us with 

questions regarding what activities might require an IDE.

We welcome the opportunity to work with external stakeholders on these 

issues. 

Alex will now discuss examples of how real world evidence has been used for 

regulatory decision making. 

Alex Hu: Okay. Thank you Josh. We believe that real world evidence can be used to 

aid FDA in our regulatory decision making under the right conditions in areas 

including but not limited to clearing or approving new devices, supporting 

labeling expansion, supplementing the total evidence required for clearances 

or approvals, and in a postmarket setting.
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 11

My name is Alex Yuzhi Hu and here I'm going to talk about four cases that 

were generalized from the actual use of real world evidence. 

Labeling expansion is a big deal and it is usually supported by trials with 

limited sample size and in many cases it can be hard to yield sufficient data 

that is needed to support such a decision. 

There was a class 3 device. We approved it based on traditional clinical trials 

with focused patient selection criteria. And, due to this reason, we approved 

indications for use that was quite focused as well. But after the device entered 

the market, it has seen an expansion of clinically acceptable use beyond what 

was approved. We discussed with the sponsor about labeling expansion but 

the scarcity of clinical study data made approval through traditional venue 

unlikely. 

To address this lack of information, we worked with the company and 

successfully identified an existing national registry that collects clinical 

information for all patients with this device and other similar devices as 

participating institutions. It was also linked with claims data through a 

validated matching algorithm such that it made possible the evaluation of 

long-term performance. Right now, a study using the data collection and 

analysis infrastructure of this registry was already initiated to support this 

labeling expansion. 

As for control group, it needs to be there for us to make various regulatory 

decisions. But it doesn't mean that a control group has to be created based on 

randomization or it has to be enrolled prospectively. Here in the next 

example, I will show you such a case. In this example, we determined that 

clinical data was needed to support an approval decisions for a significant 

change to an existing device and we decided that we would like to see the 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 12

comparison between this device and other clinical alternatives that were 

already on the U.S. market, and this rendered a use of routine care data 

possible. To our knowledge, a registry is already out there capturing data on 

all uses of medical devices with similar intended uses.

From this registry, we drew a non-randomized concurrent control group 

consists of targeted on-marketed devices as comparator. To make sure that 

the registry would provide sufficiently relevant and reliable data to the control 

population. The quality of this registry was evaluated by us and also by the 

sponsor according to the factors cited in the guidance. 

We determined that the quality is sufficient enough for making such a 

decision, The sponsor did not have to collect additional data from these

patients or influence the course of their clinical care in any way. So even 

though the patient's that received the investigational device were enrolled 

under an approved IDE, the control group did not need an IDE because the 

registry was just collecting data on patients with FDA approved devices under 

the normal course of care.

No study is perfect, there will always be a need here and there for additional 

data. In many cases, more evidence will lead to decision making that is better 

and faster. Here in this example, let's talk about where the use of real world 

evidence provides supplementary evidence. There was a groundbreaking 

class 3 device under review for a new indication, to which, the sponsor 

provided FDA with somewhat limited data from a prospective clinical trial 

particularly in the sense that it did not enough follow-up information and less 

than clear information on a control population. 

To overcome this, we identified a pre-existing data source that was already 

collecting and reporting data on the control therapies. The registry data was 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 13

used to supplement and help interpret the original study results allowing the 

FDA to come to an appropriate regulatory decision without requiring 

additional clinical trial data. In this case, real world evidence provided crucial 

clinical information that is otherwise unavailable to us. It expedited the 

decision-making process, protected the patient's health while also facilitated 

medical device development. 

As we know, postmarket requirements could serve the patients well but also 

reduce the pre-market burden while still ensuring that the statutory standard of 

reasonable assurance of safety and effectiveness is met. But how could a 

post-marketing requirement also be very helpful for the regulatory decision 

making on many other devices? In this last example, let's take a look at a 

postapproval surveillance case where the use of real world evidence made 

earlier device approval possible and much more. 

 It was a first of a kind class 3 device approved based on prospective 

randomized controlled clinical trial data. Early in the PMA review process we 

began to consider postmarket commitments. We decided to use data 

generated from routine care to support the postmarket requirement as opposed 

to resorting to a traditional stand-alone post approval clinical trial. With the 

help of FDA, a registry was launched that generated real world data that could 

meet FDA's data requirements on this case. Because the new registry is able 

to capture information on all patients receiving devices with similar design 

and indications. This data source provided not only an earlier approval 

decision for this device but also the necessary infrastructure for making 

regulatory decisions in many other similar devices. This registry has since 

been used to collect surveillance data, support indication expansion, even 

embedded prospective clinical investigations under IDE for subsequent 

devices with similar designs and indications all because we started using real 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 14

word evidence on one single case. All of this cannot happen if a traditional 

stand-alone post-approval study was required instead. 

To sum it up we believe that there is opportunity for greater use of real world 

evidence and this guidance is designed to provide the framework to help all 

stakeholders assess relevance and reliability of real world evidence. The 

center has been supporting various efforts to facilitate the development of 

infrastructures and tools to better access and use real world evidence for 

regulatory decision making including the development of NEST. Please 

contact us through pre-submission or let us know how we can help you 

through the email addresses provided below. Thank you for your time. 

Irene Aihie: We'll now take questions. 

Joshua Chetta: And while we're waiting for those questions to get queued up, we've 

mentioned a few times the pre-submission process and we wanted to point out 

that there is a guidance document: Requests for feedback on medical device 

submission, the pre-submission program and meeting with Food and Drug 

Administration staff. This lays out the process and recommended structure for 

pre-submission requests and the types of questions that should be and would 

be appropriate for that type of submission. So we encourage you to contact us 

early if you have any question about a specific submission.

Coordinator: Thank you and for audio questions, if you would like to ask a question please 

press star one and record your name at this time. One moment we have an 

audio question and thank you the first question comes from (Chris) your line 

is open. 

(Chris): Hi I was wondering if you could describe a process for post-approval 

commitments for class 2 devices.
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 15

Joshua Chetta: Thanks for the question. I think that’s a little bit out of scope for what we're 

talking about today. 

(Owen Faris): Hi this is (Owen Faris), and I'm the director of the clinical trials program in 

CDRH. So I agree with (Josh) that if we're talking generally about postapproval requirements for class 2 devices out of the scope of today's 

conversation and we won't touch on that directly. I will say that I think real 

world evidence can be used to support regulatory decisions for any class of 

device. So class 2 and class 3 devices are certainly the subject of this 

guidance and the principles apply in terms of obtaining quality data that can 

answer the right questions. 

Chris: We have another question in the room. 

Woman 1: So just a follow-up on that, is there any example that you could give on real 

world evidence of what would be acceptable for a class 2 device that doesn't 

have, for example, device registry or something like that. 

Josh Chetta: Again there have been decisions made for class 2 devices in the center using 

real world evidence. After that information, the information included in that 

submission was found to meet the kind of the parameters that we laid out here 

for quality, relevance, reliability, informed consent - all of those things, if 

necessary.

(Owen Faris): This is (Owen Faris) so I will say that you know this is an area where the use 

of real world evidence is an evolving landscape and certainly I think this gets 

to the point of your question, certainly some of the earliest experiences with 

using real world evidence and regulatory decisions came from (PMA) devices. 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 16

But there is a strong interest in the agency to support the expanded use of real 

world evidence into the class 2 space. 

We have seen early examples of that in certain places but we strongly 

encourage you - if you're a manufacture of a class 2 device and would like to 

think about how real world evidence can be used to support regulatory 

decisions in that space, please come talk with us. We're very interested in 

engaging - particularly through the pre-submission process to talk about how 

we can make that work. 

Chris: Great. Thanks for your help. 

Coordinator: We do have another audio question. It comes from (Wade) your line is open. 

(Wade): And thank you. I have two questions. The first question is that I'm working 

on oncology and can you comment on how this guidance will apply for the 

companion diagnostic in oncology. My second question is regards to that, you 

gave lots of examples of using the registry data. I'm wondering if any 

examples using some other data sources, like a EHR databases or claim 

databases which has been used before in your experience? Thank you. 

Mike Waters: Yes, hi this is Mike Waters from the Office of In Vitro Diagnostics and 

Radiologic Health (OIR). There have been a lot of efforts to help figure out 

how to leverage realworld evidence from electric health care records (EHRs) 

and registries when it comes to companion diagnostics. This is an ongoing 

active space of development. 

There have also been some submissions that have gone through the process of 

leveraging data from appropriately curated next generation sequencing

databases in the area of companion diagnostics which (at some levels) fits into 

the concept of realworld evidence. But this is like I was saying before, this is a 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 17

space that's currently under development. If you have something that you 

would like to utilize, then engaging through the pre-submission process would 

be an extremely valuable effort. 

(Owen Faris): So I'm going to take this. This is (Owen Faris) again, so I'm going to take the 

part two of your question and you know that much of the example information 

we provided was related to registry data and I think it's clear that's where our 

earliest experiences and our most extensive experience with real world data 

have been because registries are generally systematically designed to gather 

consistent high quality information that may be more complicated to gather in 

other ways but I'll say a few things about that. 

So one is that many of the registries that gather information are actually linked 

to claims data to gather later data points that the registry itself doesn't gather 

and so we do have experience with linking registries to claims data to gather 

some of that information. There's also a lot of interest in how can we start 

using electronic health records as the primary data source for real world 

evidence and how do we gather that in a high-quality way. 

So that's a broad effort. The agency's very engaged and very interested in and 

we've seen some examples of that, but I think we're going to see a lot more in 

the future. The National Evaluation System for Health Technologies which is 

just getting off the ground this year, just recently announced a couple of pilot 

projects which are directly focused on that question of how do you get high 

quality data from an electronic health care system and translate that into 

information that can be used for regulatory purposes. 

So, I definitely think that's where we're going. We're seeing some of the 

earliest signals there. But our experience with using real world data with high
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 18

quality submissions thus far has a lotof Registries that have played a major 

role.I think we're going to start seeing that shift in the future.

(Wade): Thank you.

Coordinator: Thank you. As a reminder if you would like to ask a question, please press 

star one. All questions will be taken through the phone. Again if you would 

like to ask a question, please press star one. The next question is from (Julie), 

your line is open. 

(Julie): Hello thank you. I was wondering if there was any previous examples or 

thoughts around the use of international data. So given that quality is all 

controlled and the other considerations are met, are there any thoughts on 

using for example UK data as part of a (unintelligible). 

Josh Chetta: Thanks for that question. So the center has used information from outside the 

United States to support regulatory decisions in the past and that doesn't 

change with this. We would still want to see that the all those parameters that 

discussed are met. We'd also look for an evaluation of whether the patient 

population from which the information was collected is relevant to the United 

States population. There are certain you know depending on the device space 

there may be differences in considerations that would be specific to that. The 

broad answer to your question is that yes, we do accept that information and 

we have used it in the past. 

(Owen Faris): I'll just add just a little bit. That (Josh)’s answer extends not to just real world 

data but also to clinical trial data. We're looking for high quality data that is 

relevant and applicable to the patient population of interests for U.S. 

indication. And so if that information comes from the outside of the U.S. 

whether it's a clinical trial or it's high quality real world evidence, we are 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 19

completely open to having that be part of a pre-market submission or other 

regulatory submission. 

Irene Aihie: Are there any more questions? 

Coordinator: Yes ma'am. The next question is from (Randall). Your line is open. 

(Randall): Yes hello. So my question is regarding along the same vein and it's a good 

Segway. It's regarding the quality of the data. As a sponsor company, how do 

we - what mechanisms do you recommend for us to ensure or looking for to 

ensure the quality of data is being submitted to support an IDE or PMA. 

(Owen Faris): So, I'll take a first crack. This is (Owen) and I'm director of the clinical trials 

program. You know there are a lot of nuances to that question and in many 

cases, it depends on the particular situation in front of us. But what we find 

really is of most benefit is having a conversation with us early. You know 

before - as early as possible really. Before you've opened up that data set, 

talking about your plan with us. Figuring out what we think the obstacles may 

be.

We really find a lot of value in sitting down with you and mapping out a 

strategy. We all recognize that you know while the guidance lays out, the 

ways in which a high-quality dataset can be constructed there will be 

limitations and pros and cons to any dataset and any question in front of us. 

We recognize that it will not be perfect. It almost never is and we will have 

some level of uncertainty and the question is how do we get to the acceptable 

level of uncertainty to support that particular regulatory decision. And so the 

earlier you can come and talk with us the more aligned we can be and set 

expectations appropriately the better., and I think you'll find that we're very 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 20

flexible and open to these ideas. This is a new ground that we're all breaking 

here. But, at the same time, the questions are the same questions. How do 

you provide credible data that's relevant and that's reliable to the question at 

hand and we believe there's a real opportunity to gather it in a real world 

space.But coming and talking with us and mapping out a plan is really how 

we get started. 

(Randall): So, I'd just like to add to that. So as a sponsor I have monitors I have CREs

that go out and look at the data and do source data verifications so I can stand 

behind the data that I'm providing. Because I have those mechanisms in 

place. But in these particular cases when we were using real world data and 

this data is being collected by a third party, an investigator, an academic, 

people that don't have the same mechanisms in place that don't have data 

monitoring or don't have the standard operating procedure sin place, that's 

where I'm looking for that guidance and that's where I'm struggling to 

understand which way do we go. What do we do as a sponsor? 

(Ben Eloff): Hi this is Ben Eloff, Deputy Director of the Division of Epidemiology at

FDA. The section in the guidance within reliability discusses the need for the 

concept of quality by design and that is that the registry or other data source 

being considered will generally have some operating procedures along with it. 

And understanding the design of the operating procedures and the adherence 

to the design is going to be a critical factor in understanding the reliability of 

the data and how well it has captured the actual clinical experience. This is 

different than a traditional bioresearch monitoring inspection and individual 

source data verification. But the - a well-designed well executed third party 

data collection and analysis system can be just as reliable under the right 

circumstances as a traditional standalone clinical trial.

(Randall): Thank you.
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 21

Coordinator: Thank you. The next question comes from (Robert) your line is open. Hi 

there. Thank you. 

(Robert): My question relates to the more understanding about the off label use and 

getting approval for new uses for a device. So as a sponsor what is the line 

between promoting an off label use and setting up a real world study to 

determine whether something can be cleared for new use?

(Owen Faris): Hi this is (Owen) again. So, I think the fact that you're getting into a territory 

that is tricky and you know we try to navigate part of the answer to that 

question and the guidance around when do you need an IDE. But certainly,

there are questions that this guidance doesn’t directly take on about 

promotions. So , generally, when we approve a device, it has a specific 

indication, particularly for class 3 devices at a clinical trial that indication is 

generally focused on the patient population that was enrolled in that study that 

supported its approval. 

And we all know that those devices, very frequently, get used for indications 

that are beyond the scope of what they were originally approved for and this is 

part of how we have a learning health system, right? Physicians and patients 

and other care providers learn more about how to use that device better in 

terms of the ways they use it and the patients that use it, etc. and we don't 

want to blind ourselves to learning from that information. 

With that said, it's certainly not appropriate for a manufacturer to promote for 

that use which is outside of the scope of its approved indication and that's a 

tricky water to navigate. But the kinds of questions we ask when we're 

looking at can you use a registry that gathered information that was beyond 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 22

the scope of the approved indication for example to support marketing 

applications are around, 

Well, did you influence care? Did you recruit patients for that use? Did you 

promote in a way that was inappropriate? There are lots of questions around 

that - that really have to be taken on in a case-by-case basis. It's hard to give 

you a clear roadmap in a few sentences and the answer to the question. But 

these are the kinds of considerations that go into with how we deal with those. 

(Robert): Okay. Thank you. So just to summarize I think what I heard was we should 

probably seek your input first. That would be the most appropriate way to 

handle those situations? 

(Owen Faris): Absolutely. We handle those sorts of questions every day. 

(Randall): Okay. Thank you. 

Coordinator: Thank you. The next question is from (Andy). Your line is open. Please 

check your mute button, your line is open. (Andy) we're unable to hear you. 

(Andy): Hello. 

Coordinator: Your line is open sir. 

(Andy): Well hello.

Coordinator: Yes, did you want to ask a question? You pressed star one. 

(Andy): Could someone just perhaps discuss the potential use of meta-analysis? 

Published meta-analyses and which papers that were obviously used to 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 23

comprise that we're all investigator sponsored kind of perspective clinical 

trials?

(Owen Faris): This is (Owen). I'll take the first crack at the question and see if I miss 

anything. So, I would say, in general, meta-analyses would be outside of the 

scope of this guidance in the sense that we're generally talking about a Metaanalyses of clinical trials rather than real world evidence. With that said, I 

think the same sort of principles apply that we're looking for a high quality, 

relevant, reliable data to support regulatory decision making. 

And there are times when Meta analyses can form part or all of that dataset. 

So I think you can think of similar principles applying but that's not 

specifically the scope of this guidance or these discussions. 

(Ben Eloff): This is Ben Eloff, from the Division of Epidemiology again. Indeed as 

(Owen) mentioned, meta-analyses, when used appropriately and designed 

appropriately, can provide additional information as a part of any regulatory 

decision making. With regards to the real world guidance that we're 

discussing today, where we consider the literature and presentation to be just 

that: a presentation of data. The discussion focused in the guidance as the 

data itself and how it was generated. So an meta-analysis of clinical trial data 

would not be covered under the scope of this guidance. 

The guidance and the principles in this guidance relating to the relevance and 

reliability are based on sound scientific principles that are really a part of the 

underlying understanding of any scientific data source. So, we would apply 

the same principles regardless.

(Andy): Thank you.
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 24

Coordinator: Thank you. The next question is from (Anne Marie). Ma'am your line is 

open. 

(Anne Marie): Hi. In the final guidance, FDA states that, in certain circumstances, RWD 

may be used for generating summary reports and MDRs and unique postapproval studies both of the study and the adverse event reporting are 

submitted to the Office of Surveillance and Biometrics. So, my question is, in 

what circumstances, are MDRs required, specifically if the registry date are 

provided to a class 3 manufacturer by a third party meets the study and points, 

but is de-identified when there's no possibility of one event investigation or 

two, identification and removal of duplicate adverse events that have already 

been reported, then does 21CFR803 still apply to require MDRs for these 

registry data?

(Ben Eloff): This is Ben Eloff again from the Division of Epidemiology. Thank you very 

much for that question. The bottom line is like many of these questions it 

depends on the specifics. The process for doing summary reporting, is out of 

scope of this specific guidance and is covered by a different set of procedures.

When we have used summary reporting within a registry it has been on a 

specific case-by-case basis through a waiver process, so you would have to go 

through that process. And again, we're happy to discuss this on a case-bycase basis.

(Anne Marie): Okay. Thank you.

Coordinator: Thank you. The next question we have is from (Archita). Your line is open. 

(Archita): Hi. I just have a quick question. It's actually a follow-up to one of the 

previous questions. Let's say you do have a (unintelligible) of evidence that it 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 25

is connected from our connected product as a medical device company. How 

do you think it's going to affect reimbursement if we were to use that in 

patient outcome? 

Josh Chetta: This is (Josh). Unfortunately, you’re kind of muffled. Could you repeat the 

question again? I’m sorry, we didn’t quite catch it. 

(Archita): I'm so sorry. Can you hear me all right?

Josh Chetta: Yes, that's better.

(Archita): So, I was wondering, currently, you know as a medical device company we 

have a good amount of real world data. How exactly can we use that for 

improving reimbursement as we're looking at improving patient outcomes but 

we're not actually able to figure a way for reimbursement for it?

Josh Chetta: Hi this (Josh) again. Thanks for the question. Unfortunately, I don't think we 

can address reimbursement in this discussion right now. We're focused on 

regulatory decisions that the FDA would make and those would be outside 

kind of the scope of what we do. 

(Archita): That's all right. Thank you. 

Coordinator: Thank you. The next question is from (Rob) your line is open. 

(Rob): Hi do you have any guidance or thoughts on collecting patient reported 

outcome data for post-market commitment studies using mobile devices? 

(Ben Eloff): Hi this is (Ben Eloff) from the Division of Epidemiology. With such an openended question I can provide a few different avenues for consideration. One, 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 26

patient reported outcomes are a very high priority for us here in CDRH for a 

variety of reasons and we have indeed relied upon them in post-approval 

settings and in pre-approval settings for many devices over the years.

In a real world setting, the qualification and evaluation of a PRO instrument 

and its fitness for purpose for evaluating the endpoints that it's supposed to 

evaluate are part of qualification document for that instrument would be 

submitted via the PRO guidance processes and the medical device 

development tools process. Once appropriately qualified within the space, it 

could then be used as an appropriately robust endpoint data collection tool for 

measuring that outcome. 

Coordinator: Thank you.

Irene Aihie: We'll take our next question. 

Coordinator: Yes, the next question is from (Kim) your line is open. 

(Kim): Hi. Thanks for doing the teleconference. In one of the examples, this 

morning, FDA performed a study evaluation of a registry - will this be the 

standard for the Agency going forward and if so will it be done in conjunction 

with the sponsor or will it be independent between the agency and the 

registry?

(Owen Faris): Hi this is (Owen). You know I think right now frankly we're seeing both. We 

are seeing sponsors come to us and say we'd like to use this registry or we'd 

like to take part in developing this registry. Here's the purpose that we have in 

mind for a regulatory use and can we work with FDA to talk about how we 

can make this happen? We're definitely seeing that. We're also seeing patient 

groups, physician groups saying ‘this is happening in this space right now and 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 27

we'd like to have a better way of gathering this information; FDA, we know 

you have some experience in this, Can you help us? And can you help us 

form a registry that can gather information that can answer these kinds of 

questions’. We may want to help answer and work with you know 

manufacturers with in the future. 

And so, really it can come from anywhere. There are times when 

manufacturers have the interest in that and there are times when patients or 

physician groups have interest in that information. And we're happy to 

engage with either of those groups or together. 

(Ben Eloff): Just to add to Owen’s comments, much of the engagements with real world 

data sources and qualifying these sources as evidentiary is going to be covered 

and rapidly evolving through the National Evaluation System for health 

technologies. We do not right now have a clear one size fits all standards or 

route map that would cover every conceivable source. However, what we do 

have is a long-standing commitment to partnering with all of available and 

interested stakeholders in the interest of advancing public health. 

Mike Waters: And if I could add to that, (this is Mike Waters from OIR (the Office of InVitro Diagnostics and Radiologic Health)), we have several examples and 

efforts that we're actively engaged in right now throughout CDRH to develop 

the harmonized and interoperable infrastructure to access real-world evidence 

in both the therapeutic and diagnostic spaces. In the diagnostic space, we've 

engaged multiple stakeholders including CDC, NIH, ONC, CMS, device 

manufacturers, labs, electronic healthcare record (EHR) vendors and standards 

developers to adopt and develop semantic standards and structured data 

formats to improve access to harmonized and interoperable real-world

evidence. So, if you want to help in enhancing the ability to access RWE, you 

can contact us at OIR-Policy@fda.hhs.gov or 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 28

CDRHClinicalEvidence@fda.hhs.gov and you can get involved in some of 

these developing efforts.

(Kim): Excellent. If it's okay, I have a couple of other specific questions. One is 

regarding in the irrelevance category and real world data study designs and 

study protocol and analysis plans, it goes on to say whether or not it's capable 

of being accomplished in efficiently timely manner. Could you provide more 

specifics on what that means? Are you saying its treatment may be 

device/treatment affect specific and who would determine the timeliness?

Josh Chetta: Hi, this is Josh. I think that again we’ve answered a number of times, it's 

going to be handled on a case-by-case basis but the intent is to ensure that the 

information is collected in a timely manner, the patients are and that the 

information can be used and it's going to be fit for purpose. So I don't if 

there's additional - if you had another question about that or what specifically 

you were asking. 

(Kim): I was trying to understand better what the temporal nature of that part of the 

sentence that relates to? Is it relating to actual collection - like the timely 

collection of the data itself or is it implying something about the timeliness of 

the study designs somehow? 

(Ben Eloff): This is (Ben Eloff) again, I think a good way to understand that question is the 

condition that it's attempting to mitigate. As we all know science and clinical 

practice moves fairly rapidly and evolves and the concern with using 

especially a retrospective design would be that you would be analyzing results 

that are not necessarily applicable to the current state of affairs or events. So 

understanding whether or not a trial can be performed or analysis can be 

performed in a time that's still able to address the question that is relevant. It 

is the concern that we're trying to mitigate.
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 29

(Kim): That's very clear. Thank you so much for that answer. And the last one from 

me at least is on the data accrual and reliability. Again, there's a bullet related 

to establishing the steady plan and protocol relative to the collection of the 

retrieval the real world data. Maybe this is a can to, but we just discussed on 

the relevance side I wasn't sure it's the same thing. I didn't know if implied 

prospective versus retrospective and if you have RWD that was already predetermined and there was a common definitional claim work, as well as a 

data capture form. What difference would it make if it was retro versus PRO? 

If you can comment on that. 

(Ben Eloff): Right. This is actually a little bit different than the prior case. The concern 

here that we're interested in is whether or not the data have been accessed or if 

there is an understanding or look at the data prior to the plan going into place 

that would affect the development of that plan. One can do a prospective 

evaluation of retrospectively collected data if you don't know what is 

necessarily in there. However, if the data has already been analyzed, retrieve 

and then the study plan is put in place, the scientific value of that analysis is 

reduced more to a hypothesis generating status than it would be to a definitive 

conclusion. 

(Owen Faris): This is (Owen) to add a little bit to (Ben)’s response. You know there are 

times when retrospective data that is even looked at retrospectively, is so 

compelling that it can still support a regulatory decision. But to Ben’s point 

when you already know the data that's in front of you and you say now this is 

how I'm going to look for a regulatory submission - that has a strongly 

negative impact on the sort of strength and credibility of those data. It can at 

times be overcome with such compelling information that we still can rely 

upon it. But good clinical trial design, good data analysis design - is to plan 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 30

what you're going to do before you have knowledge what those data say. And 

so the more you can do that, the greater reliance we can have on those data.

(Kim): Excellent. Thank you. 

Coordinator: Thank you. The next question we have comes from Heather. Your line is 

open. 

(Heather): Thank you. So thinking about new and novel types of data from different data 

sources, it would be really helpful for me if you could lay out some of your 

thinking around the types of documentation or policies and processes you 

would want to see in place under in particular under the sections for reliability

of the data for new digital health applications, wearable and other tools that 

could be collecting real world data to support evaluations. Thank you. 

(Ben Eloff) Hi this is (Ben Eloff again. The processes for doing these evaluations again 

are rapidly evolving and there are several working groups within the agency 

and in the (Nest) partnership that are approaching out to evaluate what 

documentation specifically to collect and so on. The general principles and 

high level principles we lay out in the guidance but as was mentioned in the 

earlier slides, we don't have a score sheet or a check list or anything like that 

at this point that can be used for full validation of any given data source or 

sources. We are hoping that through the NEST partnership and working with a 

variety of data sources and users of those data we can come up with some 

scenarios that can be reused more generalizably in the future, but the science 

is not at that point yet. 

(Heather): Thank you and I have one more follow-up if you don't mind. It's actually 

about linkages between different data sources. In some ways it's written to 

look at a single source of real world data for an evaluation and it might be too 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 31

early days to talk about it, but it would be helpful if you guys had thoughts 

about data linkages between different real world data sources and what types 

of validation processes you ideally would like to see. 

(Ben Eloff): We do. We actually have a number of examples and quite a bit of experience 

in the use of the linked datasets from a variety of different clinical areas and 

types of data. Within these specific sources, we continue to abide by 

scientific principles. The linkage between different data sources has been well 

described in the academic literature and methodology and we would rely upon 

that methodology as described and validation studies for the specific 

endpoints of interest so that we could rely upon them. 

(Owen Faris): This is (Owen). I just wanted to add a little bit to (Ben)’s

good comments that we've been sort of peripherally in our presentation today 

in answering the questions today talked about NEST, the National Evaluation 

System for Health Technologies. Maybe it's appropriate to just give a more 

direct shout out to what's happening there because NEST is really the bringing 

together of essentially all of the stakeholders in this space. So we're talking 

about FDA of course, but also manufacturers, patient groups, providers - sort 

of all the folks that are in this space of generating, colleting, interpreting, 

using real world evidence and trying to figure out best practices. Setting up 

ways to link data, developing both methods but also sources of data so that if I 

am a stakeholder and I have a question, eventually NEST will be able to help 

me figure out what data are available that can help me answer that question 

and so it is a very exciting time for real world evidence in that the National 

Evaluation System for Health Technology is just really getting off the ground 

this year. The director of NEST was named and the Board of Directors to 

support NEST was named. There is money allocated for the next five years to 

support NEST going forward and I think this is just going to be generating a 

lot of really useful resources and thinking in this area on the very kinds of 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 32

questions that you're asking: How do we do this? How do we do this well? 

What data are available to me? How can I rely upon it? These are the kind of 

questions that we'll be answering jointly as an ecosystem over the coming 

years and it's moving really quickly and its' been really in my mind very 

impressive thus far in terms of what they've done just over the past few 

months that they've been in existence. 

Alex Hu: This is (Alex Hu) speaking. So, regarding the data assurance, besides the real 

world evidence guidance, there are also many published recommendations by, 

for example, Agency of Healthcare Research and Quality, Patient-Centered

Outcome Research Institution, National Medical Device Registry Task Force, 

so on and so forth. In drafting these recommendations, FDA has been highly 

involved and these publications are laid out on the page 16 of the guidance. 

Please take a look.

(Kim): Thank you.

Coordinator: The next question is from (Napoleon). Your line is open. 

(Napoleon): Thank you. Has, can, or will the real world evidence methodology be applied 

to combination products across the divisions of FDA. How is that working at 

NEST? Thank you. 

(Owen Faris): So, this is (Owen). So, this guidance does apply to combination products and 

we are very interested across the agency in using real world evidence. There 

are a lot of activities and a lot of recent publications and outreach on using 

real world evidence across the agency for any kind of medical product. 

Obviously, combination products have particular challenges and unique 

questions and again our sort of standard answer is please come talk with us if 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 33

you have an idea for a combination product for which you wanted to use or 

develop real world evidence source, but absolutely that's on the table to do. 

(Napoleon): As a follow-up the Parental Drug Association is meeting shortly again on 

combination products and I would hope that we can encourage them to be 

involved because they have a very board view. So thank you. 

Irene Aihie: We'll take our next question. 

Coordinator: Yes ma'am. The next question is from (Sonya) your line is open. 

(Sonya) Hi thank for you the webinar. My first question is about informed consent. 

Is informed consent required for real world data collection? And the second 

one is there a difference between real world data collection and retrospective 

data collection? If yes, what's the can you elaborate on the difference?

(Owen Faris): So I may have to ask you to repeat the second part of your question. The first 

part around informed consent is a really complicated question and specifically 

has so many factors that it was really difficult for it to be included in the scope 

of this kind since there are many factors that play into whether or not 

informed consent is required. How are the data being used? Whether the data 

are identifiable. There are many other factors so there are situations where 

real world data are used in regulatory submissions that did not require 

informed consent. There are situations where it is used in which it did require 

conform consent. We specifically did not take that on in this guidance but 

we're happy to talk to you on a case-by-case basis.

Would you mind repeating the second part of your question? 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 34

(Sonya): My second question yes, I do believe that whatever type of data we're going to 

collect we need the pre-plan what we're going to do but my question is if there 

is any difference between real world data and retrospective data. If he asked 

can you elaborate on them? 

(Owen Faris): The difference between real world data and retrospective data?

(Sonya): Yes.

(Owen Faris): So real world data can be retrospective data. There can also be a plan to 

collect real world data prospectively. The question is, are you collecting it in

a way that's influencing care such that you protocolizing it then it becomes a 

little less real world or do you have a plan for collecting it going forward in a 

way that is really just seeing what is happening in the real world. 

These are the kinds of questions that we start to get into when we think about 

whether this is an investigation that might require an IDE or whether it is 

really an observation of what's happening in the real world. It can be 

prospective. There could be a plan to do this prospectively. As (Ben) alluded 

to in one of the earlier answers, it can also be a plan to analyze data that 

already exists. So there could be data that has been collected in say electronic 

health care system or in a registry and you develop a plan for how you are 

going to analyze those data. As we've been discussing, that data will be much 

stronger and will be given much more creditability if the plan is developed 

before the data are known. But it could still be retrospective. 

(Sonya): Thank you.

Coordinator: Thank you. The next question is from (Ronald) your line is open. 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 35

(Ronald): Yes, thank you. I don't know something about how real world data could be 

used in DeNovo submissions specifically. Can you comment on how the 

guidance addresses that point? 

Josh Chetta: Hi this is (Josh). So, the guidance doesn't explicitly address that question but 

it does make clear that these types of data - real world evidence may be 

considered valid scientific evidence to support any regulatory decision for any 

device type. So certainly we would be open to evaluating real world data or 

real world evidence in De Novo submissions. Again we would encourage you 

to probably contact us via the pre-submission process before you plan to 

submit that De Novo application, but there's nothing that prohibits the use of 

those types of data in a De Novo 

(Ronald): Thank you.

Coordinator: Thank you. The next question is from (David) your line is open. 

(David): Yes, hello. I have a question from the IRB perspective. You know it was 

greatly helpful that you put in the criteria for if the study affects clinical care 

then you'll probably need an IDE and if it doesn't you probably won't need an 

IDE. My question has to do with those studies that don't need an IDE. Does 

FDA have an expectation that those will be non-significant risk studies under 

abbreviated IDE requirements or exempt studies when they're on label or are 

they not clinical investigations at all? I'm just thinking of the IRB 

recordkeeping process. Thank you. 

(Owen Faris): Great question. Complicated answer. This is (Owen) again. This is again 

one of the questions that we decided we couldn't do justice to at this time to 

put it in the scope of this guidance. There is a lot of work in this space right 

now to figure out exactly the answers to the questions that you're 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 36

contemplating right now. So, I'm not sure that I can give you a specific 

answer right now. It would be very helpful to sit down with you and talk with 

you about a particular example if you have a particular question outside of this 

discussion here today. I would say the kinds of questions you're asking are 

very relevant today and are being contemplated and discussed inside the 

agency and outside the agency as well.

(David): Thank you.

Coordinator: Thank you. The next question is from (Ed), your line is open. Please strike 

your mute button your line is open. 

(Ed): Was that a test maybe?

Coordinator: Go ahead.

(Ed): Okay. Thank you. Nice job on the guidance folks and appreciate the 

informative and helpful webinar. I know you probably have limitations on 

what you can say specifically but do you have any general comments you can 

make about cases in which efforts to use real world data have not been 

acceptable as real world evidence?

Alex Hu: There were cases using real world evidence from outside of the U.S. that lead 

to major deficiencies in some of the clinical areas. This is very in detail and 

probably beyond the scope of this webinar training for the guidance. And if 

you have detailed questions, please send us an email.

(Owen Faris): Hi this is (Owen). I will chime in a little bit and I think (Ben)’s going to 

chime in after me. I think the kinds of questions - I'll answer your question 

generally in how it applies more broadly than just real world evidence. We 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 37

have questions every day that come from the agency back to sponsors about 

the quality of the data that are provided whether they're from real world 

evidence sources or whether they're from a clinical trial and regardless of 

where they're conducted, we have times when the quality of those data aren't 

enough to get us over the bar and so you know you might think of some of the 

kinds of questions that might arise with real world data. We've already talked 

about the pre-specification of your data analysis plan relative to knowledge of 

the data. That's sort of the obvious one where we would have questions about 

that.

We might have questions about missing data, data completeness in terms of 

timelines for when those data were collected and were they collected on all of 

the patients or a significant portion of the patients. Those sorts of questions 

are the kinds of things that we will be digging into when data comes in and it 

really also depends on the kind of question that's put in front of us to ask. But 

if we're talking about considering marketing application for a new device or 

an expanded use of a device, we will be digging into many elements of the 

data quality and depending on how that datasets constructed - depending on 

how relevant that real world evidence source is to the question at hand, there 

may be questions about when you ask the question versus knowledge of the 

data, missing data, completeness of the data, all those sorts of pieces. And 

frankly, is this patient population? does it represent the patient population? 

The broad patient population that may be in the same new indication, but 

these are questions that come up not just in real world evidence, they come up 

broadly and really any kind of submission where data are provided to support 

a regulatory question. 

(Ben Eloff): Hi and this is (Ben Eloff) and I'm going to approach this from a slightly 

different angle than (Owen) just did. Thinking about the acceptance of real 

world data as real world evidence is not a binary process. It would be 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 38

extremely rare for data presented to FDA to have no value whatsoever in a 

given regulatory decision. So it may be that data is presented that does not 

suffice as the entirety of the data necessary to support a given decision but to 

supplement somewhere else for that decision. But to have a source where you 

say flat out no, this has no value would be extremely, extremely and 

vanishingly rare. 

(Ed): Okay. Thanks. Thanks, gentlemen.

Coordinator: Thank you and the final question is from Gina, your line is open.

(Gina): Hello thank you (Irene), (Josh), (Alex), panelist, the wonderful webinar today. 

My question is regarding the use of real world evidence in the 510K paradigm

for class 2 devices. I know this is something that we're really excited for. The 

potential application of this to the 510K paradigm. So if you say that I'm a 

manufacturer of a class 2 510K cleared legally marketed device and I wanted 

to make a modification to that device. Let's say it's a labeling change or one 

that does not change the indications for use, it doesn't change the fundamental 

scientific technology. If I wanted to use real world data that had been 

collected using my legally marketed device, the unchanged device, my 

primary predicate device, it's my understanding it would have to be a 

traditional 510K if the real world evidence would be needed to answer patient 

safety and effectiveness questions relating to the particular change. In the 

view of the panel, if the real world evidence was not really used to answer any 

safety or effectiveness question, but say use as a validation to ensure the 

modified device still needs the user requirements, is the special 510K pathway 

still feasible for real world evidence supported modifications? I understand 

you may not be willing to (unintelligible). Sure.
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 39

Josh Chetta: This is (Josh). Thanks for that question. It's complicated and I think we're 

running out of time. To answer your question, you'd want to look at what the 

parameters of the special 510K are. the changes - how significant they were 

and whether the data provided to support that change would be applicable to 

the device depending the change. If it's a labeling change, you may be 

changing the intended use so there's a lot of questions to unpack there. We 

don't quite have time to go into all those. Again, the same answer you've 

gotten a number of times. Contact us with a pre-submission to talk about the 

particulars of the submission. 

(Gina): Thank you.

Coordinator: Thank you. I'll turn the conference back to Ms. Aihie. 

Irene Aihie: Thank you. This is Irene Aihie. We appreciate your participation and 

thoughtful questions. Today's presentation and transcripts will be made 

available on the CDRH webpage at www.fda.gov/training/cdrhlearn by 

Wednesday October 18th. If you have additional questions about today's 

presentation, please use the contact information provided at the end of this 

live presentation. As always we appreciate your feedback.

Following the conclusion of today's webinar, please complete a short 13 

question survey about your FDA CDRH webinar experience. The survey can 

be found at www.fda.gov/cdrhwebinar immediately following the conclusion 

of today's live webinar. 

Again, thank you for participating. This concludes today's webinar. 

Coordinator: Thank you and this does conclude today's conference. All parties may 

disconnect. 
FDA 

Moderator: Irene Aihie

10-10-17/1:00 pm ET 

Page 40

END