FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 1

FDA User Session - Digital Health Software Precertification (Pre-Cert) Pilot Program

Moderator: Irene Aihie

May 10, 2018

12:00 pm ET

Coordinator: Welcome, and I would like to thank you for holding, and inform you that your 

lines are in listen-only during today's conference until the question and answer 

session. At that time, if you'd like to ask a question, you will press star then 

one. Today’s call is being recorded. If you have any objections, you may 

disconnect. Now I’d like to turn to Irene Aihie. You may begin.

Irene Aihie: Hello, and welcome to today’s FDA Digital Health Software Precertification 

User Session. I’m Irene Aihie of CDRH's Office of Communication and 

Education. On July 27, 2017, the FDA announced the launch of the agency's 

software precertification pilot program. 

The FDA recognizes that we need a regulatory framework that accommodates 

the distinctive nature of digital health technology, its clinical promise, its 

unique user interface and the industry's compressed commercial cycle of new 

product introductions. 

This is the first in a series of user sessions on the digital health pre-cert pilot 

program. As part of our commitment to keep our efforts transparent, the FDA 

will share updates throughout this pilot process. Today, Bakul Patel, 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 2

Associate Director for Digital Health in CDRH, and members of the digital 

health team, will discuss the progress that the agency has made on the 

software pre-cert pilot program. 

The session will cover the program working model, which includes key 

program areas and questions that the agency hopes to gather public input on. 

The session will also include a discussion of next steps for the program. 

Throughout the presentation, we will open the lines for your questions related 

to information provided during the presentation.

Additionally, there are other center subject matter experts to assist with the 

Q&A portion of our webinar. Now, I give you Bakul.

Bakul Patel: Thank you, Irene and welcome everybody to the webinar and the first user

session that we're holding. This is in our effort to continue to be transparent 

and collaboratively build the program as we have talked about since the day 

we launched the program itself. 

Before we jump into the pre-certification program, let me walk you through 

the Digital Health Innovation Action Plan. In the innovation action plan, we 

talked about first aligning our policies to the 21st Century Cures Act, and we 

promised publications of many guidances. We did some of those guidances in

2017. 

Along with the last update we did for the pre-cert program, we also announced 

the multifunction guidance. Let me touch upon that before we jump into the 

program quickly. So we talked about the three things we are doing in the 

digital health innovation action plan, aligning ourselves with the 21st Century

Cures, building additional health units, and the streamlined - and exploring a 

streamlined review. 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 3

Today, I'm going to talk about this last component of the action plan, which 

talks about the multi-functionality guidance. Let me jump into that guidance 

itself. The guidance basically talks about how should FDA look at products 

that have many functions that may be devices that may be reviewed or maybe 

exempted, or maybe not - may not be devices. 

How should we think about that? How should we review them? It proposes a 

policy of looking at the impact of safety and effectiveness on the function that 

is it medical device that's under review. We’re looking at that very 

specifically. As it says in this particular slide, does it impact - the question 

we're asking is, does it impact or have a result on an increased risk, or have an 

adverse effect on the performance of the device in review?

So this slide basically talks about an overview of the entire guidance. We are 

encouraging people to comment on this guidance, but more importantly, let 

me just turn you over - turn over to the precertification program itself. This is

how we are thinking about and you’ve heard me talk many times in the 

previous webinar as well. 

We're looking at an organizational based streamlined regulatory approach for 

Software as a Medical Device as we start the program that relies on a 

demonstrated Culture of Quality and Organization Excellence. 

Let me share a little bit more about the program to just level set everybody. 

The concept we started off in July last year, was to look at an organization, 

look at kind of the products, understand what the risk of those products would 

be, and include the confidence we can have in those companies that are 

building this product, and afford them an extremely streamlined pre-market 

review for products that need review.
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 4

For other products that are - that could be lower risk, can go straight to market

with the commitment to collect and use the real world information that these 

digital health products, including software, can deliver. We’re looking at 

opportunities to leverage technology that brings to us these products,

technology that sort of affords us to collect information from these - from the 

users, from the products and use that to actually better create a system that 

actually will enhance everybody's knowledge. And at the end of the - enhance 

patient safety.

Thinking about that, we’re also looking at using other information and 

evidence and using the programs like Nest that we're pursuing in other parts of 

the center to inform how well the program also operates, and how can we 

continue to improve and adapt to the changing needs as we learn more about 

the program. So truly in an iterative way, we're actually building the program

collaboratively with all of you.

Let me just talk about how we are thinking about this. we're thinking about 

building the program with not only the nine participants that are bravely 

willing to sort of help us build the program in an extremely tight timeline, but 

we also are trying to engage everybody that's on the call today, and others 

who will listen to this call after the fact, and look at the working document to 

help us build this program.

Having said that, let me just walk you through what we had just released a 

couple of weeks ago on the working model. But before we get there, let me 

just give you an overview of how the session will run. I'll talk about the 

program itself, which I just did. 

But I also want us to share some of the nuance - some of the things that we put 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 5

out in the working model, give you sort of a background on that. We also 

talked about how we can think about the program into four different 

components. I’ll walk you through that. We’ll have - we’ll spend 15 minutes 

on each of the program components.

I have the team leads from the program areas that will give you a picture of 

what the program component scope is, what the goal of the program is and

turn that into a question and answer session with all of you online. And we'll 

spend 15 minutes. We’ll walk through the rest of the components in the 

program. And further, at the end of the webinar, we’ll open it up for anybody 

who wants to ask any questions about the rest of the program itself.

If that is clear, let me just walk you through the components that we just 

announced a couple of weeks ago on April 26. We announced or we 

published a working model, which is truly meant to be a work in progress that

allows you to see our current thinking as we sort of hear from stakeholders, 

hear from the pilot participants, and also hear some of the comments, and look 

at some of the comments we receive in the docket.

Taking all that into consideration, we thought it was beneficial for us to lay 

out our initial step, our initial concept (unintelligible) and what this program 

looks like. In the working model, we also highlighted a set of challenge 

questions, and these challenge questions are primarily intended to give people 

bite sized problem areas that we want help on.

This should be also looked upon as, how should we - taking those questions, if 

folks want to help us and actually solve this particular - these challenge 

questions, we would encourage you guys to sort of take a look at that and give 

us feedback on those - on solutions for those challenge questions. 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 6

The other thing we also said is about the roadmap. This is our commitment to 

being transparent. We will be updating this working model. We'll be sharing 

with you on a very frequent and periodic basis of what we are thinking, what 

we are hearing, give you an opportunity to sort of engage with us. At the 

same time, truly make it a crowd sourced collaborative building of this 

program. We want to hear from you and that's what we intend to do in this 

program.

Let me walk into the four big components and they don't stand alone. As this 

picture sort of illustrates, they are interconnected. And I'll talk about that in 

just a little bit and to just give you an idea of the four components. There is

this excellence appraisal that understands what our organization’s capabilities 

are and what their culture of quality and organizational excellence is. 

We had proposed in the working model two levels as a starting point. We are 

looking to see how well they can sustain to keep it simple. The other 

component is about determination. Once the companies are certified, what 

should they do with their products they build and which - what can they afford 

in terms of going straight to market or to be reviewed?

The streamlined review, as the name sounds, it is actually about streamlined 

review. And the last part, which is about real world performance, and you’ll 

hear a little bit more about what that means, but it's about what kind of 

information we want to understand when the product is in the market that can 

inform, not only you as developers of the program, us as agency and also help 

the program go forward.

With that, let me just revisit the five excellence principles we’ve proposed. 

We heard very clearly at the workshop we had in January, that these are the 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 7

five principles seems to be the right principles as we move forward. We’re

using this as a basis and a foundation to build the rest of the program, but 

we're really looking at these five principles as a thing that we would want to 

see in any organization/

Let me just turn it over to Cisco Vicenty and let him talk about the scope of 

work and that component. And having walked through what his thinking is 

right now and also share with you the intent of sharing what we shared in the 

working model. 

Cisco Vicenty: So thank you, Bakul. And as I get started, I just want to also acknowledge 

and thank a lot of the people who are on the team who might be listening 

online working to develop this construct, this pretty complex portion of the 

whole precertification program. 

But as we get into there, I'd like to just highlight that this piece, the excellence 

appraise list, let me focus on, you know, what are we developing that whole 

eligibility, the application process, how the FDA will evaluate or assess the 

organization's capability to develop that Software as a Medical Device 

through the lens that Bakul highlighted earlier, right, along those five

excellence principles.

And this is really about the organization demonstrating their commitment and 

their capability regarding the patient safety, the product quality, clinical 

responsibility, cybersecurity responsibility and a proactive culture that's really 

able to monitor issues, respond to those issues, and incorporate that learning 

back into that system. 

How well that demonstration is fulfilled, that's going to determine the level of 

the precertification that the organization will actually be awarded - given, 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 8

however we really structure that piece. The team is also working on 

developing how the organization sustains that precertification. And that's a 

little further down the line in the construct phase, but it's really that element of 

continual review and again, the adapted and continual learning.

A couple of things to just really highlight and note with what the goals and 

activities are within the excellence appraisal team. It’s not focused on 

compliance to any quality system efforts at all. It’s about execution. How 

well are you delivering. 

The other piece of that is the agency is looking to really enable flexibility for 

the organization to leverage their basic internal quality system framework. 

Enabling a lot of, you know, what is the tools that they are looking to 

implement and apply? How agile they can be, and the big piece of this is 

really leveraging whatever good work they're already doing.

Let's not reinvent wheels where it is unnecessary. And that will feed a little 

bit later into some of the challenge questions we're talking about. This is a 

complex effort. It is something that we are continuously working towards. 

And what we did with the excellence appraisal and the precertification 

activities, and you'll see some of these questions out there right now, is try to 

highlight where we as an agency need some additional guidance or help.

As I mentioned beforehand, we really are looking to leverage as much of the 

good work that's out there as possible. These excellence principles are a great 

point of the agency focusing what do we care about to deliver on our mission 

and to the patient population while enabling as much flexibility as possible. 

We want to leverage a lot of these existing models that might be out there, and 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 9

they all focus and get to the same things. But now we've given them where 

the focal point is. 

Bakul Patel: So Cisco, thank you for that overview. And right now, we are on the slide 

which says, challenge question 1.1. It seems to be like one of the things that 

you’ve highlighted that you wanted to learn from, and how people help you 

with solving this. Can you just walk us through that, just in 15 seconds? 

And I'm going to go to (Chuck) next so to ask, you know, we heard a lot from 

everybody. So (Chuck), from your perspective, while Cisco talks about it, can 

you share a little bit from your perspective, your engagement with us and how 

do you - how would you advise people to help with this portion?

Cisco Vicenty: So yes. with regards to the challenge questions, that's a great opportunity to 

just help us build and understand how you externally, if you’re leveraging 

another set of standards and know the appraisal model already, how does that 

apply so that we are not asking or imposing something different?

A lot of the puts that come from there actually will fulfill what we're looking 

to develop in our requirements. So help us make that connection. That’s

really what the challenge questions are about.

Bakul Patel: So (Chuck), how would you help and guide the folks who are listening on the 

line who have not been deeply engaged in this discussion, but also seek help 

from those folks who are just hearing or seeing just the working model?

(Chuck): Thanks, Bakul. Thanks, Cisco. So to the extent that folks at home are not 

necessarily intimately involved with the pre-cert process. I would propose 

looking at the working model and then actually reviewing the challenge 

questions as Cisco and Bakul have laid out.
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 10

And from that point, there's quite a bit of material that has been put out into

this space from the initial appraisal questions that I think Bakul had published. 

And then additionally, I think there were some additional conversations at the 

initial meeting in January. 

And so using that as a starting point, I think one of the things that would be 

incredibly helpful is for folks to participate in a way where they can actually 

internally assess how they would potentially satisfy some of these proposed 

excellence appraisals. 

There’s a lot of ways to approach these questions and provide an input that 

says hey, we take these challenge questions and we're going to apply them 

internally. And I think from our smaller working group, the way we’ve 

approached, this is really let’s provide the flexibility for companies, the eight 

or nine pilot participants to say, let's ask these questions more broadly and

let's find a way for us to come up with our own proposed responses to these. 

And what we found is actually - it actually fit really well within the challenge 

questions, although they were considerably less detailed I'll say in the way we 

pose these questions internally. But what we found is that if we ask these 

questions more broadly, pilot participants are willing to really put themselves 

out there and make sure that they're providing a thorough systematic look at 

how they would propose answering these questions. 

Bakul Patel: Yes. That’s really a good summary, (Chuck) and actually really good advice 

for people who are listening for the first time. At this time, I think I would 

want to open it up for everybody on the call to line up questions. but while 

people are doing that, I'm going to go back to Cisco and just say, just pick on 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 11

one of the things that - you used the words, use of standards and you want to 

give people sort of that credit back that we have talked about a lot. 

Thinking about adoption of corporate level policies at a business unit level, 

what did you mean by that?

Cisco Vicenty: So there is - oftentimes an organization will be leveraging within their own 

structure, work that is already established policy or efforts for maybe the 

whole unit, the whole company that they're in or work that another unit has 

already done, for example in terms of leveraging a standard maybe around 

cybersecurity that they follow, or other reporting standards that they need to 

do, maybe even from a business standpoint.

That is all effort and work that actually helps fulfill or demonstrate your 

capability or your commitment, and not typically something that I think the 

agency is focused on now, but will help fulfil our excellence principles. Use 

that. That’s really what that's focusing about. 

Bakul Patel: Great. Thanks, Cisco. Folks on the line, if you're listening to this 

conversation and this is supposed to be an interactive session, for the next few 

minutes we'll take questions on this particular component, then we'll move on. 

And the whole idea, I know this is the first session, so I'm going to continue 

until - unless people have questions online, to just have this conversation with 

(Chuck) and Cisco here. 

So we're actually providing the clarity that we're seeking. So let me just take 

another question at (Chuck). (Chuck) is a great pilot participant. He’s been 

helping us build the program and giving us really good thoughts and ideas. 

(Chuck), if you look at some of the things that we have talked about as part of 

the appraising the maturity of an organization and how well they produce 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 12

products, anything that in a way that sort of we can rely on having them go to 

market. 

What kind of - what lessons that you thought that we were asking that may -

that was different or something new that you sort of learned that you could 

help orient people who are listening to this conversation to even think outside 

of the question that we've asked or help them answer some of the questions.

(Chuck): Yes. So Bakul, I think that's a great way to approach this. I think from our 

standpoint, the pilot participants and I really spent a lot of time actually trying 

to think like FDA. 

And to the extent that it's difficult to put yourself in another standpoint or 

another perspective, I think one of the things that really helped us is - and I 

think it was earlier on in one of our site visits, I think one of the 

recommendations was to approach this as if we were appraising one of our 

supply chains. 

Approach this is if we were appraising another entity we wanted to do 

business. And that actually helped put that perspective for us and say okay,

well if we're going to appraise a business relationship with another vendor,

company, et cetera, how would we think about this? 

What are the things that are important to us to make sure that if they were out 

there and there was not a strong business connection naturally, how would we 

approach this?

And so from our perspective, that actually helped put us in the perspective of 

FDA, and we could actually turn that around and provide information back to 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 13

you in a way that I think was much more in line with the business language. 

And we were able to really interpret that and translate that to FDA.

And so from my perspective, looking at it like that, it was actually a very 

valuable tool. 

Bakul Patel: Great. Thank you so much, (Chuck). 

Coordinator: At this time, if you'd like to ask a question on the audio portion, please press 

star then one. That’s star then one. Thank you. 

Bakul Patel: Great. Thank you, Ed. Cisco, back to you in terms of how we think about 

and you talked about those four or five stages within the excellence appraisal. 

Can you share a little bit more depth in terms of what you’re thinking of in 

terms of maintenance or anything else that you think long term, even though 

we're not addressing it right now. But any color you can provide to any of the 

questions that you have sort of asked. 

Cisco Vicenty: Yes, I think - and especially with regards to anybody who is already working 

in the space in terms of their certification, type of activities to one of these 

other existing excellence models or maturity appraisals. The one thing that 

we are looking to do and drive towards is for maintenance in particular too is,

how do we identify the metrics, right?

Somewhere in the software - the medical device space, there is the unique 

opportunity to really get a lot more real world data, real world evidence and 

have that learning and that knowledge and that demonstration actually feed 

back into the appraisal process and that maintenance portion, right? 

So the maintenance becomes a standard of really feeding back in and not 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 14

necessarily needing the same level of insight that we collect in the initial onset 

from that standpoint.

Bakul Patel: Great. Looks like we have some questions online. Thank you, Cisco so much 

for that. We have a question online. We will go ahead and take that.

Coordinator: Our first one comes from (Tipton). Your line is open, (Tipton).

(Tipton): Hi. My question was, does this program apply to software as a standalone to 

medical device, or does it also apply to like embedded software and

combination device software too?

Bakul Patel: I can take that. This is Bakul. I can tell you, the program is designed is for 

software in general. But however, as we start the program up and build the 

program, we are focusing right now in taking everything that is - that we can 

understand from the - from just software that is a medical device, which is in 

other words used to be known as standalone and use those - all those 

characteristics and learn as much as we can. 

So the first part of our phase is about Software as a Medical Device or 

standalone software. 

(Tipton): Okay. Thank you. 

Bakul Patel: Great. Next - we can take the next question.

Coordinator: Next question comes from Joseph McBride. Your line is open.
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 15

Joseph McBride: Hello. I was wondering, what is the relationship between the precertification 

program and the Medical Device Single Audit Program? Is that - would that 

be considered part of the excellence appraisal or maturity assessment?

Cisco Vicenty: So that's actually a great question and this is a very different mode and model 

than what the Medical Device Single Audit Program is doing right now. The

goal of the Medical Device Single Audit Program is really about 

harmonization within that medical device space. 

Within the pre-cert program, right now as Bakul had mentioned beforehand 

we’re starting with Software as a Medical Device. It has a lot more potential,

but it was really geared about changing the whole dynamic and moving away 

from the compliance aspect and really focusing on the organization’s delivery 

and a different way to really keep that pulse of that organization.

Bakul Patel: Great. So I know there's a couple of other people online. We will take - if 

you don't mind, hold your questions till the end of - till we run through all the 

four components and then we can probably take those questions. So if you 

don't mind, please keep the questions on hold. And we'll do the same thing 

for the other component. We’ll work through the slides.

The next section is going to be about review determination, which is 

fundamentally talking about what should be done for review determination? 

What are the inputs to this component which sort of takes in the work that is 

going to happen in the excellence appraisal and bring it together with the 

types of products.

Imagine a place where you have very clear criteria, if you're making this kind 

of products, you appraise it at this level, you would know - as a user of the 

program, you would know whether your product or whether you as a maker of 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 16

the product, would have to go to the same lender view, or you would go 

straight to market. 

We’d also - we're also looking at defining in this work product, the adoption 

of the IMDRF framework and the definitions. So we can actually be very,

very clear at the end of the day what product ends up in what pathway. That’s

the intent of this program. 

Without belaboring this further, let me just go to the challenge questions that 

we posed. One of the concepts we have in this program is to, how do you 

explore the definition statement, which is really a combination of intended use 

and a device description and indications of use together that was done in the 

IMDRF setting.

how do we take that and make it clearer for folks that are applying to this 

program in the US to align and also to find so that enough clarity that once 

we're in the program, once the reviewers look at it, users look at it, anybody 

else looks at it, have a very clear understanding of what that product is all 

about. 

So we asked about the definition statement. We want to know, how can we 

refine it? We want to know, how can we better it? How can we align it to US

regulations? And that's the intent of this question. I'm not going to read 

through the slides here, but I just wanted to give you that flavor.

With that, let me just ask (Diane) who was one of the pilot participants, who 

can share some nuance around so the intent behind this particular component. 

So (Diane), if I - if you - if I may, what do you think? You’ve known about 

the IMDRF framework. You’ve sort of studied it to detail. 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 17

And now as we use this in this program, how would you think about it that 

way? And the kind of clarity you think you need in the framework that I laid 

out from refining, aligning to regulatory restructure, aligning to what we know 

in the current regulatory structure, but not necessarily gravitating towards the -

towards our structure, but really looking at clarity. What can be used from 

this framework that can provide clarity? So I’m going to turn it over to you,

(Diane).

(Diane): Thank you, Bakul. One of the things that in this section that I think it's 

important to stress, because as participants, or at least for me as a participant,

was it continues to evolve. The thinking around is that software is 

instrumented in such a way that you are getting real time information that you 

can act on. And that really does change the way you can look at things/

So if you look at - take a step back and you look at IMDRF and the risk 

classifications as they were developed, they’re based on the reliance of the 

data, how much of it is the decision relied upon. Is it informing, driving 

treating? And what is the state of the healthcare of the patients? Are they just 

a little sick or are they very sick?

And these considerations can also be viewed through the context of how risk 

is defined in US statutes. So FDA in table three proposed kind of how you 

would interpret IMDRF in the context of our definitions here in the US. And I 

would really encourage the public to comment on the particular proposal in 

table three, because we're looking at different paradigms that are not 

necessarily tied irrevocably to historical approaches such as substantial 

equivalence and de novo classification.

And I think it's going to be really important for all the stakeholders to 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 18

understand that although how the classification, if I can use that word, is 

determined, it may be different, but we're still meeting the same evidentiary 

standards, perhaps demonstrating in a different way, but that the standards are 

the same and that patient safety is still a primary consideration. 

So I think fleshing these out in more details and having all the stakeholders 

understand that this is in fact the case, I think will be very important to 

making everyone in the ecosystem supportive of this type of a program. 

Bakul Patel: Great. Thank you, (Diane). That was very helpful. I think it gives a good 

framework for people to think about, especially as you look at it I mean from 

outside into the program. I think the experience that you have sort of brings 

that to the table.

Let me just share one other thing that (Diane) talked about, which is important 

sort of to think through is, not only we have some challenges because we are 

dealing with this extremely fast moving technology, but also something that 

does get - can be only recognized when it’s described in the right way.

And one of the things we're looking at is how can we describe the product? 

How can we understand what does that mean for things like de novo? And 

what do we mean by substantial equivalence? How should we think about 

those things? And those are the questions you see. 

So I would encourage people to think about - to look at those questions and 

answer - give the specific answers to us. we're looking at, in this component, 

to get input from the lines where we drew in terms of where things could be

reviewed versus not. 

We’d be getting input from the other components in this program because it'll 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 19

be dependent - as by the time you're done with the program, it will be 

dependent on what outputs come out of the excellence appraisal and what 

things we're expecting out of - to be reviewed. And then what real world 

evidence. So that's why when I said before, they’re all interconnected.

I want to take this moment and open it up for the audience, people who are 

listening in to engage here and ask questions about this particular part and 

how - what kind of information would you guys need to help us provide those

solutions? Now you have seen the challenge questions. You’ve seen sort of 

the concepts. 

As (Diane) mentioned, we have a table that talks about an initial proposal of 

how we think from a risk perspective and what we could transfer the review or 

oversight to different parts of the program. So think - how would you think 

about that and what kind of information would you need to provide better 

input to us? 

So it looks like we have a few people online ready to ask questions. We'll 

take the next question which is online.

Coordinator: Our next one will come from (Pat Hew). Your line is open, sir. Please check 

your headset or mute button. Our next one will come from Naveen Agarwal. 

Your line is open, sir.

Naveen Agarwal: Good afternoon everyone. My question was actually related to the challenge 

question in the previous section that you were describing about excellence 

determination. And you mentioned that the focus is on really the execution 

and not on compliance. 

So I was just curious if FDA would consider also looking at their own internal 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 20

data from inspections or other enforcement actions and look at repeat issues or 

issues that seem to have a common underlying cause just to see if there are

patterns, because my concern is that if you rely on external assessments and 

standards, there are so many of them out there that they will be - there may

not be any consistency in reviewing the information. So I would love to hear 

your comments on that.

Bakul Patel: Yes. Naveen, great question. I think let's focus on - we’ll definitely answer 

your question. We have 15 minutes at the end of the webinar to take those 

and some of the questions we didn't get to ask before so in the previous 

component. 

But let's - I would like us to - I will - however, we hold off in answering that 

question after we’ve run through all the four components, it may actually 

answer your question. So let's go to the next caller who has a question teed 

up.

Coordinator: Our next one will come from (Ashkan Rizzoli). Your line is open. And 

again, if you'd like to ask a question on this session, please press star then one. 

That’s star then one to ask a question. Thank you.

(Ashkan Rizzoli): Okay. I'm trying this now. Is this working?

Coordinator: Yes. (Ashkan), your line is open, sir. 

(Ashkan Rizzoli): Great. My question is not about the challenge question, mostly about the 

status of the scorecard. And I was wondering how close that is to being done,

whether there's drafts of it available?
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 21

Bakul Patel: Yes. So great question. This is Bakul. I think we will talk about scorecard 

down the road. We have - we are more focusing right now on making sure 

that we have the elements to deliver into the scorecard from the - I think the 

concept we discussed at the January meeting was more about sort of how to 

do you sort of visualize that output from the appraisal and how can that be 

then shared across FDA or with FDA or with others for that matter?

So great question, (Ashkan). But why don’t we - I think we’ll - since we 

answered this one, I wanted to - wanted just, folks, to ask question about the 

review determination part and the choices that we laid out in the working 

model. If folks have questions on that, I’ll take them now.

Coordinator: One moment please.

Bakul Patel: It looks like we have somebody in the queue. While we're waiting for that to 

happen, (Diane), just to sort of give more insight into - to folks who are 

listening to this call and listening to this webinar, what would you suggest? 

Just like I asked the question to (Chuck) earlier, what would you suggest 

people should think about differently when we are co-creating this program 

together?

(Diane): So I'm going to respond more broadly than just this particular topic, because 

one of the things that I find most interesting is, especially as regulatory 

professionals, we’re often used to thinking of this is the regulation and this is 

my SOP that supports it, and here is my documentation and that is where it’s

stored and that's what I take out and show an inspector. 

And what I think I'm finding very interesting about the approach to this pilot 

is, it's not - back to what Cisco said, it's not about compliance. And the first 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 22

time he said that to me, you know, I really kind of struggled with, what does 

that mean? 

But now as the months have gone on and the interactions have increased, it's 

more about getting credit for what you're doing, when what you're doing is 

being done for the right reason i.e. It drives the quality of the product. It 

supports the culture of excellence. 

So that to me is a very different paradigm. So maybe FDA would be looking 

at slightly different aspects of the business than they would in a QSIT audit 

for example. But I think that it's very interesting that it's about what is your 

business doing? Is it - are they doing it for the right reason and how do you 

get credit for it? As opposed to this prescriptive taking and tying to a

regulation. I think it's a very exciting concept. 

Bakul Patel: Thank you, Diana. That's exactly the mindset we're looking for. We have a 

couple of people online while you were sort of sharing that thought. So we 

could take the next question. 

Coordinator: Our first one comes from Agata Anthony Your line is open, ma’am.

Agata Anthony: Hi. This is Agata Anthony from GE Healthcare. I was wondering, with 

regards to the review determination and the IMDRF definition statement, has 

the FDA and the pre-cert participants attempted to tie the existing device 

classifications to the definitions in the IMDRF definition statement? And do 

all of those categories fall into the definition of a medical device as defined by 

the act?

Bakul Patel: It’s a great question, Agata because if you look at what language we used in 

the review determination concept, and we actually use - we try to explain this 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 23

in the document itself is, we are actually revisiting the entire paradigm of ou0r 

classification when it comes down to Software As A Medical Device. 

They’re reimagining and not trying to like fit in. 

And the need for doing that is we have an opportunity here to leverage 

technology, leverage business practice and deliverable performance to make 

sure patients get the best options. So we are purposely not using the word 

submission, we are using the word review. We are purposely not using the 

word compliance, we are using the word excellence. I think that is by design. 

What we are looking at here is what should be reviewed when we know a 

company’s excellence - where they fall - how excellent they are. 

What should we review when we know we can collect this fantastic real work 

performance information that we can collect, because of the connected 

technology we have? So that’s why we are not looking at the moment to 

classify in the traditional sense, but we are looking at it from taking a step 

back and seeing what’s the best thing to do and what’s the best oversight to 

do, to have for those products? 

With that, thank you for this question and I apologize for folks who are in the 

queue to ask a question, waiting to ask a question in this section, but please 

hold your thoughts. We will take those questions at the end of the session, 

after we go through the last other two - the topics. With that, I’m going to 

turn it over to streamline review to (Adam) and have him talk through the 

couple of slides on what does it mean when you talk about streamline review 

and what’s the scope? 

(Adam): Thanks Bakul. I also want to send thanks out to the entire streamline review 

team as well as everyone who has actually joined today on this call, because 

this truly is a collaborative effort and we need your input into each one of 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 24

these pieces, to make sure that the program actually works in the best way it 

can possibly do so. 

You know, as Bakul said, we have a number of different parts going on here 

and, you know, specifically within streamline review process, we are trying to 

set up a process for how we would actually receive, evaluate and determine 

the safety and efficacy of a (SaCMD) product, from a company that actually 

goes through the precertification process. 

Now, you know, as (Diane) mentioned, you know, part of this is really to 

reduce the burden on both the manufacturers as well as ourselves and make 

sure that patients have access to high quality cutting edge technologies that are 

safe and effective. You know, when I break down the actual work that we’re 

doing in this component, you know, and I’ll do it very simply, into, you know, 

essentially we’re defining inputs and processes. 

And what I mean by that is, you know, we’re looking at the inputs that are 

actually needed for the review process itself. You know, we’re trying to 

minimize and streamline these to just what we need, to actually make that 

safety and efficacy determination. And part of this process is actually 

thinking about different ways we can get access to that information as 

opposed to the traditional routes of requiring it when a company brings a 

product in. 

You know, we’re also thinking about this, you know, setting up pathways for 

both new and existing products that we’re thinking about this for, you know 

new products that have never, you know, never been seen before, as well as 

for modifications. And, you know, ideally the idea is to leverage existing 

processes that we have already in place, as well as to map out new ones that 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 25

can make it more efficient and at the end of the day, create a more efficient 

path to market. 

And so part of this is actually identifying, you know, some type of interactive 

process we’re conducing that review. And if you look at the challenge 

questions, which are now up, we’ve largely framed these around those two 

component pieces. And if you think about it, you know, you look at some of 

the examples in here, we’re asking what product specific content would be 

expected to be reviewed in the pre-market and which elements could be 

shifted to be part of the excellence appraisal process or even all the way out to 

post-market? 

And so that’s what we’re looking for, you know, help from the community is 

to really start thinking about what those critical elements are and where and 

how we can actually collect those. 

Bakul Patel: Great. Thank you (Adam). As we’ll get folks to line up to ask questions on 

this particular component, let me just ask (Larry) who is also a pilot 

participant, to ask a similar question I asked (Chuck) and (Diane) is (Larry), 

you’ve been in the industry for a long time, how would you think about this 

from a - completely from off the scratch of the clean slate? What was your 

reaction when we - when you first heard that and how would you help others 

sort of understand what we are trying to achieve here, so they can actually 

help us? 

(Larry): Yes. Thanks Bakul. Yes. I mean this is an area where I think the rubber 

really hits the road. So this is kind of showing your work, so I would hope the 

- I would expect the public would have, you know a lot of energy around this, 

particularly after reviewing the working model. But, you know, I think as you 

mentioned, I mean there’s a lot of room here for what I call practical 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 26

creativity. I mean we do have a blank slate here, so the intent of the, you 

know, streamlined review, is, you know, really to fully leverage the 

excellence appraisal on certification. 

So I think, you know, taking that into consideration, you know, the materials 

that would be in the review for a product, as (Adam) mentioned, would be 

kind of an hour demonstration if the company does what they say they do. So 

there really shouldn’t be any debate there. And they’re following the 

processes that FDA has already assessed. 

Bakul Patel: Great. Thank you (Larry). As we take on more questions on this particular 

component, let me just go back to (Adam) as people are lining - queuing up or 

trying to formulate a question. (Adam), you asked about - or you talked a 

little bit about, you know, shifting some of the - or revisiting and looking at 

during a review there are maybe things that we may not need to see or we may 

need to see differently and how we see it. So there is a content part and there 

is an approach and how we do it part. Can you talk a little bit more about 

that? 

(Adam): Yes. And, you know, I’m not going to be all inclusive here when I say these 

things, but, you know, we’re really thinking about what it is that a company is 

bringing into us. I mean when you think about, you know, some of the 

components of a software submission today and I apologizes for using that 

word, but when you look at it today, you know, you’re thinking about the 

intended use, the device description, the risk analysis, configuration 

management, the clinical performance, labeling, etc. 

And, you know, the question that we’re asking ourselves is which of those 

pieces do we really need as part of that pre-market assessment and which 

pieces might be able to be, you know, be as (Larry) said, you know, can we 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 27

leverage some of the work that’s going on in the excellence appraisal portion 

to help us get comfortable with those components instead. Or maybe they can 

be collected in the post-market setting. 

So we’re really trying to look at the different component pieces that make up a 

submission and ask what is it that we can get down to in terms of those 

minimum key critical components that we need to be able to see and review, 

to make sure that patients are getting safe and effective devices at the end of 

the day. 

Bakul Patel: Great. So I think I’ll make the announcement again to - we are taking 

questions right now, so folks who are like we said before, who are willing or 

want to have some questions on what we put out, we probably should think 

about asking those now, or we can take them at the end of the webinar. What 

- so let me just talk about the very first bullet that you have on this slide about 

specific elements itself. There - our current guidance on software submissions 

talks about a few components that we see today. 

How would you think about that and what kind of input do you guys need 

from people as they’re thinking about this program as a whole in the review 

process? Can you and (Larry) talk through that a little bit with us. (Larry), do 

you want to go first? 

(Larry): Yes, sure. I think that - I mean to (Adam)’s point, I think, you know, 

identifying the value of being pre-certified, leveraging that and probably - I’m 

surprised to be saying this, but minimum viable review I think is kind of what 

we’re talking here, for safety and efficacy. So I think from an excellence 

appraisal, (untellable) point, I mean if you look at the principles around 

cybersecurity, patient safety which I think takes into account risk 

management, product safety which takes into account how a company does 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 28

their testing and how they approach their testing, you know, what their 

processes are. 

I think a lot of these systematic things could be leveraged in a way that no 

matter what product, you know, the company is developing, there would be 

just a kind of, you know, executive summary or some kind of summary of 

those activities. It’s just that why you did the things you did and, you know, 

demonstrating that you were (unintelligible). So I think that would be, you 

know, one example. The post market side from clinical responsibility, if 

that’s been assessed and there’s a, you know, mutual understanding in the way 

that the company approaches (unintelligible) evidence, then maybe for a 

particular product there could be, you know, a very brief - I mean each 

company is committing to doing that by being part of this program and going 

forward in pre-cert. 

So, you know, there could be a (straw) plan about that particular product, how 

they plan to kind of manage it in the post-market. Those would be, you know, 

a couple of examples. 

Bakul Patel: Right. Thanks (Larry). It looks like we have a question before (Adam) gets 

to his part of this question. We’ll ask the question. It looks like the next 

question online on queue. 

Coordinator: It comes from (Scott Steelthiel). (Scott), your line is open. 

(Scott Steeltheil): Thanks and thanks Bakul. Question for you. In the model that is documented 

that you’ve shared, there’s a spectrum of risk of product that’s outlined within 

there and how those are identified. And the document doesn’t call out this 

specifically, but the streamline review seems to be almost a one size fits all for 

any of the streamline review. Is that the case or are you considering and 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 29

looking at evidence that would be needed in the streamline review that would 

be commensurate with the risk of the product that’s being assessed? 

(Adam): (Scott), that’s actually a really great question and one that we’re still working 

through the answer to. But, you know, if you think through that process, I 

mean you could create a system where you’re looking at, you know, a variety 

of inputs even though you’re still looking at the same inherent process that 

everyone would go through. So depending on the risk you might need to have 

a little bit more information if it’s a high risk device for instance, as opposed 

to a moderate risk device. 

You know, I think there the difference is not so much in the process as it is 

how much information we’re going to probably need to be able to get 

assurance. So you can think about it in that kind of a context that risk is going 

to influence the amount of information we get. 

Bakul Patel: Yes. And I would also add to what (Adam) just said. I mean we are shooting 

- I mean as I said in the onset, we’re shooting for simplicity, which means that 

the more nuances we add and more risk based stuff we have, the more 

complex the process becomes. And the whole idea for this reimagined 

paradigm, is to keep it simple that anybody and everybody who is wanting to 

take products to patients, has very clear path. 

(Larry): Yes. And this is (Larry). I think there’s a small element of kind of, you 

know, poetic license on the manufacturer’s side that, you know, they would 

need to decide, we would need to decide what device specific materials that 

the company thinks is necessary to determine safe use. And I think - and then 

inherently there’s kind of a big watch out there that, you know, we’d 

obviously want to minimize FDA requests for additional information, but a 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 30

company has to make that decision themselves. What do I need to provide to 

insure safety and efficacy? 

Bakul Patel: So thanks (Larry). We’ll take the next question, the last question on this topic 

and we’ll move onto the next component. And then we’ll open it up for 

generally, all questions across all four components, but I wanted to give this 

opportunity for the next person on the line. 

Coordinator: Thank you. Your line is open (Sue Hiblen). 

(Sue Hiblen): Hi. I was wondering how can companies that are about to submit new 

technology, be involved to answer some of these streamline review questions? 

Bakul Patel: Go for it (Adam)? 

(Adam): Thanks Bakul. I appreciate that. So I think right now you know, we’re still in 

that formative stage for the actual process, but, you know, we do have a 

docket open, where we’d be happy to take your thoughts on how that could 

actually - what kind of a process would actually be more efficient and 

effective from your vantage point. I think as we get further on, it would be 

great to be able to pilot some of this with the community itself. 

You know, so that’s the other part of this is, you know, once we get further 

along and have a model that we could actually start testing, I think it would be 

ideal to actually start testing that with some of the products that people want 

to bring in house. 

Bakul Patel: That’s a great question and a great response. I want to just encourage people 

to - the reason we are being extremely transparent about this and being 

collaboratively trying to build this program, is not just for the people who are 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 31

having these products, but also people who are planning on doing this, people 

who have spent the time working on software, making products in other areas. 

When they see those questions and when you hear sort of what the program is 

trying to deliver, please, please take a look at that and use those questions as 

prompts to that’s where you can help it. So if you decide to help them, the 

entire program more than welcome. If you decide to work on one portion of 

the program, one question, that’s more than welcome. So please keep those 

questions coming and like (Adam) said dock it. If you have questions on the 

program itself, general questions, without solutions, ask us questions, with the 

email box. 

And we are also looking - you also see the updates going out. So stay 

engaged and provide us feedback. So great question. I’m going to pause here 

and turn it over to the next component and I apologize for folks who are about 

to ask a question, but please hang on and hold your questions until we get to 

the next component. Hopefully that answers some of the questions that we 

have been sort of getting. 

So I’m going to turn it over to (CatherineKathryn), to talk about real world 

performance. 

(CatherineKathryn): Thanks Bakul and I just want to thank the real world performance team 

and really all of the other components, because I think everyone in their talks’

today, in this session, has talked about real world performance as an input to 

monitoring their specific component’s activities. And so I think that it’s an 

important aspect that I’d like to bring to light. So the scope of the real world 

performance team is really defining the real world performance elements and 

defining and developing the methodologiesythologies needed for pre-cert

program activities. 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 32

And we’re looking at that at three different levels. So at first, the product 

level, we think that we can look at real world performance data to support 

post-launch product monitoring, to iensure ongoing safety and effectiveness at

(CMD)of SaMD products. We think that in the future that real world 

performance can also be used to make and modify product claims after a 

SaMD(CMD) product has been on the market. At another level, we think that 

real world performance data can also inform at the organization level. 

It can provide input to initial precertification. So if pre-certified companies 

are state what methods they’re using and what product aggregate real world 

performance datae they have can be used as inputs into their precertification 

status, as well as into that company or that organization level’s maintenance 

of precertification. 

Finally, we think that real world performance can be used to provide feedback 

into the pre-cert program overall. And so we think that organizational real 

world performance data could feedback into for example, the excellence 

appraisal team and we could look at modifying certain aspects of those 

excellence principles or what’s looked at. Or it could also be used in 

streamlined review to look at what elements may or may not need to be 

reviewed in the future, for specific SaMD(CMD) products. 

And something that we wanted feedback on is what we mean when we say 

real world performance. And so in this case we’re thinking about data 

relevant to the safety, effectiveness and performance of marketed 

SaMD(CMD) products and that breaks down into, in our current thinking, 

three different areas, the first of which being real world health data, which 

could be inputs and outcomes related to the intended use of the SaMD (CMD)

product. 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 33

We’re also thinking that performance data could be related to user experience 

data. So outputs derived from experience related to real world use of the 

SaMD(CMD) product. Finally, we’re also thinking about product 

performance data as an aspect of real world performance data, which could be 

outputs and outcomes related to accuracy, reliability and security of a 

SaMD(CMD) product. And this is something that we are looking for 

feedback on in terms of our terminology, and are we thinking about real world 

performance data in an accurate and comprehensive way? 

Bakul Patel: Great. Thank you (KathrynCatherine). Thank you so much for that 

background. I think this is - like you said, it was the most interesting part of 

the program that we all touched upon and we rely on as we move forward. 

Having said that, I want to open up the lines for people to ask questions about 

this component, before we get into - before I get into asking questions. 

I also want to sort of point out that (Sandra) and (Ryan) are also participants 

that are on the line. And if they have an opportunity to think about - now that 

you heard all of the four components, what would you want to say to the 

people hearing this session? So (Ryan), do you - would you want to go first? 

(Ryan): Sure. Absolutely. Thanks Bakul. You know, our experience in this program 

has been really eye opening to see how the other pilot participants are thinking 

about the program, about thinking about how it can be structured in the future. 

And I would really encourage others who are not currently pilot participants, 

from providing input because it’s just been really fascinating to see how 

people are thinking about it in different ways I guess. And the more 

perspectives you all can get, I think the better this program is going to be. 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 34

Bakul Patel: Great. Thank you. Why don’t we go to - take the next question online about 

this part? And (Sandra), I’ll come back to you right after that. 

Coordinator: At this time, if you’d like to ask a question on this session, please press star 

then 1. That’s star then 1. Thank you. 

Bakul Patel: Great. (Sandra), did you have any thoughts on this particular component that 

you want to share with the rest of the folks? 

(Jeannie): Hi Bakul, good afternoon. This is actually (Jeannie) calling in for (Sandra). 

With regards to real world performance, I think with respect to the product 

that we had on the market and (unintelligible), I think that we had a great 

opportunity to collect real world performance data and use it to demonstrate 

the efficacy and safety of the products. In addition, I like to use this real 

world data to try to validate what the digital health team was doing with the 

streamline reviews. So I do encourage everyone in the industry, to...

Bakul Patel: Great. So we have some folks on the line. Thank you (Jeannie). We have 

some folks on the line waiting on the queue to ask questions. So we’ll take 

the next one. 

Coordinator: Our first one will come from (Anjut). Your line is open. 

(Anjut): Hi. I had a question regarding complaint handling. So in this process will

there be any difference for complaint handling? And how will that affect the 

pre-certification of the company if there are complaints in the field, from the 

real world data? 

Bakul Patel: Well so if you look at what (Catherine) suggested, it’s about user experience 

and product performance. Think about it from that perspective. So people, if 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 35

they’re giving you feedback for - as we call complaints today, think about that 

as what does that mean in those terms of is the user experiencing issues? Is 

the product not performing or is it really having a topic or a condition that 

they’re thinking about it from a real world health data perspective? 

So that’s how we are thinking about this in this world where really the 

information that is gathered from the performance of the product in the field, 

is really what we want to leverage. 

(Anjut): Okay. Thank you. 

Coordinator: The next question will come from (Hanji). Your line is open. 

(Hanji): Hi. Am I audible? Bakul, this is (Hanji). My question to you is for some of 

the new technologies that we are trying out, the real world evidence may not 

still exist, right? So how much of such evidence would you really like us to 

have before we feel that we have a good starting point here? 

Bakul Patel: So, you know, (CatherineKathryn), do you want to take that question? 

(KathrynCatherine): So I think that it depends on a lot about the product that you’re talking 

about and if you’re pre-certified. But if I understand your question correctly, 

you know pre-certified companies that maybe ongoing and collecting real 

world performance data, you know, that could lead to what pre-cert status you 

have with the different levels that we’ve been talking about at this time. 

Whereas if you’re potentially a newer company with not as many products on 

the market or maybe not any SaMD (CMD) products at all, maybe you might 

be in a different tier with a pre-cert status because you haven’t had that 

demonstrated track record of gathering that real world performance data. 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 36

Bakul Patel: That’s a great answer there (KathrynCatherine). Thank you. And thank you 

for the question. I think it’s important to - and this is exactly the feedback 

(CatherineKathryn) is looking for - is, are we collecting the right information? 

Some of it might be - already been collected, but be called differently. We’re 

trying to structure that. I think that’s what we are trying to do here. And if we 

can structure that it’ll be useful for everybody. We’ll take the next question. 

Coordinator: The next question comes from (John Savak). Your line is open, sir. 

(John Savak): Yes. I had a question. This is - this sounds a lot like the types of data that 

CMS and AMA are trying to collect around 99091. Are you working in 

conjunction with those groups? 

Bakul Patel: So we - at this time, we - I mean other part of (CDRH) work with CMS all the 

time. But I think as we are exploring this and we haven’t actually tried it in 

the real world, we - it’s hard for us to sort of have that conversation. But yes, 

we - as we move forward and as we understand more exactly what can be 

done, and how it can be done, that’s the least burdensome for both FDA as 

well as the people participating in the program, I think that’s the better 

conversation to have. 

So I totally agree. There’s a lot of work being done in this area in other parts 

that can be leveraged and be benefitted - can be benefitted from. So I think 

that’s exactly what we are trying to mirror. And how we do it that’s closest 

and the most useful for not just people who are producing and us as FDA, but 

also patients at the end of the day. That’s the most important part for us. 

Next question. 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 37

Coordinator: Our next question will come from (Ashkan Rizzoli). Your line is open. 

(Ashkan Rizzoli): Yes. My question is about, on this slide, what is the difference between the 

real world health data and product performance data exactly? I’m trying to 

figure that out and I can’t quite see that clearly. And then I don’t know if I’m 

allowed to do it, I’ll sneak in a follow up question. On user experience, the 

IMDRS guidance encompasses both PDS and CDS offers, so patient decision 

support software. 

And so when we talk about user experience are we looking both at, you know, 

feedback as the nine as, you know, the UI is not aesthetically beautiful versus 

let’s say the UI for example, is actually difficult to use or causes the user to 

misuse this particular product.

(CatherineKathryn): Sure. So let me take your first question. In terms of real world health 

data, we could be talking about things related to clinical effectiveness or 

adverse events or things of that nature, so outcomes related to clinical outputs 

and things of that nature. In terms of product performance, we could be 

talking about outcomes and outcomes related to cybersecurity performance of 

the SaMD(CMD) product or the accuracy that that product has. 

So in the health data aspect there’s more of a clinical stint compared to the 

product performance which is specifically inherent to the software of the 

SaMD(CMD) product. 

Bakul Patel: That’s exactly right. Thanks (KathrynCatherine). Next question? 

Coordinator: The next question will come from (Barbara Heglund). Your line is open. 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 38

(Barbara Heglund): Yes. I was wondering if the real world could be planned for in the 

products if we could help people to plan ahead by having requirements about 

what we’re going to report, what sorts of things we’re going to capture and 

what sort of criteria we’re going to use to weight those things. 

(CatherineKathryn): So that is something we’d really be looking for feedback on. You know, 

we’re currently thinking about the different data elements or analyzed data 

that could encompass these three different types of data that we’re talking 

about. But we would really like to get your input on what you think we 

should be looking at as well. 

(Barbara Heglund): Okay. Thank you. 

Bakul Patel: Great. So this is a great segue into sort of opening it up broadly for the entire 

program. Now that you heard all the four components, four major 

components of the program itself, how we are thinking about building it, a 

little bit more deeper dive into what we had published in the working model, 

and you saw the questions that we asked in the working model and it gives 

you a color of like what we’re looking for and the depths that we’re looking 

for. 

Hopefully this was helpful for you in terms of how we are exploring and what 

kind of input we need. So I want to open up the call - the questions from 

everybody on the call, to ask questions either about any of the four 

components that we may not have gotten to or not have an opportunity to sort 

of ask or answer. And anything else about the program that you want to learn 

about. 

So this is probably a good time to open up for that. So let’s get into the next 

question that we have lined up. 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 39

Coordinator: Thank you. Again, at this time, if you’d like to ask a question please press 

star then 1. Our next one will come from (Devoni). Your line is open ma’am. 

(Devoni): Hi. Really great presentation - can everyone hear me? 

Bakul Patel: Yes. 

(Devoni): Okay. Really great presentation and really awesome program. The first 

question I had, I had a lot, but I couldn’t ask them during the session, I 

seemed to always miss the exact time I could speak about it, was about the 

new guidance that came out on I want to say like kind of like modular 

submissions and how that relates to the pre-submission program. 

Is that something where let’s say you had a medical device where there’s a 

hardware component to it and that part wouldn’t classify as a software as a 

medical device, but you have other components that are software as a medical 

device, you could separate out your kind of submission to have, you know, 

one Part B submitted as, you know, through the regular process and the rest to 

possibly follow precertification? Like to be a part of the precertification 

program? 

Bakul Patel: So the - let me just take a step back. I think you’re asking about the 

multifunctionality guidance. 

(Devoni): Yes, the multifunctionality one. Exactly. 

Bakul Patel: So the point of the multifunctionality guidance is products have many 

components built into them, some of them are devices, some of them are not 

devices and some of them are exempt from review and some of them are not 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 40

exempt from review. The guidance talks about the policy of how the FDA is 

going to use - or what criteria it’s going to use and what kind of questions 

people will ask if you have a device that has those components. 

Now you can imagine, that is not necessarily meant to be that you can split 

your first submission. It is about, you know, when you’re submitting 

something to FDA, if you have a component which is not a device, but it has 

an effect on - an adverse effect on the device component or device function, 

the medical device function within your product, that’s - the questions we’ll 

ask is what we are focusing on in that particular guidance. 

So it’s less about the regulatory path as opposed to more about how FDA is 

going to treat a submission that has those multiple things built into them. 

(Devoni): Okay. And for that even if there is separation between a PMA and a 510K 

multifunction, you would - that would just be the impact of one on the other? 

Bakul Patel: Exactly. I mean that’s the whole point of that guidance is when we are 

reviewing a certain functionality within a product, what’s the impact - adverse 

impact that can have on performance of the product that we’re reviewing, 

from those things that we are not reviewing? So that’s really what they’re...

(Devoni): And it won’t tie to the pre-cert in any way, in terms of having a hardware 

component? Let’s say your phone has a component that is considered 

hardware, software as a medical device because of its functionality. And then 

you have other aspects of your software that are purely software as a medical 

device as...

((Crosstalk))
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 41

Bakul Patel: We had to think about this - I mean there are examples in the guidance 

documents that talk specifically about that scenario that you just painted. So I 

would encourage you to sort of look at that and if that raises other questions or 

comments or things that we should consider are very highly - I encourage you 

to sort of provide that feedback to us, to the docket, on that guidance 

document. Thank you very much for your question. Let’s go to the next one. 

Coordinator: The next question comes from (Garanji Samra). Your line is open. 

(Garanji Samra): Yes. It’s on real world performance. This is a follow up on that, our postmarket performance. I’m curious if you can pull that upstream a bit, talking 

about possible solutions to that regards. So can we have something - like the 

pre or the post-market. The ideal beta site testing, you know, working with 

five or ten beta sites is not new. So - especially for new companies, right, 

they don’t have anything in post-market. 

So why can’t the industry work with, and small companies work with, you 

know, some meaningful sample of five or ten beta sites and such, collect 

meaningful evidence, you know, pre or the post-market and have that 

substantiate, you know, via the streamlined review process or otherwise? And 

we are not even talking about, you know, the software developers they use. 

Obviously (unintelligible) and we are not even talking about the (sprint) 

demos and such that happens every two weeks, four weeks, six weeks 

anyway. 

So we are not talking about that, but any sense of maturity of these software 

products going through the lifecycle, but really when they’re ready for the 

primetime working at these beta sites and collect the pre, you know, pre or the 

post-market data and be able to substantiate. So that’s one thing. 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 42

And another follow up on that is, you know, the - as you collect more and 

more field performance, even after you are commercializing, you know, the 

earlier idea of the frameworks with the metrics and dashboards, I’m 

wondering if clearing a field performance file or something like that, will 

make sense, you know, to gather the lifecycle activities before, during and 

after. 

So ideas on the lines of real market performance there. Thank you. 

Bakul Patel: Great. Thank you for those comments and they are great solutions and like 

you said, you were actually providing solutions. So we’ll take those ideas into 

consideration. But I think in general I’ll just respond back to the idea, is I 

think we do want to make sure that people use whatever information they can 

collect in either real world or beta testing or any other way they can do that, to 

get that high level of confidence that you - that we all need and everybody 

needs, not only the developer, but also the patients who we are serving. 

Thank you. Let’s go to the next question. 

Coordinator: The next question comes from (Tipran Nadic). Your line is open ma’am. 

(Tipran Nadic): Yes. My question was as far as the program is concerned, is there like an 

overall schedule on when it’s going to be rolled out officially? That’s one 

question. And on the real world data, I also wanted - I had a follow up 

question regarding is there expectations on the duration of time we have to 

collect the real world data or expectations on how many months or years the 

device has to be marketed to collect this kind of data? 

Bakul Patel: Yes. Great question. We haven’t solved all of those details yet, but our - we 

have been talking about the concepts of - two ways - one is if the products that 

we’re talking about in this space are an enabler in collecting this data, we 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 43

want those products to be instrumented that can collect the data first, because 

if we missed the opportunity to collect the data then we will never do it after 

the fact. 

So that’s one of the concepts we are trying to embed into this program, that 

people patriating in the program will do that. The second part of is taking into 

account where the products are. That’s why the criteria that (Catherine) 

shared are generic, that can be applied across other products as well. So that’s 

how we think about it. And that’s sort of explained in the working model. 

But if you have thoughts and comments and there are use cases that would be 

helpful for us to consider, we’d be very open and looking forward to those 

kind of input. 

So we’ll take one more question after this, before we wrap up this webinar. I 

think we can go on for the entire day, but I think everybody’s time is valuable, 

so we’ll take one more question next, before we wrap the webinar for today. 

Coordinator: The next question comes from (Matt Trachtenberg). Your line is open sir. 

(Matt Trachtenberg): Oh. Hi Bakul. Thanks so much for sending out these challenge questions, 

so we can understand what FDA is focusing in on. I wanted to ask a 

clarifying question about the challenge question on cybersecurity. It says that 

cybersecurity issues often circumvent intended use and how can or should this 

be considered when determining risk level? Are you able to clarify what you 

mean when you say cybersecurity issues often circumvent intended use? 

Man: Yes, hi, this is - that’s a great question. And one of the things in that 

challenge aspect that we’re really trying to get at is the agency recognizes in 

cybersecurity in particular, there’s a lot that can be done ahead of time. But at 

the end of the day all of that is only as smart as the next person who really 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 44

wants to try to prove something or do something with the system or challenge 

their own capabilities. 

So that’s really about how do we incorporate the organization’s ability to react 

to that, right, that assessment? How are they on top of that and how do they 

really take that and respond to that? That’s really what we’re trying to get at 

with that question. 

Bakul Patel: Great. Thank you everybody. But before we go forward, let me just share 

with you how - where we are in the program. We are very early in the 

program, maybe not so early in the program, but 1/3 of the way into the 

program. What we are trying to do, if you see my slides, is trying to launch a 

pre-cert 1.0 by the end of this year. We will be looking at testing the program 

just like - I know one of the callers talked about testing the program and 

testing their products and trying it out and working out some of the issues. 

So that’s what we plan on doing in 2019. And we’ll have another iteration 

that will go out at the end of 2019. But that’s sort of the general schedule. I 

also wanted to share with you a way for you to stay engaged. We will be 

updating the working model on a frequent basis. When I say frequent, it’ll be 

in the 30 to 45 day timeframe. At the same time we want to sort of give you 

additional information than what we had published before. 

We plan on taking - we actually will be taking input on a rolling basis, so 

folks who are providing input to the program through the working model, 

schedule or other tasks, we want them to be sort of doing that as soon as we 

have these ideas put out and concepts put out for comments. We want you to 

stay tuned for future user sessions as we will continue to share our thoughts or 

feedback we have received so far, and hoping that we will build upon the 

feedback we get and some of the thoughts that we heard here as well. 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 45

I’m sure many folks had questions today that we couldn’t get to. I would 

recommend that people look at the working model, provide their input and 

thoughts into the docket that we have continuously open. We also encourage 

people to ask questions about the working model. If you don’t know what 

kind of solutions you - that we are looking for, ask the email that we have on 

the screen, FDA Pre-Cert Pilot email address. And then look for the program 

updates on the Web site. 

We plan on doing this, as I said, collaboratively throughout the way, towards 

December, but we are looking for giving you information as we think, as we 

evolve, as we build the program going forward. So I want to thank everybody 

who participated in the program and I’m going to turn it back to Irene. 

Irene Aihie: Thank you. This is Irene Aihie. We appreciate your participation and 

thoughtful questions. Today’s presentation and transcript will be made 

available on the CDRH Learn Web page at 

www.FDA.gov/Training/CDRHLearn, by Friday, May 18th. If you have 

additional questions about today’s presentation, please use the contact 

information provided at the end of this live presentation. As always, we 

appreciate your feedback. 

Following the conclusion of today’s user session, please complete a short 

survey about your FDA CDRH user session experience. The survey can be 

found at www.FDA.gov/CDRHWebinar immediately following the 

conclusion of today’s live user session. Again, thank you for participation and 

this concludes today’s user session. 

Coordinator: At this time that would conclude today’s conference. You may disconnect 

and thank you for your attendance. 
FDA 

Moderator: Irene Aihie

05-10-18/12:00 pm ET

Page 46

END