FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 1

FDA- PreCertification (Pre-Cert) Pilot Update

Moderator: Irene Aihie

November 17, 2017

1:30 pm ET

Coordinator: Welcome and thank you for standing by. At this time all lines are in a listenonly mode. During the question and answer session please press Star 1 on 

your touchtone phone. I would also like to inform parties that today’s 

conference is being recorded. If you have any objections you may disconnect 

at this time. Now I’d like to turn today’s conference over to Miss Irene Aihie. 

Thank you ma’am you may begin.

Irene Aihie: Hello and welcome to today’s FDA Webinar. I am Irene Aihie of CDRHs 

Office of Communication and Education. On July 27th the FDA announced 

the launch of the Digital Health Software Precertification Pilot Program. The 

FDA recognizes that we need a regulatory framework that accommodates the 

distinctive nature of digital health technology, its clinical promise, its unique 

user interface and the industry’s compressed commercial cycle of new product 

introduction. 

This is the first in a series of Webinars on the Digital Health Pre-cert Pilot 

Program. As part of our commitment to keep our efforts transparent the FDA 

will share updates throughout this pilot process. Today Bakul Patel, Associate 

Director for Digital Health in CDRH, will present a model for demonstrating a 

culture of organizational excellence that was discussed with the pilot 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 2

participants during their kickoff meeting. Following Bakul’s presentation we 

will open the lines for your questions related to information provided through 

this presentation. Additionally there are other Center subject matter experts to 

assist with the Q&A portion of our Webinar. 

Now I give you Bakul. 

Bakul Patel: Thank you Irene. Welcome everybody to the Webinar and this is a status 

update as Irene mentioned. In an effort to be very transparent through this

collaboratively creation of the pilot program I want to start off by saying one 

of the things we heard very loud and clear as we engaged with the pilot 

participants is be clear. This pilot is not a typical pilot. We are actually 

developing the content of the pilot with the pilot participants. 

So I wanted to share today what we shared with the pilot participants as a 

kickoff and early in the past month, we learned a few things and shared 

lessons learned and provide an opportunity for folks on the phone to ask 

questions about the program itself. Like I said it’s a developing program for a 

pilot. And as we figure out, as we learn from the participants, we want to 

learn from you as well. So we welcome any input and feedback from you all 

as well and this is why we’re holding this Webinar.

To give you a background on what we covered in the kickoff meeting with the 

pilot participants: We revisited and level set our goals and objectives. And 

I’ll spend a little bit more time again even though we talked about it last time. 

We wanted to share where we were coming from and going to hear from them 

as well. And share how we are going to engage with the pilot participant and 

also look at questions we’ll be asking to the pilot participants for learning how 

to do and develop software and maintain a high quality of software product. 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 3

And understand how they would consider excellence and those common 

perspectives we have shared before. 

And I want to open that up afterwards and we did have a question and 

discussion and I will share some of those as well. But we welcome folks on 

the Webinar today to ask those questions of me as well.

So we did start off and just to reiterate we’re looking at a program that is 

company based and moving away from a traditional model of a product base. 

How can we create a company based program that will lead to a streamlined 

regulatory approach for software and medical device that relies on culture of 

quality and organization excellence.

And we have shared our intent is to bring the trust and the technology. We 

need to bring the confidence in people making the technology. So how can 

we get there and how can we realign with the timelines that software is being 

developed, how do we align with practices people do and this is exactly what 

we’re trying to learn is how do we learn those practices. And then finally 

looking at the software as a medical device principle that we have developed 

in the IMDRF framework we can align globally as well. 

We’ve put this concept out there. It’s important for us to walk through this 

again one time just to get on the same page. It’s looking at creating a program 

that will certify companies and understand how to create software as a 

medical device, what excellent principles they use and trying to understand all 

those principles as part of the program. And then, you know, based on that 

excellence what products can be brought to markets right away and what

products would are required of you.
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 4

We are in this process of understanding how this program can stay nimble and 

learning as well. This goes to the objectives and I’m going to spend a little bit 

more time here than other slides. And this is emblematic of what we did in 

the kickoff as well as the pilot discipline. I want to emphasize we want to 

create an efficient regulatory framework and efficiency’s not just for, you 

know, industry or any one stakeholder. It’s also for FDA. We are going to 

keep that in mind as we move forward we want to make sure us at FDA being 

more efficient ultimately allows products to be available to patients in a timely 

manner.

Having said that we want this pre-cert program to be easily maintained by all 

who are participating in this program. We want to rely on these excellence 

principles. We are focusing a lot more on excellence than trying to figure out 

whether we need to be very compliant or not. 

Having said that using key performance indicators is really one of the 

concepts we are putting on the table for us to understand based on what people 

do or what we learn from participants, how can we sort of scale that to 

organizations of different sizes, different development strategies, aligning 

process freedom for people to develop software at the same time allowing 

those measures to be something that the companies or the organizations 

develop on their own. But at the same time can we get them to be aligned to 

those key principles that we all care about.

We want to scale this program for a variety of sizes of organization 

management processes they use today. And look in a holistic way of not 

keeping this program static and create something that will be out of the gate 

but also have the ability to sort of learn and adapt over time to make sure the 

program phase is current. 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 5

And with that most of you have seen this. And you’ve seen a lot of things that 

we have to do going forward as building the program. This is what I call our 

backlog of things to do in the program itself and what we want to do is we’ll 

make this public. We’ll make this list public and we’ll see that things that we 

will start to solve as time goes on. And we’ll encourage folks to help in 

building this, provide input along the way and be collaborative. 

One thing I would say for sure that this is a unique way for us to co-create this 

with the stakeholders, all stakeholders, providers, industry manufacturers, 

software developers and other interested parties who can help us create this 

program that is best suited for patients at the end of the day. And we want to 

make sure that happens. 

So we can do everything all at one time. So what we shared with the pilot 

participants is we started the pilot. We are going to take a focus approach. So 

in the initial part of this pilot program the next few months they’re going to 

try to dive deep into this organization excellence piece. Where we are trying 

to learn is what does excellence really mean, how can we recognize it, how 

can we measure it in a way that’s consistent, not looking at one we are 

measuring but also and going to my principles we’ve shared how can we sort 

of know people are doing the right thing, how do we find that trust, how do 

we know it happens in a way that we think is best for public health. 

We want to be able to try to collect that information. We want to collect it. 

We will share with everybody and have a conversation largely with the entire 

stakeholder community in January. We are planning on having a public 

meeting at the end of January on January 30 and 31 so if people are listening 

save that date. We will be announcing that shortly but that’s where we are 

going to take the learning that we have from the pilot participant and have a 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 6

larger discussion, dive deeper into what we need to do next and so on. So 

that’s the space. 

Going forward as we saw there’s another big component of what the 

(unintelligible) view look like. How should we streamline that? What should 

that look like in terms of the roles for what a patient should do? What can be 

reviewed? How it can be reviewed so those questions will be asked in the 

next space. And, you know, along the same lines we will start looking at once 

we have a product in the market what’s the real-world data collection look 

like. What kind of access we can get. What can we learn from that 

collectively not just as an FDA but as a stakeholder and how can we be more 

transparent about that. 

So here’s what we are doing as we do the site visits with each of the pilot 

participants. We have assigned five to seven team members to each of the 

pilot participants with one point of contact. (Marisa), (Cathy) and (John) who 

are part of my team are going to be the point of contacts for this pilot 

participant. How they’re attending all the site visits with representation from 

premarket office, post market, compliance and we bring them on board. 

The idea here is to have a diverse perspective as we look at excellence as 

people do it in their organization in their native form so we can learn from that 

and then present it to the larger community to sort of bring it back together 

and set the right bar where excellence should be. I would have to say that we 

are doing if you see on the previous slide in addition to the pilot participants 

we want engagement from everybody including folks, like, you on the phone 

today and others also. So we encourage that to happen. 

A rough schedule and a plan that we shared with the pilot participants is we’ll 

have the kickoff meeting. We’ll start collecting information. We’ll start 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 7

analyzing that information and we’ll share that with the public as well and 

everybody who is interested in helping us. The asterisk as you see is 

something that we would start publishing on our Web site. You’ll see those 

Web sites up there come up often on the pre-certification initial health 

Webpage. This is an opportunity for us to share publicly what we are seeing, 

what we would like to see and sitting in for some kind of participants and so 

on.

This is just another presentation for how we’re pursuing the program but 

really comes down to collecting, aggregating, finding and then sharing 

publicly and getting broader input. Truly in the sense of co-creating the 

program and seeking input from everybody.

Let me share the model that we shared with the pilot participant. In order for 

us to understand excellence we put out these five principles is how do you 

provide safe patient experience, how does the organization look at quality and 

what’s excellence in that perspective. What does sponsoring mean for an 

excellent organization? What does cybersecurity responsibility mean for an 

organization and then how do we assess that a company is truly proactive in 

its culture. 

Taking these concepts we are asking folks to look at the description that we 

have and help us refine them. And we’ll go over that in the next few slides 

but this is an excellent point. The model is if a leader of an organization 

would have its scorecard and have those excellence principles and has 

something that detracts how would they detract from people such as 

organizational and processors that they have in place. And balance it out or 

understand if those people in the organization that they have in place and the 

processes they have in place truly the results and an outcome from customer 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 8

responding back to those processes and people and organizations that they 

have. And how are they moving forward and learning and growing. 

Again this is an initial model. We’re looking to refine this over time. So you 

can completely expect this to either refine further or even change it from

(unintelligible) base of input. And again this is not just with the nine 

participants. We encourage anybody who has input or feedback we 

recommend submitting your comments to the docket. 

Here’s what we want to see happen is taking those five principles, looking at 

the different perspectives. If we could sort of identify key performance 

indicators and a library of those and the best practices that can happen and that 

could get standardized over time. But being able to recognize those 

connections that people do in terms of processes and measures and how do 

they align to those is really what they’re looking for.

Now this is a completely different approach than what’s been employed in 

evaluating certain organizations. But allowing this freedom is really what the 

intention is. Howe we get there is something that we really need help from 

and we are hoping to learn that with the pilot participants and help us get to 

the next level. 

Here’s how we are looking and this is what we shared with the pilot 

participants to take that concept which I just shared in a couple slides ago is if 

we take one of those excellence principles and starting with patient safety 

what question would you ask of an organization or another organization not 

yours and what indicators or performance indicators would you want to see 

that aligns from an organization perspective and a profit perspective that lines 

up to the patient’s safety. 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 9

So we provided a list of questions as examples, as guiding questions for 

people to provide input to or at least have the thought process going forward. 

And you can see there’s other slides that talk about each one of those 

principles and have different questions answered then. Granted we had to 

clear up that there is a lot of duplication or sort of overlap in the questions you 

ask. But there is a purpose to make sure that we can ask the right questions in

different ways that can really identify an organization that’s excellent.

So skipping to some of the slides what we asked the pilot participants is how 

would you refine these excellence principles and just common validating 

perspectives of an organization that you think is excellent. And how do you 

think a person or an organization other than you can demonstrate that to a 

third party or to your leadership in your organization. And how can we look 

at an organization that is using metrics to evaluate performance of this 

excellence itself. 

So we’re looking for this very broad, very high level for feedback and trying 

to learn these things as we move to the site visit. We made it really clear that 

the site visits are not audit per inspection. It’s truly to understand current 

practices that are used in these organizations and this is the reason why we 

picked the spread we picked and is to understand different perspectives both 

large and small and the type of industry they come from to understand how 

excellence can really reflect into the model that we are looking to build. 

So here’s what we’re specifically asking which is asking folks to do. And this 

reflects some of the feedback during the conversation we had with the kickoff 

with the pilot participants is we’re specifically asking people to modify the 

descriptions of the excellence principles, identify other principles that they 

think may have been missing or should be included. And also defining, 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 10

maybe refining common values and perspectives and just giving them an 

understanding of that. 

The second big thing that we found was important for us to clear and as we 

sought input from the nine participants or learned from the nine participants 

was how can the processes that folks measure or use and measure align to 

their business needs. And I want to emphasize to their business needs rather 

than the traditional regulatory approach. 

What I mean by that is we didn’t want to collect input that showed processes 

that are aligned to their current, you know, CFR regs or something, like, that. 

But rather to why is it important to them to maintain, to have the processes, 

why is it important to them for measuring those things in their organization. 

So truly back to why people or understanding why they do what they do. And 

then taking those things and asking those questions and mapping them to that 

excellence and put into perspective what we’re looking to get to sort of get to. 

Understandably we also learned – I can share with you we have done three 

site visits so far. And we’ve learned phenomenal, great things from all of the 

three site visits. So what can out to be really clear is we had to be very 

explicit in terms of asking what we are asking and what’s said in the slide. 

But also taking and effort from going to what they do in day to day life in 

their organization and then translating it to something that can be 

generalizable for their intent and objective of tailoring it to all types of 

organizations. 

So that work is going to happen next but we ask and learn from the 

engagements that we have so far is that higher level abstraction is more 

important. But to understand that abstraction we needed to know what people 

do today. So this was an iterative process as you can imagine. We didn’t 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 11

have a great answer to begin with or to take questions. We are iterating even 

on the questions. So I would hope as you guys think through these and have 

questions that you may want to ask that we can clarify and help us, like, shape 

those questions so we can get to those objectives that we will want to see sort 

of ultimately be established. 

We shared this spreadsheet. We asked folks to fill out this spreadsheet with 

what questions would you ask in your organization that aligns the patient’s 

safety from a resource and organization perspective and why do you ask that? 

What is the value-added questions that you ask your own organization and in 

relation to that what are the key performance indicators?

Now you can take the spreadsheet and start filling from left to right or you 

could start from recognizing key performance indicators that you think are 

important from public health and a patient’s safety perspective and then ask 

the question why do we have those measures, what questions are you 

answering based on those measures and how does it line up? 

This was a structure that we gave people to sort of complete and we got good 

input back and we’re trying to compile some of the questions. We also before 

I move off that I also wanted to share that we gave in the spreadsheet which is 

also available for everybody to download is another tab which shows some 

example questions that FDA and staff here brainstormed to see what questions 

we would ask. And I would encourage everybody to use those as guiding 

questions but not necessarily answering those questions themselves but use 

them as guiding questions to complete this spreadsheet that we asked people 

to do.

Now I can imagine as you look at in your own organizations may have some 

of the components, may not have some of the components. But at the end of 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 12

the day if we get more input and it helps us and get that feedback from the 

nine pilot participants and from others the program model can actually be 

much more robust and that’s what we’re encouraging here. And that’s how 

I’m trying to share that work with you and everybody on this phone. 

We are inviting input from all stakeholders. Here’s what we’re doing. We are 

going to provide regular status updates on our Web site and you’ll see that 

first iteration already on the Web site. We are going to have, like, Irene said 

in the beginning this is the first of the series of Webinars we are going to have 

along the pilot program. We are also holding the first public meeting at the 

end of January. I would emphasize save that date. It’s going to be an 

important working meeting. We plan on working in this meeting rather than 

just talking. We will actually try to solve problems together and we invite 

everybody to do that. 

We will request everybody to submit their comments and input to the docket 

so we can take that and use that information to build a really robust program. 

What we want everybody and all the stakeholders to do we encourage 

associations, trade associations or other associations, coalitions, alliances and 

other common interest groups to form and reach out to those parties who are 

interested and collectively build the input and provide it to the docket. I think 

we are looking to as broad of input as possible from everybody and to inform 

this building of this program. 

We encourage groups to monitor our Webpage. We also want groups to stay 

engaged. We also want everybody to participate in this Webinar and in this 

public meeting. I think this is the best way we as a community can co-create 

this program that is best for the patient. So keeping that notion in mind - a 

patient we should make sure that we have the input. We as collectively FDA 

and all stakeholders have the input to create this program. 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 13

So with that if you have questions about the program itself, about the pre-cert 

program itself feel free to reach out to pre cert. If you have questions about 

what we’re asking or anything on the Web site we encourage you to ask the 

questions to the pre-cert pilot, the email address. If it’s anything outside of 

that feel free to reach out to (Dice) or the visual health email box that you 

already know. We will be recording this Webinar and posting the recording 

as well as the transcript going forward on the Webpage that you see here.

Having said that I’m going to open it up for questions and this is the time for 

you to ask question about the program, how we are engaging with the pilot 

participants, what questions you would have individually that would help us 

provide that input we’d be happy, I would be happy to answer those questions.

Irene Aihie: Operator do we have any questions?

Coordinator: We will now begin the question and answer session. If you’d like to ask a 

question please press Star 1. Please unmute your phone and record your name 

clearly when prompted. Your name is required to introduce your question. 

To withdraw your request press Star 2. One moment please for your first 

question. 

Bakul Patel: So while people are queueing up for the question I’ll just throw out something 

that we learned on the site visits and interacting with the pilot participants. 

One of the things that we had to clear up with the pilot participants is how 

would you ask the question of others that we are asking of you is a good way 

to think about how to approach providing feedback. 

One example that came up was, like, if folks are evaluating their vendors what 

mechanisms do you use to evaluate your vendors, what questions, what key 

excellence questions you’re looking, what principles you’re looking for those 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 14

vendors. Again for the purpose of your business might be to get the great 

product out but from an overall high level perspective how would you think 

about asking that question of your vendors for picking the right partner with 

your organization. So thinking about that is something that we discovered 

was really useful to talk through.

Irene Aihie: We’ll take our first question. 

Coordinator: Your first question comes from (David Amore). Your line is open. 

(David Amore): Hey Bakul. Thanks again to you and your team, Cathy and everybody else for 

a really excellent and collaborative session to speak with, so it was a fantastic 

opportunity to collaborate with you folks. And I mean it’s really an important 

effort of developing this new digital health framework. 

So that being said I think one of the things that I wanted to ask generally here today is as we and 

the industry go through this program would you say it’s more important that 

we think about how companies today currently meet the existing regulatory 

framework or more generally how they instill confidence in healthcare 

providers, payers, users, patients and other stakeholders in our company, some 

products by really demonstrating that excellent software development practice 

as an organization holistically? 

Bakul Patel: I would say the latter (David). I think that’s a great point. It’s less about the 

existing regulatory framework. It’s more about what’s the business doing best 

for making sure they’re delivering the right product in the right way with the 

right excellence. And that’s really – and the excellence can be not just only 

for FDA purposes but also, like, you said it’s about for the stakeholders we 

care about in the ecosystem is the payers, the providers, the patients included, 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 15

right? So it’s about being how can their excellence recognize that everybody 

in this healthcare ecosystem. 

(David Amore): Great thank you.

Bakul Patel: Thank you.

Coordinator: Your next question comes from (Arina Angel). Your line is open. 

(Arina Angel): Hi thank you so much for the excellent presentation. I have a question from 

the perspective of real world provider in this position, like, (unintelligible). 

And my question is (unintelligible) engage or (unintelligible) work with 

providers either by (unintelligible) a couple of (unintelligible) ask in addition 

(unintelligible) communication that have (unintelligible) physicians aren’t 

able (unintelligible) and maintain relationship. And, you know, 

(unintelligible) in a different way that data (unintelligible)...

Bakul Patel: Can I just pause you for a sec. We could hear only every other syllable. 

You’re cutting out so would you be able to repeat your question or get a 

different connection?

(Arina Angel): I will try to get a different connection. I apologize and I will try to maybe 

submit it in writing. I have another participant to have an opportunity. 

Bakul Patel: So you’re better now so before you were not. So if you want to ask quickly a 

question before your line gets...

(Arina Angel): Okay. Sorry for the confusion. I had a question if there’s any plan or possibly 

to engage individual providers who are not a part of an industry or academia 

who have a source of data, hard to reach patients. Say agoraphobia patients 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 16

and also could provide an insight to improve precision in caring to some type 

of patients with mobile devices to interpret this data to look at them. 

Bakul Patel: Yes this maybe I’ll just answer what I understood. I think you’re asking is 

there a way for individual patients and providers to share information that can 

lead to better outcome. I’m hoping that’s your question.

(Arina Angel): No. Sorry I will submit it in writing. I apologize for the connection. 

Bakul Patel: Yes please do. 

(Arina Angel): Thank you. 

Irene Aihie: We’ll take our next question.

Coordinator: (Susan Sheldon) your line is open.

(Susan Sheldon): Hi thank you for the excellent presentation. I was wondering how did you 

select your initial participants for this pilot program and if there is room for 

other people to participate at this point?

Bakul Patel: So first of all I think I would say that this program is a development program 

developing – you’re working with the nine participants to develop this 

program. And the thought process we went through a public process. We 

asked for people to submit their names and interest in participating. And we 

eventually selected the process and fundamentally I’ll just summarize the 

selection process into one sentence. With the goal of developing this program 

together with the nine participants and all stakeholders they’re looking to 

learn the most from each of the nine pilot participants. And this is how you 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 17

can see the spread and the diversity in the types of participants we selected 

and that was the thinking and rationale behind the selections. 

(Susan Sheldon): Okay thanks.

Bakul Patel: Okay. 

Coordinator: Your next question comes from (Timothy Holick). Your line is open. 

(Timothy Holick): Hi can you hear me okay? 

Bakul Patel: Yes.

(Timothy Holick): Okay great. I was just curious about the public meetings you said you were 

going to be trying to work specific topics and work through comments. Does 

this Webinar kind of go by the announcement and for the patient for that or 

could we expect more detail about what you’d like submitted for discussion or 

work there?

Bakul Patel: Oh what I said in this Webinar was purely a save the date. There will be a 

formal announcement. And there will be much more detail published. We are 

still formulating the agenda, etc and we’ll look for – and again it will be 

informed by the interactions we have and the next level of questions as we’re 

uncovering and learning through this pilot program. So there will be a lot 

more details to be coming. So if you have thoughts there will be opportunity 

and an avenue for you to provide that as well.

Man: Okay great. Is it okay if I ask a more substantive question too or should I just 

requeue up in the question?
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 18

Bakul Patel: If it's about the public Webinar so you can send your emails to FDA-precert

pilot and we'll get to you. If there is anything specific that what I presented 

today I would be happy to answer.

Man: Sure. Yes I just wanted to ask so far with your kickoff meetings – without 

getting into specifics have you kind of encountered the collaborative spirit you 

seem to be aiming for with the announcement or do - or are people still seem 

to kind of hold things close to the chest when discussing with the FDA?

Bakul Patel: I can tell you without hesitation I think the eagerness to share, eagerness to 

help in creating and building this program has been phenomenal. I’ve been 

completely humbled by the eagerness and the willingness for people to help 

build this program in the right way. So I would say it far exceeds, you know, 

anybody’s expectation in how open and willing this - the nine participant - the 

three we have visited so far and to the kickoff we have heard everybody 

willing to co-create this program with us.

Man: That’s great. Thanks for your time.

Bakul Patel: Thank you.

Coordinator: Next question comes from (Karen Keely). Your line is open.

(Karen Keely): Hi there. It's (Karen Keely). I'm a biotech consultant. Can you hear me okay?

Bakul Patel: Yes.

(Karen Keely): Hi. Thank you for all the information. It’s been very informative. I do have an 

ecosystem question. I work a lot with rare and orphan diseases, clinical trials 

and also in the autism community. Down the road are you hoping to focus 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 19

these devices in the healthcare ecosystem on any of those groups because I 

can say that that would be extremely needed and very beneficial?

Bakul Patel: Absolutely. I think as you can – as you saw we didn't pick any specific disease 

type with this program.

(Karen Keely): Right.

Bakul Patel: This program is supposed to be very agnostic. What we are trying to create 

here is an extremely efficient and streamlined regulatory program that can get 

to those disease types and those areas of public health that need most 

attention. We are trying to get innovation and happen in the space in a way 

that...

(Karen Keely): Right.

Bakul Patel: ...reaches patients that we care about. I think that's exactly the intent behind 

this so your question is spot on from that perspective.

(Karen Keely): Oh I’ll be following it closely because it’s very interesting. Thank you so 

much.

Bakul Patel: I would encourage you to attribute as well as follow.

(Karen Keely): I will absolutely. You’ll be hearing from me. Thank you.

Bakul Patel: Thank you.

Coordinator: Your next question comes from (Jeff Met). Your line is open.
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 20

(Jeff Met): Good afternoon. Can you hear me?

Bakul Patel: Yes.

(Jeff Met): Excellent. Well thank you for the Webinar and for the information. As it 

relates to the KPIs can you say whether you are looking to leverage industry 

standards for software development such as those meant for quality 

management or risk management as part of this pilot? And specifically I know 

that the Association for the Advancement of Medical Instrumentation AAMI 

is working on set standards for health IT software. And I’m just curious to 

know if you’re looking to leverage those kinds of standards as part of this 

work?

Bakul Patel: So standards we would leverage anything and anything that’s relevant in 

keeping organization excellence. That would be my short answer. But I would 

say yes is those standards are followed within our organization is those - and 

some that reflect - is reflected in either certification or other way. We are 

actually taking a very broader holistic look at an organization other - outside 

of in the (call your) systems regulations - I'll call it the systems or just risk 

management. We are looking at how an organization truly excels in making 

sure to provide safe, efficient experience and maintaining high product quality 

and so on and so forth as we highlighted in those excellence reports.

(Jeff Met): Okay thank you.

Coordinator: Your next question comes from (Gesa Rau). Your line is open.

(Gesa Rau): Hi. Thank you. Good to hear from you (Bicole). Thank you very much for a 

great presentation and a very open invitation to participate. My question is a 

follow-up from the very first question you answered relating to focus of this 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 21

effort being more on the process and less on the product. And given however 

that digital health can include both hardware and software certainly we can't 

completely move away from the regulatory - existing regulatory processes as 

(proposed) - just the question just before this one. So I was wondering if you 

could comment on a perspective here that to the extent we have a precedent in 

the traditional medical device regulatory environment of contract 

manufacturers having to meet certain requirements and being certified in 

certain ways regardless of the actual product. And the concept there is that 

their process has been certified to meet certain requirements and therefore it 

allows certain low risk products to simply go straight to market as long as they 

work with certified contract manufacturers. Is that an analogy that you think 

would be applicable over here?

Bakul Patel: I think so we are looking at (early) in the concept is very similar. If we look at 

what we are trying to do is we can – we are trying to identify people who we 

can trust make good products in a way that can actually be health recognizable 

and drive towards excellence rather than just meeting a bar. And let me 

explain that right? 

So if you can think about an organization that you're out for saying or you’re 

getting services from or developing software if you are looking for an 

excellence in that organization the question is what excellence are you looking 

for and how does that excellent lineup to the patient safety and public health 

means? And that’s what we're trying to get to. So if that - if it takes the 

concept that you just mentioned I think it applies but you're trying to deal 

even broader than that and apply to everybody who wants to do – who wants 

to bring products to the market in this space.

(Gesa Rau): Great. Thank you.
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 22

Coordinator: Your next question comes from (Pasello). Your line is open.

(Pasello Barlett): Yes hi. My name is (Passelo Barlett). First of all thank you very much for this 

great program and engaging the entire industry and stakeholders. My question 

is sort of stepping back from the specific program focused on the company. 

How are you all thinking about still ensuring efficacy of the actual product 

and the review process for efficacy and I think continuing to provide what 

sometimes is perceived by the industry and by patients is some validation that 

this stuff actually works because I don’t see that really jumping off this 

process?

Bakul Patel: Yes great question. I think I’ll actually expand on what you just asked. I think 

maybe even reading into where you’re coming from. I think the intent of this 

program and definitely the objective of this program is to not change our need 

to have safe and effective products. I think in fact we are looking at a way 

where we can find the people who can do that evaluation. When we have one 

of the principles, referenced principals about being clinically responsible I 

think that speaks to the effectiveness of the product. The high quality product 

excellence principle leads to a safe product.

So as you think about those different principles that we highlighted as part of 

certifying or pre-certifying the organization that is baked into that - their 

concept. However I think like you mentioned we are not talking about, you 

know, lowering the bar on the 50 year effectiveness but rather looking at 

people doing all – anything – anybody who’s making products or least in the 

virtual health space for this program in the space rather it's reviewed or not 

we're looking for folks and organizations who are actually good at doing those 

things.
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 23

(Pasello Barlett): So that you're looking to the third part certification to review the efficacy of 

the products that are submitted?

Bakul Patel: We are looking out the process that the companies follow to do and to make 

sure that people in the organization is doing their - so that's the certification 

part. And then if that today it requires us to review those projects we are 

looking at how best to review them. I think that’s where the efficacy and the 

safety review can happen. But that’s again risk based and also making sure 

that we have trust in those companies, making products that are even not 

being reviewed by FDA so that review part is still in the same paradigm and 

the concept that we have today.

(Pasello Barlett): Got it. Thank you.

Bakul Patel: Thank you.

Coordinator: The next question comes from (Steven Barnell). Your line is open.

(Steven Barnell): Hi (Bicole) and thank you to - for yourself (Kathy) and the team for a very 

collaborative meeting. So I would just like to emphasize the importance to - of 

building this together with industry and the approach you're taking. I think it 

deserves to be recognized the transparent approach you're taking both with the 

nine and with the broader community. One observation from the meeting this 

week and the process to date is, you know, this -- and it’s obvious to everyone 

-- is no trivial task. And many of these measures can be quantified for some of 

the things around product quality and companies have many measures of 

clinical responsibility and product quality. 

Others are much more subjective and qualitative if you like culture and how 

proactive you are. And I think one of the things that we really need to go wide 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 24

and if you like brainstorm with across the entire community is how we can 

really define measures for these more subjective or qualitative things. And I 

wondered if you could speak to that a little bit?

Bakul Patel: I’d love to and I thank you to you. And I think like you said I think some of us 

got – we do anticipate some of the measures will be qualitative. We also 

anticipate some of those measures will be (computative). I think some of the 

qualitative measures will have to be – any measures would have to have a 

balance sort of outcome measure that sort of shows that those qualitative 

measures do actually matter. For example you used the word culture and 

understanding a proactive culture may have a measure that may be purely 

qualitative but as a result of that qualitative measure is that – is the presence 

of proactiveness in an organization or a presence of high touch

communication does that result at the end of the day in product quality? And 

how can we use proxies, how can we use measures that may not be 

necessarily direct but may be adjacent or may be a way to sort of indicate that 

being that particular aspect exists is really what we need to look for?

So I'll – I think as people do diligence for acquisition, people do diligence for 

vendor selection, people - things like Morningstar reports, things like Dunn & 

Bradstreet, stuff like evaluation those methods are - and culture evaluation 

that have – that there's a science and a field that sort of exists can we learn 

from those? And it’s a great pointer about can we leverage those things that 

are already there in the capital industry, in other industries that we can use that 

to then identify objectively whether a company is truly excellent or not is 

really what we're after. So great point and I think we would welcome other 

thoughts, other things that people may have been doing in other areas, maybe 

not necessarily in this healthcare space but maybe other spaces, other sectors 

that we can bring sort of the sector would be phenomenal. And we encourage 

anybody who has input or ideas apart from on that that and we will take it.
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 25

(Steven Barnell): Thank you. I totally agree.

Bakul Patel: Thank you.

Coordinator: You next question comes from (Carl Washburn). Your line is open. 

(Carl Washburn): Happy Friday and thank you for a fantastic Webinar. Even though we're not a 

partner in the pre-cert development program we are benefiting greatly from 

the thought leadership, the transparency and the openness that you’re showing 

throughout the process including today’s Webinar and the public meeting. So 

back to the point that (Steven) was making how often do you see the team 

presenting Webinars and is there any framework for broadening the discussion 

for those who are not pre-cert members who, you know, look into some of the 

discussions or discussion board or some other way to participate other than 

these Webinars which are fantastic?

Bakul Patel: So great question (Carl). I think so if your team is going (unintelligible) what 

is exactly what I am addressing. I’m encouraging caution the (lances) through 

the (unintelligible) group, encourage your group to sort of be on top of this, 

the discussion. And the discussion and the content we are looking at as we 

sort of learn more we are going to put – we are putting up on the Web. The 

spreadsheet I’m showing is already up there. So I would encourage you guys 

to look and follow what’s happening in the preset program as we are building, 

as we are learning, as we're collecting information and whatever. I encourage 

every group to sort of come bring together their class and add to the inputs 

that we are seeking. And the docket mechanism is exactly what we want 

people to sort of provide that input towards. Engage us in asking what 

questions we're asking that we can clarify what we're asking to the pilot 

participants. If that (house) is helpful in you to provide input to us. So and 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 26

exactly to the point. We want everybody in this community to help us build 

this together.

(Carl Washburn): Thank you.

Bakul Patel: Maybe I didn’t answer your one question about how frequent we're going to 

do this. So as you can see they're a little bit time constrained between now and 

the end of the year so probably have this only Webinar now and followed up 

with a public meeting or my or intent as Irene said in the beginning it’s going 

to be – we’re intending this Webinar is to be a series. So as we have more 

information we will periodically keep people updated in addition to putting 

content on the Web.

(Carl Washburn): Yes the point I want to make is even today’s Webinar fantastic information 

you presented. But we're all benefiting from listening to each other’s 

questions and so thank you for creating this forum to allow us to communicate 

with each other.

Bakul Patel: Absolutely.

Coordinator: Your next question comes from (Wallace White). Your line is open.

(Wallace White): Thank you. I’m curious to see how this will work out and how this program 

can be shaped well to work both for consumer product companies and for 

medical device companies. I know both of them are involved in the (path) 

which is good. For the medical device companies who are entering digital 

health but already have established processes to be compliant with QSARN 

ISO-1345 certified how will we make sure this doesn’t become a whole 

second layer that they have to add to their processes that actually becomes 

more burdensome? And also related to that do you expect for the pre-cert 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 27

programmed to apply also to suppliers such as 1345 certification can apply to 

a design supplier?

Bakul Patel: Well I think you’re making a really good point about, you know, what we are 

trying to achieve. And like I said in the beginning this is not about aligning to 

1345 or not aligning to Phase 20. In fact we are on purpose asking people as 

we build the components of the program and the excellent principles and 

recognizing those is really going back to the right thing the companies do and 

organizations do for their business.

If we can identify those right business objectives and that alignment to those 

five principles I think the secondary question is how that sort of aligns to 

those other standards and requirements. And that is not something they're 

exploring right now. That's a task we will take down the road. But that’s 

where we are going back to the base and the fundamental principle can we 

recognize an excellent organization? 

Can we give people credit back to all the great things they’re doing? How can 

we do that? And if there’s something lacking how can we shine a light so 

people can fill those gaps? And that’s really what they’re looking for. I 

actually in the process forgot your second part of your question. Would you 

mind repeating it?

(Wallace White): Sure, whether the pre-cert program and a certification itself would extend to 

suppliers such as groups could do software development rather than strictly 

being to at least what I’ll say is the equivalent of the manufacture record to the 

company that sells the product and takes on responsibility for its adverse 

events and so forth?

Bakul Patel: So I don’t know if you are one of those people or not but I think there is...
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 28

(Wallace White): Yes. 

Bakul Patel: ...is definitely an opportunity for us all in the entire, you know, chain of 

building products to get on the same page. And we are hoping this framework 

would enable us to get everybody in the entire chain be on the same page with 

the same principles. So we are all shooting from the same direction. So I’m 

hoping someday that these principles can be applied at the same level that a 

design house can also be – can also show that they're meeting. There’s some 

inefficiencies in that process.

(Wallace White): Okay that makes sense. Thank you very much.

Bakul Patel: Thank you.

Coordinator: Your next question comes from Matt Trachtenberg. Your line is open.

Matt Trachtenberg: Hi (Bicole). This is Matt Trachtenberg with BD. I appreciate you sharing 

your - the spreadsheet. We have a different appraisal questions and I'll take a 

look at them. It seems like a majority of them are how questions, so how do 

you prioritize something? How do you become aware of an issue? And I'd like 

to better understand how you see us going from all those how questions which 

could have many different varied responses to these metrics and their KPIs?

Bakul Patel: So that’s a great observation. You can tell that was Version 001 of the 

questions that we created and truly have to be used as a guiding set of 

questions. But I can tell you that you spotted something that we also – we're 

also trying to evolve and reiterate towards Nexus really asking the question do 

you have such and such process and why do you have it? If you measure 

something the question to ask is why does it matter and what value that 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 29

measure adds to your business. And then if it adds value to the business how 

does it line up or how does it align to the rational principles? 

So I think reframing and learning working through that is something that we 

will end up evolving over time. And I guess the question I’m – I would ask 

back to you and everybody on the phone is how would you – how to frame the 

questions in the right way knowing that the intent we have and the objectives 

we have for as I'm creating this program how would you do that? And that’s 

something that it would be greatly that for us to sort of understand is what 

resonates well? I think this is purely one way of looking at it if it but if you’re 

hearing sort of what we want to achieve and measure we would love to hear 

feedback from folks is how should we be asking this question.

Matt Trachtenberg: Understood. Thank you.

Woman: We'll take our next question. 

Coordinator: Your next question comes from (Garret Marin). Your line is open.

(Garret Marin): Hi. Thanks for the time to speak today and thanks for a great presentation. I 

have a question about we understand medical devices are not included in this 

present program. And specifically I’m in the insulin pump industry. So I’m 

curious what you think about the mobile applications and the software 

applications that connect with and interact with devices and how they might 

be included in the future?

Bakul Patel: We are starting small. I can be very honest with you. I think and to your point 

about an observation of medical devices not included that’s not entirely true. 

We have two companies who are very medical device, traditional medical 

device companies that are in the product program. And I would also say there 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 30

are other organizations in the nine pilot programs and they're making medical 

device means they're - they may be making purely software as a medical 

device but that’s where we're starting as a starting point. We are not talking 

about software in a medical device yet that could be down the road. 

But we wanted to make sure that we had the scope right, we had the way to 

sort of paint it for this area first before we expand it to other areas. So and I -

to be seen down the road but we wanted to make sure that we didn't expand 

too much that we don’t achieve the goals of this program from the get-go. So 

it’s really it's scoping and managing of the issue - the issue sort of approach 

we are taking.

(Garret Marin): Okay that extends to applications that interact with those medical devices but 

are on a separate device like a smart phone or a mobile app type thing. That is 

excluded as well at this time?

Bakul Patel: So currently we are focusing on when software becomes a medical device on 

its own. That’s really what we're focusing on. So maybe you’re referring to 

things that may not necessarily be that but they may be an accessory or 

something else to the current other medical devices. That’s currently not the 

focus for us but we're looking at only software that can evolve, iterate and be 

developed extremely quickly using that as a baseline.

(Garret Marin): Got it. Thank you very much.

Bakul Patel: Thank you.

Coordinator: Your next question comes from (Nancy Stade). Your line is open.
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 31

(Nancy Stade): Hi there (Bakul). Thanks very much for the great presentation. So I’m hearing 

you describe a very broad program but I can think of a couple of ways the 

program could be challenged. And I hope you speak to just two of them. And 

the first is you alluded to earlier but I want to drill down a little bit. Consumer 

manufactures entering traditional device space. And I want to think about 

ways they might have difficulty establishing something like patient focus. So 

that’s one whether you've seen that or whether you anticipate any unique kind 

of challenges there.

And then the second is, you know, what about novel technologies, 

technologies that may be less familiar to FDA in their traditional reviews such 

as things like machine learning or multi-analyzed algorithms? Do you 

anticipate any limits to who can benefit from the program based on novelty or 

complexity of the software product?

Bakul Patel: Great questions (Nancy). I think the first question is about folks who may be 

entering this space may not necessarily have all the components that we are 

highlighting as the excellent principles. But I - so if we take a step back what 

we're intending to do is with the nine participants understanding what those 

challenges would be. And perhaps you and folks like you can help us sort of 

figure out if people have those challenges what’s the right way to those gaps 

that people can fill in? 

Or the intent here is we inform people of their – of what is needed to create 

good high-quality product and how our organization can support that highquality product that would be beneficial. So that’s the intent behind 

highlighting and staying at this very high level excellence principal and 

highlighting what could be potentially be a gap for your organization who 

may not be similar or - and have them fill the gap as they may find it. So 

that’s really what the intent of this – and hopefully you can build it with the 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 32

nine and others who are helping us build this together. And that’s really what 

we're looking for.

The second part is I think we're making this – we are - since we are focusing 

clearly on software those novel technologies are ways those novel algorithms 

can develop it’s completely included in this way of what - how we’re 

approaching the program. So it's not pick off whether it’s developed through 

machine learning, the algorithms develop the machine learning but then there 

are unique challenges that we still have to address in terms of what does 

clinical evaluation look like for those kind of products and how should we be 

evaluating. And those kind of – those questions will still be – they'll still exist 

and you still have to solve those. But where they’re going is – going with this 

is that the organization knows what to do best for whatever the digital health 

technologies they're making. And then they have a way to show they have 

infrastructure and people, processes and show valuably that their meeting 

those five principles it would still be okay to be pre-certified. And you have to 

figure out what that means and how many levels we can have for a precertification, et cetera, so to be defined but the intent is to include those novel 

technologies as well.

(Nancy Staid): Thanks very much (Bakul).

Bakul Patel: Thank you (Nancy).

Coordinator: Your next question comes from (Jacob Lamb). Your line is open.

(Jacob Lamb): Hi, good afternoon. I think we all appreciate the proactive excellence principal 

in terms of quality of the product, effectiveness, post-market surveillance, et 

cetera. I’m curious whether the pilot participants raised particular concerns 

with FDA. And that is the fears companies have in being too proactive. I’ve 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 33

seen and lived that fear in when and how to ask users and patients for 

feedback. The fear isn't based on a worry of receiving very negative feedback. 

The fear is more resource based and whether the company would be prepared 

or paralyzed in following-up on anything and everything that could be 

construed as their product being deficient in anticipated performance. 

So with the possibility of software devices now soliciting feedback from 

patients on a scale not previously possible I’m curious how or whether the 

agency thanks this may be addressed by the proactive principal that we’d like 

to instill?

Bakul Patel: Great question and I can tell you we haven't completely thought through that 

process yet. I think we're – what we are trying to get to is one of the constructs 

in this in this approach is we do want to be – we want people to be proactive. 

We want people to know what’s happening. As you can see from our policies 

on cyber security and in terms of software we want people to sort of know 

what’s happening before it happens so they can prevent further adverse 

conditions emerge. How do we sort of get there? I think that’s a question.

Now what the regulatory paradigm would look afterwards after we all agree in 

the world of software you can learn a lot more and how do you leverage that? 

The question is when you know and you don’t do - don’t respond to those 

things in a proactive way what is the question that the company needs to 

have? In my mind I'll summarize this and say in my mind if somebody knows 

and has a good handle on understanding what is going on with their product 

you would expect an excellent organization to proactive and corrected and 

that’s really what we're looking. Now there’s of course a balance and we need 

to figure out what that balance is and that this is where I think as we get down 

deeper into understanding post market real-world data collection that would 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 34

be a discussion to be entertained. And I think that would be a good way to sort 

of figure out whether that one should be.

(Jacob Lamb): Thank you.

Bakul Patel: Thank you.

Woman: We'll take our next question.

Coordinator: The next question comes from (Wallace White). Your lines is open.

(Wallace White): Thank you again. Bakul, I think earlier you said in your presentation that you 

were aiming for excellence rather than meeting a bar which is an interesting 

and good point. I'm reminded a little bit of 20 years ago or so when I drove 

through Montana back when they didn’t have a 55 mile-per-hour interstate 

limit and they posted signs up safe and reasonable or something like that. And 

it actually made me think when I drove is 55 miles per hour safe and 

reasonable for where I am right now? And in other places you could drive 

more faster. Are you implying there is that we need to better than people 

who've gone under CFRA20 in some cases do not simply meet the bar but 

truly seek out excellence in these kinds of devices and products?

Bakul Patel: The short answer is. I think and what I'll - I can tell you that I’m finding and I 

– and there's a hypothesis where folks are doing a lot more than what we're 

asking already. I think people are aspiring much harder much more than, you 

know, just simply meeting a standard of some kind. So I think in the world of 

software and that is – that evolves over time and we want to make sure that 

these products like typical software has to be updated. 
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 35

And that’s of course a lot of proactiveness, means a lot of collaboration. That 

needs a lot of vigilance. How do you sort of build that in and that’s really 

what we're shooting for. So I think your analogy is perfect and exactly what 

we're looking.

(Wallace White): Okay thank you.

Bakul Patel: Thank you.

Coordinator: As a reminder to ask a question please press Star 1 and please record your 

name clearly when prompted. One moment please for the next question. And I 

am showing no further questions. I would like to call back over to Irene Aihie.

Irene Aihie: Thank you. This is Irene Aihie. We appreciate your participation and 

thoughtful questions. Today’s presentation and transcript will be made 

available on the CDRH Learn Web page at www.fda.gov/training/cdrhlearn 

by Thursday, November 30. 

If you have additional questions about today’s presentation please use the 

contact information provided at the end of the slide presentation. As always 

we appreciate your feedback. Following the conclusion of today’s Webinar 

please complete a short 13 question survey about your FDA CDRH Webinar 

experience. The survey can be found at www.fda.gov/cdrhwebinar 

immediately following the conclusion of today’s Webinar. Again thank you 

for participating and this concludes today’s Webinar.

Coordinator: Thank you for joining today’s conference call. You may disconnect at this 

time.
FDA

Moderator: Irene Aihie 

 11-17-17/1:30 pm ET

Page 36

END