Welcome To Today’s Webinar

Thanks for joining us! 

We’ll get started in a few minutes 

Today’s Topic: 

Assessing the Credibility of Computational Modeling and Simulation in 

Medical Device Submissions, Final Guidance 

January 11, 2024
Assessing the Credibility of Computational 

Modeling and Simulation in Medical Device 

Submissions, Final Guidance

Pras Pathmanathan, PhD

Division of Biomedical Physics 

Office of Science and Engineering Laboratories

Center for Devices and Radiological Health 

U.S. Food and Drug Administration
Final Guidance

• Assessing the Credibility of Computational 

Modeling and Simulation in Medical Device 

Submissions 

– www.fda.gov/regulatory-information/search-fdaguidance-documents/assessing-credibilitycomputational-modeling-and-simulation-medicaldevice-submissions

3
Learning Objectives

• Define computational modeling and simulation 

(CM&S) and state scope of guidance 

• Describe key points and approach of guidance 

• Outline the framework for credibility assessment 

4
CM&S and Scope of Guidance

5
What is CM&S?

Hybrid methods 

§ First - principles model with 

data - driven sub - model(s) 

§ Train ML model to firstprinciples model results

Data - driven models 

§ Statistical methods 

§ Machine learning (ML)

First - principles models 

§ Physics - based models 

§ Mechanistic models

Mathematical models

6
What is CM&S?

Hybrid methods 

§ First - principles model with 

data - driven sub - model(s) 

§ Train ML model to firstprinciples model results

Data - driven models 

§ Statistical methods 

§ Machine learning (ML)

First - principles models 

§ Physics - based models 

§ Mechanistic models

Mathematical models

7
CM&S in Regulatory Submissions

In Silico 

Device Testing

Simulate device to 

address safety / 

effectiveness 

question

CM&S in device 

software 

Device takes in 

patient data and 

simulates patient 

Heartflow

In Silico 

Clinical Trial

Simulate device on 

‘virtual cohort’ of 

simulated patients

OSEL VICTRE project

CM&S - based 

MDDT

CM&S tool relevant 

to multiple devices

8
Scope

In Scope

§ First principles - based models 

§ For hybrid models: 

§ First-principles model components

Out of Scope

§ Standalone statistical or data - driven models 

§ Models with no simulation, such as anatomical models 

§ How to perform modeling studies 

§ Technical details for how to perform credibility assessment 

§ Specific level of credibility needed for regulatory submissions

9
Key Definitions

Paraphrased from Guidance and American Society of Mechanical Engineers (ASME) 

Verification & Validation (V&V) 40 - 2018 Standard: 

§ Credibility – The trust, based on all available evidence, on the predictive capability of a 

computational model 

§ Context of Use (COU) – The role and scope of the computational model in answering the 

question of interest

Mathematical 

model 

Computational 

model

Verification: 

Was this implemented correctly? 

What is the numerical error?

Real world 

observations

Validation: 

Is the computational model an 

accurate representation of the real 

world? 10
Key Points and Approach

11
Key Points and Approach

§ Guidance is consistent with ASME V&V40 - 2018 

§ Risk-informed credibility assessment 

§ Emphasis on question of interest, context of use and model risk 

§ Guidance includes additional recommendations on information to 

provide in a regulatory submission

§ Provides a general framework for model credibility assessment 

§ Intended to be applicable to wide variety of models, and all 

applications and types of regulatory submission 

§ Not prescriptive

12
Key Points and Approach (cont’d)

§ Framework extends 

approach of ASME V&V40-

2018 

§ ASME V&V40 implicitly 

assumes validation 

against prospective wellcontrolled bench tests

13
Overview of Framework

14
Framework

Guidance 

Figure 1

15
Framework

16
Step 1: state the Question of Interest

“the specific question, decision, or concern that is being addressed”

§ Should be about the real world 

§ Not about the model 

§ Should not be overly broad (“Is the device safe?”)

Device testing example

Is the device resistant to fatigue fracture under anticipated worst - case 

radial loading conditions?

17
Step 2: state the Context of Use

“the role and scope of the computational model in answering the question of 

interest”

§ what is modeled and how model outputs used to answer the question of 

interest 

§ type of modeling, key inputs and outputs 

§ whether other information (such as bench, animal, or clinical) will be used to 

answer the question of interest 

Device testing example

Combine computational modeling predictions and empirical fatigue testing 

observations to estimate device fatigue safety factors under anticipated worstcase radial loading conditions [...]

18
Step 3: assess Model Risk

“the possibility that the computational model and the simulation results may 

lead to an incorrect decision that would lead to an adverse outcome”

§ Decision consequence 

§ significance of an adverse event 

following an incorrect decision 

§ essentially “Risk” as defined in ISO 

14971 

§ Therefore, recommend 

manufacturers consider probability 

of occurrence and severity of harms 

Serious injury 

or death

Patient 

inconvenience

19
Framework

20
Step 4: Identify Credibility Evidence to be collected

“any evidence that could support the credibility of a computational model”

Categorization

1 Code verification results 

2 Model calibration evidence

3 Bench test validation results 

4 In vivo validation results 

5 Population - based validation 

results

6 Emergent model behavior

7 Model plausibility evidence

8 Calc. verification/UQ using 

COU conditions

§ Details and examples in Section VI.B 

§ Recommendations in Appendix 1

§ Recommend submissions covers: 

§ Code verification results (#1) 

§ Calculation verification results 

(#3, #4 or #8) 

§ Validation (#3 - #5) or other evidence 

pertaining to ability to reproduce 

real-world behavior (#2, #6, #7) 

21
Step 4: Identify Credibility Evidence to be collected

“any evidence that could support the credibility of a computational model”

Categorization

1 Code verification results 

2 Model calibration evidence

3 Bench test validation results 

4 In vivo validation results 

5 Population - based validation 

results

6 Emergent model behavior

7 Model plausibility evidence

8 Calc. verification/UQ using 

COU conditions

§ Details and examples in Section VI.B 

§ Recommendations in Appendix 1

§ Recommend submissions covers: 

§ Code verification results (#1) 

§ Calculation verification results 

(#3, #4 or #8) 

§ Validation (#3 - #5) or other evidence 

pertaining to ability to reproduce 

real-world behavior (#2, #6, #7) 

22
Step 5: Credibility Factors

§ Define credibility factors for planned evidence (some recommended factors provided) 

§ For each factor 

§ Define a gradation of activities 

§ Choose a target level based on the risk assessment

Example

3 Bench 

validation

Appendix 1 

recommends 

using relevant 

ASME V&V40 

factors

Gradation

(a) A single sample was used 

(b) Multiple samples were 

used, but not enough to be 

statistically relevant. 

(c) A statistically relevant 

number of samples were used.

23
Step 5: Credibility Factors

§ Define credibility factors for planned evidence (some recommended factors provided) 

§ For each factor 

§ Define a gradation of activities 

§ Choose a target level based on the risk assessment

Example

3 Bench 

validation

Appendix 1 

recommends 

using relevant 

ASME V&V40 

factors

Gradation

(a) A single sample was used 

(b) Multiple samples were 

used, but not enough to be 

statistically relevant. 

(c) A statistically relevant 

number of samples were used.

24
Framework

25
Step 6: Rationale for Adequacy

Will the credibility evidence support using the model for the COU given risk 

assessment?

§ Step 6: Prospective adequacy assessment 

§ Rationale for why planned evidence with expected

results will be sufficient 

§ Recommend Q - sub to present plan 

26
Framework

27
Step 8: Rationale for Adequacy

Does the credibility evidence support using the model for the COU given risk 

assessment?

§ Step 8: Post - study adequacy assessment 

§ Decision based on all available evidence and engineering/clinical 

judgement 

§ Considerations 

§ All relevant model features tested? 

§ If credibility goals not met, consider rationale for why results still 

adequate 

§ How do predictions compare to decision/safety thresholds? 

§ Discuss limitations 28
Step 9: Credibility Assessment Report

§ Recommend self - contained 

report on model credibility 

§ Distinct from simulation 

study results 

§ Recommended structure in 

Appendix 2 

§ Also provides 

recommended structure 

for Q - Submissions 

1. Executive Summary

2. Background

3. Device Description

4. Proposed Indications for Use

5. Description of Computational Model

6. Model Credibility Assessment

a. Summary of overall approach 

b. Question of Interest 

c. COU 

d. Model Risk Assessment 

e. Credibility Evidence. For each: 

i. Categorization of evidence 

ii. Description of evidence 

iii.Chosen credibility factors, gradations, goals/achieved 

level 

iv.Methods 

v. Results 

f. Post - study Adequacy Assessment 

7. Credibility Assessment Limitations

8. Conclusions 29
Resources

Slide 

Number

Cited Resource URL

3 Assessing the Credibility of 

Computational Modeling and 

Simulation in Medical Device 

Submissions

www.fda.gov/regulatory-information/search-fda-guidancedocuments/assessing-credibility-computational-modeling-andsimulation-medical-device-submissions 

10 ASME V&V40-2018 www.asme.org/codes-standards/find-codes-standards/v-v-40-

assessing-credibility-computational-modeling-verificationvalidation-application-medical-devices

30
Summary

§ Guidance is relevant to first principles (such as physics - based) 

models or first principles components of hybrid models 

§ Guidance provides a general framework relevant to all modeling 

fields and submission types 

§ Guidance framework is a nine - step process 

§ Steps 1 - 3: define how model will be used and assess risk of using model 

§ Steps 4 - 6: prospective planning and possible Q - Submission 

§ Steps 7 - 9: execution, justification, report 

31

Additional Panelists

Brent Craven 

Senior Science Advisor 

Division of Applied Mechanics 

Office of Science and Engineering 

Laboratories

Kenneth Aycock 

Interdisciplinary Engineer 

Division of Applied Mechanics 

Office of Science and Engineering 

Laboratories

Finn Donaldson 

Team Lead 

Peripheral Interventional Devices

Office of Health Technology 2 

Office of Product Evaluation and 

Quality

33
Let’s Take Your Questions

• To Ask a Question: 

• Raise your hand in Zoom 

• Moderator will announce your name and invite you to ask your question 

• Unmute yourself when prompted in Zoom to ask your question 

• When Asking a Question: 

• Ask one question only 

• Keep question short 

• No questions about specific submissions 

• After Question is Answered: 

• Mute yourself and lower your hand 

• If you have more questions - raise your hand again

34
Thanks for Joining Today!

• Presentation and Transcript will 

be available at CDRH Learn 

• www.fda.gov/Training/CDRHLearn

• Additional questions about 

today’s presentation

• Email: DICE@fda.hhs.gov

• Upcoming Webinars 

• www.fda.gov/CDRHWebinar

35