Risha Gidwani, DrPH 

June 2014

Sensitivity Analyses for 

Decision Modeling
 Why sensitivity analyses? 

 Modeling Uncertainty

– One-way sensitivity Analyses

– Tornado Diagrams

– Scenario Analyses

– Probabilistic Sensitivity Analyses

Content 
Output of a Decision Model 

Type of Model Output

Budget Impact Model Cost per strategy

Cost Benefit Model

Net social benefit = 

Incremental Benefit (cost) – Incremental 

Costs

Cost-Effectiveness Model IIIIIIII =

∆ cccccccc

∆ heeeeee eeeeeeee

Cost-utility Model IIIIIIII =

∆ cccccccc

∆ QQQQQQQQ
Cost-effectiveness Model quadrants

∆ Cost

II I 

II

I

IV

WTP 

II ∆ I 

III IV

∆

WTP

Poll: Which 

quadrant 

represents a costeffective strategy? 
Cost-effectiveness Model quadrants

∆ Cost

II I 

II

I 

IV

WTP 

I ∆

I 

I 

II

I 

IV

∆

WT

P 

Quadrant I: 

Quadrant II: 

Quadrant IV:

 More costly and more effective 

(?)

 More costly and less effective

 (No!)

Quadrant III: 

 Less costly and less effective 

(?)

 Less costly and more effective

 (Yes!)
Cost-effectiveness Model output

∆ Cost

II I 

II

I 

IV

WTP

II ∆ I 

III IV

∆

WTP

Variation in your ICER may cause 

your decision to change
 Evaluate how uncertainty in model inputs 

affects the model outputs 

– Base-case model  ICERs 

– Sensitivity Analyses  Variation in ICER

Why sensitivity analysis?

Statistical Analysis Cost-Effectiveness Analysis

Mean ICER (Base-Case)

Variation around Mean Variation around ICER
Varying point estimates 

(TreeAge model)

8 
General Approach, Sensitivity 

Analysis 

1. Change model input

2. Recalculate ICER

3. If new ICER is substantially different from 

old ICER  model is sensitive to that 

parameter

In this case, it is very important to be 

accurate about this parameter!

9 
 Cost

 Health Effect

– Life Years Saved

– Utilities 

– Cases of Disease Avoided

– Infections Cured

 Probabilities

 Discount Rate

Types of inputs
Types of Uncertainty

Briggs et al. 2012 Model Parameter Estimation and Uncertainty: A Report of the ISPOR-SMDM Modeling Good Research Practices 

Task Force – 6. Value in Health, 15: 835-842.

Variation between 

identical patients

Term Models AKA

Analagous term in 

regression Example

Stochastic

Uncertainty

- First-order uncertainty

- microsimulation

Error term 19% of Medicare beneficiaries 

readmitted to the hospital within 30 days. 

Person 1 = readmitted, Persons 2, 3, 4, 5 

= not readmitted

Parameter 

Uncertainty

Uncertainty in 

estimation of 

parameter of 

interest

- Second-order uncertainty

- PSA

Standard Error of the 

estimate

Toss a fair coin 100 times. You get 55

“heads” and 45 “tails”

Heterogeneity Differences in 

patient

characteristics

- Observed heterogeneity

- variability

Beta-coefficients/test 

of sig. amongst 

different levels of a 

covariate

-

Drug is cost-effective for people with 

moderate disease, but is not cost

effective for people with mild or 

advanced disease
Types of Sensitivity 

Analyses 
 One-way sensitivity Analyses 

 Tornado Diagrams

 Scenario Analyses

 Probabilistic Sensitivity Analyses

Types of Sensitivity Analyses

Often 

Deterministic 
 Deterministic (DSA): model input is 

specified as multiple point estimates and 

varied manually 

 Probabilistic (PSA): model inputs are 

specified as a distribution and varied 

Types of Sensitivity Analyses
DSA versus PSA

DSA PSA

Base case $100 $100 

Input $80, $90, $110, $120

Results ICER A (when cost is $80)

ICER B (when cost is $90)

ICER C (when cost is $110)

ICER D (when cost is $120)

The mean ICER when we vary 

the base-case using a normal 

distribution with a mean of $100 

and standard deviation of $10 is 

X, using 1000 iterations

15 

Example: Cost input, cost of outpatient visit
DSA PSA

Markov Cohort X X 

Individual-level Markov Model X X 

Discrete-Event Simulation X X 

DSA, PSA and Model structure
Sensitivity Analyses in 

TreeAge 
PE/DVT example
PE/DVT example – 

Hypothetical Probabilities
PE/DVT example – 

Hypothetical full inputs 
Model results, with point 

estimates
One-Way Sensitivity Analyses
One-way sensitivity analysis

 Vary one input (parameter) at a time, and 

see how model results are affected 

 Example: probability of AE_chemo 

– Base-case: 0.65

– Sensitivity analysis: range from 40-80% 

 Run 5 models, each with the following input:

– .040, 0.50, 0.60, 0.70, 0.80 

23 
 Best practice: 

 1. Insert variables, not point estimates

 Example: probability of AE, chemoprophylaxis

– “0.65” (Point estimate) 

– “p_AE_chemo” (Variable) 

2. Then, define variables as:

 Point estimates (DSA) or

 Distributions (PSA)

 Example: definition of probability of Adverse Event, chemoprophylaxis

– Defining variable as a point estimate: “p_AE_chemo” = 0.65” 

– Defining variable as a distribution: “p_AE_chemo” = dist_AE_chemo” 

Inputting variables to run a 

sensitivity analysis
PE/DVT example – 

Probabilities as Point Estimates
PE/DVT example – 

Probabilities as Variables and Variables 

defined as Point Estimates 
 Define your range 

One-way sensitivity analyses
Output, one-way 

sensitivity analyses

28 
 Can get range from 95% Confidence Interval 

reported

 Varying a parameter an arbitrary range, 

such as ± 50% -- not a great practice 

– This will demonstrate model sensitivity, but does 

not reflect uncertainty

 Expert Opinion

Inputs for a one-way 

sensitivity analysis
Series of One-way Sensitivity 

Analyses 

1. Vary probability of chemoprophylaxisrelated adverse event 

a. Compare these ICERs to base-case ICER

2. Vary cost of treating adverse event 

a. Compare these ICERs to base-case ICER

3. Vary probability of death from PE/DVT 

 a. Compare these ICERs to base-case ICER

4. Etc. 

30 
 Generally, a series of one-way sensitivity 

analyses will underestimate uncertainty in a 

cost-effectiveness ratio:

– The ICER is based off of multiple parameters, not 

just one

– Here, you are assuming that uncertainty exists only 

in one parameter

– Solution: Probabilistic Sensitivity Analyses!

Caution 
 You should still do one-way sensitivity 

analyses! 

 Easy way to understand which parameters 

matter

But...
 Tell you which of your one-way sensitivity 

analyses had the greatest impact on model 

results

 Bar: a one-way sensitivity analysis 

 Width of bar represents impact on model 

results 

Tornado diagrams
Conducting a tornado diagram
Tornado Diagram (Net Benefits)

35 
Tornado Results (ICER) – 

recommended graph to view
 The high value for p_PEDVT_mechan results in 

chemoprophylaxis now being the preferred strategy

 Tells us we need to be more precise with our estimate of 

PE/DVT associated with mechanical prophylaxis

 Other variables don’t impact out model conclusions

Tornado diagram, text report -
 Just a series of one-way sensitivity 

analyses, with results presented on top of 

one another 

 There is not just uncertainty in one 

parameter – there is uncertainty in most, 

if not all, parameters

Limitations of Tornado diagrams 
Scenario Analyses
 Interested in subgroups

– Cost-effectiveness of chemical versus mechanical 

prophylaxis in 85+ only

 Change risk of PE/DVT, risk of AE, risk of death from 

PE/DVT/AE

 Changes the point estimate of multiple 

parameters

 Do not incorporate uncertainty !

Scenario analyses
Probabilistic Sensitivity Analyses
 Vary multiple parameters simultaneously 

 Each variable comes from a distribution

 Model is run many times (1,000, 10,000, etc.)

– Each model iteration plucks a value from that 

distribution and uses it as the model input

Probabilistic sensitivity analysis

c 

Frequency
 Values are sampled with replacement! 

 Values sampled based on their likelihood of 

occurrence 

 Results (comparing strategy A to B): 

– Mean CostA & variation in CostA 

– Mean CostB & variation in CostB 

– Mean Health EffectA & variation in Health EffectA 

– Mean Health EffectB & variation in Health EffectB 

PSA 
 Costs: log-normal, normal

 Probabilities: beta

 Utilities: beta

Choosing distributions for your 

PSA – general guidance
 Need to define variables in terms of distributions, rather than 

point estimates

Inputting variables into your 

PSA 

Point estimates
Defining distributions in a PSA

Distributions
Creating distribution-based 

definitions

1. Create the distribution: d_AE_chemo

Define the distribution in terms of its shape 

 normal, beta, etc

– Define the parameters for that distribution

 mean/variance, alpha/beta, etc.

2. Assign the distribution to a variable: 

prob_AE_chemo = d_AE_chemo
Running a PSA

 Define all variables (model inputs) as 

distributions 

 Determine your number of iterations 

48 
 Cost-effectiveness planes (CE scatterplot)

 Cost-effectiveness acceptability curve

 Net benefits 

Ways to show uncertainty in the 

ICER
CE Scatter Plot
 In this hypothetical example (with entirely made-up 

data) Mechanical Prophylaxis is cost-effective 

compared to Chemo Prophylaxis 99.9% of the time

– Costs less AND provides more health benefit

“ICE Report”
Cost-Effectiveness Plane, if in multiple 

quadrants 
 IIIIIIII =

CCCCCCCC AA−CCCCCCCC BB

EEEEEEEEEE AA−EEEEEEEEEE BB

= −40,000

−1

= $40,000�QQ QQQQ

 IIIIIIII =

CCCCCCCC AA−CCCCCCCC BB

EEEEEEEEEE AA−EEEEEEEEEE BB

=

40,000

1

= $40,000�QQ QQQQ

Ways one should not show 

uncertainty in the ICER

∆ Effect

II I 

III IV

∆

∆

WTP

- Show only the numeric value of the ICER and Confidence Interval
 Previously, I had to specify my WTP

 What if you don’t know what that is?

– Or different decision makers have different 

WTP? 

 Use a Cost-Effectiveness Acceptability Curve

– Percentage of iterations that favor each 

strategy, over a range of WTP

Willingness to pay (WTP)
Cost-effectiveness acceptability 

curves – hypothetical 
 Combine information on costs, outcomes, and 

willingness to pay

– Net Monetary Benefits

 Positive number indicates technology is costeffective 

 Use when you are very certain about your 

WTP 

Net Benefits
 Net Monetary Benefits 

NMB = (∆ Effect * WTP) – ∆ Cost

(-0.11 * $50,000) – $1,057 = $-6,557

Net Monetary benefits
TreeAge- Net Monetary benefits
1. Cost-effectiveness planes/quadrant

2. Cost-effectiveness acceptability curve

3. Net monetary benefits (only if you are 

certain on your WTP) 

3 ways to show uncertainty in the 

ICER
 More distributions = more iterations

 Stop when the simulations generate mean values 

(without seeding) that are very similar

How many iterations in a PSA?
100 iterations
PSA Summary

 Looks at model results when multiple sources 

of uncertainty are evaluated simultaneously

 Results presented in terms of:

– C-E planes (quadrants)

– C-E acceptability curves

– Net Monetary Benefits

 Required in order to publish in a peer- reviewed journal!

65 
Joint Parameter Uncertainty 
The model will assume no covariance between 

parameters unless you specify otherwise

Joint Parameter uncertainty

Probability of response at 

26 weeks

Probability of response at 

52 weeks

Probability of Response Probability of Response
 Define one variable in terms of the other 

 X = Y+ (Y*0.2)

 Use a table to link variables, have PSA identify Index

 Variable X = if(PSA = 1; Table 1[Index; 1]; 0.55)

 Variable Y = if(PSA = 1; Table 1[Index; 2]; 0.65)

 

Accommodating Joint Parameter 

uncertainty 

Index X Y 

1 0.60 0.67 

2 0.480 0.89 

3 0.89 0.93 

• If the PSA indicator is turned on:

• go to Table 1, choose the row 

(Index) corresponding with the 

model cycle we are in and use 

the value in column 1

• otherwise, use a value of 0.55
SUMMARY
Summary 

 All model inputs have uncertainty

 Test how this uncertainty affects model results

– Do so by varying model inputs

 Tornado diagrams: first-pass understanding of the most 

important variables in your model

 Need to run a PSA in order to fully evaluate the 

combination of uncertainty in all/most model inputs on 

robustness of model results

– Be careful to accommodate joint parameter uncertainty

70 
References 

 General Overview:

– Hunink M, Glasziou P, Siegel J, et al. “Chapter 11: 

Variability and Uncertainty” in Decision Making in Health 

and Medicine: Integrating Evidence and Values. Cambridge, 

UK: Cambridge Press, 2004. 339-363. 

 Best Practices: 

– Briggs et al. Model Parameter Estimation and Uncertainty: A 

Report of the ISPOR-SMDM Modeling Good Research 

Practices Task Force – 6. Value in Health, 2012, 15: 835-

842. 

71 
QUESTIONS?