# Presentation: 388998

## Advanced Sensitivity Analyses: Probabilistic, Correlated and Scenario

- July 26, 2006
- Mendel E. Singer, Ph.D.
- Case School of Medicine
- 216-368-1951
- mendel@case.edu

## Today’s Agenda

- Role of Sensitivity Analysis
- Different Types of Sensitivity Analysis
- 1-way, 2-way
- Scenario
- Correlated
- Probabilistic
- Why and when to do each type
- Examples
- How to’s – in theory and in TreeAge

## Structure of a CEA Paper

- Introduction/Background
- Why the reader should continue reading
- Methods
- Relevance
- Reference case, Strategies
- Credibility
- Assumptions
- Results
- Baseline
- Main message
- Sensitivity Analysis
- Remove the reader’s doubts
- Discussion
- If they are still reading, you have your chance

## 1st Order Uncertainty

- Variability across patients
- Not all patients alike
- Different characteristics lead to different results
- Different mean values for a parameter
- All Patients Alike
- E.g. simulated cohort of identical patients or patient subgroups
- Probabilistic vs. Deterministic
- Variability in outcomes – results not always the same

## 2nd Order Uncertainty

- Parameter value/distribution not known with certainty
- Origin of the estimate
- Clinical trials
- Retrospective study
- Variation in results across published studies
- Expert opinion
- Variability around the parameter mean

## Structural Uncertainty

- Uncertainty about model assumptions
- Affects the structure of the tree
- Examples
- Some possible but improbable events excluded
- Certain sequences not considered
- Various scenarios “averaged”
- E.g. dose titration has many permutations
- No repeat surgery for a chronic illness
- Patients who stop treatment don’t restart treatment

## Types of Sensitivity Analysis

- 1-Way
- 2-Way
- Scenario
- Correlated
- Probabilistic

## 1-Way Sensitivity Analysis

- How to:
- Reanalyze the model for a range of clinically plausible values to see if either extreme has an impact on the results
- Why?
- Uncertainty in original estimate, e.g. 95%CI
- Misspecification bias
- Problem definition doesn’t match source of parameter value
- Weakness in study that is source of parameter value
- 2nd-order (parameter) uncertainty
- 1st-order uncertainty not well addressed

## 1-Way Sensitivity Analysis (2)

- Good for removing doubts about a “problem” parameter
- Good for identifying research priorities
- Easy to understand
- When?
- Expected in every decision analysis/cost-effectiveness analysis

## 2-Way Sensitivity Analysis

- How to:
- Select 2 parameters
- Consider combinations of values for the 2 parameters, keeping the values for each within a clinically plausible range
- For each combination determine the point of indifference between the strategies, i.e. threshold value
- Graph the point of indifference for each combination
- Are there regions where the conclusion changes?

## 2-Way Sensitivity Analysis (2)

- Why?
- May not be sensitive to 1 variable, but may be to a combination
- Often thematic
- Different patient subgroup
- Same source for 2 parameters – may be systematic bias
- 2 components both poorly known or arbitrary
- E.g. Vaccine efficacy vs Cost of vaccine for coming vaccine
- Uncertainty in original estimates, e.g. 95%CI
- Misspecification bias
- Problem definition doesn’t match source of parameter value
- Weakness in study that is source of parameter value

## 2-Way Sensitivity Analysis (3)

- 2nd-order (parameter) uncertainty
- 1st-order uncertainty somewhat addressed
- Difficult to understand for oral presentations or for audience in experienced in methods
- When to do?
- Usually only done when there is a strong reason (thematically linked parameters)

## 3-way sensitivity analysis

- Can be done with 3-D graphs, OR
- Done with animation on a computer to simulate the 3rd dimension
- TreeAge can do it
- Many people can grasp it, though the simulation won’t work in print
- I cannot grasp it, so learn it from someone else! Sorry, my brain only works in 2 dimensions

## Scenario Sensitivity Analysis

- How to:
- Create a descriptive scenario of interest that leads to changes in parameter values and/or changes in the structure of the model
- Change the tree and parameter values as needed
- Rerun the baseline analysis

## Scenario Sensitivity Analysis (2)

- Why?
- Alternative problem definition
- E.g. different setting or patient population
- Need to change more than 2 parameters
- Don’t need combinations of values of 2 parameters, just 1 specific combination
- Important clinical scenario which either:
- 1- and 2-way sensitivity analyses won’t address
- Forms a valid, alternative baseline analysis

## Scenario Sensitivity Analysis (3)

- 2nd-order (parameter) uncertainty
- 1st-order uncertainty to a degree
- Structural uncertainty
- Good for removing doubts about a small variation on the problem definition
- Useful when there are really *several* related problems to address
- Easy to understand
- When to do?
- Only done when there is a clear reason

## Scenario Sensitivity Analysis: Examples

- Patient with a common comorbid condition that was excluded in baseline
- Can affect response rates, complication rates, surgical mortality rates, etc...
- Different setting  multiple parameter changes
- VA outpatient vs. managed care outpatient
- Presentation to physician vs. emergency dept.
- Different country
- Different practice patterns, costs, genetics

## Scenario Sensitivity Analysisin TreeAge

- No special techniques since we all have to do is change some default values and rerun the main cost-effectiveness analysis.
- Here’s what I suggest:
- Save the tree under a new file name
- Edit the values to reflect those of the scenario sensitivity analysis.
- Run the cost-effectiveness analysis
- Save your work.

## Correlated Sensitivity Analysis

- How to:
- Select >=2 parameters  that are thematically linked such that as one parameter changes in one direction, the others are known to change in the same or opposite direction, but not stay the same
- As the main variable changes within its plausible range, each correlated variable changes in the specified direction by the same proportionate amount in the specified range
- Reanalyze like 1-way sensitivity analysis on the main variable, with all correlated variables changing “behind the scenes”
- E.g. main variable changes to halfway between baseline and upper limit, correlated variables also move halfway between its baseline value and the limit of the range.

## Correlated Sensitivity Analysis (2)

- Why?
- Possible systematic bias
- From same underlying study
- May have had a healthier or sicker population
- May have possible bias in methods leading to results that are biased in one direction
- Same method for each parameter, possible bias
- E.g. utilities all done in providers as proxies for patients

## Correlated Sensitivity Analysis (3)

- 2nd-order (parameter) uncertainty
- 1st-order uncertainty not very well
- Structural uncertainty – not really
- Good for dealing with cases where there is clear reason for concern that a systematic bias is present, or when there is clear reason to believe that some readers will think there is a systematic bias and this approach can alleviate those fears and restore reader confidence
- Easy to understand
- When to do?
- Any time there is a valid or perceived fear of systematic bias

## Correlated Sensitivity Analysis: Examples

- Utility values for stages of a disease
- Answer what if the impact of the illness has systematically been under-or over-estimated
- Surgical mortality inherent in the procedure, not due to patient differences
- Affects surgical mortality rates for different paths in tree, e.g. before or after some other intervention or complication

## Probabilistic Sensitivity Analysis

- How to:
- For each parameter’s point estimate, create a probability distribution for the mean value
- Create any needed correlations
- Create a random sample of parameter values using the distributions
- Run an incremental cost-effectiveness analysis using this sample’s parameter values
- Repeat for 1000 samples and analyze results
- Plot the individual samples showing the CE quadrants
- Proportion of time a strategy is cost-effective calculated at different values of willingness-to-pay

## Probabilistic Sensitivity Analysis (2)

- Why?
- Many parameters, all with uncertainty
- Impact of variation simultaneously in all variables is unanswered by traditional methods of sensitivity analysis
- Traditional incremental CEA based on mean values creates a single value, which can be misleading since it ignores the inherent variation due to uncertainty in all parameter estimates

## Probabilistic Sensitivity Analysis (3)

- 2nd-order uncertainty – does this well
- Does not address 1st-order uncertainty and structural uncertainty
- Does not address misspecification
- Complements rather than replaces traditional 1- and 2-way sensitivity analyses
- When to do?
- Almost always. Can be exceptions, and sometimes it is obvious it won’t change anything.
- Becoming expected, and some journals and many referees require it

## Probability Distributions

- Uniform
- Good for numbers based on arbitrary choice or expert opinion
- Normal
- Good when based on solid estimates
- Negative numbers allowed
- Beta
- Restricts values to 0,1
- Allows all kinds of shapes
- Log-Normal
- Good for skewed data
- Nonnegative
- Gamma
- Nonnegative
- Different shapes

## WARNING!!PROCEED AT RISK!

- What follows is my own personal approach to conducting probabilistic sensitivity analysis.
- Others use different distributions.
- I am only showing you what I do, and will demonstrate the steps to actually doing it.
- Any dangers to your personal or professional health are not the responsibility of the presenter or the VA.

## Key Idea

- For each parameter, take the baseline value as the mean for your distribution
- Use the upper and lower limits from your 1-way sensitivity analysis to form the limits of an estimated 95% confidence interval
- Approximate the width of the confidence interval as mean +/- 2 standard deviations (thank you Andy Briggs).

## Costs

- If based on solid estimates (e.g. Medicare reimbursement rates) for known specific health care utilizations, I used a Normal distribution with confidence interval +/- 25% of the mean value.
- 25% of mean = 2 standard deviations
- Standard deviation = .125 * mean

## Costs (2)

- If costs are based on studies, such as annual cost of care for a chronic illness, then I use wider confidence intervals and use a Gamma distribution. I often halve and double these kinds of costs for the high and low values for the range. The range (high – low) is about 4 std dev’s.
- Std dev = (high-low) / 4
- Parameters of gamma distribution:
-    Alpha     = (mean/sd)2
-    Lambda = mean/sd2

## Probabilities and Utilities

- I use a beta distribution to keep it between 0 and 1. It is also flexible in shape.
- sd = (high – low) / 4
- n = (mean * (1-mean) / sd2) – 1
- r = mean * n
- Parameters of Beta distribution:
-    alpha = r
-    beta = n - r

## Problem Definition

- Reference Case
- 60-year old male
- 4 cm abdominal aortic aneurysm
- Otherwise, patient is in good health
- Surgery vs Watchful Waiting
- Time horizon: 5 years
- Effectiveness Measure: Survival
- Alive = 1
- Dead = 0

## State Transition Diagram

**Aneurysm**

**Acute**
**Expansion**

**Well**

**Dead**

- If Death from other causes was considered, one might show all states with an arrow to death, or merely note this in text below the diagram.
- Not shown are arrows from a state to itself, which is valid for all states except **Acute Expansion**.

## Slide 34

## Costs

- Cost of Annual Screening, $579
- Normal distribution
- Mean = 579
- sd = .125 * mean = .125*579 = 72
- Cost of elective surgery, $49,381
- Normal distribution
- Mean = 49,381
- sd = .125 * 49,381 = 6,173

## Annual Discount Rate

- Baseline = 3%
- Range 1% - 5%
- Uniform distribution
- Low value = 0.01
- High Value = 0.05

## Probabilities

- Annual probability of expansion
- Baseline = .033     Range = .016 - .049
- sd = (high-low)/4 = (.049 - .016)/4 = .00825
- n = (mean*(1-mean)/sd2)–1
- ((.033)(.967)/.008252)-1= 467.85
- r = mean * n = .033 * 468 = 15.44
- Parameters of Beta distribution:
-    alpha = r = 15.44
-    beta = n – r = 467.85 – 15.44 = 452.41

## We first need to enter the distributions to be used in the PSA.

- Select **Distributions** from the **Values** menu.

## Since there are no distributions yet, the box is blank.

- Click on **New** to create the first distribution.

## You are shown the different distributions, each depicted with a typical shape. It defaults to the Normal dbn. This can be seen by the green shading in the diagram for Normal.

## For the selected distribution, the pertinent parameters defining the distribution are shown on the right, where you can set these values.

## For the cost of annual screening, the parameters of the Normal distribution were determined to be: mean = 579 and std deviation of 72, so this was entered. Then click OK.

- For the cost of annual screening, the parameters of the Normal distribution were determined to be: mean = 579 and std deviation of 72, so this was entered. Then click **OK**.
- We will need to create a different distribution for each parameter.

## Each distribution has an associated index. This number will show up when the distribution is used in the model. However, the name and description will show up in the list of distributions when you need to select which distribution to use.

- I like to name the distribution after the parameter it is associated with.

## The name must be distinct from any name already used for a variable. But you don’t have to enter a name. I tend to not create a name at all, but instead, for the description I enter exactly the name of the variable I created in TreeAge.

- Now click **OK** to finish creating this distribution, and go on to the other variables.

## The distribution we just created now shows up in the list of distributions. We will need to enter distributions for each of the parameters. Let’s create one more together, but this time let’s do the probability of acute expansion. It requires a beta distribution.

## Like before, the default distribution is the Normal dbn. We need to click on Beta to select the distribution type. Then the correct parameters will appear.

- Like before, the default distribution is the Normal dbn. We need to click on **Beta** to select the distribution type. Then the correct parameters will appear.

## After selecting the Beta distribution, the Beta diagram is shaded green. The parameters have been changed to n and r. BUT...also note that the choice for integer parameters is selected, but we want real-numbered parameters.

- After selecting the **Beta** distribution, the Beta diagram is shaded green. The parameters have been changed to n and r. BUT...also note that the choice for **integer parameters** is selected, but we want **real-numbered parameters**.

## After selecting real-numbered parameters, the parameters for the Beta distribution are changed to alpha and beta. We need to change the default values to 15.44 and 452.41.

- After selecting **real-numbered parameters**, the parameters for the **Beta **distribution are changed to alpha and beta. We need to change the default values to 15.44 and 452.41.

## Now we can click OK and then finish up creation of this distribution.

- Now we can click **OK **and then finish up creation of this distribution.

## At this screen, the index was defaulted for me. I then typed in the description as the name of the corresponding variable.

- The order you enter the variables is the order for the index, which is used to list the distributions when selecting a distribution to use in the model.
- You might want the order to be alphabetical. In this case, that means this should be index 8. So, I’ll change it before clicking **OK** to be done.

## Back to the tree. We need to change the values of the variables so that instead of a default value, we supply the distribution. The default values all show up in the variables window of the root node. First, we have to see the variables window. Select the root node and then.....

## Select Show variables window from the Values menu.

- Select **Show variables window** from the **Values** menu.

## Now we can select each variable one at a time, and change the value assignment from a value to its corresponding distribution. Let’s select cAnnualScreening.

- Now we can select each variable one at a time, and change the value assignment from a value to its corresponding distribution. Let’s select **cAnnualScreening**.

## The default baseline value of 579 is showing. We want to select the option on the right to Insert Distribution,

- The default baseline value of 579 is showing. We want to select the option on the right to **Insert Distribution**,

## Select the distribution cAnnualScreening, and click Use.

- Select the distribution **cAnnualScreening**, and click **Use**.

## The distribution is now being used for the value of cAnnualScreening. Note that the distribution is identified by the index number, and not the name or description for the distribution. Then click OK.

- The distribution is now being used for the value of **cAnnualScreening**. Note that the distribution is identified by the index number, and not the name or description for the distribution. Then click **OK**.
- We can then use the same technique to enter the rest of the distributions.

## When all done, the variables window looks like this. Not very informative. After doing this, we may want to see the variables, so select Preferences from the Edit menu.

- When all done, the variables window looks like this. Not very informative. After doing this, we may want to see the variables, so select **Preferences** from the **Edit** menu.

## Select the option to Show definitions. This has the advantage of letting you see the variables, but it also means the tree takes up a lot more space. This means you don’t see as much of the tree at once.

- Select the option to **Show definitions**. This has the advantage of letting you see the variables, but it also means the tree takes up a lot more space. This means you don’t see as much of the tree at once.

## Slide 59

## Before running the analysis, we should check to see if the numeric formatting is OK. Select Numeric Formatting from the Edit menu.

- Before running the analysis, we should check to see if the numeric formatting is OK. Select **Numeric Formatting** from the **Edit** menu.

## For each category, Cost, Effectiveness and Cost/Eff, select it and set the number of decimal places to 3.

- For each category, **Cost**, **Effectiveness** and **Cost/Eff**, select it and set the number of decimal places to 3.

## Before running the PSA, it is a good idea to first run the cost-effectiveness analysis again to see if you get the same answers as before. TreeAge will use the mean values of the distributions for the parameter values if you do an ordinary cost-effectiveness analysis. 

- Before running the PSA, it is a good idea to first run the cost-effectiveness analysis again to see if you get the same answers as before. TreeAge will use the mean values of the distributions for the parameter values if you do an ordinary cost-effectiveness analysis.
- Whew! Same answer!

## To run the PSA, be sure the root node is selected and choose from the menus: 

- To run the PSA, be sure the root node is selected and choose from the menus:
**Analysis/Monte Carlo Simulation/Sampling (PSA)**.

## The option Sample all distributions was selected by default, and that IS what we want!

- The option **Sample all distributions** was selected by default, and that IS what we want!
- The number of samples also defaulted to what we want – namely, 1000. This arbitrary, but commonly used.
- I then select **More Options**. This will allow me to set the seed for the random number generator.

## I had to select the option at the top, Seed random number generator. This is the number used to start the algorithm TreeAge uses to create random numbers. It is chosen randomly, sort of, each time. This causes a problem – we will get different answers each time. That can make it hard to detect if we have changed something inadvertently. 

- I had to select the option at the top, **Seed random number generator**. This is the number used to start the algorithm TreeAge uses to create random numbers. It is chosen randomly, sort of, each time. This causes a problem – we will get different answers each time. That can make it hard to detect if we have changed something inadvertently.
- Therefore, I always choose to fix the seed. I chose 1.

## I had to select the option at the top, Seed random number generator. This is the number used to start the algorithm TreeAge uses to create random numbers. It is chosen randomly, sort of, each time. This causes a problem – we will get different answers each time. That can make it hard to detect if we have changed something inadvertently. 

- I had to select the option at the top, **Seed random number generator**. This is the number used to start the algorithm TreeAge uses to create random numbers. It is chosen randomly, sort of, each time. This causes a problem – we will get different answers each time. That can make it hard to detect if we have changed something inadvertently.
- Therefore, I always choose to fix the seed. I chose 1. Easy to remember.

## Now we are ready. Click on Begin.

- Now we are ready. Click on **Begin**.

## At the bottom we are told that the statistics shown are for Elective Surgery. We can use the drop down list to see the same statistics for Watchful Waiting. Instead, lets try to see the Acceptability curve. Click on Graph.

- At the bottom we are told that the statistics shown are for **Elective Surgery**. We can use the drop down list to see the same statistics for **Watchful Waiting**. Instead, lets try to see the Acceptability curve. Click on **Graph**.

## Select Acceptability Curve from the drop-down list.

- Select **Acceptability Curve** from the drop-down list.

## Here, click on No. We want to see the pairwise comparison.

- Here, click on **No**. We want to see the pairwise comparison.

## Here I selected a range from 20,000 to 100,000 for willingness to pay. By using 8 intervals, we will get results in increments of 10,000.

- For the baseline, I selected **Watchful Waiting**.
- For comparator(s) I selected **Elective Surgery**.

## Whoops!

- Problem here.
- The **Elective Surgery** arm is dominated.
- PSA isn’t going to help. It will still lose even at the extreme end of the willingness to pay range.
- So, for a better example of what it should look like, here is a graph from an upcoming paper of mine.
- The problem is looking at whether to use growth factors as a first step before dose reduction of ribavirin in Hepatitis C patients experiencing anemia during combination therapy.

## For each willingness to pay threshold, the graph plots 2 points: the proportion of the time each strategy is cost-effective among the 1000 samples. These 3 values always add up to 1, so the 2 curves always intersect at p=.5.

- At $50,000/QALY or greater the **darbopoetin alpha** strategy is cost-effective in over 60% of the samples, and appears to be cost-effective compared to **Dose Reduction**.

## To do correlated sensitivity analysis, we go to the properties window of one of the variables. I chose the probability of death from elective surgery. Click on Correlations.

- To do correlated sensitivity analysis, we go to the properties window of one of the variables. I chose the probability of death from elective surgery. Click on **Correlations**.

## We need to select all the variables to be correlated. I will choose just the pDieSurgeryNoRupture, which is already shown above as selected. I then click on the >> button to add it to the list of correlated variables.

- We need to select all the variables to be correlated. I will choose just the pDieSurgeryNoRupture, which is already shown above as selected. I then click on the **>>** button to add it to the list of correlated variables.

## TreeAge then prompts us to set whether the correlation is positive or negative. That is, when one variable changes, does the other change similarly or in the opposite direction. When done, click OK, and do so again when you return to the list (unless you want to add another correlated variable.

- TreeAge then prompts us to set whether the correlation is positive or negative. That is, when one variable changes, does the other change similarly or in the opposite direction. When done, click **OK**, and do so again when you return to the list (unless you want to add another correlated variable.
- That’s it! Now, when you do sensitivity analysis on one of these variables, the other one will automatically change, too. I’d like to do more, but tick, tick, tick.....

## Slide 77

## Thank  you !

- Questions or follow-up?
- Contact Mendel Singer at: 216-368-1951  or mendel@case.edu