Analysis Alongside A Randomized Trial

Todd Wagner, PhD

May 2009

Objectives
        - At the end of the class, you should

Understand how to set up your datasets

Be familiar with analytical methods 

Want to hear the class on decision modeling

Dominance
        - No need to calculate an incremental cost effectiveness ratio if:

Intervention is more effective and less expensive than control

Intervention is more expensive and less effective than control
        - But:

Check whether dominance exists within subgroups

Does dominance persists after including uncertainty

CostEXP - CostCONTROL

_____________________

QALYEXP -QALYCONTROL

Incremental Cost-Effectiveness Ratio (ICER)
        - Calculate in the absence of dominance

Cost Data
        - Have costs consistent with the stated perspective
        - Societal

Health care utilization

Patient costs

Caregiver costs

Intervention costs (direct plus indirect)

Common Hurdle
        - Many of the parameters in the analysis are based on assumptions (e.g., wage rate, mileage costs)
        - Consider whether these assumptions are biased toward/against the intervention 
        - Ideally want to include uncertainty

One way sensitivity analysis

Statistical methods (bootstrapping, Cooks D)

Another hurdle
        - Include disease-related utilization or all health care utilization?
        - How do you define disease-related?
        - Recommend: look at all utilization for the CEA

Labor Outcomes
        - Productivity employment is not in the cost estimate
        - Anyone remember why?
        - If labor outcomes are important, still collect them but report them separately.

Dataset
        - You need to create a “long” dataset
        - You need to have an group indicator for experiment (1) and control (0)
        - You need to have cost and outcome information
        - You need subgroup identifiers

Data

Analysis

. tabstat totcost followup if exp==1, by(papres) stat(mean) format(%3.2f)

Summary statistics: mean

  by categories of: papres (initial pap results)

    papres |   totcost  followup

-----------+--------------------

ASCUS/AGUS |    347.31      0.57

     LGSIL |    373.97      0.64

     HGSIL |    404.72      0.87

-----------+--------------------

     Total |    355.45      0.61

--------------------------------

. tabstat totcost followup if exp==0, by(papres) stat(mean) format(%3.2f)

Summary statistics: mean

  by categories of: papres (initial pap results)

    papres |   totcost  followup

-----------+--------------------

ASCUS/AGUS |     74.92      0.32

     LGSIL |     73.98      0.30

     HGSIL |    104.94      0.43

-----------+--------------------

     Total |     77.28      0.32

--------------------------------

Hypothesis Testing

. reg totcost exp

      Source |       SS       df       MS              Number of obs =     348

-------------+------------------------------           F(  1,   346) =  292.64

       Model |   6728200.2     1   6728200.2           Prob > F      =  0.0000

    Residual |  7954890.66   346  22991.0135           R-squared     =  0.4582

-------------+------------------------------           Adj R-squared =  0.4567

       Total |  14683090.9   347  42314.3829           Root MSE      =  151.63

------------------------------------------------------------------------------

     totcost |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]

-------------+----------------------------------------------------------------

         exp |   278.1664   16.26051    17.11   0.000     246.1845    310.1483

       _cons |   77.27982   11.62933     6.65   0.000     54.40675    100.1529

------------------------------------------------------------------------------

Logistic regression                               Number of obs   =        348

                                                  LR chi2(1)      =      28.42

                                                  Prob > chi2     =     0.0000

Log likelihood = -226.30785                       Pseudo R2       =     0.0591

------------------------------------------------------------------------------

    followup | Odds Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]

-------------+----------------------------------------------------------------

         exp |   3.225974   .7243906     5.22   0.000     2.077419    5.009538

------------------------------------------------------------------------------

Conclusion
        - Intervention is more costly and more effective
        - Next Steps: 

Include uncertainty

Check for subgroups and

calculate the ICER

ICER Calculation

Intervention Group

    papres |   totcost  followup

-----------+--------------------

ASCUS/AGUS |    347.31      0.57

     LGSIL |    373.97      0.64

     HGSIL |    404.72      0.87

-----------+--------------------

     Total |    355.45      0.61

--------------------------------

Control Group

    papres |   totcost  followup

-----------+--------------------

ASCUS/AGUS |     74.92      0.32

     LGSIL |     73.98      0.30

     HGSIL |    104.94      0.43

-----------+--------------------

     Total |     77.28      0.32

--------------------------------

ICER=(355.45-77.28)

		      (0.61-0.32)

ICER=982.1837 

Is that good or bad?

Uncertainty
        - You need to calculate the confidence regions around this parameter estimate

Variable |   Reps   Observed       Bias   Std. Err.   [95% Conf. Interval]

---------+-------------------------------------------------------------------

   cerp2 |   1000   982.1837   24.04897   152.2913     683.336  1281.031  (N)

         |                                             787.438  1369.206  (P)

         |                                            787.3935  1366.588 (BC)

-----------------------------------------------------------------------------

                              N = normal, P = percentile, BC = bias-corrected

Confidence Regions
        - Ratios are complex to interpret

More effective and more expensive

More effective and less expensive

Less effective and less expensive

Less effective and more expensive

Acceptability Curves
        - Acceptability curves show the information based on willingness to pay for the outcome.
        - Shape of the curve is dependent on the bootsrapped estimates
        - Allows decision makers with different thresholds to interpret the data

Source: Henry Glick

Threshold Value forCost-Effectiveness Ratio
        - Rule of thumb:

	U.S. health care system adopts interventions with ratio of less than $50,000 ($100,000) per Quality Adjusted Life Year.

Variable |   Reps   Observed       Bias   Std. Err.   [95% Conf. Interval]

---------+-------------------------------------------------------------------

   cerp2 |   1000   982.1837   24.04897   152.2913     683.336  1281.031  (N)

         |                                             787.438  1369.206  (P)

         |                                            787.3935  1366.588 (BC)

-----------------------------------------------------------------------------

                              N = normal, P = percentile, BC = bias-corrected

9

Adopt this Pap Smear Intervention?
        - Poll
        - Yes
        - No
        - Don’t know
        - Don’t care

Limitations with RCTs
        - Proxy outcomes
        - Length of follow-up
        - Generalizability

Ideal Study
        - RCT comparing behavioral intervention and usual care control
        - Follow participants for life
        - Advantages: know all costs and all benefits
        - Disadvantages: expensive and possibly useless results at end of study

Usual Study
        - Outcomes measured at end of study
        - Use models to estimate lifetime costs and benefits in the CEA

Effectiveness
        - Preferred metric for CEA is quality adjusted life years (QALYs)
        - Most behavioral interventions use an “intermediate outcome”

e.g., receipt of mammography
        - Few behavioral interventions use QALYs because the study would have to be huge and/or very long in duration

CEA with Intermediate Outcomes
        - Easy and sufficient for publication
        - Hard to interpret ICER

Can’t compare two CEAs with different intermediate outcomes

Can’t compare CEA to other CEA from another clinical area
        - Sometimes only feasible approach

Intermediate to QALYs
        - “Translate” intermediate outcome to QALYs
        - Either build a model de novo or use an existing model
        - Requires a lot of resources
        - Most useful, but most challenging

Lost in Translation
        - Gap between ideal and usual study
        - Models fill the gap
        - Behavioral models have unique challenges

Partial behavior change is missing from current models

Definition: people who progressed in their  “stage of change” but did not successfully change their behavior at the end of the study

Wagner TH, Goldstein MK. Behavioral interventions and cost-effectiveness analysis. Prev Med. Dec 2004;39(6):1208-1214.

Outcome Responsiveness
        - Many trials use indirect QALY assessments (HUI, QWB, EQ-5D)
        - These measures are often not responsive to the intervention effect at the end of the trial.
        - How do you interpret the results when the main outcome shows an effect and the QALYs do not?

QALY Responsiveness
        - Try to understand why QALYs were not sensitive
        - Analyze data for subgroup effects
        - Analyze other outcomes that may be sensitive to change.

Disease specific quality of life

Willingness to pay
        - If you think there is an effect, ignore the p-value on the QALYs and model the data using the mean and variance estimates

Another Approach
        - Our CEA did not leverage the trial data
        - We extracted parameters, created confidence regions and then (hopefully) put these parameters into a decision model

Net Monetary Benefits
        - What we did

Collect data to estimate ΔC and ΔE

Calculate an estimate:

ΔC / ΔE (ICER = incremental cost-effectiveness ratio)
        - Another option: 

λ ΔE – ΔC (INB = incremental net-benefit)

Hoch JS, Briggs AH, Willan AR. Something old, something new, something borrowed, something blue: a framework for the marriage of health econometrics and cost-effectiveness analysis. Health Econ. Jul 2002;11(5):415-430.

Data

NMB set up
        - You have cost data and effectiveness data for each person
        - Use this information in a regular regression framework
        - You need to estimate λ

λ is the WTP for the outcome

Rerun the analysis for different λ values

NMB Limitations
        - Based on a linear model
        - Does not necessarily translate into non-linear or latent models
        - I suspect this is possible if you estimate predicted probabilities for groups and then carry out the analysis for the groups.

Generalizability
        - Recall that RCTs may not enroll a generalizable population
        - For many RCTs in VA, you can compare participants to non-participants

Generate propensity scores

Use the propensity score as a weight in the RCT analysis– place more weight on people who are a lot like the non-participants

Next talk
        - May 27, 2009  How Can Cost Effectiveness Analysis Be Made More Relevant to US Health Care?   Paul Barnett, Ph.D.