# Presentation: 529370

## Advanced EBM Meta-Analysis and Decision Theory

- Atif Zafar, M.D.
- Indiana University

## Outline of Presentation

- Introduction
- Review:  *The Hierarchy of Evidence*
- Basics of Diagnosis and Therapy Articles
- Case-Control Studies
- Cohort Studies
- Case Series and Case Reports
- Meta-Analysis
- Decision Theory and Economic Analysis

## Introduction

- What is Evidence Based Medicine (EBM)?
- “Evidence-based medicine is the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients”
- 				-- David L. Sacket.
- http://cebm.jr2.ox.ac.uk/ebmisisnt.html

## Introduction

- Useful Web Sites for Topics on EBM:
- _[http://cebm.jr2.ox.ac.uk/](http://cebm.jr2.ox.ac.uk/)_
- _[http://www.cebm.utoronto.ca/](http://www.cebm.utoronto.ca/)_
- _[http://www.cche.net/CHE/home.asp](http://www.cche.net/CHE/home.asp)_
- _[http://www.med.ualberta.ca/ebm/main.](http://www.med.ualberta.ca/ebm/main.htm)__[htm](http://www.med.ualberta.ca/ebm/main.htm)_
- http://library.downstate.edu/ebm/4toc.htm
- Essential Reference:
- “EBM: How to Practice and Teach Evidence Based Medicine”
- 		David Sackett et. Al.
- 		Churchill-Livingston
- 		ISBN: 0-443-06240-4

## Introduction

- _Why the sudden interest in EBM in the last 10-20 years?_
*	*_Four realizations about everyday practice:_
- Our daily need for valid information about diagnosis, prognosis, therapy and prevention
- Inadequacy of traditional sources of information because they are :
- (1) out of date (textbooks)
- (2) frequently wrong (experts)
- (3) ineffective (CME),
- (4) too overwhelming in volume, validity and applicability 	 	 (journals)

## Introduction

- _Why the sudden interest in EBM in the last 10-20 years?_
-    _Four realizations about everyday practice:_
- Disparity between our diagnostic skills and clinical judgement (which increase with experience) and our up-to-date knowledge and clinical performance (which decrease with time)
- Our inability to devote more than a few seconds per patient for finding and assimilating this evidence (1/2 hour per week)

## Introduction

- _5 developments helped to solve these dilemmas_:
- Development of *strategies* for effectively tracking down and *appraising* this evidence
- Creation of *systematic reviews* and *summaries* of the effects of health care
- Creation of *EBM journals*
- Creation of *information systems* for *efficient and timely delivery* of this information to us in seconds
- Identification and implementation of effective strategies for *life-long learning* and for improving our performance

## Introduction

- What questions does EBM address?
- _Basic EBM: Questions related to patient disease management_:
- What causes a disease?
- How do you diagnose a disease?
- How do you treat a disease?
- What is the prognosis of someone with the disease?
- _Advanced EBM: Integrative Studies looking at populations_:
- Decision Analysis and Economic Analysis
- Systematic Reviews (aka Meta-Analysis)
- Guidelines and Health Care Recommendations

## Introduction

- _Hierarchy of Evidence Pyramid_

**Reference:  **http://servers.medlib.hscbklyn.edu/ebmdos/3ebm100.htm

## Introduction

- Most articles on disease management will “fit” into one of these 4 categories
- Diagnosis
- Therapy
- Prognosis
- Harm
- 90% will be in Diagnosis and Therapy as will your questions about disease management in a real setting.

## Review Material

- Two questions generate 90% of literature search efforts and reviews in everyday practice:
- How do you diagnose a disease?
- How do you treat a disease?

## Articles on Diagnosis and Therapy

- 3 fundamental questions to ask when reading an article on diagnosis **or** therapy:
*I. Are these results valid?*
*II. What were the results of the study?*
*III. Will these results help me in caring for my patient?*

## Therapy

- _I. Are the results of a study valid_?
*Was the assignment of patients to treatments ****randomized****?*
*Were all the patients who entered the trial properly accounted for at its conclusion? (was follow-up complete and were patients analyzed in the groups to which they were randomized?)*
*Were the patients and health care workers ****blind**** to treatment?*
*Were the groups similar at the start of the trial?*
*Aside from the experimental intervention, were the groups treated equally?*

## Therapy

*Was the assignment of patients to treatments ****randomized****?*
*Randomization reduces bias so that comparisons between 2 groups are fair except perhaps the elements in question.*
*Were all the patients who entered the trial properly accounted for at its conclusion? (was follow-up complete and were patients?*
*This speaks to the issue of loss to follow-up. A 20% drop-out rate is unacceptable. A 10% or better drop-out rate is acceptable.*
*Loss to follow-up can dramatically reduce the statistical significance of a study.*

## Therapy

***Were patients analyzed in groups to which they were randomized****?*
*	**In randomized trials, as in real life, subjects often are noncompliant. Readers might think that these patients should be excluded from the analysis because they did not complete the trial. These noncompliant patients often fare worse than those who take their medicines. Excluding these noncompliant patients from the analysis makes the study look better because the other may be destined to have a better outcome. This destroys the value of the randomization. Thus it is important to analyze groups as randomized. This principle is called an ****Intention to Treat Analysis****. *

## Therapy

*Were the patients and health care workers ****blind**** to treatment?*
*	**Blinding is an important consideration in randomized trials because subjects who know they are on a particular treatment may have an opinion about its efficacy and may report symptoms or therapeutic effects differently as compared to if they did not know they were on a specific therapy. Similarly, study personnel who know that a certain person is on a specific treatment may evaluate the results a little differently. Both of these can distort the results and cause bias.*

## Therapy

- Were the groups similar at the start of the trial?
- 	In order to evaluate the efficacy of a specific intervention it is necessary that two comparison groups be similar except for the intervention planned. Otherwise we are comparing apples to oranges. Investigators usually list these baseline characteristics in Table - 1 of a paper. Randomization does not always produce groups that are similar, especially if the sample size is small and chance may place those with a better prognosis in one group versus another. Thus it is important to consider that appropriate power (the number of subjects needed to be able to detect a statistically significant result) is provided in the methods section. If the groups are not similar at the start of the trial, investigators need to have adjusted for these differences in their analysis.

## Therapy

*Aside from the experimental intervention, were the groups treated equally?*
*	**It is important to realize that 2 experimental groups may receive different treatment depending on where they are being evaluated. For example, a subject seen in a tertiary care center may receive closer follow-up and perhaps be on other medicines in addition to the ones for the study. These co-interventions can bias the results and investigators need to be careful about this.*

## Therapy

- II. What were the results?
- How large was the treatment effect?
- What is the precision of measurement of the treatment effect?

## Therapy

- How large was the treatment effect?
- 	The size of the treatment effect is often reported in a number of ways. Most often, randomized trials monitor whether an outcome happens or does not happen. One way to express this dichotomous outcome is via the absolute risk reduction (ARR), also known as the risk difference. This is the difference in the absolute number of occurrences of an event between the treatment group and the control group. For example, consider the following: Suppose 30% of a control group suffered an MI but only 10% of a treatment group suffered an MI.
- 	How can we express this using the ARR?

## Therapy

- How large was the treatment effect?
- ARR = Risk (control) - Risk (treatment) = 30% - 10% = 20%.
- Another way to express this is using the **relative risk**. This is the risk of events in those getting the therapy relative to that among the controls.
-    RR = Risk (Treatment) / Risk (Control) = 10%/30% = .33
- 	Finally, we may use the term relative risk reduction which is the most commonly used method to report treatment effects.
-    RRR = (1 - RR) x 100% = (1 - .33) x 100% = 67%.
- This means that the new treatment reduced the chance of an event by 67% relative to that occurring in the control group.

## Therapy

- What is the precision of measurement of the treatment effect?
- 	Precision in statistics is measured using 2 indices. One is called the *point estimate* and the other the *interval estimate*.
-  	Imagine a study where we want to estimate the average age of everyone in the US. We could go and ask each person their age but this would be too cumbersome. We could take samples (smaller subsets of the whole) of the US population and estimate the average age of the sample and then conclude that the average age of the US population is "close" to the average age of the sample. If we take more than one sample then the average age of the individual samples combined gets closer and closer to the average age of the US population. The average age of the samples is called a point estimate. If we take many samples then we get a range of average ages between which the true average age of the US population lies. This range is called the interval estimate or confidence interval of the study. We have arbitrarily chosen 95% as the bound limit for confidence intervals because statisticians attribute 5% of an observed effect to chance.

## Therapy

- III. Will the results help me in caring for my patients?
- Generalizability:  Refers to the ability to be able to replicate the results elsewhere
- Clinical and Statistical Significance
- Will I do any *harm*?
- Speaks to the notion of a ***number needed to treat***.

## Therapy

- Number Needed to Treat:
- 	The impact of a particular treatment is related not only to its benefits but its potential risks as well. Thus a drug may save 1/100 lives but has such a severe side effect profile that it may not be worth the intolerability. This observation can be formulated in the concept known as the Number Needed to Treat (NNT). The NNT is defined as the reciprocal of the Absolute Risk Reduction (ARR).
-      NNT = 1/ARR
- 	An NNT of 100 or less is generally acceptable, 50 or less is considered good and <10 is excellent, taking into account the side effect profile and costs.

## Diagnosis

- Same 3 questions as for therapy:
- Was there an independent “blind” comparison to a reference standard?
- Was the sample size appropriate?
- Did the results of the test being developed influence the decision to do the reference test?
- Were methods presented reasonably reproducible?
- Are likelihood ratios presented or calculable?
- Will the results make me change my practice?

## Diagnosis

- Was there an independent “blind” comparison to a reference standard?
- 	When using a diagnostic test we are attempting to understand the truth about a patient having a particular condition or not having it. Thus when we evaluate the validity of a diagnostic test we must compare it with the truth. Ideally, we want diagnostic tests to be safe, inexpensive and accurate. This, however, is rarely achieved in actual practice and so we must judge the utility of a test based on these three ideals. When we evaluate the validity of a test we need some best representation of the truth. This can come from a variety of sources. We could apply a test to subjects with a known disease to see how well the test fares. Or we could prospectively evaluate a test with the best estimate of the truth, also called a gold standard.

## Diagnosis

- Was the sample size appropriate?
- 	A diagnostic test is useful when it can distinguish the target disorder in the spectrum of patients it will be applied in. For example, an elevated amylase may not be a valid marker all the time for pancreatitis, unless the patient has other clinical symptoms such as abdominal pain and nausea. It could also be caused by renal failure or in macroamylasemia or in someone with bowel wall ischemia. Thus, randomly applied to patients, it does not indicate pancreatitis. Thus when diagnostic tests are evaluated for validity they should be performed on patients who will actually qualify for the test based on symptoms or other clinical indications.

## Diagnosis

- Were methods presented reasonably reproducible?
- 	In an article we must look for the specifics of how the testing was done, the selection criteria, the test protocol, the testing technology etc. This will allow us to *reproduce* the test at our own institutions.

## Diagnosis

- What were the results? Are likelihood ratio’s presented?
- 	The starting point for the application of any diagnostic test is the clinical estimation of the likelihood of disease. We would not perform a cardiac cath in a young patient with no history of chest pain. This concept is called the pretest probability. Thus, the pretest probability of an MI in a 20 year old patient with chest pain is much less than the pretest probability of an MI in a 70 year old patient with chest pain. Each historical and physical exam element that we do is itself a diagnostic test that either increases or decreases the pretest probability of a disease.
- 	Once we know the pretest probability of a disease we can evaluate the post-test probability (the probability of having disease given a positive test) of the disease. Each diagnostic test we do has an associated operating characteristic called a **likelihood ratio** (LR). The likelihood ratio is the probability that a test is positive in someone with disease as compared to someone without disease.
- 	Likelihood Ratio (+) = Sensitivity / (1 – Specificity)

## Diagnosis

- Will the results change my practice?
- Reproducibility:  Speaks to the chance that a study results may not be replicable elsewhere?
- Are the results applicable to my patients?
- Will the results change management?
- Will the patients be better off as a result?

## Diagnosis

- Operating Characteristics:
- Positive/Negative Predictive Value
- P(disease is present/absent given they test +/-)
- Sensitivity and Specificity
- P(test is +/- given that they have/don’t have disease)

## Meta-Analysis

- This is the “best evidence” we have
- Combines multiple randomized controlled trials into one analysis
- The cornerstone is the “Forest Plot” which deals with odds ratio’s.
- 	ODDS Ratio = Probability/(1-Probability)

## Meta-Analysis

- The *hierarchy of evidence: *
*Meta-Analysis*
*Randomized Controlled Trial*
*Cohort Study*
*Case-Control Study*
*Case Series*

## Meta-Analysis

- Forest Plot
- Depicts the Odds Ratio versus sample size
- Allows easy comparison between multiple studies
- Center line -> odds ratio = 1 (no effect line)
- Cross this line -> study is not significant

## Meta-Analysis

- _Example Forest Plot_:

**NO EFFECT LINE**

- Negative Effect

- Positive Effect

## Meta-Analysis

- Criteria for Study Evaluation:
- (1) study design
- (2) combinability
- (3) control of bias
- (4) statistical  analysis
- (5) *sensitivity analysis*
- (6) problems of applicability.

## Meta-Analysis

- Sensitivity Analysis
- Done to ensure that all trials included in the study actually belong in the analysis (I.e. they are similar)
- This characteristic is called *homogeneity*.
- Sensitivity Analysis looks for homogeneity

## Meta-Analysis

## Meta-Analysis

- Pitfalls specific to meta-analysis include:
- 1.It's rare that the results of the different studies precisely agree, and often the number of patients in a single study is not large enough to come up with a decisive conclusion.
- 2.If the authors are interested in supporting a particular conclusion, they can include studies that support that conclusion and omit studies that do not. Do the authors explain in their paper exactly on what basis they included studies , and do their reasons make sense?
- 3.Studies that show some kind of positive effect tend to be published more often than those that do not. This means that if the authors include only published studies, several weak positive studies may seem to add up to a strong positive result. Do weak negative studies exist? This effect is known as Publication bias.

## Case Control Studies

- Studies where patients who *already have* a certain disease are compared *with those who do not.*
***Less**** *reliable than either RCT’s or Cohort Studies
- Reason:  Just because there is a statistical relationship between 2 states does not imply causation (I.e. Association does not imply causation).

## Case-Control Studies

- Advantages:
- They can be done quickly. By asking patients about their past history, researchers can quickly discover effects that otherwise would take many years to show themselves.
- Researchers don't need special methods, control groups, etc. They just take the people who show up at their institution with a particular condition and ask them a few questions.

## Case-Control Studies

## Cohort Studies

- A Cohort Study is a study in which patients who presently have a certain condition and/or receive a particular treatment are followed over time and compared with another group who are not affected by the condition under investigation.
- For instance, since a randomized controlled study to test the effect of smoking on health would be unethical, a reasonable alternative would be a study that identifies two groups, a group of people who smoke and a group of people who do not, and follows them forward through time to see what health problems they develop.
- Cohort studies are not as reliable as randomized controlled studies, since the two groups may differ in ways other than in the variable under study. For example, if the subjects who smoke tend to have less money than the non-smokers, and thus have less access to health care, that would exaggerate the difference between the two groups.

## Cohort Studies

## Comparisons

| Cohort | Case-Control |
| --- | --- |
| Cohort Specification | Study base specification |
| Drug exposure: base+F/U | Disease (Case) IDENT |
| Confounders/Effect Modifiers: Base+F/U | Control Selection |
| Disease (outcome) Assessment | Drug Exposure: Current + Past |
|  | Confounders/Effect Modifiers: Current+Past |
| Analysis | Analysis |

## Case Series/Case Reports

- Weakest of all of the studies
- Best for *rare* disorders and outcomes
- Case series and case reports consist either of collections of reports on the treatment of individual patients, or of reports on a single patient.
- For example: one of your patients has a condition that you have never seen or heard of before and you are uncertain what to do. A search for case series or case reports may reveal information that will assist in a diagnosis. However, for any reasonably well-known condition you should be able to get better evidence. Case series and case reports, since they use no control group with which to compare outcomes, have no statistical validity .

## Case Series/Case Reports

## Decision Analysis

***EXAMPLES***:
- Need to come to a decision despite uncertain prospects:
-      **heart disease**
-           59 yo man with CAD, s/p CABG, recurrent pain
-           operate again (higher mortality, less chance work)
-           or treat with medication
-      **influenza**
-           65 yo women with diabetes
-           35 yo women with no other disease
-           vaccinate or not
**     ****diagnose pulmonary embolism**
-           V/Q scan is easy to do but not certain
-           pulmonary arteriogram is definitive but dangerous

## Decision Analysis

- Measuring the **utility** or ***value of several alternatives*** is especially difficult in medicine because of the emotional implications of the alternatives
- Goal is to figure out the utility of each alternative, and then choose the one with the highest utility ("best")
- 	**utility** = measure of preference of a given alternative
- 	**utility** can be measured in terms of
- 		years of life
- 		years of life corrected for disability
- 	                (eg, how many healthy years = years without a leg)
- 		cost (actual expense, put monetary value on years)
- 		    (if it cost $100,000 to save a life with prenatal care)

## Decision Analysis

- Simple example:
- 	Assume 2 treatments for a disease exist, and all patients die within 4 years
- 	problem is that deaths are spread out over time
- 		 **      **_**Probability of Death**_
- 	     years (**util**)   treatment A      treatment B
-      ---------------------------------------------------
- 	       1 	     0.2               0.05
- 	       2         	     0.4               0.15
- 	       3         	     0.3               0.45
- 	       4        	     0.1               0.35
- 	     **"expected value"** of A and B
- 	      multiply probability by utility for each alternative sum over all alternatives
- 	        A = .2*1 + .4*2 + .3*3 + .1*4 = 2.3
- 	        B = .05*1 + .15*2 + .45*3 + .35*4 = 3.1
- 	        therefore choose treatment B

## Decision Analysis

- But expected value does not tell all:  EQUAL here
-                    _Probability of Death_
-      years (util)   treatment A      treatment B
-      ------------------------------------------------
-        1            	   0.0               0.5
-        2            	   0.0               0.0
-        3            	   1.0               0.0
-        4            	   0.0               0.0
-        5            	   0.0               0.5

## Decision Analysis

- 1. Draw decision tree
-     organize problem into a series of decisions & alternatives often can follow temporal order
- 	    _**Outcome nodes**__ (dots)_
- 	          possible outcome for the chosen path
- 	          eg, cure
- 	     _**Chance nodes**__ (circles)_
- 	          wherever more than one outcome may occur
- 	          eg, death vs cure
- 	     _**Decision nodes**__ (boxes)_
- 	          wherever a choice must be made among alternative here is 		    often more than one way to draw the same tree

## Decision Analysis

- 2. label outcomes
- 	 need probabilities for branches at chance nodes
- 	 need utilities at outcome nodes
- 3. average out
- 	expected value at each chance node = sum of (product of probability x value for each  branch)
- 4. fold back at decision node, prune all but most preferred branch; assign it the expected value of that branch
- 	 (ie, make the best choice)
- 5. repeat for next decision if more than one in tree

## Questions