Evidence Synthesis for 

Decision Modeling: 

Meta-Analysis

Risha Gidwani, DrPH

June 11, 2014 
Probabilities in a Decision Model

 You have a model, now you need inputs for your transition 

probabilities 
Cost-Effectiveness Inputs

Greving et all., Cost 3 -effectiveness of preventive treatment of intracranial 

aneurysms: New data and uncertainties. Neurology 2009; 73: 258.
Ways to derive model inputs

 Transforming existing data inputs

 Creating data inputs: synthesizing available data

– Meta-Analysis 

– Mixed Treatment Comparisons

– Meta Regression
Meta-Analysis 

 Multiple studies have evaluated the question of 

interest

 Create a single pooled estimate from these 

multiple studies

 Premise: the pooled estimate based on multiple 

studies will be higher quality than the estimate 

provided by a single study
Multiple Studies Published

Which to select?

Answer: All that are relevant to your research question! Then 

(you may be able to) synthesize into a single pooled estimate
Copyright © 2014 American Medical 

Association. All rights reserved.

From: Association Between Omega-3 Fatty Acid Supplementation and Risk of Major Cardiovascular Disease 

Events: A Systematic Review and Meta-analysis

JAMA. 2012;308(10):1024-1033. doi:10.1001/2012.jama.11374

Error bars indicate 95% CIs of the relative risk (RR) estimates. The size of the squares correspond to the study weight in the 

random-effects meta-analysis. Diamonds represent the meta-analysis summary effect estimate. ICD indicates implantable 

cardioverter-defibrillator; PUFAs, polyunsaturated fatty acids.

Figure Legend: 

Raw Data Summary Stats Forest Plot Study Weights
Meta-Analysis: 

Step 1: Study-specific estimate

 Step 1: a summary statistic is calculated for each study 

Tx A: 30

Tx B: 20

10 Comparative Data

Mortality: 

5% 0.05 Non-comparative Data
Meta-Analysis: 

Step 2: Weight the study-specific 

estimate

 Step 2: Summary statistic for study is (almost always) 

weighted

 Can weight each study in a different ways

– Inverse-variance method is often used 

 Smaller variance (larger) studies get more weight

– Quality weights: Cochrane recommends against their use
Meta-Analysis: 

Step 3: Create a single pooled 

estimate

 Step 3: Individual weighted estimates are then averaged to 

create a pooled point estimate 

 Meta-analysis is the computation of a weighted mean estimate

– of means

– of probabilities

– of ORs

– of RRs

– etc. 
Meta-Analysis: 

Step 4: Calculate variance 

around the pooled estimate

Step 4: Calculation of variation around pooled point estimate

Meta-analysis is the computation of a (weighted) mean estimate 

along with an estimate of variation around this mean
What meta-analysis does NOT do

 Does NOT combine 2 by 2 tables from each study to construct 

an overall 2 by 2 table, and then calculate summary statistics

Exposed Unexposed

Disease 15 20 

No

Disease

4 1 

Exposed Unexposed

Disease 30 6 

No

Disease

12 4 

Exposed Unexposed

Disease 45 26

No

Disease

16 5 

RR = 1.05 
Creating a pooled estimate (RR)

Study A

Log Relative 

Risk

Relative Risk

Relative Risk Log Relative

Risk

Study B

Log Relative 

Risk

Summary 

Log risk ratio

Relative Risk

Summary 

Risk ratio

Study C
Creating a pooled estimate, 

Mean

Study A 

Mean 

Mean 

Study B

Mean 

Pooled

Mean 

Study C
Steps in a Meta-Analysis 

1. Systematic Literature Search

2. Title + Abstract Review

3. Data Extraction of Selected Studies

4. Separate OS and RCTs

5. Convert all outcomes to the same scale 

6. Evaluate heterogeneity of Selected Studies

7. Conduct Meta-Analysis

Quantitative
1. Systematic Literature Search

 Determine inclusion and exclusion criteria a priori 

 Database search

– Save your MeSH/other search strings

 Search reference sections of articles you keep

 Search www.clinicaltrials.gov for RCTs

 Gray literature

– Not peer-reviewed
2. Title + Abstract Review

1. Read through all titles, discard those that are irrelevant

2. Read through all abstracts, discard those that are irrelevant

3. Full-text review of remaining studies, 

– Discarding those that are irrelevant

– Keep track of WHY you discarded studies for which you 

did a full-text review

 Example: “High risk” on Cochrane Risk of Bias tool

4. Create a PRISMA diagram
PRISMA diagram
3. Data Extraction of Selected Studies 





PRISMA template: http://www.prisma-statement.org/2.1.2%20-%20PRISMA%202009%20Checklist.pdf

Your own template

– Author, Year

– Journal

– Study Design (RCT, OS, Case-control, etc) 

– Treatment Arm 1

 If medication, add a column for dosage

– Treatment Arm x

 If medication, add a column for dosage

– Sample size, Arm 1

– Sample size, Arm x

– Important Demographic characteristics (% female, mean age, mean BMI, etc) 

– Follow-up time (3 months, 12 months, etc) 

– Measurement of outcome (OR, RR, probability, means, median, etc) 

– Measurement of variation (SD, SE, variance, IQR, range, etc) 

– ITT, Per Protocol results, or both

– Value of outcome, Treatment Arm 1 

– Value of outcome, Treatment Arm x

– Value of variation, Treatment Arm 1

– Value of variation, Treatment Arm x
Good research practices, Data 

Extraction

 All categorical variables should be recorded in the same way

– RCT ≠ Randomized Controlled Trial

 Test your template with a small number of studies, revise the 

template as needed. 

 Data extraction can be tricky – rushing will cause many 

headaches down the road
4. Separating out OS and RCTs
Questions 

 Why separate out RCTs and Observational 

Studies? 

 Why conduct a meta-analysis on an 

Observational Study?

22 
4. Separating out OS and RCTs

 Observational Studies have systematic 

differences between groups, RCTs do not

– Relative effect is extracted from each study

 RCTs: may not be generalizable to the 

population that is in your cost-effectiveness 

analysis
5. Converting outcomes to the same scale

 All outcomes should be in the same scale (binary for 

a decision model) 

– May require the involvement of a PhD statistician – point estimate and 

variation

 OR and RR 

– work in the log scale

 Continuous data 

– work in standardized means if data are not all reported on the same 

scale

 Risk Difference 

– work in absolute scale

Borenstein M, Hedges LV. Converting Among Effect Sizes. In: Introduction to Meta-Analysis. West Sussex, United Kingdom: John 

Wiley & Sons, Ltd; 2009: 45-49.
6. Evaluate Heterogeneity of 

Selected Studies

 This step is critical! If data are too sparse, of low quality, or 

studies are too heterogeneous – you cannot continue to a metaanalysis and must end at a systematic literature review! 

 Informal 

– Review completed data extraction template

 Formal

– Statistical tests

– Graphical assessments
Informal Assessment 

of Heterogeneity

 Evaluate: 

– Differences in study population

– Differences in length of follow-up

– Differences in way outcomes are measured

– Differences in intervention
Formal Assessment of 

Heterogeneity 

 There will almost always be some difference in the 

effect sizes from different studies

 Homogeneity: Difference in effect size due to random 

variation (sampling error)

 Heterogeneity: Difference in effect sizes exceeds that 

which can be expected from sampling error alone – Can exist when effect sizes are in different directions, or when magnitude of effect sizes 

differs 
Formal Assessment of 

Heterogeneity: Statistical Tests: 

 Cochrane’s Q: tests null hypothesis that true treatment effects 

are the same in all the studies 

– H1: at least one effect differs from the rest

 Problem: power to detect heterogeneity is low when you have 

≤ 10 studies) 

– You can have heterogeneity but fail to reject null hypothesis

 Recommend using p < 0.10 as significance level 

– Conversely, if you have studies with large sample sizes, you can reject 

the null hypothesis even when effect sizes do not differ much 

 So, don’t put a lot of stock in the Q statistic
Formal Assessment of 

Heterogeneity: Statistical Tests 

 I-squared: 

– Tells you percentage of total variation across studies that is due to 

heterogeneity (rather than chance)

– Reflects the extent of overlap in CIs

 Uses the Q statistic

 Rough guide to interpreting the I2 statistic

– 0-25%: low heterogeneity

– 25-50%: moderate heterogeneity 

– 50-75%: high heterogeneity

 Also look at the confidence intervals around the I2 statistic
Formal Assessment of 

Heterogeneity: Forest Plots

30 Collins R, Yusuf S, Peto R. Overview of randomised trials of diuretics in pregnancy. Br Med J (Clin Res Ed). 1985 Jan 

5;290(6461):17-23.
Formal Assessment of 

Heterogeneity: Forest Plots

 Consistent effect sizes 

– focus on pooled estimate 

 Variations in effect sizes 

– can report pooled estimate, but note the true effect could be 

higher or lower

 Substantial variations in effect sizes 

– focus on variation rather than pooled effect

31 
Summary: heterogeneity

 Do an informal assessment: examine your 

data extraction table

 Formal assessment: forest plots, I2 

32 
If you have heterogeneity

 Excluding studies is frowned upon! 

– You have to have an excellent reason to do so 

– Test excluding these studies in sensitivity analyses

– Analyze groups of studies (grouping should be determined 

a priori)

– Using random effects models (more on this later)

– Conduct a meta-regression

 No clear guidelines exist for how much heterogeneity 

“sinks the ship”
Recap 

1. Conducted a systematic literature search

2. Completed title and abstract review

3. Extracted data from selected studies

4. Separated RCTs from OS 

5. Converted all outcomes to the same scale

6. Evaluated heterogeneity of studies

– No heterogeneity, or Heterogeneity will be handled 

(subgroup, random-effects analysis, meta-regression) 

34 
7. Conducting Meta-Analysis 

a. Determine fixed versus random effects

b. Decide how to pool your studies
Fixed vs. Random-Effects

Fixed Effects Random Effects

Assumes Variance among studies 

is due to sampling error

There is some fixed 

underlying true effect.

Variance among studies is due to 

both sampling error and because 

true effect could vary from study to 

study (e.g., because of different 

participants, different ways 

intervention was administered, etc.)

Variance Within-study Within-study and between-study (τ2)

CIs Narrower Wider

Inference The true effect is X The mean of the effects is X

Small Studies Are less precise, given less 

weight

Given more weight than in a FE 

analysis
Random Effects Distribution

 Random effects are often more suitable -- there are 

almost always differences between studies 

 But, random effects are not always more 

conservative!

– If small studies are systematically different than 

large studies then increasing weight of smaller 

studies by doing a RE analysis will overestimate 

treatment effect.
Pooling studies

Greenland S, Salvan A. (1990). Bias in the one-step method for pooling study results. Stat. Med. 9: 

247-52.

Fleiss JL. (1993). The statistical bias of meta-analysis. Stat Methods Med Res 2:121-45.

Fleiss JL. (1981). Statistical Methods for Rates and Proportions. 2nd ed. New York: Wiley.

Cornell JE, Mulrow CD, Localio R, et al. Random-Effects Meta-analysis of Inconsistent Effects: A 

Time for Change. Ann Intern Med. 2014;160:267-270.

Pooling Option Use when you have

Mantel-Haenszel OR no 0 cells, RR, risk difference

Peto method OR, 0 cells

Inverse-variance (FE) Continuous Data, low heterogeneity

DerSimonian and Laird 

(Inverse Variance with RE)

Continuous Data, low heterogeneity, multiple studies 

Knapp-Hartung (RE) Continuous Data, heterogeneity, 6 or more studies

Profile Likelihood (RE) Continuous Data, heterogeneity, suspect asymmetry 

in distribution of tau-squared 

Bayesian approach (RE) Continuous Data, heterogeneity, sparse data and/or 

few studies
Problems with DerSimonian 

and Laird and inverse-variance

 Shuster, Statistics in Medicine 2010

– Raised problems with inverse-variance and DerSimonian and Laird

 The inverse-variance/DerSimonian and Laird approaches assume that 

the point estimate and the variance are INDEPENDENT

 Binomial distribution, variance is not independent of the point estimate 

[variance = (p*q)/n] 

 Cornell et al., Annals of Internal Medicine 2014

 Assumes that we have estimated between-study variance exactly 

narrow CIs, low p-value

 Is the default weighting method in RevMan (used by 

Cochrane Collaboration)

39 
Publication Bias

 Studies with significant results are 

more likely to be published

– Meta-analysis will overestimate effect

 Larger studies more likely to be published

– If results of smaller studies are systematically different 

from larger studies:

 Random effects will be more problematic 

– gives greater weight to smaller studies than fixed 

effects do
Assessing Publication Bias

 Funnel plots 

– Asymmetry is problematic

 Unless quality of studies varies with size

– Publication bias can still exist even if there 

is symmetry

41 
Funnel Plots for Publication Bias

 Symmetric Funnel Plot 

 Asymmetric Funnel Plot

Sterne JA, Sutton AJ, Ioannidis JP, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of 

randomised controlled trials. BMJ. 2011 Jul 22;343:d4002.
Funnel Plot Asymmetry

 Large sample sizes – easier to find significant effects

 Asymmetric funnel plot: heterogeneity, or quality 

varies with size

 Don’t just look at the funnel plot – evaluate it in 

context of the other info you have about the studies, 

such as quality of study or heterogeneity of 

intervention

 Note: For a funnel plot to be useful, have to have 

studies with various sizes.

43 
What do to with Publication Bias

 Cumulative meta-analysis, ordered by precision

 Glesser and Olkin: estimate the number of missing studies

 Weighted distribution theory-based selection methods

 Trim-and-Fill method

 Copas and Li method

Sutton AJ, Abrams KR. Publication Bias. In: Methods for Meta-Analysis in Medical 

Research. West Sussex, England: John Wiley & Sons, Ltd; 2000: 109-132. 
Meta-Analysis and CEA

Point estimate 

– input in CEA

CI for CEA 

sensitivity 

analyses
Software Programs

 STATA

 SAS

 R 

 RevMan (Cochrane) 

 CMA

 OpenBugs/WinBugs 

Be careful with plug-and-chug software!
ADVANCED TOPICS

48 
Advanced Topics

 Meta-Regression 

 Mixed Treatment Comparisons 

 Individual-Patient Data (IPD) MetaAnalysis 

49 
Meta-regression 

 Regression: adjust for differences at a patient-level

 Meta-Regression: adjust for differences at a studylevel

 Not recommended when # of studies is small

– Regression: at least 10 events per covariate

– Meta-regression: no established rule
Mixed Treatment Comparisons

 Statistical method for estimating the relative treatment effect of 

interest using a network of evidence

(ѲAB) = (ѲA_Placebo) – (ѲB_Placebo) 

Var (ѲAB) = Var (ѲA_Placebo)+ Var (ѲB_Placebo) 

51 

ѲB_Placebo

Tx A Tx B 

Placebo ѲA_Placebo

ѲAB
Individual-Patient Data MetaAnalysis 

 “Regular” meta-analysis uses the 

summary statistic from each study 

 - 8 studies = 8 data inputs

 IPD meta-analysis uses the individual 

patient data from each study

 - 8 studies with 50 patients each = 400 data inputs

52 
SUMMARY

53 
Summary 

 Meta-analysis: single pooled estimate + variance 

from (usually) weighting and combining 

individual effects from multiple studies

 Requires a systematic literature review

 Considerations: 

– Assessment of study heterogeneity

– Fixed versus random effects

– How to pool individual studies

– Publication bias

54 
Further Reading

– Borenstein M, Hedges LV. Introduction to Meta-Analysis. 

West Sussex, United Kingdom: John Wiley & Sons Ltd; 

2009. 

– Sutton AJ, Abrams KR. Methods for Meta-Analysis in 

Medical Research. West Sussex, England: John Wiley & 

Sons, Ltd; 2000.

– Higgins JPT, Green S (editors) Cochrane Handbook for 

Systematic Reviews of Interventions Version 5.1.0. The 

Cochrane Collaboration, 2011. Available from 

http://handbook.cochrane.org/ 
REFERENCE EQUATIONS

56 
Inverse Variance (fixed effects)

 Pooled treatment effect: 

TT�. =

∑ wwiiTTii

∑ wwii

 Weight: 

wwii =

1

vvii

 Variance 

vvvvvv TT�. =

1

∑ wwii
Inverse Variance with random effects 

(DerSimonian and Laird, Knapp-Hartung, 

Profile likelihood, Bayesian)

 Pooled treatment effect is calculated in the same way as the fixed effect 

analysis

 However, the weight now includes the within-study variance and betweenstudies variance.

 The four approaches differ in their calculation of tau-squared (the between 

studies variance)

TT�.RRRRRR =

∑ wwii

∗

TTii

∑ wwii

∗ 

 wwii

∗

=

1

1

wwii � + ττ2 

vvvvvv TT�.RRRRRRRR =

1

∑ wwii

∗ 
Mantel-Haenszel (OR, RR, RD 

no 0 cells)

 

TT�

MMMM(OO ) =

∑ iiddii

nnii

∑ bbiiccii

nnii

TT�

MMMM(OO ) =

67×86

309 +⋯+ 117×3188

87×69

309 +⋯+

12×56

188

 = 0.95
Peto Method (OR)

 Also for odds ratios 

 Is a modification of the Mantel-Haenszel method, but can be 

used when you have cells with 0 values 

TT�

PPPPPP (OO ) = ee

∑(OOii− ii)

∑ vvii
Questions? 

61