Meta-Analysis Workshop

David B. Wilson, PhD
May 24–27, 2021
George Mason University
Department of Criminology, Law & Society
dwilsonb@gmu.edu

1

The End-Game

Forest-Plot of Standardized Mean Differences and 95% ConfidenceIntervals for the Effects of Cognitive-Behavioral Programs on Recidivism
Porporino & Robinson, 1995
Johnson & Hunter, 1995
Robinson, D., 1995
Porporino, Fabiano, & Robinson, 1991
Little, Robinson, & Burnette, 1991a
Little & Robinson, 1989
Little, Robinson, & Burnette, 1994
Burnett, 1996
Ross, Fabiano, & Ewles, 1988
Random Effects Mean
−.4
0
.4
.8
1.2
Standardized Mean Difference Effect Size

1.6

2

2

Overview• Historical background• Logic of Meta-analysis• Effect sizes• Common types• Computing standardized mean difference effect sizes• Computing odds ratio effect sizes• Basic meta-analysis methods• Random-effects versus fixed-effect model• Moderator analysis• Analog to the ANOVA• Meta-analytic regression• Forest plots• Publication bias
• Cutting edge methods• Network meta-analysis• Robust standard errors for statistically dependent effect sizes• Regression coefficient and fully multivariate models
3Historical BackgroundA Great Debate• Eysenck 1952: Psychotherapy doesn’t work
• Dizzying array of mixed results followed• Glass (with Smith) averaged results from 375 studies• Glass coined the term meta-analysis

4Deep Roots• Pearson (1904): averaged correlations between inoculation for
typhoid fever and mortality• Fisher (1944): independent studies individually may not besignificant, yet the aggregate may be improbable• W. G. Cochran (1953): developed methods of averaging means
across studies (basis of modern methods)• A. Wicker (1967) averaged correlations between attitudes andbehavior• Concurrent with Smith and Glass (1977) were
• Hunter and Schmidt (1977) Validity generalization• Rosenthal and Rubin (1978) Interpersonal expectancy effects

5

Logic of Meta-analysis

Logic of Meta-analysis• Narrative review methods:• Focuses on statistical significance• Lacks transparency and replicability• Weakness of focusing on statistical significance:• A significant effect is a strong conclusion• A non-significant effect is a weak conclusion• How do you balance a collection of significant andnon-significant effects?

6Logic of Meta-analysis• Meta-analysis:• Focuses on direction and magnitude of effect• Approaches task as a research endeavor• Examines pattern of evidence across studies• average effect• consistency of effects• relationship between study features and effects

7

Research Suitable toMeta-analysis

Forms of Research Findings Suitable to Meta-analysis• Central tendency• Prevalence rates• Pre-post contrasts• Growth rates• Group contrasts (e.g., boys versus girls)• Experimentally created groups (factorial designs, RCTs, etc.)8Forms of Research Findings Suitable to Meta-analysis• Association between variables• Measurement (psychometric) research• Individual differences research• Correlation between personality constructs• Regression models• Fully multivariate (SEM, factor analysis)9Concept of Effect Size andEssential Features

Effect Size: The Key to Meta-analysis• The effect size makes meta-analysis possible• It is the “dependent variable”• It standardizes findings across studies such that they can bedirectly compared10

Effect Size: The Key to Meta-analysis• Any standardized index can be an “effect size” (e.g., standardized
mean difference, correlation coefficient, odds ratio) as long as itmeets the following criteria:• comparable across studies (generally requires standardization)• represents the magnitude and direction of the relationship of interest• independent of sample size

11

Meta-analysis as a Systematic
Review

Elements of a Systematic Review• Explicit inclusion/exclusion criteria• Exhaustive search for all eligible studies• Coding and data extraction• Assessing study quality• Summarizing the evidence (ideally using meta-analysis)

12

The Replication Continuum

You must be able to argue that the collection of studies you aremeta-analyzing examine the same relationship. This may be at a broad
level of abstraction, such as the relationship between criminal justiceinterventions and recidivism or between school-based prevention
programs and problem behavior. Alternatively it may be at a narrow levelof abstraction and represent pure replications.
The closer to pure replications your collection of studies, the easier it is toargue comparability.

13Effect Sizes

The Effect Size: The Heart and Soul of Meta-analysis• Meta-analysis shifts the focus from statistical significance to thedirection and magnitude of the effect• Key to this is the effect size
• It is the dependent variable of meta-analysis• Encodes research findings on a numerical scale• Different types of effect sizes for different research situations• Each type may have multiple methods of computation

14

Overview• Main types of effect sizes• Logic of the standardized mean difference• Methods of computing the standardized mean difference• Logic of the odds ratio and risk ratio• Methods of computing the odds ratio• Adjustments, such as for baseline differences• Issues related to the variance estimate

15

Most Common Effect Size Indexes• Standardized mean difference (d or g)• Group contrast (e.g., treatment versus control)• Inherently continuous outcome construct• Odds ratio and Risk ratio (OR and RR)• Group contrast (e.g., treatment versus control)• Inherently dichotomous (binary) outcome construct• Correlation coefficient (r )• Two inherently continuous constructs

16Less Common Effect Size Indexes• Raw (unstandardized) mean difference• Proportion or prevalence (usually converted to a logit)• Standardized gain score• Standardized regression coefficient17

Necessary Characteristics of Effect Sizes• Numerical values produced must be comparable across studies• Must be able to compute its standard error• Must not be a direct function of sample size

18

The Standardized Mean
Difference Effect Size

The Standardized Mean Difference• Compares the means of two groups• Standardizes this differenced=sspooled =X1 − X2
spooled(n1 − 1) s12 + (n2 − 1) s22
n1 + n2 − 2

19

.5

Visual Example of d-type Effect Size

Control
Treatment0.1Frequency
.2
.3

.4

d = .50−3 −2.5 −2 −1.5 −1

−.5

0

.5
1
z−score

1.5

2

2.5

3

3.5

20

Small Sample Size Bias Adjustment

d is slightly upwardly bias when based on small samples. This can be
fixed with Hedges’ small sample size bias adjustment:
g = 1−

3
4N − 9
d = 1−

3
4(n1 + n2 − 2) − 1)dThe result is Hedges’ g. The difference between Cohen’s d and Hedges’
g is this adjustment.
It is common practice to always use this adjustment for standardized
mean differences.

21

Standard Error and Variance of g

The standard error (and variance) of g is mostly a function of sample size:

s
seg =

g2
n1 + n2+
n1 n2
2 (n1 + n2 )vg = seg2

22

Methods of Computing d• Lots of methods of computing d• Goal is to reproduce what you would get with the means, standard
deviations, and sample sizes• Some methods are straightforward, others not

23Computing d from a t-test

Formula for d:d= q

X1 − X2(n1 −1)s12 +(n2 −1)s22
n1 +n2 −2

Formula for t:t= q

X1 − X2(n1 −1)s12 +(n2 −1)s22
n1 +n2 −2Therefore:

r
d =t

q

n1 +n2n1 n2n1 + n2n1 n224Computing d from a p-value from a t-test• An exact p-value from a t-test can be convert into a t-value,
assuming you have the degrees-of-freedom• Then proceed as with prior slide25Computing d from dichotomous outcomes• Some studies may report a dichotomous outcome• d (and g) can be estimated from these data
• Several methods
1. Logit
2. Cox Logit
3. Probit
4. Arcsine (no longer recommended)• 1–3 have a similar logic and produce similar results unless theoutcome has a low or high base rate (i.e., near 0 or 1).

26

Logit Method of Computing d• Logistic distribution similar to the normal distribution• Logged odds ratios conform to the logistic distribution• Standard deviation of the logistic distribution isπslogistic = √ = 1.814
3• Thus, we can rescale a logged OR into dd=ln(OR )
1.814

• Monte Carlo simulations suggest that this performs fairly well
• However, there are some advantages to the Cox logit method (divide
by 1.65)ln(OR )
d=
1.65
27

Probit Method• The probit method uses the standard normal distribution rather thanthe logistic distribution• Converts each proportion (i.e., successes in each group) into thecorresponding z-score with p area under the left portion of the curve.
• Effect size is the difference between these to probits (z-values)
d = probit (p1 ) − probit (p2 )d = z1 − z228More Complicated Estimates of d

• Can think of d as having two parts• Numerator: mean difference• Denominator: pooled standard deviation• Trick is to get reasonable estimates of each
• You often have one but not the other29

Estimates of numerator of d

• Covariate adjusted means• Difference in gain score means• Unstandardized regression coefficient for treatment dummy• Standardized regression coefficient for treatment dummy converted
to an unstandardized estimate

30

Estimates of denominator of d
• Gain score standard deviationsgain
spooled = p2( 1 − r )



r =



s12 t 2 + s22 t 2 − X 1 − X 2 n
2s1 s2 t 2(1)• Overall standard deviation for the outcome• Standard error√s = se n − 1• One-way ANOVA

p
MSw
r
MSB
spooled =Fspooled =31

See Online Effect Size Calculator

https://campbellcollaboration.org/escalc/html/
EffectSizeCalculator-Home.php

32

Odds Ratio and Risk Ratio
Effect Sizes

Logic of the Odds Ratio and Risk Ratio• Odds ratio is the ratio of odds
• An odds is the probability of failure over the probability of success• Risk ratio is the ratio of risks (probabilities)• A risk is simply the probability of failure• An odds ratio or risk ratio of 1 indicates no difference between thegroups• Odds ratios are more difficult to interpret than risk ratios• Risk ratios are sensitive to base event rate; odds ratios are not• Odds ratios are symmetrical with respect to the outcome (this is why
they can be used in case-control studies)

33

Logic of the Odds Ratio and Risk Ratio

Odds ratios and risk ratios contrast two groups on a dichotomous
outcomeTreatment
Control

OutcomeSuccess Failureabcdwhere a, b, c, and d, are cell frequencies

34

Computing the Odds Ratio and Risk Ratio

Odds ratio is computed as:
OR =ad
p1 / (1 − p1 )=bc
p2 / (1 − p2 )Risk ratio is computed as

RR =a/(a + b)
p1=
c /(c + d )p235Computing the Standard Error and Variance of the Odds ratio
and Risk Ratio

Standard error of the logged odds ratio:rseln(OR ) =

1
1
1
1
+ + +
a
b
c
d

Standard error of the logged risk ratio:sseln(RR ) =1 − p1
1 − p2
+
n1 p1
n2 p2

Variance:v = se236Computing the Odds Ratio from Other Data

• Only so many was you can report binary data
• Makes computing odds ratios and risk ratios relatively easy
• Generally have
• Full two-by-two table• Percent of events (successes/failures) in each group• Proportion of events (successes/failures) in each group• Actual odds ratio or risk ratio• Convert d-type effect size
• Chi-square37Computing the Odds ratio from a χ2

Odds ratio can be computed from a χ2 if you have the overall event rate
for the sample (total number of successes or failures) and the sample
size in each group
The frequency for cell a of the two by two can be computed as follows"
a = N px pyr

χ2 px py (1 − px )(1 − py )N#where px is the proportion of the sample in one of the two conditions, py
is the proportion of the overall sample with a positive (or negative)
outcome, and N is the total sample size.

38

Prevalence Rate as Effect Sizes

Prevalance Rate as Effect Size
• Prevalence can be expressed as a proportion• Logit of a proportion makes for a good effect sizelogit = ln

p



1−p

• The standard error of the logit issselogit =11+np
n (1 − p )• Final results can be back-converted into a proportionp=elogitelogit + 1
39Issues Related to Variance
Estimation

Issues Related to Variance Estimation• Method of computing effect size may affect variance estimate• For example• d computed from dichotomous data
• numerator of d is adjusted for baseline or other covariates• Study data may involve clusters (there are corrections formulas forthis situation)• Possible solution: rescale standard error from correct model
provided in study (if available)
sed =

d
z

• where z is either a z or t test associate with the treatment effectfrom a complex statistical model.

40

Coding Protocol

Development of the Coding Protocol• Like a survey but for studies• A good coding protocol has transparency and replicability• Goals:• Descriptive: characteristics of the studies, particularly ways in which
the studies differ from each other• Extraction of findings: computation of effect sizes and related
information

41

Data Structure• Meta-analytic data is almost always hierarchical• Effect sizes nested within• Measures nested within• Samples or subsamples nested within• Studies• Create separate coding forms and data tables for each level• Idea is that of a relational database• Must have identifying “keys” to connect the rows across tables• Saves coding time, data cleanup time, and data manipulation time42

Where to Code• Paper forms• Excel (please don’t)• Computer database (great choice!)• Specialized software (e.g., Comprehensive Meta-analysis, RevMan)

43

Double-Coding• Reliability of coding is essential• Best practice is to double-code everything and resolve
any differences

44Basic Meta-Analysis

Overview of Method• Goal:• Describe the distribution, including its mean
• Establish a confidence interval around the mean
• Test that the mean differs from zero
• Test whether studies tell a consistent story (are homogeneous)• Explore the relationship between study features and effect size

45

Determining the Mean Effect Size

• Problem: some effect sizes are more accurate than others• What we need is an index of precision• Standard error is a direct measure of precision• Hedges and Olkin’s solution:• Weight by the inverse variance• Provides a statistical basis for:• Standard error of the mean effect size
• Confidence intervals• Homogeneity testing46

Some Preliminary Transformations• Small sample size bias correction for the standardized meandifferences:
ES g = 1 − 4N3−9 ESd

• Fisher’s Zr transform of correlations (ES r ):ES Zr = 21 log

1+r
1−r



• Log transform of odds ratios ES OR :ES ln(OR ) = log (ES OR )• Log transform of risk ratios ES RR :ES ln(RR ) = log (ES RR )47

Standard Errors• Standardized mean difference g:
seg =

q

ES 2
n1 +n2+ 2(n1 +g n2 )n1 n2• Correlation Zr :seZr = √n1−3• Logged odds ratio ln(OR ):seln(OR ) =q1+ b1 + 1c + d1a

• Logged risk ratio ln(RR ):seln(RR ) =q1−p1+ 1n−2 pp22
n1 p1

48

Inverse Variance Weights• Inverse variance weight w:
w = se12 = v149

Almost ready• At this point, we have for each study:• An effect size
• An inverse variance weight• Problem: statistical models assume independence• Only include one effect size per study (or independent sample)• Multiple analyses for different subsets of independent effects• Different outcome constructs• Different time periods

50Inverse Variance Weighted Mean

Meta-analytic mean effect size is:ES =P
wi ESiP
wi

where ESi is the effect size for each study (i) and wi is the inverse
variance weightStandard error of the mean effect size is:seES =

q

P1

wi

51

Some Basic Inferential Statistics

Confidence intervals can be constructed in the usual manner:
ES lower = ES − seES 1.96
ES upper = ES + seES 1.96

And a z-test can be performed as:z = seESES

52

Example: Cognitive-Behavioral Programs for Adult Offenders
AuthorBurnett, 1996
Johnson & Hunter, 1995
Little & Robinson, 1989
Little et al 1991
Little et al 1994
Porporino & Robinson, 1995
Porporino et al 1991
Robinson, D., 1995
Ross et al 1988

Sample Sizeg Effect Size

w

60
98
180
152
1,381
757
63
2,125
45

0.45
0.11
0.23
0.22
0.34
0.04
0.16
0.11
1.28

5.684
15.93425.04927.852150.485
65.52911.953187.177
6.441

Note: These studies are a subset of studies included in Wilson et al.
(2005) and represent two specific treatment programs (Moral Reconation and Reasoning and Rehabilitation) and studies that were randomized or used high quality quasi-experimental designs.
53

Computations by Hand

Sum

g

w

g×w

0.45
0.11
0.23
0.22
0.34
0.04
0.16
0.11
1.28

5.684
15.93425.04927.852150.485
65.52911.953187.177
6.441

2.55780
1.75274
5.76127
6.1274451.16490
2.62116
1.91248
20.58947
8.244482.94

496.104

100.7317P

gw
100.7317=
g= P= 0.203w
496.104

s
se =

1
P =
w

r

1
= 0.045496.104g
0.203== 4.511se
0.045
g lower 95 = 0.203 − 1.96(0.045) = 0.115
z=

g upper 95 = 0.203 + 1.96(0.045) = 0.291

54

Example: Cognitive-Behavioral Programs for Adult Offenders
Stata output from “masum.ado”. masum g, w(invw) model(FE)No. of obs =9
Homogeneity Analysis
Minimum obs =
0.040
Q =
14.1885
Maximum obs =
1.280
df =
8
Minimum weight =5.6840p =0.07698
Maximum weight =187.1770
I^2 =43.62%-------------------------------------------------------------------------|
Mean-95%CI+95%CIsezp----------------+--------------------------------------------------------Model FE| 0.2030
0.11510.29100.04494.5225 0.0000-------------------------------------------------------------------------Fixed (Common) Effect model
Version 2021.04.18 of masum written by David B. Wilson

55

Homogeneity Testing• Homogeneity analysis tests whether the assumption that all of theeffect sizes are estimating the same population mean is areasonable assumption.
• If homogeneity is rejected, the distribution of effect sizes is assumed
to be heterogeneous.
• Single mean ES not a good descriptor of the distribution• There are real between study differences, that is, studies estimate
different population mean effect sizes.
• Three options:• model between study differences• fit a random effects model• do both56

Homogeneity Q Statistic• Q is simply a weighted sums-of-squares:Q=P

wi (ES i − ES )2• There are easier computational formulas:Q=P

wi ES i 2 −(P
2wi ES i )P
wi

• It is distributed as a chi-square with k − 1 degrees of freedom,
where k is the number of effect sizes

57

Higgins’ I 2 Statistic• Q is statistically under powered when the number of studies is small,unless the sample sizes of those studies is very large• Jonathan Higgins proposed an alternative• It is based on Q but doesn’t focus on statistical significance
2

I =



Q − dfQ
× 100• This reflects the percentage of the total variance in effects due toheterogeneity rather than sampling error

58

Higgins’ I 2 Statistic

Rough guide to interpreting I 2 :I2Excess variability

0% to 40%
30% to 60%
50% to 90%
75% to 100%might not be important
may represent moderate heterogeneity
may represent substantial heterogeneity
may represent considerable heterogeneity59Computations by Hand

Q=
g

P

w

g×w

g2 × w

0.45
0.11
0.23
0.22
0.34
0.04
0.16
0.11
1.28

5.684
15.93425.04927.852150.485
65.52911.953187.177
6.441

2.55780
1.75274
5.76127
6.1274451.16490
2.62116
1.91248
20.58947
8.244481.15100.19281.3251
1.348017.3960
0.10480.30602.264810.55292.94

496.104

100.7317

34.6416

X

wi ES i 2 −(P
w ES )2
Pi i
wi(100.7317)2
496.104
Q = 34.6416 − 20.4531

Q = 34.6416 − −Q = 14.19
df = 9 − 1 = 8,

p < .05

Q − dfQ
14.19 − 82
I = 100 ×
14.19
I 2 = 100 ×

I 2 = 43.6%

60

Example: Cognitive-Behavioral Programs for Adult Offenders
Stata output from “masum.ado”. masum g, w(invw) model(FE)No. of obs =9
Homogeneity Analysis
Minimum obs =
0.040
Q =
14.1885
Maximum obs =
1.280
df =
8
Minimum weight =5.6840p =0.07698
Maximum weight =187.1770
I^2 =43.62%-------------------------------------------------------------------------|
Mean-95%CI+95%CIsezp----------------+--------------------------------------------------------Model FE| 0.2030
0.11510.29100.04494.5225 0.0000-------------------------------------------------------------------------Fixed (Common) Effect model
Version 2021.04.18 of masum written by David B. Wilson

61

Random Effects versus Fixed
Effect Models

Random versus Fixed Effect Models• Fixed effect model assume:• there is one true population effect that all studies are estimating• all of the variability between effect sizes is due to sampling error• Random effects model assume:• there are multiple (i.e., a distribution) of population effects that thestudies are estimating• variability between effect sizes is due to sampling error + variability inthe population of effects

62

Fixed versus Random: Which to Use?

• A random-effects model becomes a fixed-effect model when
distributions is homogeneous• Assumptions of fixed effects model rarely plausible• Consequence: standard error that is too small; confidence intervals
that are too narrow• Bottom-line: Use the random-effects model unless you can make astrong a priori case for a fixed effect model

63

Computing a Random Effects Model• Fixed effects model: weights are a function of sampling error• Random effects model: weights are a function ofsampling error + study level variability• We call this study level variability τ2• Thus, we need a new set of weightsw=

1
se2 + τ264

Computing a Random Effects Model• First, compute τ2 (random effects variance component):

τ2 = P Q −dfPQwi 2
wi − P w

i• Second, re-compute the inverse variance weights:
wi = sei 21+τ2• Third, re-compute meta-analytic results using new weight

65

Random Effects Computations by Hand

Q − df Q
P 2
τ2 = Pw
wi − P wi i

τ2 =14.19 − 8
.76496.104 − 63848
496.104

τ2 = 0.0168

66

Random Effects Computations by Hand

Step 1.Compute v if you don’t already have such a column.
g

w

v

0.45
0.11
0.23
0.22
0.34
0.04
0.16
0.11
1.28

5.684
15.93425.04927.852150.485
65.52911.953187.177
6.441.175932.062759.039922.035904.006645.015260.083661.005343.155255

67

Random Effects Computations by Hand

Step 2. Add tau 2 to each v .
g

w

v

tau 2

v + tau 2

0.45
0.11
0.23
0.22
0.34
0.04
0.16
0.11
1.28

5.684
15.93425.04927.852150.485
65.52911.953187.177
6.441.175932.062759.039922.035904.006645.015260.083661.005343.155255

.0168
.0168
.0168
.0168
.0168
.0168
.0168
.0168
.0168

.192732.079559.056722.052704.023445.032060.100461.022143.172055

68

Random Effects Computations by Hand

Step 3. Compute new weight (1/(v + tau 2 ). In ths example, these are
DerSiminian and Laird weights (hence the “DL”)
g

w

v

tau 2

v + tau 2

wDL

0.45
0.11
0.23
0.22
0.34
0.04
0.16
0.11
1.28

5.684
15.93425.04927.852150.485
65.52911.953187.177
6.441.175932.062759.039922.035904.006645.015260.083661.005343.155255

.0168
.0168
.0168
.0168
.0168
.0168
.0168
.0168
.0168

.192732.079559.056722.052704.023445.032060.100461.022143.172055

5.18854
12.56931
17.62992
18.97387
42.65269
31.19111
9.9541145.16195
5.81208

69

Random Effects Computations by Hand

Step 4 and 5. Multiply the new weights by the effect size and sum
columns.

Sum

g

w

v

tau 2

v + tau 2

wDL

g × wDL

0.45
0.11
0.23
0.22
0.34
0.04
0.16
0.11
1.28
2.94

5.684
15.93425.04927.852150.485
65.52911.953187.177
6.441.175932.062759.039922.035904.006645.015260.083661.005343.155255

.0168
.0168
.0168
.0168
.0168
.0168
.0168
.0168
.0168

.192732.079559.056722.052704.023445.032060.100461.022143.172055

5.18854
12.56931
17.62992
18.97387
42.65269
31.19111
9.9541145.16195
5.81208
189.1336

2.33481.38264.05494.174314.5019
1.24761.59274.96787.439541.696170

Random Effects Computations by Hand

Step 6. Re-compute mean effect size and standard error as before.
Comment:
Because the variance is larger, the weight is smaller. Also, acrossstudies, the weights will be more similar. With a large enough τ2 theweights will become virtually equal.

71

Example: Cognitive-Behavioral Programs for Adult Offenders
Stata output from “masum.ado”. masum g, w(invw) model(DL)No. of obs =Minimum obs =Maximum obs =Minimum weight =Maximum weight =

9
0.040
1.280
5.187445.0727

Homogeneity Analysis
Q =14.1885df =8p =0.07698I^2 =43.62%
tau^2 =
0.01684se (tau^2) =0.02158
tau =
0.12978-------------------------------------------------------------------------|
Mean-95%CI+95%CIsezp----------------+--------------------------------------------------------Model DL| 0.2205
0.07790.36310.07283.0303 0.0024-------------------------------------------------------------------------Random effects: Dersimonian-Laird
Version 2021.04.18 of masum written by David B. Wilson
72

Numerous Estimators for τ2

There are numerous estimators for the random effects variancecomponent.
• Dersimonian-Laird method-of-moments most widely used
• Doesn’t always perform well when the number of studies is smalland heterogeneity is high• Restricted maximum likelihood (REML)• Widely recommended alternative• Requires iteration; not suitable for hand calculation• Is now widely available in most software programs• Roughly 7 other estimators with various pros and cons73

Comparison of Random Effects and Fixed Effect Models• Biggest difference you will notice is the significance levels andconfidence intervals• As heterogeneity increases• p-value for the mean effect size will increase• confidence interval will get wider• A mean that was significant under a fixed effect model might not besignificant under random effects model• Random effects model is more conservative• Mean effect size may differ between models if sample size and effectsize are related74Summary of Basic Meta-analysis• Apply any transformations on effect sizes• Small sample size bias correction (d)• Fisher’s Zr transformation (r )• Natural log (OR and RR).

• Compute inverse variance weight• Compute mean, standard error, and confidence intervals• Compute homogeneity statistics (Q and I 2 )• Assume random effects unless you can make a strong a priori casefor a fixed effect model• Next up, moderator analysis

75Moderator Analysis

Moderator Analysis• Modeling between study variability• Categorical models (analogous to a one-way ANOVA)• Regression models• Fixed and random effects versions of each (latter often called“mixed” models)• Number of studies needed76Analog to the ANOVAAnalog to the ANOVA• Useful for a single categorical independent variable• Produce a separate mean effect size for each category• Recall that Q is a sum-of-squares• The total sum-of-squares (Q) can be partitioned• Variability between groups (QBetween )• Residual variability within groups (QWithin )77

Analog to the ANOVA• QBetween analogous to an F -test between means• QWithin assesses whether residual distribution homogeneous78Analog to the ANOVA Equations

Recall that the overall (or total) Q is computed as
QTotal =QTotal =XX

wi (ES i − ES )2P
2
w ES )
Pi i
wi ES i −
wi
2

(

79

Analog to the ANOVA Equations

To compute the QWithin , we need to compute the Q within each group andthen sum across groups
QWithin =X

wij (ES ij − ESj )280

Analog to the ANOVA Equations

To compute the QBetween , simply subtract QWithin from QTotal
QBetween = QTotal − QWithin
Recall that Q is distributed as a χ2 . The degrees-of-freedom arecomputed just as they are in a one-way ANOVA:
dfWithin = k − jdfBetween = j − 1
dfTotal = k − 1
where k is the number of effect sizes and j is the number of categories orgroups.

81

Analog to the ANOVA Interpretation• QBetween reflects the differences in the means across categories(analogous to the F from a one-way ANOVA• QWithin indicates whether the distributions within categories arehomogeneous (overall)• For the random effects model, QWithin should be based on a fixed effectmodel or a significance test of τ2• Test is often statistically under-powered (particularly in the random
effects case)• The random effects version estimates τ2 based on the within groups
heterogeneity82Analog to the ANOVA in Stata

Using my Stata “ado” file maanova, you can run an analog to the ANOVA.
For a simple fixed effect model

maanova g iv, w(invw) model(FE)where g is the effect size, iv is the categorical or grouping variable andinvw is the inverse variance weight.

83

Analog to the ANOVA in Stata

For a mixed (random) effects model, add the model option. You canspecify either a method-of-moments estimator of τ2 or the maximum
likelihood estimator as model(mm) and model(ml) repsectively.

maanova g iv, w(invw) model(FE)maanova g iv, w(invw) model(DL)maanova g iv, w(invw) model(REML)
maanova g iv, w(invw) model(REML) print(EXP)
If you are analyzing logged odds ratios or risk ratios, add theprint(exp) option to get the mean effects reported as odds ratios orrisk ratios, as shown in the last example above.

84

Random Effects Analog to the ANOVA

Experimental versus Quasi-experimental Studies in theDomestic Violence (Page 1 of 2). maanova d random, w(w) model(DL)No. of obs =Minimum effect size =Maximum effect size =Minimum weight =Maximum weight =Common tau^2 =SE tau^2 =Common tau =
I^2 =

14
-0.4671
1.52862.88785.61750.16260.10000.403376.20%----------------+------------------------------------------------------------Cateogry|
Mean-95%CI+95%CIsezpk----------------+------------------------------------------------------------Total|
0.27840.0251
0.5316 0.1292 2.1543 0.0312
14----------------+------------------------------------------------------------0.0000|
0.2912-0.05860.6409 0.1785 1.6315 0.1028
7
1.0000|0.2643-0.10290.6314 0.1873 1.4107 0.1583
7----------------+--------------------------------------------------------------

85

Random Effects Analog to the ANOVA

Experimental versus Quasi-experimental Studies in theDomestic Violence (Page 2 of 2)----------------+------------------------------Cateogry| Q-within p(Q-within)df----------------+------------------------------0.0000| 42.2240
0.00006
1.0000|8.18730.22476----------------+------------------------------Q-within based on fixed effect weights
Analog-to-the-ANOVA Table----------------+-----------------------------------Source
|
Q
p(Q)
df
----------------+-----------------------------------Between (Model) |0.01080.9172
1
Within (Error) |50.41130.000012----------------+-----------------------------------Random effects: Dersimonian-Laird
Q-within based on fixed effect weights

86

Meta-analytic Regression

Meta-analytic Regression• Conceptually identical to multiple regression• Effect size is the dependent variable• Study moderator variables are the independent variables• Can handle multiple variables simultaneously• Don’t use standard OLS regression procedures (even if weighted)• Must use specialized software

87

Meta-analytic Regression in Stata

Using my Stata “ado” file mareg, you can run a meta-analytic multiple
regression model following the examples below
For a simple fixed effect model

mareg g iv1 iv2, w(invw) model(FE)where g is the effect size, iv1 and iv2 are the independent variables,
and invw is the inverse variance weight (you can alternatively specify
var(variance) or se(standard error).

88

Meta-analytic Regression in Stata

For a mixed (random) effects model, add the model option. You canspecify one of 8 estimators for τ2 . The two most common are therestricted maximum likelihood estimator (model(REML)) and theDersimonian and Laird method-of-momenents estimator model(DL),repsectively.

mareg g iv1 iv2, w(invw) model(DL)mareg g iv1 iv2, w(invw) model(REML)89

Meta-analytic Regression. mareg g txvar1 txvar2 random, var(v) model(DL)No. of obs =Minimum effect size =Maximum effect size =Minimum weight =Maximum weight =tau^2 =se (tau^2) =
tau =
I^2 =

38
-0.6400
0.96848.486420.9782
0.04700.01930.216992.51%

Random effects: Dersimonian-Laird----------------+------------------------------------------------------------Variable|bsezP(z) [-----95% CI-----]----------------+------------------------------------------------------------txvar1|
0.37730 0.14363 2.62690 0.00862 0.09579 0.65881
txvar2|
0.19846 0.08198 2.42072 0.01549 0.03777 0.35915
random
|0.07296 0.08327 0.87619 0.38093 -0.09025
0.23617_cons|-0.67358 0.23890 -2.81949 0.00481 -1.14182 -0.20534----------------+------------------------------------------------------------Overall Model Statistics----------------+-----------------------------------Source
|
Q
p(Q)
df
----------------+-----------------------------------Model|
14.7194
0.0021
3
Error| 453.7534
0.000034----------------+-----------------------------------Q-within based on fixed effect weights

90

Publication Selection Bias

Publication Selection Bias
• Statistically significant effects are more likely to be published thannonsignificant effects• Has many forms• outcomes select for inclusion in a report• analyses selected for inclusion in a report• author’s desire to write-up a study• reviewers’ recommendations for publication• editor’s decision to accept for publication• Existence of this bias is well established• Affects all form of reviewing the literature• Essential that you search for and included unpublished studies thatmeet eligibility criteria• Essential that you explore the risk of publication selection bias inyour data
91

Methods of Assessing Publication Selection Bias

• Statistic approaches to assessing publication bias
• Funnel plot: Scatterplot of the effect size against the inverse standard
error of effect size
• Trim-and-fill method (Tweedie and Duvall)• Fail-safe N (Not recommended!)92

Funnel Plot

• As sample size increases, smaller effects become statisticallysignificant• Thus, smaller effects from smaller studies are more likely to be
missing• Statistical significance is directly a function of the standard error• Funnel plot is a scatter plot if the inverse of the standard error (large
values reflect larger samples) and effect size
• Expectation is that the plot should be symmetric around the meaneffect size
• I.e., it should look like an upside-down funnel• Asymmetry is suggestive of publication selection bias

93

Funnel Plot in Stata

There is an installable command called metafunnel that will produce
funnel plots.
If you have not already done so, run the following command:

ssc install metafunnel
Using the sexoffender.dta data file, generate a funnel plot with thefollowing commend:

metafunnel lgores vlgores, var
Does this look asymmetrical?

94

Trim and Fill Method• Method for assessing the impact of publication selection bias on
results• Based on the same idea as the funnel plot
• Nonparametric method for adjusting the mean effect size
• Rectifies the funnel plot asymmetry• Assumes the effects to the left of the mean are more likely to be
missing• Initially trims effect sizes iteratively to make funnel plot symmetric• Replaces trimmed effect sizes with pairs on the left side of the mean

95

Trim and Fill Method

There is an installable command called metatrim that will produce
funnel plots.
If you have not already done so, run the following command:

ssc install metatrim
Using the sexoffender.dta data file, run a trim and fill analysis with thefollowing command:

metatrim lgores vlgores, var
What affect did this procedure have on the estimated overall mean effect
size?

96

Forest PlotsForest Plots• Visual representation of results• Row for each study that shows• study label• sample size; may include other information• effect size (dot, square, diamond)• confidence interval (horizontal line)• Row for the overall mean results• effect size (dot, square, diam-and)• confidence interval (horizontal line)

97

Example Forest-Plot

Forest-Plot of Standardized Mean Differences and 95% ConfidenceIntervals for the Effects of Cognitive-Behavioral Programs on Recidivism
Porporino & Robinson, 1995
Johnson & Hunter, 1995
Robinson, D., 1995
Porporino, Fabiano, & Robinson, 1991
Little, Robinson, & Burnette, 1991a
Little & Robinson, 1989
Little, Robinson, & Burnette, 1994
Burnett, 1996
Ross, Fabiano, & Ewles, 1988
Random Effects Mean
−.4
0
.4
.8
1.2
Standardized Mean Difference Effect Size

1.6

2

98

Creating a Forest-Plot in Stata

The “ado” file maforest can create basic forest plots.
The basic syntax is:maforest effect se
where effect is the effect size, se is the standard error of the effect size√
(se = v = 1/w). If you have not already compute se, you must
compute these before using the maforest command.

99Creating a Forest-Plot in Stata
Syntax for maforest

maforest effect se
Options:xmin(num)xmax(num)
xtics(numlist)
sortby(varlist)
col1(varname)col1lbl("string")col2(varname)col2lbl("string")
xlabel("string")
logxaxis
font(large|medium|small|tiny)mean(fe|mm|ml)

Minimum value for the effect size axis
Minimum value for the effect size axis
List of values for the effect size axis
Variable to sort rows
Variable (such a author name) for column 1Column label for column 1
Variable (such a sample size) for column 2Column label for column 2Label for axis (default is “Effect Size”
Expoential to the axis (effect sizes are logged)
Font size
Add mean effect100Forest Plot Stata Examples

Use the cbt.dta data file.
The variable you will need are:authortotalngse

Label with the author names
Total sample size
Hedges’ g effect sizeStandard error of the effect size

101Forest Plot Stata Examples

maforest se
maforest g se, col1(author) col2(totaln)
maforest g se, col1(author) col2(totaln)
xtics(-3,-2,-1,0,1,2,3)
col1lbl(Author) col2lbl(Sample Size)
See exercise handout for more.

102

Features of a Quality
Meta-analysis

What to look for in a quality meta-analysis• Explicit inclusion and exclusion criteria• Inclusion and exclusion criteria well justified• Not restricted to published studies• Search strategy well explicated• Search includes multiple sources (e.g., databases, hand-searches,
contact with authors, etc.)• Used a detailed coding protocol• Assessed coder reliability (e.g., double-coding)103What to look for in a quality meta-analysis• What to look for in a quality meta-analysis• Maintained statistical independent in the analysis of effect sizes• Use proper meta-analytic methods (e.g., inverse variance weighting)• Tested for heterogeneity in effect sizes• Used a random effects model or was clearly justified in using a fixed
effect model• Used proper methods of testing moderator effects (e.g.,analog-to-the-ANOVA, meta-regression)• Assessed for publication bias
• If included methodologically “flawed” studies, performed sensitivity
analysis on results

104

Cutting Edge Methods

Cutting Edge Methods• Network Meta-analysis• Meta-analysis of dependent effect sizes• Regression coefficients and fully multivariate meta-analysis

105

Network Meta-analysis

Network Meta-analysis• Extension of regular meta-analysis to two or more treatment
conditions• Allows to the mixing of treatment versus treatment with treatment
versus control studies• Makes use of all available comparisons in the network to estimate
the relative effectiveness of each treatment• Well suited to comparing the effectiveness of multiple drugs fortreating a common condition• There are Stata commands for network meta-analysis

106

Network Meta-analysis

Example of a simple network of treatments

107

Network Meta-analysis
Example of direct and indirect effects

108

Network Meta-analysis

Example showing how indirect effects are estimated (conceptually)109

Dependent Effect Sizes

Dependent Effect Sizes• Hedges, Tipton, and Johnson (2010) developed robust varianceestimation method• Allows for dependent effect sizes• That is, multiple effect sizes per study or sample!

• Does not require knowledge of the covariance structure• Has been implemented in Stata and R• ssc install robumeta

110Dependent Effect Sizes• Method includes meta-regression models• Works similar to mareg but requires a study ID• Can test for moderator effects both between and within studies• Difference in effect sizes between boys and girls

111

Regression Coefficients andMultivariate Models

Regression Coefficients• Numerous examples in the literature• Debate among meta-analysis scholars about best methods• Challenges• What effect size?
• How comparable must the models be?
• How should you compute the effect size variance?

• Possible effect sizes• Standardized regression coefficient (generally preferred)• Semi-partial correlation coefficient (similar to above)• Partial correlation coefficient (models must include the same variables)• Modified Cohen’s d (assumes independent variable is dichotomous)• For logistic regression models, unstandardized regression coefficient

112

Multivariate Models

If you are interested in fully multivariate models and not just a bivariate
relationship from a multivariate model, you can meta-analyze theunderlying correlation matrix of interest.
Then use the matrix for your model (regression, factor analysis, structural
equation modeling, etc.).
Complication: not all studies will have all correlations of interest.

113

Questions and Open
Discussion