Formulas Used by the“Practical Meta-Analysis
Effect Size Calculator”
David B. Wilson
George Mason University
April 27, 2017

1

Introduction

This document provides the equations used by the on-line Practical MetaAnalysis Effect Size Calculator, available at:
http://www.campbellcollaboration.org/resources/effect_size_input.php
and
http://cebcp.org/practical-meta-analysis-effect-size-calculator
The calculator is a companion to the book I co-authored with Mark Lipsey, titled “Practical Meta-Analysis,” published by Sage in 2001 (ISBN-10: 9780761921684).
This calculator computes four effect size types:
1. Standardized mean difference (d)2. Correlation coefficient (r)3. Odds ratio (OR)
4. Risk ratio (RR)
In addition to computing effect sizes, the calculator computes the associatedvariance and 95% confidence interval. The variance can be used to generate theweight for inverse-variance weighted meta-analysis (w = 1/v) or the standard√
error (se = v).

2

Notation

Notation commonly used through this document is presented below. Other notation will be defined as needed.

1Formulas for “Practical Meta–Analysis Online Calculator”

d
n1
n2
N
s1
s2
s
se1
se2
sgain
X1
X2
Xij
∆1
∆2
vd
sed
r
Zr
t
F
fij
χ2
b
β
df
SS

3

Standardized mean difference effect size
Sample size for group 1
Sample size for group 2
Total sample sizeStandard deviation for group 1Standard deviation for group 2
Full sample standard deviation
Standard error for the mean of group 1Standard error for the mean of group 2Gain score standard deviation
Mean for group 1
Mean for group 2
Mean for subgroup i of group j
Mean gain score for group 1
Mean gain score for group 2
Variance of dStandard error of dCorrelation coefficient
Fisher’s Zr transformation of r
t-value from a Student’s t-test
F-value from an ANOVA
Frequency for the ith row and jth column of a contingency table
Chi-squared value
Unstandardized regression coefficientStandardized regression coefficient
Degrees-of-freedom
Sum-of-squares

The Standardized Mean Difference Effect Size

The standardized mean difference effect-size (Cohen’s d or Hedges’ g)1 is widelyused in meta-analysis and more generally as a descriptive statistic in primarystudies. The fundamental relationship represented by this effect-size is a dichotomous independent variable and a continuous (scaled) dependent variable.
For example, d would be an appropriate effect size index for a study of the effectiveness of a cognitive-behavioral program with two experimental conditions(treatment versus control) and a scaled measure of depression. Alternatively,
1

Cohen’s d and Hedges’ g are conceptually the same but Hedges’ g is more precise
for effect sizes based on small sample sizes. Cohen’s d is upwardly biased in absolutevalue when based on a small sample size. Hedges’ g corrects for this with the equation
presented in 3.1.3. The term Cohen’s d however is often applied to both versions of thiseffect size. This online calculator does not apply Hedges’ sample size size bias correction.
Future version may build in this correction.

David B. Wilson2Formulas for “Practical Meta–Analysis Online Calculator”

it would be appropriate for representing the difference between naturally occurring groups, such as boys and girls, on a scaled dependent variable such as
reading comprehension. The standardization allows for the comparison of effect sizes across studies with different operationalizations of the same construct.
In these two examples, the measures of depression and reading comprehension.
Note that the term “effect” does not necessarily imply causation. Effect sizes are
merely an index of the empirical relationship of interest, causal or otherwise.
Although d is defined by 4, there are numerous ways to compute d depending on the statistical information available in a given manuscript. Some of theseestimation methods for d are algebraically equivalent to 4 whereas others represent an approximation.

3.1

Some Preliminary Equations

Below are the equations for computing the variance of d, the confidence intervals
around d, and for the small sample size bias correction.

3.1.1

Variance of d

The variance of the standardized mean difference effect size (d) is:vd =n1 + n2
g2
+
.
n1 n2
2 (n1 + n2 )(1)This estimate of vd is used for all methods of computing d unless otherwise
noted. In general, it is used for all computations of d based on means or on
statistics derived from means, such as a t-test. In cases where d is based on abinary (dichotomous) dependent variable, an alternative estimate of vd is usedthat is specific to that method of approximating d.

3.1.2

95% Confidence Interval of d

The 95% confidence interval of d is computed in the standard fashion using thestandarized normal distribution. The lower-bound of the interval is computedas√dlower95 = d − 1.959964 vd ,and the upper-bound is computed as√
dupper95 = d + 1.959964 vd .

3.1.3

Small Sample Size Bias Correction

The standardized mean difference effect size d is slighly upwardly biased in absolute value when based in small sample sizes. This bias is effectively removed

David B. Wilson3Formulas for “Practical Meta–Analysis Online Calculator”

by multiplying d by the correction factor J.
J=1−

3
4N − 9(2)

Thus, Hedges’ g is computed asg = d × J.

(3)

The use of this adjustment is recommended if you are using these effect sizes
for the purpose of meta-analysis. Future versions of the calculator will help automate this process. However, it is fairly trivial to apply this correction to alleffect sizes that are part of a data file through a transformation statement (suchas compute in SPSS or generate in Stata).

3.2

Means and Standard Deviations

The definitional equation for the standardized mean difference (d) effect size isbased on the means, standard deviations, and sample sizes for the two groups
being contrasted. The equation is:d=

X1 − X2,
spooled(4)

where spooled issspooled =s21 (n1 − 1) + s22 (n2 − 1).
n1 + n2 − 2(5)All other equations for d are either algebraic alternatives or approximations of
this quantity.

3.3

Student’s t-test (Two Independent Samples)

The standard formula for an independent t-test is equation 4 with an additionalterm in the denominator based on group sample sizes. Thus, d can easily be
computed from t. For unequal sample sizes, the equation is:r
n1 + n2
d=t
.
(6)
n1 n2
For equal sample sizes, the equation is:
2t
d= √ .
N(7)

Note that this equation cannot be used for a dependent or paired t-test. The twomeans being differenced must be from two independent samles. Also note that
this equation cannot be used for t-values from other statistical procedures, suchas the t associated with a regression coefficient from a multivariate regression
model.
David B. Wilson4Formulas for “Practical Meta–Analysis Online Calculator”

3.4

Significance level (p-value) from a Student’s t-test

In cases where the t-value for an independent t-test is not reported the exact
p-value is reported, the t-value associated with that p-value and sample size
can be determined. This is done by using an asymptotic approximation to thequantile function for the inverse of the two-tailed Student’s t distribution. This is
implemented using Algorithm 396 (see Hill, 1970). This method is accurate to six
decimal places so long as the t has 5 or more degrees-of-freedom. The algorithm
is as follows with k representing the degrees of freedom and p representing thep-value:1k − .5
48
b= 2
a

a=

a
− 98)a − 16)a + 96.36brπd = ((94.5/(b + c) − 3)/b + 1) (a )k2
x = dpc = ((20700

2

y = xkIf y 6 (a + .5) theny = ((1/(((k + 6)/(ky) − 0.089d − 0.822)(k + 2)3)+
0.5/(k + 4))y − 1)(k + 1)/(k + 2) + 1/ypt = kyElse if y > (a + .5) thenp
x=zStandard normal deviate of p divided by 22
y = x2c = (((.05dx − 5)x − 7)x − 2)x + b + c
y = (((((0.4y + 6.3)y + 36)y + 94.5)/c − y − 3)/b + 1)xy = ay2
If y > .002 in previous line, thent=p(expy −1) k

Else
t=

p
(.5y2 ) k

Equation 6 or 7 is then used to compute d.

David B. Wilson5Formulas for “Practical Meta–Analysis Online Calculator”

3.5

One-way ANOVA (F-test) with 2 Independent Groups

A one-way ANOVA contrasting the means of two groups produces an F that isequal to t2 . The d effect size for a one-way ANOVA with a single degree-offreedom in the numerator (i.e., two means) and unequal sample sizes is computed as:sd=F (n1 + n2 ).
n1 n2This formula simplifies if the group sample sizes are equal:
r
F
d=2
.
N
This equation is not appropriate for F-values based on three or more groups
or categories (i.e., three or more means). See section 3.20 for a method of handling one-way ANOVAs with 3 or more groups or categories.

3.6

Means and Standard Errors

The standard error of a mean can be converted into a standard deviation of theraw data as follows:√s = se n − 1 .
Equation 3.2 can then be used to compute d.

3.7

2 by 2 Frequency Table (Contingency Table)

A study with two conditions, such as treatment and control, and a dichtomous
dependent variable often reports the findings in a 2 by 2 frequency table (alsocalled a contingency table). A standardized mean difference effect size can be estimated from such data. Note that if the outcome of interest for a meta-analysis
is typically dichotomous then one should use the odds ratio or risk ratio as theeffect-size of choice. The conversions presented in this section are intended forsituations in which most studies measure the construct of interest on a scale but
a few do not. The methods below allow for comparison of findings across suchstudies and for the synthesis of the scaled and dichotmous findings in a single
meta-analysis. It is important to note, however, that these methods are approximations to what would have been observed hand the dependent variable beenmeasured on a scale.
Three different methods are provided below. All three produce similar results in most situations—differences increase as the event rate approaches 0 or 1.
The first two are based on a conversion of the logged odds ratio. The logic of thisis based on the similarity of the logistic and standardized normal√ distributions.
The former has a mean of zero and a standard deviation of π/ 3, whereas thelatter has a mean of zero and a standard deviation of 1. The shapes are similarDavid B. Wilson6Formulas for “Practical Meta–Analysis Online Calculator”

but not identical, with the logistic distribution being somewhat leptokurtic. The
Hasselblad and Hedges conversion simply rescales the logged odds ratio basedon the difference between the standard deviation of these two distributions. The
Cox method tweaks the conversion to account for the slightly leptokurtic natureof the logistic distribution, improving the performance of the conversion whenthe event rate approaches 0 or 1. The probit method is based entirely on the normal distribution. This approach finds the z associated with the area under theleft portion of the curve that equals the proportion experiencing the event foreach group. For example, a success rate (or failure rate) of .50 would correspondwith a z of 0 (half of the distribution is to the left of 0). Similarly, a success rate of.025 would correspond to a z of -1.96. This value is called the probit of a proportion. The standardized mean difference effect size is computed as the difference
between these two probits. Each of these methods has a specific equation for thevariance of the effect size, provided below.
The Hasselblad and Hedges method starts by computing the odds ratio. Labeling the cell frequencies of a 2 by 2 table as a, b, c, and d, reading from left toright, top to bottom, the odds ratio is
OR =

ad
.
bc

(8)

The odds ratio is then converted to d using√
3= ln (OR) × 0.551 .
d = ln (OR)π
The variance of d using this method is
vd = vln(OR)3= vln(OR) × 0.304 ,π2

where vln(OR) is1
1 1 1+ + + .
(9)a b c dThe Cox method also starts with the odds ratio (equation 8) and then converts
to d usingln (OR).
d=
1.65
The variance of d using this method isvln(OR) =vd =vln(OR),
1.652

where vln(OR) is computed as above in equation 9.
Note that the above becomes undefined if any of the four cell frequencies
equals zero. To avoid this, .5 is added to any cell frequency equal to zero.
The probit method computes d as
d = probit (p1 ) − probit (p2 ) .
David B. Wilson(10)7Formulas for “Practical Meta–Analysis Online Calculator”

The variance of d using this method is22

2πp1 (1 − p1 ) ez1
2πp2 (1 − p2 ) ez2
vd =
+
.
n1
n2
The calculator determines the probit using the Odeh and Evans’ (1974) algorithm. This method is accurate to within 5 decimal places (see Brophy, 1985). The
algorithm first computes y as:p
y = −2ln(p) .
Using y, the probit value (zp ) is then approximated as
zp = y − ((((0.0000453642210148y + 0.0204231210245)y+ 0.342242088547)y + 1)y + 0.322232431088)/((((0.0038560700634y + 0.10353775285)y+0.531103462366)y + 0.588581570495)y+0.099348462606) .

3.8

(11)

Binary Proportions

A dichotomous or binary dependent variable may also be reported as a simple
proportion or percent succeeding (or failing) in each of the two groups or conditions. The frequencies for a 2 by 2 contingency table can be computed from these
proportions using the group sample sizes.
f11 = p1 n1
f12 = (1 − p1 ) n1
f11 = p2 n2
f22 = (1 − p2 ) n2
The d type effect-size is then computed as in section 3.7 for a 2 by 2 frequency
table.

3.9

Point-Biserial Correlation

The point-biserial correlation is a correlation coefficient between a dichotomous
variable, such as a condition dummy code, and a continuous or scaled variable.
This correlation is conceptually and algebraically very similar to d and can beconverted to d with the formulas below. Separate equations are provided forequal and unequal sample sizes between the two conditions.
For unequal sample sizes, d is computed as
r
.
d= p
2
(1 − r ) (p (1 − p))David B. Wilson(12)8Formulas for “Practical Meta–Analysis Online Calculator”

For equal sample sizes, d is computed as
2r
d= √
.
1 − r2(13)

A study may report the p-value for a point-biserial correlation coefficient.
The significance test on which this p-value is based is the independent t-test. As
such, the method presented in section 3.4 is used to find the t-value and thenequation 6 or 7 is used to compute the effect-size. Note that the p-value froma correlation based on two continuous variables (a Pearson’s correlation coefficient) or two dichotomous variables (a phi coefficient) should not be handled inthis way.
The variance for d based on a point-biserial correlation is computed as inequation 1.

3.10

Phi Coefficient

The relationship between a dichotomous independent variable (such as treatment condition) and a dichotomous dependent variable (such as success or failure) might be presented in a study as a phi coefficient. A study may report thissimply as r. However, if the underlying data is dichotomous for both the independent and dependent variable, then the r is a phi coefficient and should betreated as such.
The effect-size d is approximated as
2r
d= √
.
1 − r2(14)

Equation 1 for the variance of d is biased for d approximated from a phi
coefficient. Given the relationship between phi and χ2 , a variance estimate canbe computed that is consistent with the significance test for phi. This is done by
first converting phi into a chi2 using
χ2 = r2 ∗ n .

(15)

The variance is then determining as
v=

d2
.
chi2

(16)

Note that this estimator for dv fails when phi equals zero. Also, this approximation of d and dv should only be used when 2 by 2 frequency or proportion data
are not reported or cannot be determined. Computing d from 2 by 2 frequency
or proportion data as described in sections 3.7 and 3.8 provides a better estimate.

David B. Wilson9Formulas for “Practical Meta–Analysis Online Calculator”

3.11

Chi-Square (χ2 ) based on 2 by 2 Frequency Data

This method is algebraically the same as that used for the phi coefficient (section3.10). As shown in equation 15, there is a simple algebraic relationship between
phi and chi-square. Thus, d is computed from χ2 as
s
χ2
d=2
.
(17)
N − χ2
As with the phi coefficient, the variance is estimated such that it is consistent
with the chi-squared test.
d2
v = 2.
(18)
χThis estimator fails when chi-squared equals zero. The chi-square must have one
degree-of-freedom. If it has two or more degrees-of-freedom, then it is not basedon 2 by 2 frequency data.
Also note that this estimator for d should only be used when the 2 by 2 frequency or proportion data are not reported or cannot be determined. Computing
d from 2 by 2 frequency or proportion data as described in sections 3.7 and 3.8
provides a better estimate.

3.12

Exact p-value for a Chi-Squared based on 2 by 2 Frequency Data

The chi-squared is the most commonly used significance test for a 2 by 2 frequency table. The chi-squared distribution with one degree-of-freedom is thesquared standardized normal distribution. Thus, the chi-squared associated witha p-value from a chi-squared computed on a 2 by 2 frequency table can be determined using equation 11 and squaring the result, that is,χ2 = z2p .
The effect size d is then computed using the methods of section 3.11. Notethat the chi-squared must have only 1 degree-of-freedom. Also, this methodshould only be used when other data are not provided.

3.13

Frequency Distribution

A study may report a frequency distribution by group for an ordinal, interval,
or ratio scale but not report means, standard deviations, or a t-test that would
allow for the easy computation of d. Assuming equal intervals (and order) forthe categories of the frequency distribution, we can compute a mean and standard deviation for each group by assigning the values 1, 2, 3, etc. to the rows of
the frequency distribution. For an ordinal distribution, the assumption of equalDavid B. Wilson10Formulas for “Practical Meta–Analysis Online Calculator”

intervals is not satisfied and as such this method should be used cautiously withrecognition that it may result in a biased estimate.
The means for each group are computed asP
fi1 i
X1 = P
,
fi1
P
fi2 i
X2 = P,fi2

andwhere fij are the frequencies for each row, i, and each group, j.
The standard deviations are computed assPPPfi1 fi1 i2 − ( fi1 i)2,
s1 =P( fi1 )2
and

sP
s2 =

fi2

P

fi2 i2 − (P( fi2 )2P

fi2 i)2.

If only proportions are provided, then the frequencies are computed as
fij = pij (nj ).
The d effect size is then computed using 4.

3.14

Unstandardized Regression Coefficient

The unstandardized regression coefficient for treatment dummy variable (a variable coded as 0 and 1 or 1 and 2 to indicate group membership) represents themean difference between the two groups adjusted for the other independent
variables included in the model. As such, the unstandardized regression coefficient (b) can be used in the numerator of equation 4 in computing d (i.e.,
it is the covraiate adjusted mean difference). What is needed is an estimate ofthe denominator, spooled . The difference between the standard deviation of thethe dependent variable (sy ) and the pooled within groups standard deviation(spooled ) is that the latter excludes variability attributable to group membership(treatment). Thus, spooled can be determined viavuu s2 (N − 1) − b2 n1 n2
t y
n1 +n2spooled =
.
(19)
N−2
This equation removes the variance associate with treatment from the total variance. Using this estimate of spooled , the effect size is computed asd=David B. Wilsonb.
spooled(20)

11Formulas for “Practical Meta–Analysis Online Calculator”

3.15

Standardized Regression Coefficient

The standardized regression coefficient, (β), can also be used to compute d but
it first must be converted into an unstandardized coefficient (b). The relationship between the standardized and unstandardized coefficient is a function of
the standard deviations for both the dependent (y) and independent (x) variables. The standard deviation of the dependent variable is often reported but thestandard deviation for a treatment dummy variable rarely is. However, because
the sample size for each group is generally available, the standard deviation of
the dummy variable for x can be computed assn1 − n21 /N
sx =
.
(21)
N−1
The unstandardized regression coefficient is then computed as
b=β

sy
,
sx

(22)

and d is estimated as using the equations of section (3.14).

3.16

Means and Full Sample Standard Deviation

The pooled standard deviation used in the denominator of d is simply the total
sample standard deviation for the dependent variability with the variance due
to group membership removed. Thus, the pooled standard deviation can becomputed from the total or full sample standard deviation as follows
v


u
2
2
X1 +X2 −2X1 X2
u 2
t sy (N − 1) −Nspooled =.
(23)
N
The d is then computed with equation 4
d=

3.17

X1 − X2.
spooled(24)

Mean Gain Scores and Gain Score Standard Deviations

Gain scores represent a challenge for computing d. Simply treating the meanand standard deviation of gain scores as simple means and standard deviations
and proceeding with equation 4 will produce misleading results. The standardized mean difference version of d is standardized based on the variability acrossindividuals on the dependent variable (i.e., variability in the raw scores). Using
the standard deviation of gains scores in place of the standard deviation of theraw dependent variable scores standardizes on variability in change or gains,
David B. Wilson

12Formulas for “Practical Meta–Analysis Online Calculator”

a rather different thing altogether. Essentially, using the standard deviation ofgains produces a version of d that is on a different scale (i.e., standardizing on
gains) and is not directly comparable to the standardized mean difference effectsize that is standardizing on differences across individuals.
There are two reasons why we might be trying to convert gain score statistics
into a standardized mean difference type d. First, this might be the only statistical data reported. Second, we might want to use a mean difference that is
adjusted for baseline scores (a difference in differences). A workable solution is
to convert the standard deviation of the gain scores into a standard deviation of
raw scores. Unfortunately, the correlation between the time 1 and time 2 scores isneeded to make this conversion and this is often not available (note that the alphacoefficient for internal consistency that is often reported as a measure of reliability is not an estimate of the time 1 versus time 2 correlation). In some situations
you may have reasonable estimates of this correlation from other sources, suchas from psychometric studies, and using these approximations may be preferable to omitting the study from the meta-analysis altogether. However, if you doso, sensitivity analyses are recommended to assess the affect of these approximations on overall results.
The conversion of a standard deviation of gain scores (sgain ) to a standard
deviation of raw scores (s) is
sgain
s= p
(25)
2 (1 − r)
The pooled standard deviation is then computed using equation 5 and d iscomputed as∆1 − ∆2d=.
(26)
spooledThe variance d based on gain scores is computed as
v=

3.18

2 (1 − r1 ) 2 (1 − r2 )
d2
+
+
n1
n2
2 (n1 + n2 )Mean Gain Scores, Pre and Post-test Standard Deviations, and a Paired t-test or Pre-Post r

A study may present pretest and post-test means, standard deviations, and sample sizes. Using either the pretest or post-test data, d can be computed using 4.
However, a mean difference that is adjusted for pretest differences, a difference
in differences, effect size may be wanted and can be computed asd=∆1 − ∆2spooled(27)

where ∆1 and ∆2 are the pre-post mean differences for each group, computed as∆j = Xpost − Xpre .
David B. Wilson

(28)
13

Formulas for “Practical Meta–Analysis Online Calculator”

The pooled standard deviation is based on all four raw score standard deviations. First, the pooled pre-post standard deviation for each group is determinedasss2pre + s2post
sj =
.
(29)
2
Using sj for each group, the spooled is computed using equation 5.
Although an estimate of the variance of d can be determined using 1, thevariance of d is affected by the precision (or imprecision) of all four means ofwhich it is a function. The proper estimate of vd must therefore take this into
account and can be estimated asv=

2(1 − r1 ) 2(1 − r2 )
d2
+
+
n1
n2
2(n1 + n2 )(30)

where r1 and r2 are the correlations between the pretest and post-test scoreswithin each group. These values are rarely reported but can be determinedfrom paired t-tests. The latter are often reported in such situations. The pretest/posttest correlation can be determined for each group as2
s21 t2 + s22 t2 − X2 − X1 nr=
2s1 s2 t2(31)

where s1 and s2 were the pre-test and posttest standard deviations, and t is thepaired t-test.

3.19

Means and Standard Deviations with Subgroups

A common situation in conducting a meta-analysis is finding a study that reports means and standard deviations for subgroups within each condition but
does not report the overall results. For example, a study may report the resultsof a treatment and control group comparison separately for boys and girls. The
mean for each condition can easily be found by computing a weighted mean.
The pooled standard deviation is less straightforward and there are two options,depending on the situation. If the subgroups represent a breakout on a sample
characteristic, such as gender or risk-level, then simply pooling the standard deviations, as with formula 5, will underestimate the full within-group standard
deviation because variability associated with the subgroup variable has been
removed (e.g., variability in the outcome associated with gender). Thus, it isimportant to compute the variability associated with this subgroup and add itback into the pooled standard deviation. There are situations, however, where
the subgroups represent a manipulated variable that adds variability rather thansubtracts it. For example, in a two-by-two factorial design where each factor isa true independent variable (i.e., manipulated by the researcher). In such a case,this additional variability associated with a manipulated factor should not be
added to the pooled within-groups standard deviation.
David B. Wilson

14Formulas for “Practical Meta–Analysis Online Calculator”

The weighted mean for each condition (i.e., treatment and control) is computed as:
P
Xj nj
X= P
,
nj
where j represents each subgroup.
The pooled within-group standard deviation for each condition (i.e., treatment and control), ignoring any variance removed due to the subgroup varable,issP
s2j nj − 1 .
Pspooled =nj − 1The subgroup variable is added back into the within group variance using
the following formula:
v

uP 2
2
P
u s nj − 1
X 2
Xj nj
j
t
 +
P
P
spooled =Xj nj −.
nj
nj − 1
The within-groups pooled standard deviation is then computed using 5.

3.20 F-test (ANOVA) with 3 or More Groups
The F-value from a one-way ANOVA with three or more groups can be used tocompute d so long as the means and sample sizes on which the F is based are alsoavailable. Thus, this method provides a way to determine the pooled standard
deviation when the standard deviation for each group is missing.
In the case of a one-way ANOVA, F is the ratio of the MS-between groupmeans to the MS-within (or MS-error) and the MS-within is the pooled withingroup variance. We can determine the MS-within asr
MS between
MS within =,(32)
Fwhere MS between is found throughP
MS between =2P

Xj nj −

X j nj )
( P

k−1

nj

2

,

(33)

where k is the number of groups. d is then computed using 4. If two or moremeans are selected to represent either the treatment or control group, then theweighted mean across the multiple groups is used. For example, if two groups
are selected as being part of the treatment condition, the weighted mean for thetreatment group is computed asP
Xj nj
X1 = P.
(34)
nj
David B. Wilson

15Formulas for “Practical Meta–Analysis Online Calculator”

3.21

Means and an Analysis-of-Covariance (ANCOVA)

An analysis of covariance tests the effect of treatment or group membership on
the dependent variable adjusted for a continuous covariate, such as a pretest orother characteristic of the study participants. The F-value for the treatment effectcannot be converted as described in section 3.5 and using that method will overestimate d. However, if sufficient information is provided, d can be computed asfollows. First, spooled is computed ass
MS error df error − 1spooled =,(35)
1 − r2
df error − 2
where r is the correlation between the covariate√
and the dependent variable. In
the case of multiple covariates, the multivariate R2 can be used.
The means and group sample sizes are needed to compute d using equation
4. Thus this method is only useful when the standard deviations missing but
the full ANCOVA results are reported along with the correlation between thecovariate and dependent variable. In practice, this is rarely the case.

3.22

Means and a Two-way ANOVA

As with the ANCOVA, an F-value from a two-way ANOVA cannot be treatedas an F-value from a one-way ANOVA. Doing so will over-estimate d. Computing d requires information from the full sums-of-squares table and uses thisinformation to determine the pooled within groups standard deviation. A twoway ANOVA produces three F-values, one for each main effect (often referred
to generically as factors A and B), and one for the interaction between the two
facotrs. Assuming that the effect of interest is factor A (e.g., the factor representing the treatment versus control group), you must decide whether to incorporate
the variance associated with factor B into the pooled standard deviation. If factor B is a blocking factor (i.e., not manipulated) such as gender, risk-level, etc.,
then it makes sense to include this variability in the pooled standard deviation.
If factor B is a manipulated factor such as another treatment manipulation, thenit may make sense to exclude this variability in the pooled standard deviation.
The default for the online calculator is to include the variability of factor B, thenon-focal factor.
The pooled standard deviation with the variability of factor B is computed ass
SS B + SS AB + SS error
spooled =.
(36)
df AB + df AB + df erroras

The pooled standard deviation without the variability of factor B is computeds
SS AB + SS error
spooled =.
(37)
df AB + df error
Using the appropriate estimate of spooled , d is computed using equation 4.

David B. Wilson

16Formulas for “Practical Meta–Analysis Online Calculator”

4

The Odds Ratio and Risk Ratio Effect Size

The odds ratio (OR) and risk ratio (RR) are effect sizes suitable for a two groupcomparison (e.g., experimental and control) and a dichotomous outcome (e.g.,
success and failure). These effect sizes are widely used in meta-analyses withinpublic health and medicine and are become increasingly common in the social
sciences. These effect sizes are computed from raw frequencies, proportions orpercents. It is also possible to transform an r or χ2 based on a 2 by 2 frequency
table into an OR or RR so long as the marginal distributions for the independent(grouping) and dependent (outcome) variables are available.

4.1

Frequency Data

Both the odds ratio (OR) and the risk-ratio (RR) are easily computed from 2 by 2
frequency data as follows
ad
OR =
,
(38)
bc
a/(a + b)
RR =
,
(39)
c/(c + d)
where a, b, c, and d, are the cell frequencies of a 2 by 2 table reading from left toright and top to bottom.
Given the asymmetric nature of these effect sizes, meta-analysis is performedon the natural log of each. The variance for the logged OR and logged RR are

4.2

vln(OR) =

1
1 1 1
+ + +
a b c d(40)

vln(RR) =

b/a
d/c
+
a+b c+d(41)

Proportion or Percent Data

Although OR and RR can be computed directly from proportions, the effect size
calculator recreates the 2 by 2 frequencies and uses the equations above. Thishas the advantage that these frequencies are needed to compute the varianceestimates. These equations assume that these are the within group proportions
or percentages (i.e., proportion successful in each group). The cell frequencies
are computed from proportions as
a=

p1
,
n1

1 − p1
,
n1
p2
c=
,
n2

b=

David B. Wilson

(42)
(43)
(44)
17

Formulas for “Practical Meta–Analysis Online Calculator”d=1 − p2.
n2(45)Percents are converted to proportions through division by 100.

4.3

Phi and Chi-squared

The odds ratio and risk ratio can be computed from a phi coefficient (i.e., a correlation based on 2 by 2 frequency data) or a chi-squared with one degree-offreedom (i.e., chi-square based on 2 by 2 frequency data). However, the marginalsof the 2 by 2 table are needed. Generally the sample sizes for the two conditions
are known, providing the marginals for the independent variable. The overall
event-rate (success or failure rate) is needed to determine the dependent variable marginals.
Using the chi-squared, the top-left cell frequency, a, can be determined as"#r
χ2 px py (1 − px )(1 − py )a = N px pyNwhere px is the proportion of the sample in one of the two conditions, py is the
proportion of the overall sample with a positive (or negative) outcome, and N is
the total sample size.
Using the phi coefficient (r), the top-left cell frequency, a, can be determinedasqa = N px py + r px py (1 − px )(1 − py ) .
The remaining cell frequencies can be determined as follows:b = n1 − a,c = N(py ) − a,d = N − (a + b + c).
The odds ratio and risk ratio and associated variances are computed using
equations 38, 39, 40, and 41 using the frequencies of the 2 by 2 table determined
above.

4.4

Odds Ratio Based on d

An odds ratio can be approximated from a standardized mean difference effectsize, d. This can be useful when the dominant method of measuring the outcomeof interest produces dichotomous data but a subset of relevant studies measure
the outcome on a scale. This conversion allows for the combining of effect size
based on scale measures along with those based on dichotomous data. Notethat this is an approximation and assumes that the outcome measure is normally
distributed, although simulations show that it performs reasonably well withDavid B. Wilson18Formulas for “Practical Meta–Analysis Online Calculator”

moderately skewed data. Two methods are presented below, the Hasselblad andHedges logit method (denoted simply as ORlogit ) and the modified logit method
proposed by Cox (denoted as ORCox ). The Cox method tweaks the conversion
to account for the more peaked nature of the logistic distribution. This method
performs slightly better when the event rate approaches 0 or 1. The conversions
and associated variance estimates are computed asdπ
ORlogit = √ ,
3

(46)

vd π2
,
(47)
3
ORCox = d1.65,(48)
vdvCox =.
(49).367
The risk ratio depends on the marginal distribution of the outcomes and assuch d cannot be converted into a risk ratio.
vlogit =5

Correlation Coefficient Effect Size

The correlation coefficient is a commonly used effect size for meta-analysis. It isbest suited to research domains in which the two variables of interest are continuous in nature, even if some studies dichotomies one or both of the variables.
The equation for computing the Pearson’s correlation coefficient can be readyfound in most any introductory statistics book and as such is not reproduced
here.
A complication with using the correlation as the effect size in a meta-analysis
that relies on the inverse-variance weight method is the inability to compute a
variance for r. A solution is Fisher’s z transformation of r (denote as zr below).
This is a variance stabilizing transformation and as r approaches -1 or 1, zr approaches infinity. The zr transformation is


1
1+r
zr = ln
.
(50)
2
1−r
The variance of zr is simply
vzr =

1
.
N−3

(51)

Results, such as a mean zr and associated confidence intervals, can be converted back in r using
e2zr − 1
r = 2z.
(52)
e r +1David B. Wilson

19Formulas for “Practical Meta–Analysis Online Calculator”

5.1

Correlation from a j by k Frequency Table

Before computers were widely used to compute statistics there were computationally efficient methods of computing common statistics, such as Pearson’s
correlation coefficient, from a contingency table, rather than the raw data. Thismethod can be used to compute a correlation from a contingency table (a j by kfrequency table, where j is the number of rows and k is the number of columns).
This method assumes that each variable is at least ordinal and that the table
maintains the ordered nature of the data.
P
N fjk − fjk jfjk kr = rh
2 i
2 i h PPPP
N fjk k2 −fjk k
N fjk j2 −fjk jwhere j represents the rows and k represents the columns.

5.2

Correlation from d

The standardized mean difference effect size, d, can easily be converted into a
correlation coefficient using the following formula:dr= p,d2 + 1/(p(1 − p))(53)

where p is defined as the proportion of the total sample in either one of the twogroups. This is computed as
n1
.
(54)
p=
n1 + n2
The conversion produces the point-biserial correlation coefficient. If the sample
sizes in the two groups are equation, the equation simplifies todr= p.
4(d2 + 1)(55)

The variance for zr based on d is
vzr =

vd
,
vd + 1/(p(1 − p))(56)

where vd is the variance of the d being converted.

5.3

Correlation from a 2 by 2 Frequency Table

A correlation coefficient based on a 2 by 2 frequency (contingency) table is aphi coefficient, although using the Pearson’s correlation coefficient equation willproduce the same result. In the equation below, a, b, c, and d, are the cell frequencies of the 2 by 2 table, reading from left to right and top to bottom. Phi (r)is computed asad − bc
r= p.
(57)(a + b)(c + d)(a + c)(b + c)
David B. Wilson20Formulas for “Practical Meta–Analysis Online Calculator”

Because the variance of zr based on the above is affected by both the overall
sample size and the marginal distributions, a more precise variance estimate iscomputed by rescaling the variance for a logged odds ratio. Thus, the variancefor r based on phi is computed as
vr = vln(OR)z2rln (OR)2(58)

where vln(OR) is computed using equation 40 and ln (OR) is computed using
equations 38. This provides a more accurate estimate of vzr r then 51.

5.4 χ2 and Sample Size
A chi-squared for a relationship between two dichotomous variables can be converted into r. The r in this case is a φ coefficient as is computed as
r
χ2
r=
.
(59)
N
This value is then transformed into zr using equation 50. The variance of theFisher’s zr is estimated asz2(60)vzr = r2χand produces confidence intervals consistent with the χ2 test.
Note that this conversion is appropriate only for a χ2 from a 2 by 2 frequency
table. A χ2 of such a table will have 1 degree-of-freedom. A χ2 with 2 or moredegrees-of-freedom cannot be converted with this equation and does not reflect
the linear relationship between the two variables (i.e., you cannot convert it toan r).

5.5

t-test and Exact p-value From a t-test

The t-test and exact p-value from a t-test can easily be converted to r so long as
you have the sample size on which the test was based. r based on t is computed
as
t
r= √
.
(61)
2
t +N−2
If only the p-value is reported, r can be determined using the equation above
after computing the t-value associated with the reported p-value using the method
shown in section 3.4.

References
Borenstein, M. (2009). Effect sizes for studies with continuous outcome
data. In H. M. Cooper, L. V. Hedges, & J. Valentine (Eds.), Handbookof research synthesis (2nd ed.). New York: Russell Sage Foundation.
David B. Wilson21Formulas for “Practical Meta–Analysis Online Calculator”

Brophy, A. L. (1985). Aproximation of the inverse normal distributionfunction. Behavior Research Methods, Instruments, & Computers, 17,415–417.
Fleiss, J. L. (1994). Measures of effect size for categorical data. In H. M.
Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (p.
245-260). New York: Russell Sage Foundation.
Fleiss, J. L., & Berlin, J. A. (2009). Measures of effect size for categorical
data. In H. M. Cooper, L. V. Hedges, & J. Valentine (Eds.), Handbookof research synthesis (2nd ed.). New York: Russell Sage Foundation.
Hasselblad, V., & Hedges.(1995). Meta-analysis of screening and diagnostic tests. Psychological Bulletin, 117, 167-178.
Hill, G. W. (1970). Algorithm 396: Student’s t-quantiles [S14]. Communications of the ACM, 13, 619–620.
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand
Oaks, Calif: Sage Publications.
Odeh, R. E., & Evans, J. O.(1974). Algorithm AS 70: The percentage pointsof the normal distribution. Applied Statistics, 23, 96–97.
Rosenthal, R. (1994). Parametric measures of effect size. In H. M. Cooper& L. V. Hedges (Eds.), The handbook of research synthesis (p. 231-244).
New York: Russell Sage Foundation.
Sánchez-Meca, J., Marín-Martínez, F., & Chacón-Moscoso, S. (2003, December). Effect-size indices for dichotomized outcomes in metaanalysis. Psychological Methods, 8, 448-467.

David B. Wilson

22