# Publication Bias in Meta-Analysis: Confidence Intervals for Rosenthal’s Fail-Safe Number

Authors: Konstantinos C. Fragkos; Michail Tsagris; Christos C. Frangos
Journal: International Scholarly Research Notices
 (2014)

---

## Abstract

The purpose of the present paper is to assess the efficacy of confidence intervals for Rosenthal’s fail-safe number. Although Rosenthal’s estimator is highly used by researchers, its statistical properties are largely unexplored. First of all, we developed statistical theory which allowed us to produce confidence intervals for Rosenthal’s fail-safe number. This was produced by discerning whether the number of studies analysed in a meta-analysis is fixed or random. Each case produces different variance estimators. For a given number of studies and a given distribution, we provided five variance estimators. Confidence intervals are examined with a normal approximation and a nonparametric bootstrap. The accuracy of the different confidence interval estimates was then tested by methods of simulation under different distributional assumptions. The half normal distribution variance estimator has the best probability coverage. Finally, we provide a table of lower confidence intervals for Rosenthal’s estimator.


---

## Body

## 1. Introduction

Meta-analysis refers to methods focused on contrasting and combining results from different studies, in the hope of identifying patterns among study results, sources of disagreement among those results, or other interesting relationships that may come to light in the context of multiple studies [1]. In its simplest form, this is normally done by identification of a common measure of effect size, of which a weighted average might be the output of a meta-analysis. The weighting might be related to sample sizes within the individual studies [2, 3]. More generally there are other differences between the studies that need to be allowed for, but the general aim of a meta-analysis is to more powerfully estimate the true effect size as opposed to a less precise effect size derived in a single study under a given single set of assumptions and conditions [4]. For reviews on meta-analysis models, see [2, 5, 6]. Meta-analysis can be applied to various effect sizes collected from individual studies. These include odds ratios and relative risks; standardized mean difference, Cohen’s d, Hedges’ g, and Glass’s Δ; correlation coefficient and relative metrics; sensitivity and specificity from diagnostic accuracy studies; and P
-values. For more comprehensive reviews see Rosenthal [7], Hedges and Olkin [8], and Cooper et al. [9].


## 2. Publication Bias

Publication bias is a threat to any research that attempts to use the published literature, and its potential presence is perhaps the greatest threat to the validity of a meta-analysis [10]. Publication bias exists because research with statistically significant results is more likely to be submitted and published than work with null or nonsignificant results. This issue was memorably termed as thefile-drawer problem by Rosenthal [11]; nonsignificant results are stored in file drawers without ever being published. In addition to publication bias, other related types of bias exist including pipeline bias, subjective reporting bias, duplicate reporting bias, or language bias (see [12–15] for definitions and examples).The implication of these various types of bias is that combining only the identified published studies uncritically may lead to an incorrect, usually over optimistic, conclusion [10, 16]. The ability to detect publication bias in a given field is a key strength of meta-analysis because identification of publication bias will challenge the validity of common views in that area and will spur further investigations [17]. There are two types of statistical procedures for dealing with publication bias in meta-analysis: methods for identifying the existence of publication bias and methods for assessing the impact of publications bias [16]. The first includes the funnel plot (and other visualisation methods such as the normal quantile plot) and regression/correlation-based tests, while the second includes the fail-safe (also called file-drawer) number, the trim and fill method, and selection model approaches [10, 14, 18]. Recent approaches include the test for excess significance [19] and the p-curve [20].The most commonly used method is the visual inspection of a funnel plot. This assumes that the results from smaller studies will be more widely spread around the mean effect because of larger random error. The next most frequent method used to assess publication bias is Rosenthal’s fail-safe number [21]. Two recent reviews examining the assessment of publication bias in psychology and ecology reported that funnel plots were the most frequently used (24% and 40% resp.), followed by Rosenthal’s fail-safe number (22% and 30%, resp.).


### 2.1. Assessing Publication Bias by Computing the Number of Unpublished Studies

Assessing publication bias can be performed by trying to estimate the number of unpublished studies in the given area a meta-analysis is studying. The fail-safe number represents the number of studies required to refute significant meta-analytic means. Although apparently intuitive, it is in reality difficult to interpret not only because the number of data points (i.e., sample size) for each ofk studies is not defined, but also because no benchmarks regarding the fail-safe number exist, unlike Cohen’s benchmarks for effect size statistics [22]. However, these versions have been heavily criticised, mainly because such numbers are often misused and misinterpreted [23]. The main reason for the criticism is that, depending on which method is used to estimate the fail-safe N, the number of studies can greatly vary.Although Rosenthal’s fail-safe number of publication bias was proposed as early as 1979 and is frequently cited in the literature [11] (over 2000 citations), little attention has been given to the statistical properties of this estimator. This is the aim of the present paper, which is discussed in further detail in Section 3.Rosenthal [11] introduced what he called thefile drawer problem. His concern was that some statistically nonsignificant studies may be missing from an analysis (i.e., placed in a file drawer) and that these studies, if included, would nullify the observed effect. Bynullify, he meant to reduce the effect to a level not statistically significantly different from zero. Rosenthal suggested that rather than speculate on whether the file drawer problem existed, the actual number of studies that would be required to nullify the effect could be calculated [26]. This method calculates the significance of multiple studies by calculating the significance of the mean of the standard normal deviates of each study. Rosenthal’s method calculates the number of additional studies N
R, with the mean null result necessary to reduce the combined significance to a desired α level (usually 0.05).The necessary prerequisite is that each study examines a directional null hypothesis such that the effect sizesθ
i from each study are examined under θ
i
≤
0 or (θ
i
≥
0). Then the null hypothesis of Stouffer [27] test is
(1)
H
0
:
θ
1
=
⋯
=
θ
k
=
0
.
The test statistic for this is
(2)
Z
S
=
∑
i
=
1
k
Z
i
k
,
with z
i
=
θ
i
/
s
i, where s
i are the standard errors of θ
i. Under the null hypothesis we have Z
S
~
N
0,1 [7].So we get that the number of additional studiesN
R, with mean null result necessary to reduce the combined significance to a desired α level (usually 0.05 [7, 11]), is found after solving
(3)
Z
α
=
∑
i
=
1
k
Z
i
N
R
+
k
.
So, N
R is calculated as
(4)
N
R
=
∑
i
=
1
k
Z
i
2
Z
α
2
-
k
,
where k is the number of studies and Z
α is the one-tailed Z score associated with the desired α level of significance. Rosenthal further suggested that if N
R
>
5
k
+
10, the likelihood of publication bias would be minimal.Cooper [28] and Cooper and Rosenthal [29] called this number the fail-safe sample size or fail-safe number. If this number is relatively small, then there is cause for concern. If this number is large, one might be more confident that the effect, although possibly inflated by the exclusion of some studies, is, nevertheless, not zero [30]. This approach is limited in two important ways [26, 31]. First, it assumes that the association in the hidden studies is zero, rather than considering the possibility that some of the studies could have an effect in the reverse direction or an effect that is small but not zero. Therefore, the number of studies required to nullify the effect may be different than the fail-safe number, either larger or smaller. Second, this approach focuses on statistical significance rather than practical or substantive significance (effect sizes). That is, it may allow one to assert that the mean correlation is not zero, but it does not provide an estimate of what the correlation might be (how it has changed in size) after the missing studies are included [23, 32–34]. However, for many fields it remains the gold standard to assess publication bias, since its presentation is conceptually simple and eloquent. In addition, it is computationally easy to perform.Iyengar and Greenhouse [12] proposed an alternative formula for Rosenthal’s fail-safe number, in which the sum of the unpublished studies’ standard variates is not zero. In this case the number of unpublished studies n
α is approached through the following equation:
(5)
Z
α
=
∑
i
=
1
k
Z
i
+
n
α
M
α
n
α
+
k
,
where M
α
=
-
ϕ
(
z
α
)
/
Φ
(
z
α
) (this results immediately from the definition of truncated normal distribution) and α is the desired level of significance. This is justified by the author that the unpublished studies follow a truncated normal distribution with x
≤
z
α.  Φ
· and ϕ
· denote the cumulative distribution function (CDF) and probability distribution function (PDF), respectively, of a standard normal distribution.There are certain other fail-safe numbers which have been described, but their explanation goes beyond the scope of the present article [35]. Duval and Tweedie [36, 37] present three different estimators for the number of missing studies and the method to calculate this has been named Trim and Fill Method. Orwin’s [38] approach is very similar to Rosenthal’s [11] approach without considering the normal variates but taking Cohen’s d   [22] to compute a fail-safe number. Rosenberg’s fail-safe number is very similar to Rosenthal’s and Orwin’s fail-safe number [39]. Its difference is that it takes into account the meta-analytic estimate under investigation by incorporating individual weights per study. Gleser and Olkin [40] proposed a model under which the number of unpublished studies in a field where a meta-analysis is undertaken could be estimated. The maximum likelihood estimator of their fail-safe number only needs the number of studies and the maximum P value of the studies. Finally, the Eberly and Casella fail-safe number assumes a Bayesian methodology which aims to estimate the number of unpublished studies in a certain field where a meta-analysis is undertaken [41].The aim of the present paper is to study the statistical properties of Rosenthal’s [11] fail-safe number. In the next section we introduce the statistical theory for computing confidence intervals for Rosenthal’s [11] fail-safe number. We initially compute the probability distribution function of N
^
R, which gives formulas for variance and expectation; next, we suggest distributional assumptions for the standard normal variates used in Rosenthal’s fail-safe number and finally suggest confidence intervals.


## 2.1. Assessing Publication Bias by Computing the Number of Unpublished Studies

Assessing publication bias can be performed by trying to estimate the number of unpublished studies in the given area a meta-analysis is studying. The fail-safe number represents the number of studies required to refute significant meta-analytic means. Although apparently intuitive, it is in reality difficult to interpret not only because the number of data points (i.e., sample size) for each ofk studies is not defined, but also because no benchmarks regarding the fail-safe number exist, unlike Cohen’s benchmarks for effect size statistics [22]. However, these versions have been heavily criticised, mainly because such numbers are often misused and misinterpreted [23]. The main reason for the criticism is that, depending on which method is used to estimate the fail-safe N, the number of studies can greatly vary.Although Rosenthal’s fail-safe number of publication bias was proposed as early as 1979 and is frequently cited in the literature [11] (over 2000 citations), little attention has been given to the statistical properties of this estimator. This is the aim of the present paper, which is discussed in further detail in Section 3.Rosenthal [11] introduced what he called thefile drawer problem. His concern was that some statistically nonsignificant studies may be missing from an analysis (i.e., placed in a file drawer) and that these studies, if included, would nullify the observed effect. Bynullify, he meant to reduce the effect to a level not statistically significantly different from zero. Rosenthal suggested that rather than speculate on whether the file drawer problem existed, the actual number of studies that would be required to nullify the effect could be calculated [26]. This method calculates the significance of multiple studies by calculating the significance of the mean of the standard normal deviates of each study. Rosenthal’s method calculates the number of additional studies N
R, with the mean null result necessary to reduce the combined significance to a desired α level (usually 0.05).The necessary prerequisite is that each study examines a directional null hypothesis such that the effect sizesθ
i from each study are examined under θ
i
≤
0 or (θ
i
≥
0). Then the null hypothesis of Stouffer [27] test is
(1)
H
0
:
θ
1
=
⋯
=
θ
k
=
0
.
The test statistic for this is
(2)
Z
S
=
∑
i
=
1
k
Z
i
k
,
with z
i
=
θ
i
/
s
i, where s
i are the standard errors of θ
i. Under the null hypothesis we have Z
S
~
N
0,1 [7].So we get that the number of additional studiesN
R, with mean null result necessary to reduce the combined significance to a desired α level (usually 0.05 [7, 11]), is found after solving
(3)
Z
α
=
∑
i
=
1
k
Z
i
N
R
+
k
.
So, N
R is calculated as
(4)
N
R
=
∑
i
=
1
k
Z
i
2
Z
α
2
-
k
,
where k is the number of studies and Z
α is the one-tailed Z score associated with the desired α level of significance. Rosenthal further suggested that if N
R
>
5
k
+
10, the likelihood of publication bias would be minimal.Cooper [28] and Cooper and Rosenthal [29] called this number the fail-safe sample size or fail-safe number. If this number is relatively small, then there is cause for concern. If this number is large, one might be more confident that the effect, although possibly inflated by the exclusion of some studies, is, nevertheless, not zero [30]. This approach is limited in two important ways [26, 31]. First, it assumes that the association in the hidden studies is zero, rather than considering the possibility that some of the studies could have an effect in the reverse direction or an effect that is small but not zero. Therefore, the number of studies required to nullify the effect may be different than the fail-safe number, either larger or smaller. Second, this approach focuses on statistical significance rather than practical or substantive significance (effect sizes). That is, it may allow one to assert that the mean correlation is not zero, but it does not provide an estimate of what the correlation might be (how it has changed in size) after the missing studies are included [23, 32–34]. However, for many fields it remains the gold standard to assess publication bias, since its presentation is conceptually simple and eloquent. In addition, it is computationally easy to perform.Iyengar and Greenhouse [12] proposed an alternative formula for Rosenthal’s fail-safe number, in which the sum of the unpublished studies’ standard variates is not zero. In this case the number of unpublished studies n
α is approached through the following equation:
(5)
Z
α
=
∑
i
=
1
k
Z
i
+
n
α
M
α
n
α
+
k
,
where M
α
=
-
ϕ
(
z
α
)
/
Φ
(
z
α
) (this results immediately from the definition of truncated normal distribution) and α is the desired level of significance. This is justified by the author that the unpublished studies follow a truncated normal distribution with x
≤
z
α.  Φ
· and ϕ
· denote the cumulative distribution function (CDF) and probability distribution function (PDF), respectively, of a standard normal distribution.There are certain other fail-safe numbers which have been described, but their explanation goes beyond the scope of the present article [35]. Duval and Tweedie [36, 37] present three different estimators for the number of missing studies and the method to calculate this has been named Trim and Fill Method. Orwin’s [38] approach is very similar to Rosenthal’s [11] approach without considering the normal variates but taking Cohen’s d   [22] to compute a fail-safe number. Rosenberg’s fail-safe number is very similar to Rosenthal’s and Orwin’s fail-safe number [39]. Its difference is that it takes into account the meta-analytic estimate under investigation by incorporating individual weights per study. Gleser and Olkin [40] proposed a model under which the number of unpublished studies in a field where a meta-analysis is undertaken could be estimated. The maximum likelihood estimator of their fail-safe number only needs the number of studies and the maximum P value of the studies. Finally, the Eberly and Casella fail-safe number assumes a Bayesian methodology which aims to estimate the number of unpublished studies in a certain field where a meta-analysis is undertaken [41].The aim of the present paper is to study the statistical properties of Rosenthal’s [11] fail-safe number. In the next section we introduce the statistical theory for computing confidence intervals for Rosenthal’s [11] fail-safe number. We initially compute the probability distribution function of N
^
R, which gives formulas for variance and expectation; next, we suggest distributional assumptions for the standard normal variates used in Rosenthal’s fail-safe number and finally suggest confidence intervals.


## 3. Statistical Theory

The estimatorN
^
R of unpublished studies is approached through Rosenthal’s formula:
(6)
N
^
R
=
∑
i
=
1
k
Z
i
2
Z
α
2
-
k
.
Let Z
i, i
=
1,2
,
…
,
i
,
…
,
k, be i.i.d. random variables with E
Z
i
=
μ and Var
⁡
Z
i
=
σ
2. We discern two cases:(a)
kis fixed or
(b)
k is random, assuming additionally that k
~
Pois
λ. This is reasonable since the number of studies included in a meta-analysis is like observing counts. Other distributions might be assumed, such as the Gamma distribution, but this would require more information or assumptions to compute the parameters of the distribution.In both cases, estimators of μ, σ
2, and λ can be calculated without distributional assumptions for the Z
i with the method of moments or with distributional assumptions regarding the Z
i.


### 3.1. Probability Distribution Function ofN
^
R

#### 3.1.1. Fixedk

We compute the PDF ofN
^
R by following the next steps.Step 1.
Z
1
,
Z
2
,
…
,
Z
i
,
…
,
Z
k in the formula of the estimator N
^
R  (6) are i.i.d. distributed with E
Z
i
=
μ and Var
⁡
Z
i
=
σ
2. Let S
=
∑
i
=
1
k
Z
i and according to the Lindeberg-Lévy Central Limit Theorem [42], we have
(7)
k
S
k
-
μ
⟶
d
N
0
,
σ
2
⟹
S
⟶
d
N
k
μ
,
k
σ
2
.
So the PDF of S is
(8)
f
S
s
=
1
2
π
k
σ
2
exp
⁡
-
s
-
k
μ
2
2
k
σ
2
.Step 2.
The PDF of Rosenthal’sN
^
R can be retrieved from a truncated version of (8). From (3), we get that
(9)
S
=
Z
α
N
^
R
+
k
.
We advocate that Rosenthal’s equations (3) and (9) implicitly impose two conditions which must be taken into account when we seek to estimate the distribution of N
R:
(10)
S
≥
0
,
(11)
N
^
R
≥
0
.
Expression (10) is justified by the fact that the right hand side of  (9) is positive, so then S
≥
0. Expression (11) is justified by the fact that N
R expresses the number of studies, so it must be at least 0. Hence, expressions (10) and (11) are satisfied when S is a truncated normal random variable.
Thetruncated normal distribution is a probability distribution related to the normal distribution. Given a normally distributed random variable X with mean μ
t and variance σ
t
2, let it be that X
∈
(
a
,
b
)
,
-
∞
≤
a
≤
b
≤
∞. Then X conditional on a
<
X
<
b has a truncated normal distribution with PDF: f
X
(
x
)
=
(
1
/
σ
)
ϕ
(
(
x
-
μ
t
)
/
σ
t
)
/
(
Φ
(
(
b
-
μ
t
)
/
σ
t
)
-
Φ
(
(
a − μ
t
)
/
σ
t
)
), for a
≤
x
≤
b and f
X
(
x
)
=
0 otherwise [43].
Let it beS
*, such that S
*
≥
Z
α
k. So the PDF of S
* then becomes
(12)
f
S
*
s
*
=
1
Φ
λ
*
2
π
k
σ
2
exp
⁡
-
s
*
-
k
μ
2
2
k
σ
2
,
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
i
h
h
h
h
h
s
*
≥
Z
α
k
,
where λ
*
=
(
k
μ
-
Z
α
)
/
σ.Then, we have(13)
f
N
^
R
n
R
=
f
S
*
s
*
d
S
*
d
N
R
⟹
9
,
12
f
N
^
R
n
R
=
Z
α
2
Φ
λ
*
2
π
k
σ
2
n
R
+
k
×
exp
⁡
-
Z
α
n
R
+
k
-
k
μ
2
2
k
σ
2
,
n
R
≥
0
.
The characteristic function is
(14)
ψ
N
^
R
t
=
E
exp
⁡
i
t
N
R
=
Φ
μ
1
+
λ
*
/
σ
1
Φ
λ
*
·
Z
α
exp
⁡
k
2
μ
2
i
t
/
(
Z
α
2
-
2
k
σ
2
i
t
)
-
k
i
t
Z
α
2
-
2
k
σ
2
i
t
1
/
2
,
where i
=
-
1, μ
1
=
2
k
k
μ
σ
i
t
/
(
Z
α
2
-
2
k
σ
2
i
t
), σ
1
2
=
Z
α
2
/
(
Z
α
2
-
2
k
σ
2
i
t
).From (14) we get
(15)
E
N
^
R
=
k
2
μ
2
+
k
σ
2
Z
α
2
-
k
+
ε
,
where ε
=
(
ϕ
λ
*
/
Φ
λ
*
)
·
(
k
σ
(
k
μ
+
Z
α
)
/
Z
α
2
).Also,(16)
Var
⁡
N
^
R
=
2
k
2
σ
2
2
k
μ
2
+
σ
2
Z
α
4
+
δ
*
,
where
(17)
δ
*
=
ϕ
λ
*
Φ
λ
*
k
2
σ
3
5
k
μ
+
Z
a
2
Z
α
4
h
h
h
h
h
h
h
-
ϕ
λ
*
Φ
λ
*
+
λ
*
k
3
/
2
σ
2
k
μ
+
Z
a
2
Z
α
4
.Proofs for expressions (14), (15), and (16) are given in the Appendix.Comments. Consider the following:(1)
For a significantly largek we have that Φ
λ
*
≈
1. So (13) becomes
(18)
f
N
^
R
n
R
=
Z
α
2
2
π
k
σ
2
n
R
+
k
×
exp
⁡
-
Z
α
n
R
+
k
-
k
μ
2
2
k
σ
2
,
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
n
R
≥
0
.
Also we get
(19)
E
N
^
R
=
k
2
μ
2
+
k
σ
2
Z
α
2
-
k
,
(20)
Var
⁡
N
^
R
=
2
k
2
σ
2
2
k
μ
2
+
σ
2
Z
α
4
.
(2)
A limiting element of this computation is thatN
^
R takes discrete values because it describes number of studies, but it has been described by a continuous distribution.


#### 3.1.2. Randomk

It is assumed thatk
~
Pois
λ. So taking into account the result from the distribution of N
^
R for a fixed k we get that the joint distribution of k and N
^
R is
(21)
f
N
^
R
,
n
n
R
,
k
=
f
N
^
R
n
R
∣
k
=
k
·
p
k
=
k
⟹
f
N
^
R
,
n
n
R
,
k
=
Z
α
2
Φ
λ
*
2
π
k
σ
2
n
R
+
k
×
exp
⁡
-
Z
α
n
R
+
k
-
k
μ
2
2
k
σ
2
-
λ
·
λ
k
k
!
,
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
n
R
≥
0
,
k
=
0,1
,
2
,
…
.


### 3.2. Expectation and Variance for Rosenthal’s EstimatorN
^
R

(a)
Whenk is fixed, expressions (19) and (20) denote the expectation and variance, respectively, for N
^
R. This is derived from the PDF of N
^
R; an additional proof without reference to the PDF is given in the Appendix.
(b)
Whenk is random with k
~
Pois
λ, the expectation and variance of N
^
R are
(22)
E
N
^
R
=
λ
2
μ
2
+
λ
μ
2
+
σ
2
Z
α
2
-
λ
,
Var
⁡
N
^
R
=
4
λ
3
+
6
λ
2
+
λ
μ
4
+
4
λ
3
+
16
λ
2
+
6
λ
μ
2
σ
2
+
2
λ
2
+
3
λ
σ
4
×
Z
α
4
-
1
-
2
·
2
λ
2
+
λ
μ
2
+
λ
σ
2
Z
α
2
+
λ
.Proofs are given in the Appendix.


### 3.3. Estimators forμ, σ
2, and λ

Having now computed a formula for the variance which is necessary for a confidence interval, we need to estimateμ, σ
2, and λ. In both cases, estimators of μ, σ
2, and λ can be calculated without distributional assumptions for the Z
i with the method of moments or with distributional assumptions regarding the Z
i.


#### 3.3.1. Method of Moments [44]

Whenk is fixed, we have
(23)
μ
^
=
∑
i
=
1
k
Z
^
i
k
,
σ
^
2
=
∑
i
=
1
k
Z
^
i
2
k
-
∑
i
=
1
k
Z
^
i
k
2
.
When k is random, we have
(24)
λ
^
=
k
,
μ
^
=
∑
i
=
1
k
Z
^
i
k
,
σ
^
2
=
∑
i
=
1
k
Z
^
i
2
k
-
∑
i
=
1
k
Z
^
i
k
2
.


#### 3.3.2. Distributional Assumptions for theZ
i

If we suppose that theZ
i follow a distribution, we would replace the values of μ and σ
2 with their distributional values. Below we consider special cases.Standard Normal Distribution. The Z
i follow a standard normal distribution; that is, Z
i
~
N
0,1. This is the original assumption for the Z
i [11]. In this case we have
(25)
λ
^
=
k
,
μ
=
0
,
σ
2
=
1
.
Although the origin of the Z
i is from the standard normal distribution, the studies in a meta-analysis are a selected sample of published studies. For this reason, the next distribution is suggested as better.Half Normal Distribution. Here we propose that the Z
i follow a half normal distribution HN0,1, which is a special case of folded normal distribution. Before we explain the rational of this distribution, a definition of this type of distribution is provided. A half normal distribution is also a special case of a truncated normal distribution.Definition 1.
Thefolded normal distribution is a probability distribution related to the normal distribution. Given a normally distributed random variable X with mean μ
f and variance σ
f
2, the random variable Y
=
X has a folded normal distribution [43, 45, 46].Remark 2.
The folded normal distribution has the following properties:(a)
probability density function (PDF):(26)
f
Y
y
=
1
σ
f
2
π
exp
⁡
-
-
y
-
μ
f
2
2
σ
f
2
+
1
σ
f
2
π
exp
⁡
-
y
-
μ
f
2
2
σ
f
2
,
h
h
h
h
h
h
l
l
h
h
h
h
h
h
h
h
h
h
h
for
y
≥
0
,
(b)
(27)
E
Y
=
σ
f
2
π
exp
⁡
-
μ
f
2
2
σ
f
2
+
μ
f
1
-
2
Φ
-
μ
f
σ
f
,
Var
⁡
Y
=
μ
f
2
+
σ
f
2
-
σ
f
2
π
exp
⁡
-
μ
f
2
2
σ
f
2
+
μ
f
1
-
2
Φ
-
μ
f
σ
f
2
.Remark 3.
Whenμ
f
=
0, the distribution of Y is ahalf normal distribution. This distribution is identical to thetruncated normal distribution, with left truncation point 0 and no right truncation point. For this distribution we have the following.(a)
f
Y
y
=
2
/
σ
f
π
exp
⁡
(
-
y
2
/
2
σ
f
2
), for y
≥
0.
(b)
E
Y
=
σ
f
2
/
π, Var
⁡
Y
=
σ
f
2
1
-
2
/
π.Assumption 4.
TheZ
i in Rosenthal’s estimator N
R are derived from a half normal distribution, based on a normal distribution N
0,1.Support. When a researcher begins to perform a meta-analysis, the sample of studies is drawn from those studies that are already published. So his sample is most likely biased by some sort of selection bias, produced via a specific selection process [47]. Thus, although when we study Rosenthal’s N
R assuming that all Z
i are drawn from the normal distribution, they are in essence drawn from a truncated normal distribution. This has been commented on by Iyengar and Greenhouse [12] and Schonemann and Scargle [48]. But at which point is this distribution truncated? We would like to advocate that the half normal distribution, based on a normal distribution N
0,1, is the one best representing the Z
i Rosenthal uses to compute his fail-safe N
R. The reasons for this are as follows.(1)
Firstly, assuming that allZ
i are of the same sign does not impede the significance of the results from each study. That is, the test is significant when either Z
i
>
Z
α
/
2 or Z
i
<
Z
1
-
α
/
2 occurs.
(2)
However, when a researcher begins to perform a meta-analysis of studies, many timesZ
i can be either positive or negative. Although this is true, when the researcher is interested in doing a meta-analysis, usually the Z
i that have been published are indicative of a significant effect of the same direction (thus Z
i have the same sign) or are at least indicative of such an association without being statistically significant, thus producing Z
i of the same sign but not producing significance (e.g., the confidence interval of the effect might include the null value).
(3)
There will definitely be studies that produce a totally opposite effect, thus producing an effect of opposite direction, but these will definitely be a minority of the studies. Also there is the case that these other signedZ
i are not significant.Hence, in this case
(28)
λ
^
=
k
,
μ
=
2
π
,
σ
2
=
1
-
2
π
.Skew Normal Distribution. Here we propose that the Z
i follow a skew normal distribution; that is, Z
i ~ SNξ
,
ω
,
α.Definition 5.
The skew normal distribution is a continuous probability distribution that generalises the normal distribution to allow for nonzero skewness. A random variableX follows a univariate skew normal distribution with location parameter ξ
∈
R, scale parameter ω
∈
R
+, and skewness parameter α
∈
R [49], if it has the density
(29)
f
X
x
=
2
ω
ϕ
x
-
ξ
ω
Φ
α
x
-
ξ
ω
x
∈
R
.
Note that if α
=
0, the density of X reduces to the N
(
ξ
,
ω
2
).Remark 1.
The expectation and variance ofX are [49]
(30)
E
X
=
ξ
+
ω
δ
2
π
,
where
δ
=
α
1
+
α
2
,
Var
⁡
X
=
ω
2
1
-
2
δ
2
π
.Remark 2.
The methods of moments estimators forξ
,
ω, and δ are [50, 51]
(31)
ξ
~
=
m
1
-
a
1
m
3
b
1
1
/
3
,
ω
~
2
=
m
2
-
a
1
2
m
3
b
1
2
/
3
,
δ
~
=
a
1
2
+
m
2
b
1
m
3
2
/
3
-
1
/
2
,
where a
1
=
2
/
π, b
1
=
4
/
π
-
1
a
1, m
1
=
n
-
1
∑
i
=
1
n
X
i, m
2
=
n
-
1
∑
i
=
1
n
X
i
-
m
1
2, and m
3
=
n
-
1
∑
i
=
1
n
X
i
-
m
1
3. The sign of  δ
~ is taken to be the sign of m
3.Explanation. The skew normal distribution allows for a dynamic way to fit the available Z-scores. The fact that there is ambiguity on the derivation of the standard deviates from each study from a normal or a truncated normal distribution creates the possibility that the distribution could be a skew normal, with the skewness being attributed that we are including only the published Z-scores in the estimation of Rosenthal’s [11] estimator.Hence, in this case and taking the method of moments estimators ofξ, ω, and δ, we get
(32)
λ
^
=
n
,
μ
^
=
ξ
~
+
ω
~
δ
~
2
π
,
σ
^
2
=
ω
~
2
1
-
2
δ
~
2
π
,
where a
1
=
2
/
π, b
1
=
4
/
π
-
1
a
1, m
1
=
n
-
1
∑
i
=
1
n
Z
i, m
2
=
n
-
1
∑
i
=
1
n
Z
i
-
m
1
2, and m
3
=
n
-
1
∑
i
=
1
n
Z
i
-
m
1
3.


### 3.4. Methods for Confidence Intervals

#### 3.4.1. Normal Approximation

In the previous section, formulas for computing the variance ofN
^
R were derived. We compute asymptotic 1
-
α
/
2% confidence intervals for N
R as
(33)
N
^
R
low
,
N
^
R
up
=
N
^
R
-
Z
1
-
α
/
2
Var
⁡
^
N
^
R
,
N
^
R
+
Z
1
-
α
/
2
Var
⁡
^
N
^
R
,
where Z
1
-
α
/
2 is the (
1
-
α
/
2
)th quantile of the standard normal distribution.The variance ofN
^
R for a given set of values Z
i depends firstly on whether the number of studies k is fixed or random and secondly on whether the estimators of μ, σ
2, and λ are derived from the method of moments or from the distributional assumptions.


#### 3.4.2. Nonparametric Bootstrap

Bootstrap is a well-known resampling methodology for obtaining nonparametric confidence intervals of a parameter [52, 53]. In most statistical problems one needs an estimator of a parameter of interest as well as some assessment of its variability. In many such problems the estimators are complicated functionals of the empirical distribution function and it is difficult to derive trustworthy analytical variance estimates for them. The primary objective of this technique is to estimate the sampling distribution of a statistic. Essentially, bootstrap is a method that mimics the process of sampling from a population, like one does in Monte Carlo simulations, but instead drawing samples from the observed sampling data. The tool of this mimic process is the Monte Carlo algorithm of Efron [54]. This process is explained properly by Efron and Tibshirani [55] and Davison and Hinkley [56], who also noted that bootstrap confidence intervals are approximate, yet better than the standard ones. Nevertheless, they do not try to replace the theoretical ones and bootstrap is not a substitute for precise parametric results, but rather a way to reasonably proceed when such results are unavailable.Nonparametric resampling makes no assumptions concerning the distribution of, or model for, the data [57]. Our data is assumed to be a vector Z
obs of k independent observations, and we are interested in a confidence interval for θ
^
Z
obs. The general algorithm for a nonparametric bootstrap is as follows.(1)
Samplek observations randomly with replacement from Z
obs to obtain a bootstrap data set, denoted by Z
*.
(2)
Calculate the bootstrap version of the statistic of interestθ
^
*
=
θ
^
Z
*.
(3)
Repeat steps (1) and (2) several times, sayB, to obtain an estimate of the bootstrap distribution.In our case(1)
compute a random sample from the initial sample ofZ
i, size k,
(2)
computeN
R
* from this sample,
(3)
repeat these processesb times.Then the bootstrap estimator of N
R is
(34)
N
R
_
bootstrap
=
∑
N
R
*
b
.
From this we can compute also confidence intervals for N
R
_
bootstrap.In the next section, we investigate these theoretical aspects with simulations and examples.


## 3.1. Probability Distribution Function ofN
^
R

### 3.1.1. Fixedk

We compute the PDF ofN
^
R by following the next steps.Step 1.
Z
1
,
Z
2
,
…
,
Z
i
,
…
,
Z
k in the formula of the estimator N
^
R  (6) are i.i.d. distributed with E
Z
i
=
μ and Var
⁡
Z
i
=
σ
2. Let S
=
∑
i
=
1
k
Z
i and according to the Lindeberg-Lévy Central Limit Theorem [42], we have
(7)
k
S
k
-
μ
⟶
d
N
0
,
σ
2
⟹
S
⟶
d
N
k
μ
,
k
σ
2
.
So the PDF of S is
(8)
f
S
s
=
1
2
π
k
σ
2
exp
⁡
-
s
-
k
μ
2
2
k
σ
2
.Step 2.
The PDF of Rosenthal’sN
^
R can be retrieved from a truncated version of (8). From (3), we get that
(9)
S
=
Z
α
N
^
R
+
k
.
We advocate that Rosenthal’s equations (3) and (9) implicitly impose two conditions which must be taken into account when we seek to estimate the distribution of N
R:
(10)
S
≥
0
,
(11)
N
^
R
≥
0
.
Expression (10) is justified by the fact that the right hand side of  (9) is positive, so then S
≥
0. Expression (11) is justified by the fact that N
R expresses the number of studies, so it must be at least 0. Hence, expressions (10) and (11) are satisfied when S is a truncated normal random variable.
Thetruncated normal distribution is a probability distribution related to the normal distribution. Given a normally distributed random variable X with mean μ
t and variance σ
t
2, let it be that X
∈
(
a
,
b
)
,
-
∞
≤
a
≤
b
≤
∞. Then X conditional on a
<
X
<
b has a truncated normal distribution with PDF: f
X
(
x
)
=
(
1
/
σ
)
ϕ
(
(
x
-
μ
t
)
/
σ
t
)
/
(
Φ
(
(
b
-
μ
t
)
/
σ
t
)
-
Φ
(
(
a − μ
t
)
/
σ
t
)
), for a
≤
x
≤
b and f
X
(
x
)
=
0 otherwise [43].
Let it beS
*, such that S
*
≥
Z
α
k. So the PDF of S
* then becomes
(12)
f
S
*
s
*
=
1
Φ
λ
*
2
π
k
σ
2
exp
⁡
-
s
*
-
k
μ
2
2
k
σ
2
,
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
i
h
h
h
h
h
s
*
≥
Z
α
k
,
where λ
*
=
(
k
μ
-
Z
α
)
/
σ.Then, we have(13)
f
N
^
R
n
R
=
f
S
*
s
*
d
S
*
d
N
R
⟹
9
,
12
f
N
^
R
n
R
=
Z
α
2
Φ
λ
*
2
π
k
σ
2
n
R
+
k
×
exp
⁡
-
Z
α
n
R
+
k
-
k
μ
2
2
k
σ
2
,
n
R
≥
0
.
The characteristic function is
(14)
ψ
N
^
R
t
=
E
exp
⁡
i
t
N
R
=
Φ
μ
1
+
λ
*
/
σ
1
Φ
λ
*
·
Z
α
exp
⁡
k
2
μ
2
i
t
/
(
Z
α
2
-
2
k
σ
2
i
t
)
-
k
i
t
Z
α
2
-
2
k
σ
2
i
t
1
/
2
,
where i
=
-
1, μ
1
=
2
k
k
μ
σ
i
t
/
(
Z
α
2
-
2
k
σ
2
i
t
), σ
1
2
=
Z
α
2
/
(
Z
α
2
-
2
k
σ
2
i
t
).From (14) we get
(15)
E
N
^
R
=
k
2
μ
2
+
k
σ
2
Z
α
2
-
k
+
ε
,
where ε
=
(
ϕ
λ
*
/
Φ
λ
*
)
·
(
k
σ
(
k
μ
+
Z
α
)
/
Z
α
2
).Also,(16)
Var
⁡
N
^
R
=
2
k
2
σ
2
2
k
μ
2
+
σ
2
Z
α
4
+
δ
*
,
where
(17)
δ
*
=
ϕ
λ
*
Φ
λ
*
k
2
σ
3
5
k
μ
+
Z
a
2
Z
α
4
h
h
h
h
h
h
h
-
ϕ
λ
*
Φ
λ
*
+
λ
*
k
3
/
2
σ
2
k
μ
+
Z
a
2
Z
α
4
.Proofs for expressions (14), (15), and (16) are given in the Appendix.Comments. Consider the following:(1)
For a significantly largek we have that Φ
λ
*
≈
1. So (13) becomes
(18)
f
N
^
R
n
R
=
Z
α
2
2
π
k
σ
2
n
R
+
k
×
exp
⁡
-
Z
α
n
R
+
k
-
k
μ
2
2
k
σ
2
,
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
n
R
≥
0
.
Also we get
(19)
E
N
^
R
=
k
2
μ
2
+
k
σ
2
Z
α
2
-
k
,
(20)
Var
⁡
N
^
R
=
2
k
2
σ
2
2
k
μ
2
+
σ
2
Z
α
4
.
(2)
A limiting element of this computation is thatN
^
R takes discrete values because it describes number of studies, but it has been described by a continuous distribution.


### 3.1.2. Randomk

It is assumed thatk
~
Pois
λ. So taking into account the result from the distribution of N
^
R for a fixed k we get that the joint distribution of k and N
^
R is
(21)
f
N
^
R
,
n
n
R
,
k
=
f
N
^
R
n
R
∣
k
=
k
·
p
k
=
k
⟹
f
N
^
R
,
n
n
R
,
k
=
Z
α
2
Φ
λ
*
2
π
k
σ
2
n
R
+
k
×
exp
⁡
-
Z
α
n
R
+
k
-
k
μ
2
2
k
σ
2
-
λ
·
λ
k
k
!
,
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
n
R
≥
0
,
k
=
0,1
,
2
,
…
.


## 3.1.1. Fixedk

We compute the PDF ofN
^
R by following the next steps.Step 1.
Z
1
,
Z
2
,
…
,
Z
i
,
…
,
Z
k in the formula of the estimator N
^
R  (6) are i.i.d. distributed with E
Z
i
=
μ and Var
⁡
Z
i
=
σ
2. Let S
=
∑
i
=
1
k
Z
i and according to the Lindeberg-Lévy Central Limit Theorem [42], we have
(7)
k
S
k
-
μ
⟶
d
N
0
,
σ
2
⟹
S
⟶
d
N
k
μ
,
k
σ
2
.
So the PDF of S is
(8)
f
S
s
=
1
2
π
k
σ
2
exp
⁡
-
s
-
k
μ
2
2
k
σ
2
.Step 2.
The PDF of Rosenthal’sN
^
R can be retrieved from a truncated version of (8). From (3), we get that
(9)
S
=
Z
α
N
^
R
+
k
.
We advocate that Rosenthal’s equations (3) and (9) implicitly impose two conditions which must be taken into account when we seek to estimate the distribution of N
R:
(10)
S
≥
0
,
(11)
N
^
R
≥
0
.
Expression (10) is justified by the fact that the right hand side of  (9) is positive, so then S
≥
0. Expression (11) is justified by the fact that N
R expresses the number of studies, so it must be at least 0. Hence, expressions (10) and (11) are satisfied when S is a truncated normal random variable.
Thetruncated normal distribution is a probability distribution related to the normal distribution. Given a normally distributed random variable X with mean μ
t and variance σ
t
2, let it be that X
∈
(
a
,
b
)
,
-
∞
≤
a
≤
b
≤
∞. Then X conditional on a
<
X
<
b has a truncated normal distribution with PDF: f
X
(
x
)
=
(
1
/
σ
)
ϕ
(
(
x
-
μ
t
)
/
σ
t
)
/
(
Φ
(
(
b
-
μ
t
)
/
σ
t
)
-
Φ
(
(
a − μ
t
)
/
σ
t
)
), for a
≤
x
≤
b and f
X
(
x
)
=
0 otherwise [43].
Let it beS
*, such that S
*
≥
Z
α
k. So the PDF of S
* then becomes
(12)
f
S
*
s
*
=
1
Φ
λ
*
2
π
k
σ
2
exp
⁡
-
s
*
-
k
μ
2
2
k
σ
2
,
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
i
h
h
h
h
h
s
*
≥
Z
α
k
,
where λ
*
=
(
k
μ
-
Z
α
)
/
σ.Then, we have(13)
f
N
^
R
n
R
=
f
S
*
s
*
d
S
*
d
N
R
⟹
9
,
12
f
N
^
R
n
R
=
Z
α
2
Φ
λ
*
2
π
k
σ
2
n
R
+
k
×
exp
⁡
-
Z
α
n
R
+
k
-
k
μ
2
2
k
σ
2
,
n
R
≥
0
.
The characteristic function is
(14)
ψ
N
^
R
t
=
E
exp
⁡
i
t
N
R
=
Φ
μ
1
+
λ
*
/
σ
1
Φ
λ
*
·
Z
α
exp
⁡
k
2
μ
2
i
t
/
(
Z
α
2
-
2
k
σ
2
i
t
)
-
k
i
t
Z
α
2
-
2
k
σ
2
i
t
1
/
2
,
where i
=
-
1, μ
1
=
2
k
k
μ
σ
i
t
/
(
Z
α
2
-
2
k
σ
2
i
t
), σ
1
2
=
Z
α
2
/
(
Z
α
2
-
2
k
σ
2
i
t
).From (14) we get
(15)
E
N
^
R
=
k
2
μ
2
+
k
σ
2
Z
α
2
-
k
+
ε
,
where ε
=
(
ϕ
λ
*
/
Φ
λ
*
)
·
(
k
σ
(
k
μ
+
Z
α
)
/
Z
α
2
).Also,(16)
Var
⁡
N
^
R
=
2
k
2
σ
2
2
k
μ
2
+
σ
2
Z
α
4
+
δ
*
,
where
(17)
δ
*
=
ϕ
λ
*
Φ
λ
*
k
2
σ
3
5
k
μ
+
Z
a
2
Z
α
4
h
h
h
h
h
h
h
-
ϕ
λ
*
Φ
λ
*
+
λ
*
k
3
/
2
σ
2
k
μ
+
Z
a
2
Z
α
4
.Proofs for expressions (14), (15), and (16) are given in the Appendix.Comments. Consider the following:(1)
For a significantly largek we have that Φ
λ
*
≈
1. So (13) becomes
(18)
f
N
^
R
n
R
=
Z
α
2
2
π
k
σ
2
n
R
+
k
×
exp
⁡
-
Z
α
n
R
+
k
-
k
μ
2
2
k
σ
2
,
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
n
R
≥
0
.
Also we get
(19)
E
N
^
R
=
k
2
μ
2
+
k
σ
2
Z
α
2
-
k
,
(20)
Var
⁡
N
^
R
=
2
k
2
σ
2
2
k
μ
2
+
σ
2
Z
α
4
.
(2)
A limiting element of this computation is thatN
^
R takes discrete values because it describes number of studies, but it has been described by a continuous distribution.


## 3.1.2. Randomk

It is assumed thatk
~
Pois
λ. So taking into account the result from the distribution of N
^
R for a fixed k we get that the joint distribution of k and N
^
R is
(21)
f
N
^
R
,
n
n
R
,
k
=
f
N
^
R
n
R
∣
k
=
k
·
p
k
=
k
⟹
f
N
^
R
,
n
n
R
,
k
=
Z
α
2
Φ
λ
*
2
π
k
σ
2
n
R
+
k
×
exp
⁡
-
Z
α
n
R
+
k
-
k
μ
2
2
k
σ
2
-
λ
·
λ
k
k
!
,
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
n
R
≥
0
,
k
=
0,1
,
2
,
…
.


## 3.2. Expectation and Variance for Rosenthal’s EstimatorN
^
R

(a)
Whenk is fixed, expressions (19) and (20) denote the expectation and variance, respectively, for N
^
R. This is derived from the PDF of N
^
R; an additional proof without reference to the PDF is given in the Appendix.
(b)
Whenk is random with k
~
Pois
λ, the expectation and variance of N
^
R are
(22)
E
N
^
R
=
λ
2
μ
2
+
λ
μ
2
+
σ
2
Z
α
2
-
λ
,
Var
⁡
N
^
R
=
4
λ
3
+
6
λ
2
+
λ
μ
4
+
4
λ
3
+
16
λ
2
+
6
λ
μ
2
σ
2
+
2
λ
2
+
3
λ
σ
4
×
Z
α
4
-
1
-
2
·
2
λ
2
+
λ
μ
2
+
λ
σ
2
Z
α
2
+
λ
.Proofs are given in the Appendix.


## 3.3. Estimators forμ, σ
2, and λ

Having now computed a formula for the variance which is necessary for a confidence interval, we need to estimateμ, σ
2, and λ. In both cases, estimators of μ, σ
2, and λ can be calculated without distributional assumptions for the Z
i with the method of moments or with distributional assumptions regarding the Z
i.


### 3.3.1. Method of Moments [44]

Whenk is fixed, we have
(23)
μ
^
=
∑
i
=
1
k
Z
^
i
k
,
σ
^
2
=
∑
i
=
1
k
Z
^
i
2
k
-
∑
i
=
1
k
Z
^
i
k
2
.
When k is random, we have
(24)
λ
^
=
k
,
μ
^
=
∑
i
=
1
k
Z
^
i
k
,
σ
^
2
=
∑
i
=
1
k
Z
^
i
2
k
-
∑
i
=
1
k
Z
^
i
k
2
.


### 3.3.2. Distributional Assumptions for theZ
i

If we suppose that theZ
i follow a distribution, we would replace the values of μ and σ
2 with their distributional values. Below we consider special cases.Standard Normal Distribution. The Z
i follow a standard normal distribution; that is, Z
i
~
N
0,1. This is the original assumption for the Z
i [11]. In this case we have
(25)
λ
^
=
k
,
μ
=
0
,
σ
2
=
1
.
Although the origin of the Z
i is from the standard normal distribution, the studies in a meta-analysis are a selected sample of published studies. For this reason, the next distribution is suggested as better.Half Normal Distribution. Here we propose that the Z
i follow a half normal distribution HN0,1, which is a special case of folded normal distribution. Before we explain the rational of this distribution, a definition of this type of distribution is provided. A half normal distribution is also a special case of a truncated normal distribution.Definition 1.
Thefolded normal distribution is a probability distribution related to the normal distribution. Given a normally distributed random variable X with mean μ
f and variance σ
f
2, the random variable Y
=
X has a folded normal distribution [43, 45, 46].Remark 2.
The folded normal distribution has the following properties:(a)
probability density function (PDF):(26)
f
Y
y
=
1
σ
f
2
π
exp
⁡
-
-
y
-
μ
f
2
2
σ
f
2
+
1
σ
f
2
π
exp
⁡
-
y
-
μ
f
2
2
σ
f
2
,
h
h
h
h
h
h
l
l
h
h
h
h
h
h
h
h
h
h
h
for
y
≥
0
,
(b)
(27)
E
Y
=
σ
f
2
π
exp
⁡
-
μ
f
2
2
σ
f
2
+
μ
f
1
-
2
Φ
-
μ
f
σ
f
,
Var
⁡
Y
=
μ
f
2
+
σ
f
2
-
σ
f
2
π
exp
⁡
-
μ
f
2
2
σ
f
2
+
μ
f
1
-
2
Φ
-
μ
f
σ
f
2
.Remark 3.
Whenμ
f
=
0, the distribution of Y is ahalf normal distribution. This distribution is identical to thetruncated normal distribution, with left truncation point 0 and no right truncation point. For this distribution we have the following.(a)
f
Y
y
=
2
/
σ
f
π
exp
⁡
(
-
y
2
/
2
σ
f
2
), for y
≥
0.
(b)
E
Y
=
σ
f
2
/
π, Var
⁡
Y
=
σ
f
2
1
-
2
/
π.Assumption 4.
TheZ
i in Rosenthal’s estimator N
R are derived from a half normal distribution, based on a normal distribution N
0,1.Support. When a researcher begins to perform a meta-analysis, the sample of studies is drawn from those studies that are already published. So his sample is most likely biased by some sort of selection bias, produced via a specific selection process [47]. Thus, although when we study Rosenthal’s N
R assuming that all Z
i are drawn from the normal distribution, they are in essence drawn from a truncated normal distribution. This has been commented on by Iyengar and Greenhouse [12] and Schonemann and Scargle [48]. But at which point is this distribution truncated? We would like to advocate that the half normal distribution, based on a normal distribution N
0,1, is the one best representing the Z
i Rosenthal uses to compute his fail-safe N
R. The reasons for this are as follows.(1)
Firstly, assuming that allZ
i are of the same sign does not impede the significance of the results from each study. That is, the test is significant when either Z
i
>
Z
α
/
2 or Z
i
<
Z
1
-
α
/
2 occurs.
(2)
However, when a researcher begins to perform a meta-analysis of studies, many timesZ
i can be either positive or negative. Although this is true, when the researcher is interested in doing a meta-analysis, usually the Z
i that have been published are indicative of a significant effect of the same direction (thus Z
i have the same sign) or are at least indicative of such an association without being statistically significant, thus producing Z
i of the same sign but not producing significance (e.g., the confidence interval of the effect might include the null value).
(3)
There will definitely be studies that produce a totally opposite effect, thus producing an effect of opposite direction, but these will definitely be a minority of the studies. Also there is the case that these other signedZ
i are not significant.Hence, in this case
(28)
λ
^
=
k
,
μ
=
2
π
,
σ
2
=
1
-
2
π
.Skew Normal Distribution. Here we propose that the Z
i follow a skew normal distribution; that is, Z
i ~ SNξ
,
ω
,
α.Definition 5.
The skew normal distribution is a continuous probability distribution that generalises the normal distribution to allow for nonzero skewness. A random variableX follows a univariate skew normal distribution with location parameter ξ
∈
R, scale parameter ω
∈
R
+, and skewness parameter α
∈
R [49], if it has the density
(29)
f
X
x
=
2
ω
ϕ
x
-
ξ
ω
Φ
α
x
-
ξ
ω
x
∈
R
.
Note that if α
=
0, the density of X reduces to the N
(
ξ
,
ω
2
).Remark 1.
The expectation and variance ofX are [49]
(30)
E
X
=
ξ
+
ω
δ
2
π
,
where
δ
=
α
1
+
α
2
,
Var
⁡
X
=
ω
2
1
-
2
δ
2
π
.Remark 2.
The methods of moments estimators forξ
,
ω, and δ are [50, 51]
(31)
ξ
~
=
m
1
-
a
1
m
3
b
1
1
/
3
,
ω
~
2
=
m
2
-
a
1
2
m
3
b
1
2
/
3
,
δ
~
=
a
1
2
+
m
2
b
1
m
3
2
/
3
-
1
/
2
,
where a
1
=
2
/
π, b
1
=
4
/
π
-
1
a
1, m
1
=
n
-
1
∑
i
=
1
n
X
i, m
2
=
n
-
1
∑
i
=
1
n
X
i
-
m
1
2, and m
3
=
n
-
1
∑
i
=
1
n
X
i
-
m
1
3. The sign of  δ
~ is taken to be the sign of m
3.Explanation. The skew normal distribution allows for a dynamic way to fit the available Z-scores. The fact that there is ambiguity on the derivation of the standard deviates from each study from a normal or a truncated normal distribution creates the possibility that the distribution could be a skew normal, with the skewness being attributed that we are including only the published Z-scores in the estimation of Rosenthal’s [11] estimator.Hence, in this case and taking the method of moments estimators ofξ, ω, and δ, we get
(32)
λ
^
=
n
,
μ
^
=
ξ
~
+
ω
~
δ
~
2
π
,
σ
^
2
=
ω
~
2
1
-
2
δ
~
2
π
,
where a
1
=
2
/
π, b
1
=
4
/
π
-
1
a
1, m
1
=
n
-
1
∑
i
=
1
n
Z
i, m
2
=
n
-
1
∑
i
=
1
n
Z
i
-
m
1
2, and m
3
=
n
-
1
∑
i
=
1
n
Z
i
-
m
1
3.


## 3.3.1. Method of Moments [44]

Whenk is fixed, we have
(23)
μ
^
=
∑
i
=
1
k
Z
^
i
k
,
σ
^
2
=
∑
i
=
1
k
Z
^
i
2
k
-
∑
i
=
1
k
Z
^
i
k
2
.
When k is random, we have
(24)
λ
^
=
k
,
μ
^
=
∑
i
=
1
k
Z
^
i
k
,
σ
^
2
=
∑
i
=
1
k
Z
^
i
2
k
-
∑
i
=
1
k
Z
^
i
k
2
.


## 3.3.2. Distributional Assumptions for theZ
i

If we suppose that theZ
i follow a distribution, we would replace the values of μ and σ
2 with their distributional values. Below we consider special cases.Standard Normal Distribution. The Z
i follow a standard normal distribution; that is, Z
i
~
N
0,1. This is the original assumption for the Z
i [11]. In this case we have
(25)
λ
^
=
k
,
μ
=
0
,
σ
2
=
1
.
Although the origin of the Z
i is from the standard normal distribution, the studies in a meta-analysis are a selected sample of published studies. For this reason, the next distribution is suggested as better.Half Normal Distribution. Here we propose that the Z
i follow a half normal distribution HN0,1, which is a special case of folded normal distribution. Before we explain the rational of this distribution, a definition of this type of distribution is provided. A half normal distribution is also a special case of a truncated normal distribution.Definition 1.
Thefolded normal distribution is a probability distribution related to the normal distribution. Given a normally distributed random variable X with mean μ
f and variance σ
f
2, the random variable Y
=
X has a folded normal distribution [43, 45, 46].Remark 2.
The folded normal distribution has the following properties:(a)
probability density function (PDF):(26)
f
Y
y
=
1
σ
f
2
π
exp
⁡
-
-
y
-
μ
f
2
2
σ
f
2
+
1
σ
f
2
π
exp
⁡
-
y
-
μ
f
2
2
σ
f
2
,
h
h
h
h
h
h
l
l
h
h
h
h
h
h
h
h
h
h
h
for
y
≥
0
,
(b)
(27)
E
Y
=
σ
f
2
π
exp
⁡
-
μ
f
2
2
σ
f
2
+
μ
f
1
-
2
Φ
-
μ
f
σ
f
,
Var
⁡
Y
=
μ
f
2
+
σ
f
2
-
σ
f
2
π
exp
⁡
-
μ
f
2
2
σ
f
2
+
μ
f
1
-
2
Φ
-
μ
f
σ
f
2
.Remark 3.
Whenμ
f
=
0, the distribution of Y is ahalf normal distribution. This distribution is identical to thetruncated normal distribution, with left truncation point 0 and no right truncation point. For this distribution we have the following.(a)
f
Y
y
=
2
/
σ
f
π
exp
⁡
(
-
y
2
/
2
σ
f
2
), for y
≥
0.
(b)
E
Y
=
σ
f
2
/
π, Var
⁡
Y
=
σ
f
2
1
-
2
/
π.Assumption 4.
TheZ
i in Rosenthal’s estimator N
R are derived from a half normal distribution, based on a normal distribution N
0,1.Support. When a researcher begins to perform a meta-analysis, the sample of studies is drawn from those studies that are already published. So his sample is most likely biased by some sort of selection bias, produced via a specific selection process [47]. Thus, although when we study Rosenthal’s N
R assuming that all Z
i are drawn from the normal distribution, they are in essence drawn from a truncated normal distribution. This has been commented on by Iyengar and Greenhouse [12] and Schonemann and Scargle [48]. But at which point is this distribution truncated? We would like to advocate that the half normal distribution, based on a normal distribution N
0,1, is the one best representing the Z
i Rosenthal uses to compute his fail-safe N
R. The reasons for this are as follows.(1)
Firstly, assuming that allZ
i are of the same sign does not impede the significance of the results from each study. That is, the test is significant when either Z
i
>
Z
α
/
2 or Z
i
<
Z
1
-
α
/
2 occurs.
(2)
However, when a researcher begins to perform a meta-analysis of studies, many timesZ
i can be either positive or negative. Although this is true, when the researcher is interested in doing a meta-analysis, usually the Z
i that have been published are indicative of a significant effect of the same direction (thus Z
i have the same sign) or are at least indicative of such an association without being statistically significant, thus producing Z
i of the same sign but not producing significance (e.g., the confidence interval of the effect might include the null value).
(3)
There will definitely be studies that produce a totally opposite effect, thus producing an effect of opposite direction, but these will definitely be a minority of the studies. Also there is the case that these other signedZ
i are not significant.Hence, in this case
(28)
λ
^
=
k
,
μ
=
2
π
,
σ
2
=
1
-
2
π
.Skew Normal Distribution. Here we propose that the Z
i follow a skew normal distribution; that is, Z
i ~ SNξ
,
ω
,
α.Definition 5.
The skew normal distribution is a continuous probability distribution that generalises the normal distribution to allow for nonzero skewness. A random variableX follows a univariate skew normal distribution with location parameter ξ
∈
R, scale parameter ω
∈
R
+, and skewness parameter α
∈
R [49], if it has the density
(29)
f
X
x
=
2
ω
ϕ
x
-
ξ
ω
Φ
α
x
-
ξ
ω
x
∈
R
.
Note that if α
=
0, the density of X reduces to the N
(
ξ
,
ω
2
).Remark 1.
The expectation and variance ofX are [49]
(30)
E
X
=
ξ
+
ω
δ
2
π
,
where
δ
=
α
1
+
α
2
,
Var
⁡
X
=
ω
2
1
-
2
δ
2
π
.Remark 2.
The methods of moments estimators forξ
,
ω, and δ are [50, 51]
(31)
ξ
~
=
m
1
-
a
1
m
3
b
1
1
/
3
,
ω
~
2
=
m
2
-
a
1
2
m
3
b
1
2
/
3
,
δ
~
=
a
1
2
+
m
2
b
1
m
3
2
/
3
-
1
/
2
,
where a
1
=
2
/
π, b
1
=
4
/
π
-
1
a
1, m
1
=
n
-
1
∑
i
=
1
n
X
i, m
2
=
n
-
1
∑
i
=
1
n
X
i
-
m
1
2, and m
3
=
n
-
1
∑
i
=
1
n
X
i
-
m
1
3. The sign of  δ
~ is taken to be the sign of m
3.Explanation. The skew normal distribution allows for a dynamic way to fit the available Z-scores. The fact that there is ambiguity on the derivation of the standard deviates from each study from a normal or a truncated normal distribution creates the possibility that the distribution could be a skew normal, with the skewness being attributed that we are including only the published Z-scores in the estimation of Rosenthal’s [11] estimator.Hence, in this case and taking the method of moments estimators ofξ, ω, and δ, we get
(32)
λ
^
=
n
,
μ
^
=
ξ
~
+
ω
~
δ
~
2
π
,
σ
^
2
=
ω
~
2
1
-
2
δ
~
2
π
,
where a
1
=
2
/
π, b
1
=
4
/
π
-
1
a
1, m
1
=
n
-
1
∑
i
=
1
n
Z
i, m
2
=
n
-
1
∑
i
=
1
n
Z
i
-
m
1
2, and m
3
=
n
-
1
∑
i
=
1
n
Z
i
-
m
1
3.


## 3.4. Methods for Confidence Intervals

### 3.4.1. Normal Approximation

In the previous section, formulas for computing the variance ofN
^
R were derived. We compute asymptotic 1
-
α
/
2% confidence intervals for N
R as
(33)
N
^
R
low
,
N
^
R
up
=
N
^
R
-
Z
1
-
α
/
2
Var
⁡
^
N
^
R
,
N
^
R
+
Z
1
-
α
/
2
Var
⁡
^
N
^
R
,
where Z
1
-
α
/
2 is the (
1
-
α
/
2
)th quantile of the standard normal distribution.The variance ofN
^
R for a given set of values Z
i depends firstly on whether the number of studies k is fixed or random and secondly on whether the estimators of μ, σ
2, and λ are derived from the method of moments or from the distributional assumptions.


### 3.4.2. Nonparametric Bootstrap

Bootstrap is a well-known resampling methodology for obtaining nonparametric confidence intervals of a parameter [52, 53]. In most statistical problems one needs an estimator of a parameter of interest as well as some assessment of its variability. In many such problems the estimators are complicated functionals of the empirical distribution function and it is difficult to derive trustworthy analytical variance estimates for them. The primary objective of this technique is to estimate the sampling distribution of a statistic. Essentially, bootstrap is a method that mimics the process of sampling from a population, like one does in Monte Carlo simulations, but instead drawing samples from the observed sampling data. The tool of this mimic process is the Monte Carlo algorithm of Efron [54]. This process is explained properly by Efron and Tibshirani [55] and Davison and Hinkley [56], who also noted that bootstrap confidence intervals are approximate, yet better than the standard ones. Nevertheless, they do not try to replace the theoretical ones and bootstrap is not a substitute for precise parametric results, but rather a way to reasonably proceed when such results are unavailable.Nonparametric resampling makes no assumptions concerning the distribution of, or model for, the data [57]. Our data is assumed to be a vector Z
obs of k independent observations, and we are interested in a confidence interval for θ
^
Z
obs. The general algorithm for a nonparametric bootstrap is as follows.(1)
Samplek observations randomly with replacement from Z
obs to obtain a bootstrap data set, denoted by Z
*.
(2)
Calculate the bootstrap version of the statistic of interestθ
^
*
=
θ
^
Z
*.
(3)
Repeat steps (1) and (2) several times, sayB, to obtain an estimate of the bootstrap distribution.In our case(1)
compute a random sample from the initial sample ofZ
i, size k,
(2)
computeN
R
* from this sample,
(3)
repeat these processesb times.Then the bootstrap estimator of N
R is
(34)
N
R
_
bootstrap
=
∑
N
R
*
b
.
From this we can compute also confidence intervals for N
R
_
bootstrap.In the next section, we investigate these theoretical aspects with simulations and examples.


## 3.4.1. Normal Approximation

In the previous section, formulas for computing the variance ofN
^
R were derived. We compute asymptotic 1
-
α
/
2% confidence intervals for N
R as
(33)
N
^
R
low
,
N
^
R
up
=
N
^
R
-
Z
1
-
α
/
2
Var
⁡
^
N
^
R
,
N
^
R
+
Z
1
-
α
/
2
Var
⁡
^
N
^
R
,
where Z
1
-
α
/
2 is the (
1
-
α
/
2
)th quantile of the standard normal distribution.The variance ofN
^
R for a given set of values Z
i depends firstly on whether the number of studies k is fixed or random and secondly on whether the estimators of μ, σ
2, and λ are derived from the method of moments or from the distributional assumptions.


## 3.4.2. Nonparametric Bootstrap

Bootstrap is a well-known resampling methodology for obtaining nonparametric confidence intervals of a parameter [52, 53]. In most statistical problems one needs an estimator of a parameter of interest as well as some assessment of its variability. In many such problems the estimators are complicated functionals of the empirical distribution function and it is difficult to derive trustworthy analytical variance estimates for them. The primary objective of this technique is to estimate the sampling distribution of a statistic. Essentially, bootstrap is a method that mimics the process of sampling from a population, like one does in Monte Carlo simulations, but instead drawing samples from the observed sampling data. The tool of this mimic process is the Monte Carlo algorithm of Efron [54]. This process is explained properly by Efron and Tibshirani [55] and Davison and Hinkley [56], who also noted that bootstrap confidence intervals are approximate, yet better than the standard ones. Nevertheless, they do not try to replace the theoretical ones and bootstrap is not a substitute for precise parametric results, but rather a way to reasonably proceed when such results are unavailable.Nonparametric resampling makes no assumptions concerning the distribution of, or model for, the data [57]. Our data is assumed to be a vector Z
obs of k independent observations, and we are interested in a confidence interval for θ
^
Z
obs. The general algorithm for a nonparametric bootstrap is as follows.(1)
Samplek observations randomly with replacement from Z
obs to obtain a bootstrap data set, denoted by Z
*.
(2)
Calculate the bootstrap version of the statistic of interestθ
^
*
=
θ
^
Z
*.
(3)
Repeat steps (1) and (2) several times, sayB, to obtain an estimate of the bootstrap distribution.In our case(1)
compute a random sample from the initial sample ofZ
i, size k,
(2)
computeN
R
* from this sample,
(3)
repeat these processesb times.Then the bootstrap estimator of N
R is
(34)
N
R
_
bootstrap
=
∑
N
R
*
b
.
From this we can compute also confidence intervals for N
R
_
bootstrap.In the next section, we investigate these theoretical aspects with simulations and examples.


## 4. Simulations and Results

The method for simulations is as follows.(1)
Initially we draw random numbers from the following distributions:
(a)
standard normal distribution,
(b)
half normal distribution(
0,1
),
(c)
skew normal distribution with negative skewness SN(δ
=
-
0.5
,
ξ
=
0
,
ω
=
1),
(d)
skew normal distribution with positive skewness SN(δ
=
0.5
,
ξ
=
0
,
ω
=
1).
(2)
The numbers drawn from each distribution represent the number of studies in a meta-analysis and we have chosenk = 5, 15, 30, and 50. When k is assumed to be random, then the parameter λ is equal to the values chosen for the simulation, that is, 5, 15, 30, and 50, respectively.
(3)
We compute the normal approximation confidence interval with the formulas described in Section3 and the bootstrap confidence interval. We also discern whether the number of studies is fixed or random. For the computation of the bootstrap confidence interval, we generate 1,000 bootstrap samples each time. We also study the performance of the different distributional estimators in cases where the distributional assumption is not met, thus comparing each of the six confidence interval estimators under all four distributions.
(4)
We compute the coverage probability comparing with the true value of Rosenthal’s fail-safe number. When the number of studies is fixed the true value of Rosenthal’s number is(35)
E
N
^
R
=
k
2
μ
2
+
k
σ
2
Z
α
2
-
k
.
When the number of studies is random [from a Poisson (λ) distribution] the true value of Rosenthal’s number is
(36)
E
N
^
R
=
λ
2
μ
2
+
λ
μ
2
+
σ
2
Z
α
2
-
λ
.
We execute the above procedure 10,000 times each time. Our alpha-level is considered 5%.This process is shown schematically in Table1. All simulations were performed in R and the code is shown in the Supplementary Materials (see Supplementary Materials available online at http://dx.doi.org/10.1155/2014/825383).Table 1
Schematic table for simulation plan.
Variance formula for normal approximation confidence intervals
Bootstrapconfidence interval
Realvalue of N
R
Distributional
Moments
k
=
5,15,30,50 studiesDraw Z
i from N
(
0,1
),
HN
(
0,1
), SN(
δ
=
-
0.5
,
ξ
=
0
,
ω
=
1
), SN(
δ
=
0.5
,
ξ
=
0
,
ω
=
1
)
Fixedk (standard normal distribution values μ
=
0, σ
2
=
1; half normal distribution μ
=
2
/
π, σ
2
=
1
-
2
/
π; skew normal with negative skewness μ
=
-
1
/
2
π, σ
2
=
1
-
1
/
2
π; skew normal with positive skewness μ
=
1
/
2
π, σ
2
=
1
-
1
/
2
π)  Var
⁡
[
N
^
R
]
=
2
k
2
σ
2
(
2
k
μ
2
+
σ
2
)
Z
α
4
Fixedk
(
μ
^
=
∑
i
=
1
k
Z
^
i
k
,
σ
^
2
=
∑
i
=
1
k
Z
^
i
2
k
-
(
∑
i
=
1
k
Z
^
i
k
)
2
)     Var
⁡
[
N
^
R
]
=
2
k
2
σ
2
(
2
k
μ
2
+
σ
2
)
Z
α
4
Using theN
R
* we compute the N
R
_bootstrap and respectively the standard error needed to compute the confidence interval
Fixedk (standard normal distribution values μ
=
0, σ
2
=
1; half normal distribution μ
=
2
/
π, σ
2
=
1
-
2
/
π; skew normal with negative skewness μ
=
-
1
/
2
π, σ
2
=
1
-
1
/
2
π; skew normal with positive skewness μ
=
1
/
2
π, σ
2
=
1
-
1
/
2
π)
E
[
N
^
R
]
=
k
2
μ
2
+
k
σ
2
Z
α
2
-
k
Randomk (standard normal distribution values μ
=
0, σ
2
=
1; half normal distribution μ
=
2
/
π, σ
2
=
1
-
2
/
π; skew normal with negative skewness μ
=
-
1
/
2
π, σ
2
=
1
-
1
/
2
π; skew normal with positive skewness μ
=
1
/
2
π, σ
2
=
1
-
1
/
2
π; λ
=
5,15,30,50)
Var
⁡
[
N
^
R
]



=
(
4
λ
3
+
6
λ
2
+
λ
)
μ
4
+
(
4
λ
3
+
16
λ
2
+
6
λ
)
μ
2
σ
2
Z
α
4
+
(
2
λ
2
+
3
λ
)
σ
4
Z
α
4
-
2
·
(
2
λ
2
+
λ
)
μ
2
+
λ
σ
2
Z
α
2
+
λ
Random
k
(
μ
^
=
∑
i
=
1
k
Z
^
i
k
,
σ
^
2
=
∑
i
=
1
k
Z
^
i
2
k
-
(
∑
i
=
1
k
Z
^
i
k
)
2
,
λ
=
5,15,30,50
(
μ
^
=
∑
i
=
1
k
Z
^
i
k
,
σ
^
2
=
∑
i
=
1
k
Z
^
i
2
k
-
(
∑
i
=
1
k
Z
^
i
k
)
2
)
Var
⁡
[
N
^
R
]



=
(
4
λ
3
+
6
λ
2
+
λ
)
μ
4
+
(
4
λ
3
+
16
λ
2
+
6
λ
)
μ
2
σ
2
Z
α
4
+
(
2
λ
2
+
3
λ
)
σ
4
Z
α
4
-
2
·
(
2
λ
2
+
λ
)
μ
2
+
λ
σ
2
Z
α
2
+
λ
Using theN
R
* we compute the N
R
_bootstrap and respectively the standard error needed to compute the confidence interval
Randomk (standard normal distribution values μ
=
0, σ
2
=
1; half normal distribution μ
=
2
/
π, σ
2
=
1
-
2
/
π; skew normal with negative skewness μ
=
-
1
/
2
π, σ
2
=
1
-
1
/
2
π; skew normal with positive skewness μ
=
1
/
2
π, σ
2
=
1
-
1
/
2
π; λ
=
5,15,30,50)
E
[
N
^
R
]
=
λ
2
μ
2
+
λ
(
μ
2
+
σ
2
)
Z
α
2
-
λWe observe from Table2 and Figure 1 that the bootstrap confidence intervals perform the poorest both when the number of studies is considered fixed or random. The only case in which they perform acceptably is when the distribution is half normal and the number of studies is fixed. The moment estimators of variance perform either poorly or too efficiently in all cases, with coverages being under 90% or near 100%. The most acceptable confidence intervals for Rosenthal’s estimator appear to be in the distribution-based method and to be much better for a fixed number of studies than for random number of studies. We also observe that, for the distribution-based confidence intervals in the fixed category, the half normal distribution HN(0,1) produces coverages which are all 95%. This is also stable for all number of studies in a meta-analysis. When the distributional assumption is not met the coverage is poor except for the cases of the positive and negative skewness skew normal distributions which perform similarly, possibly due to symmetry.Table 2
Probability coverage of the different methods for confidence intervals (CI) according to the number of studiesk. The figure is organised as follows: the Z
i are drawn from four different distributions (standard normal distribution, half normal distribution, skew normal with negative skewness, and skew normal with positive skewness).
DrawZ
i from
Values ofμ and σ
2 from the standard normal distribution
Values ofμ and σ
2 from the half normal distribution HN(0, 1)
Values ofμ and σ
2 from the skew normal distribution with negative skewness SN(δ
=
-
0.5
,
ξ
=
0
,
ω
=
1)
Values ofμ and σ
2 from the skew normal distribution with positive skewness SN(δ
=
0.5
,
ξ
=
0
,
ω
=
1)
k
=
5
k
=
15
k
=
30
k
=
50
k
=
5
k
=
15
k
=
30
k
=
50
k
=
5
k
=
15
k
=
30
k
=
50
k
=
5
k
=
15
k
=
30
k
=
50
Standard normaldistribution
Fixedk
Distribution Based CI
0.948
0.950
0.948
0.952
0.994
0.110
0.002
0.000
0.985
0.999
1.000
1.000
0.982
0.998
1.000
1.000
Moments Based CI
0.933
0.996
1.000
1.000
0.529
0.088
0.005
0.000
0.842
0.686
0.337
0.120
0.842
0.686
0.337
0.120
Bootstrap CI
0.929
0.996
1.000
1.000
0.514
0.089
0.005
0.000
0.830
0.680
0.337
0.120
0.830
0.680
0.337
0.120
Randomk
Distribution Based CI
0.966
0.956
0.951
0.955
0.999
1.000
0.084
0.001
0.998
1.000
1.000
1.000
0.990
0.999
1.000
1.000
Moments Based CI
1.000
1.000
1.000
1.000
0.535
0.094
0.006
0.000
1.000
0.702
0.338
0.122
1.000
0.702
0.338
0.122
Bootstrap CI
0.929
0.996
1.000
1.000
0.429
0.074
0.004
0.000
0.804
0.649
0.322
0.115
0.804
0.649
0.322
0.115
Half normal distribution HN(0, 1)
Fixedk
Distribution Based CI
0.635
0.021
0.000
0.000
0.945
0.952
0.951
0.948
0.864
0.624
0.279
0.053
0.841
0.483
0.142
0.014
Moments Based CI
0.861
0.187
0.000
0.000
0.771
0.880
0.911
0.927
0.885
0.657
0.126
0.003
0.885
0.657
0.126
0.003
Bootstrap CI
0.858
0.217
0.000
0.000
0.775
0.884
0.913
0.929
0.887
0.672
0.138
0.003
0.887
0.672
0.138
0.003
Randomk
Distribution Based CI
0.720
0.027
0.000
0.000
0.989
0.995
0.996
0.997
0.966
0.915
0.762
0.459
0.901
0.578
0.198
0.027
Moments Based CI
1.000
1.000
0.130
0.000
0.806
0.937
0.971
0.984
1.000
1.000
0.995
0.358
1.000
1.000
0.995
0.358
Bootstrap CI
0.858
0.217
0.000
0.000
0.715
0.859
0.899
0.920
0.885
0.698
0.152
0.004
0.885
0.698
0.152
0.004
Skew normal distributionwith negative skewness SN(δ
=
-
0.5
,
ξ
=
0
,
ω
=
1)
Fixedk
Distribution Based CI
0.872
0.666
0.399
0.174
0.980
0.472
0.184
0.048
0.953
0.970
0.977
0.981
0.944
0.948
0.949
0.957
Moments Based CI
0.917
0.979
0.944
0.858
0.597
0.375
0.200
0.074
0.845
0.860
0.882
0.895
0.845
0.860
0.882
0.895
Bootstrap CI
0.912
0.978
0.945
0.857
0.586
0.377
0.199
0.074
0.840
0.857
0.881
0.894
0.840
0.857
0.881
0.894
Randomk
Distribution Based CI
0.903
0.688
0.409
0.178
0.996
1.000
0.687
0.306
0.987
0.997
0.998
0.999
0.965
0.964
0.965
0.968
Moments Based CI
1.000
1.000
0.999
0.968
0.609
0.399
0.237
0.103
1.000
0.872
0.886
0.902
1.000
0.872
0.886
0.902
Bootstrap CI
0.912
0.978
0.945
0.857
0.514
0.342
0.181
0.066
0.818
0.845
0.874
0.889
0.818
0.845
0.874
0.889
Skew normal distributionwith positive skewness SN(δ
=
0.5
,
ξ
=
0
,
ω
=
1)
Fixedk
Distribution Based CI
0.880
0.673
0.402
0.164
0.982
0.471
0.186
0.050
0.956
0.972
0.976
0.979
0.948
0.952
0.951
0.955
Moments Based CI
0.923
0.980
0.947
0.852
0.596
0.372
0.201
0.076
0.850
0.865
0.874
0.896
0.850
0.865
0.874
0.896
Bootstrap CI
0.918
0.978
0.946
0.846
0.583
0.372
0.200
0.077
0.841
0.862
0.873
0.896
0.841
0.862
0.873
0.896
Randomk
Distribution Based CI
0.911
0.696
0.415
0.169
0.996
1.000
0.683
0.314
0.989
0.996
0.998
0.999
0.967
0.967
0.964
0.966
Moments Based CI
1.000
1.000
0.999
0.964
0.606
0.399
0.236
0.105
1.000
0.875
0.880
0.905
1.000
0.875
0.880
0.905
Bootstrap CI
0.918
0.978
0.946
0.846
0.514
0.335
0.180
0.068
0.819
0.850
0.868
0.893
0.819
0.850
0.868
0.893This figures shows the probability coverage of the different methods for confidence intervals (CI) according to the number of studiesk. The figure is organised as follows: the Z
i are drawn from four different distributions (standard normal distribution, half normal distribution, skew normal with negative skewness, and skew normal with positive skewness) which are depicted in ((a)–(d), (e)–(h), (i)–(l), and (m)–(p)), respectively. The different values of μ and σ
2 for the variance correspond to the standard normal distribution ((a), (e), (i), and (m)), half normal distribution ((b), (f), (j), and (n)), skew normal with negative skewness ((c), (g), (k), and (o)), and skew normal with positive skewness ((d), (h), (l), and (p)).
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(k)
(l)
(m)
(n)
(o)
(p)In the next sections, we give certain examples and we present the lower limits of confidence intervals for testing whetherN
R
>
5
k
+
10, according to the suggested rule of thumb by [11]. We choose only the variance from a fixed number of studies when the Z
i are drawn from a half normal distribution HN(0,1).


## 5. Examples

In this section, we present two examples of meta-analyses from the literature. The first study is a meta-analysis of the effect of probiotics for preventing antibiotic-associated diarrhoea and included 63 studies [24]. The second meta-analysis comes from the psychological literature and is a meta-analysis examining reward, cooperation, and punishment, including analysis of 148 effect sizes [25]. For each meta-analysis, we computed Rosenthal’s fail-safe number and the respective confidence interval with the methods described above (Table 3).Table 3
Confidence intervals for the examples of meta-analyses.
Fixed number of studies
Random number of studies
Bootstrap based CI
Distributionbased CI
Momentbased CI
Distributionbased CI
Momentbased CI
Study 1 [24]Rosenthal’s N
R
=
2124
(2060, 2188)
(788, 3460)
(2059, 2189)
(369, 3879)
(740, 3508)
Study 2 [25]Rosenthal’s N
R
=
73860
(73709, 74012)
(51618, 96102)
(73707, 74013)
(40976, 106745)
(51662, 96059)We observe that both fail-safe numbers exceed Rosenthal’s rule of thumb, but some lower confidence intervals, especially in the first example, go as low as 369, which only slightly surpasses the rule of thumb (5
*
63
+
10
=
325 in this case). This is not the case with the second example. Hence the confidence interval, especially the lower confidence interval value, is important to establish whether the fail-safe number surpasses the rule of thumb.In the next section, we present a table with values according to which future researchers can get advice on whether their value truly supersedes the rule of thumb.


## 6. Suggested Confidence Limits forN
R

We wish to answer the question whetherN
R
>
5
k
+
10 for a given level of significance and the estimator N
^
R, which is the rule of thumb suggested by Rosenthal. We formulate a hypothesis test according to which
(37)
H
0
:
N
R
≤
5
k
+
10
H
1
:
N
R
>
5
k
+
10
.
An asymptotic test statistic for this is
(38)
T
&
=
N
^
R
-
5
k
-
10
Var
⁡
N
^
R
⟶
d
N
0,1
,
under the null hypothesis.So we reject the null hypothesis if(
N
^
R
-
5
k
-
10
)
/
Var
⁡
N
^
R
>
Z
α
⇒
N
^
R
>
Z
α
Var
⁡
N
^
R
+
5
k
+
10.In Table4 we give the limits of N
R above which we are 95% confident that N
R
>
5
k
+
10. For example if a researcher performs a meta-analysis of 25 studies, the rule of thumb suggests that over 5
·
25
+
10
=
135 studies there is no publication bias. The present approach and the values of Table 4 suggest that we are 95% confident for this when N
R exceeds 209 studies. So this approach allows for inferences about Rosenthal’s N
^
R and is also slightly more conservative especially when Rosenthal’s fail-safe number is characterised from overestimating the number of published studies.Table 4
95% one-sided confidence limits above which the estimatedN
R is significantly higher than 5
k
+
10, which is the rule of thumb suggested by Rosenthal [11]. k represents the number of studies included in a meta-analysis. We choose the variance from a fixed number of studies when the Z
i are drawn from a half normal distribution HN(0, 1), as this performed best in the simulations.
k
Cutoff point
k
Cutoff point
k
Cutoff point
k
Cutoff point
1
17
41
369
81
842
121
1394
2
26
42
380
82
855
122
1409
3
35
43
390
83
868
123
1424
4
45
44
401
84
881
124
1438
5
54
45
412
85
894
125
1453
6
63
46
423
86
907
126
1468
7
71
47
434
87
920
127
1483
8
79
48
445
88
934
128
1498
9
86
49
456
89
947
129
1513
10
93
50
467
90
960
130
1528
11
99
51
479
91
973
131
1543
12
106
52
490
92
987
132
1558
13
112
53
501
93
1000
133
1573
14
118
54
513
94
1014
134
1588
15
125
55
524
95
1027
135
1603
16
132
56
536
96
1041
136
1619
17
140
57
547
97
1055
137
1634
18
147
58
559
98
1068
138
1649
19
155
59
571
99
1082
139
1664
20
164
60
582
100
1096
140
1680
21
172
61
594
101
1109
141
1695
22
181
62
606
102
1123
142
1711
23
190
63
618
103
1137
143
1726
24
199
64
630
104
1151
144
1742
25
209
65
642
105
1165
145
1757
26
218
66
654
106
1179
146
1773
27
228
67
666
107
1193
147
1788
28
237
68
679
108
1207
148
1804
29
247
69
691
109
1221
149
1820
30
257
70
703
110
1236
150
1835
31
266
71
716
111
1250
151
1851
32
276
72
728
112
1264
152
1867
33
286
73
740
113
1278
153
1883
34
296
74
753
114
1293
154
1899
35
307
75
766
115
1307
155
1915
36
317
76
778
116
1322
156
1931
37
327
77
791
117
1336
157
1947
38
338
78
804
118
1351
158
1963
39
348
79
816
119
1365
159
1979
40
358
80
829
120
1380
160
1995


## 7. Discussion and Conclusion

The purpose of the present paper was to assess the efficacy of confidence intervals for Rosenthal’s fail-safe number. We initially defined publication bias and described an overview of the available literature on fail-safe calculations in meta-analysis. Although Rosenthal’s estimator is highly used by researchers, its properties and usefulness have been questioned [48, 58].The original contributions of the present paper are its theoretical and empirical results. First, we developed statistical theory allowing us to produce confidence intervals for Rosenthal’s fail-safe number. This was produced by discerning whether the number of studies analysed in a meta-analysis is fixed or random. Each case produces different variance estimators. For a given number of studies and a given distribution, we provided five variance estimators: moment- and distribution-based estimators based on whether the number of studies is fixed or random and on bootstrap confidence intervals. Secondly, we examined four distributions by which we can simulate and test our hypotheses of variance, namely, standard normal distribution, half normal distribution, a positive skew normal distribution, and a negative skew normal distribution. These four distributions were chosen as closest to the nature of theZ
i
s. The half normal distribution variance estimator appears to present the best coverage for the confidence intervals. Hence, this might support the hypothesis that Z
i
s are derived from a half normal distribution. Thirdly, we provide a table of lower confidence intervals for Rosenthal’s estimator.The limitations of the study initially stem from the flaws associated with Rosenthal’s estimator. This usually means that the number of negative studies needed to disprove the result is highly overestimated. However, its magnitude can give an indication for no publication bias. Another possible flaw could come from the simulation planning. We could try more values for the skew normal distribution, for which we tried only two values in present paper.The implications of this research for applied researchers in psychology, medicine, and social sciences, which are the fields that predominantly use Rosenthal’s fail-safe number, are immediate. Table4 provides an accessible reference for researchers to consult and apply this more conservative rule for Rosenthal’s number. Secondly, the formulas for the variance estimator are all available to researchers, so they can compute normal approximation confidence intervals on their own. The future step that needs to be attempted is to develop an R-package program or a Stata program to execute this quickly and efficiently and make it available to the public domain. This will allow widespread use of these techniques.In conclusion, the present study is the first in the literature to study the statistical properties of Rosenthal’s fail-safe number. Statistical theory and simulations were presented and tables for applied researchers were also provided. Despite the limitations of Rosenthal’s fail-safe number, it can be a trustworthy way to assess publication bias, especially under the more conservative nature of the present paper.



---

*Source: 825383-2014-12-03.xml*