arXiv:1508.00934v3 [] 19 Dec 2016

Admissibility in partial conjunction
testing
Jingshu Wang
Stanford University

Art B. Owen
Stanford University

December 2016
Abstract
Meta-analysis combines results from multiple studies aiming to increase
power in finding their common effect. It would typically reject the null hypothesis of no effect if any one of the studies shows strong significance. The
partial conjunction null hypothesis is rejected only when at least r of n component hypotheses are non-null with r = 1 corresponding to a usual metaanalysis. Compared with meta-analysis, it can encourage replicable findings
across studies. A by-product of it when applied to different r values is a confidence interval of r quantifying the proportion of non-null studies. Benjamini
and Heller (2008) provided a valid test for the partial conjunction null by ignoring the r − 1 smallest p-values and applying a valid meta-analysis p-value
to the remaining n − r + 1 p-values. We provide sufficient and necessary conditions of admissible combined p-value for the partial conjunction hypothesis
among monotone tests. Non-monotone tests always dominate monotone tests
but are usually too unreasonable to be used in practice. Based on these findings, we propose a generalized form of Benjamini and Heller’s test which allows
usage of various types of meta-analysis p-values, and apply our method to an
example in assessing replicable benefit of new anticoagulants across subgroups
of patients for stroke prevention.

1

Introduction

When a null hypothesis is tested in n different settings, a meta-analysis can be used
to obtain a combined p-value based on all of the test results. It gains power as
0

This work was supported by the US National Science Foundation under grant DMS-1407397.

1

Table 1: four hypothetical cases for five ordered p-values.
Case p(1)

p(2)

p(3)

p(4)

p(5)

A
B

10−200
10−10

0.4
10−9

0.5
10−8

0.6
10−7

0.7
10−6

C
D

10−100
0.048

10−100
0.048

10−100
0.048

0.049
0.048

0.8
0.8

the combined p-value is usually more significant than any of the individual p-value
in each setting. However, the combined p-value in meta-analysis is only valid for
the global null where the null hypothesis is true in every setting, thus it is possible
that the null is then rejected largely on the basis of just one extremely significant
component hypothesis test. Such a rejection may be undesirable as it could arise
from some irreproducible property of the setting in which that one component test
was made.
Refering to Table 1, cases A and B illustrate the potential problem of metaanalysis. Both a Fisher and a Stouffer meta-analysis would find case A more significant than case B, while the only significant setting in case A may largely due
to a technical or statistical bias. The random effect model in meta-analysis has
been widely accepted for consideration of heteogeneity across studies (Higgins et al.,
2009). However, it still assumes that the effects across studies are similar and does
not explicitly guarantee replication nor being robust to extreme bias.
Researchers in functional magnetic resonance imaging (fMRI) have adopted conjunction (logical ‘and’) testing (Price and Friston, 1997; Friston et al., 1999; Nichols
et al., 2005) in which a hypothesis must be rejected in all n settings where it is tested.
The n settings may correspond to related tasks or they may correspond to independent subjects. For example in Table 1, a conjunction test would only reject case B
which shows consistent replication. However, conjunction tests lose power for large
n as they are based on the largest of n p-values. A compromise is to require evidence
that at least r out of n null hypotheses are false, for some user specified r. Such tests
of the ‘partial conjunction (PC) null hypothesis’ were used in Friston et al. (2005)
and then studied by Benjamini and Heller (2008). The extremes r = 1 and r = n
correspond to the usual meta-analysis tests and conjunction testing respectively.
Partial conjunction testing is useful in areas beyond neuroimaging. It is the tool
for replicability analysis, which finds effects that present in more than one studies,
and has been applied in systematic review for healthcare prevention (Shenhav et al.,
2015) and genome-wide association studies (Heller et al., 2014). PC test also has
2

the potential usage in finding common gene regulation patterns across tissues for
eQTL data (Flutre et al., 2013). Besides, it can be applied to gene set enrichment
analysis to avoid selection of gene sets whose significance depend on only one single
gene (Wang et al., 2010).
A Benjamini-Heller partial conjunction (BHPC) test works as follows. One sorts
the observed p-values yielding p(1) ≤ p(2) ≤ · · · ≤ p(n) , ignores the smallest r − 1 of
them, and then applies a valid p-value combination rule to the remaining n − r +
1 p-values. Benjamini and Heller (2008) show that BHPC tests are valid for the
partial conjunction null when the n hypotheses are independent. They also consider
some dependent test conditions as well as the consequences of using PC tests in the
Benjamini-Hochberg procedure.
Cases C and D illustrate an interesting property of the BHPC tests. Suppose
that we need to reject at least four null hypotheses to have a meaningful finding.
Then a BHPC test finds that case D is stronger evidence (smaller p-value) than
case C, because BHPC is based only on p(4) and p(5) . In case C we are extremely
confident of three rejections and are banking on the fourth one to be correct. In case
D by contrast, none of the four smallest p-values is much better than borderline. It
appears to have about four times as many ways to disappointing us. This comparsion
between case C and D reveals a counter-intuitive property of the BHPC tests, that
we study further.
Here we investigate the power properties of BHPC tests focussing on admissibility. Under the assumption that the component p-values are either independent or
have a positive dependency structure, we characterize the complete class of tests for
monotone admissibility, which is a generalized form of BHPC p-values (GBHPC pvalues). the only admissible PC tests among monotone tests are either of the BHPC
form, or its generalization (GBHPC), which uses combined p-values constructed
by

n
taking the maximum of the meta-analysis p-value of each of the r−1
subsets of
n − r + 1 hypotheses. Under mild assumptions, a sufficient condition for the monotoneadmissibility of a GBHPC p-value is that each of the meta-analysis p-value for
n
subsets is admissible. GBHPC p-values are also called r-values in Shenhav
r−1
et al. (2015).
The monotonicity condition, which means that the combined p-value is a nondecreasing function of the individual p-values, is necessary for us to discuss admissibility for partial conjunction hypotheses with r > 1. If we relax this condition, then
BHPC tests become inadmissible. Because non-monotone tests are quite unreasonable scientifically in most cases, this is not a strong criticism of BHPC. We side with
Perlman and Wu (1999) in rejecting the admissibility criterion not the test, when
methods lacking face-value validity are included in comparisons.
3

The admissibility properties show supreme of BHPC p-values, but how can we
then explain its puzzling behavior alluded to for cases C and D in Table 1? An
explanation is that unlike the combined p-values in meta-analysis, the PC p-values
measure the strength of replicability (the true proportion of non-null studies) instead
of the magnitude of effect size. a PC p-value can be much smaller when the true
number (r0 ) rather than the effect size of non-null studies is large. Compared with
case D, the three extreme p-values of case C in Table 1 gain us much stronger evidence
of a large effect size but not much more evidence on r0 ≥ 4. Thus, the PC p-values
for both case C and case D are similar.
Section 2 presents our notation and some background on partial conjunction
tests and admissibility. Section 3 proposes the GBHPC p-values and presents the
main theorems on monotone admissible partial conjunction p-values. Section 4 is a
simulation study for the power comparison of several GBHPC p-values under various
hypotheses configurations. Compared with BHPC p-values, GBHPC p-values have
the advantage that it can be constructed from more sophisticated meta-analysis pvalues. We illustrate this benefit in Section 5 in an application of GBHPC p-values
for assessing replicable benefit and safety concerns of new oral anticoagulants across
subgroups of patients for stroke prevention. Section 6 has our conclusions and states
some future work.

2
2.1

Preliminaries
Definitions and notations

The problem begins with n null hypotheses to test, H0i for i = 1, . . . , n. Each H0i is
the hypothesis to test for an individual setting/study. The corresponding alternative
hypotheses are H1i . The i’th hypothesis refers to a parameter θi . If H0i holds
then θi ∈ Θ0i , while H1i specifies that θi ∈ Θ1i . The parameter space for the i’th
hypothesis is Θi = Q
Θ0i ∪ Θ1i and of course Θ0i ∩ Θ1i = ∅. The parameter space of
(θ1 , . . . , θn ) is Θ = i Θi .
To each hypothesis, there corresponds a p-value, pi . there may be a loss of
information in reducing a data set to one p-value. Yet often that loss is small and
very commonly the researchers who gathered the original data share only their pvalues for reasons that may include privacy of their subjects.
We use pi to denote the numerical value of the p-value for the i’th hypothesis. It
is the observed value of a corresponding random variable Pi . The sorted p-values are
p(1) ≤ p(2) ≤ · · · ≤ p(n) and P(1) ≤ P(2) ≤ · · · ≤ P(n) are the sorted random variables.
Probability and expectation for functions of Pi are given by Pθi and Eθi respectively.
4

We let θ = (θ1 , . . . , θn ) and P = (P1 , . . . , Pn ). Probability and expectation for
functions of P are given by Pθ and Eθ . Each pi is a valid P-value according to the
definition below:
Definition 1 (Validity). A valid component p-value satisfies supθi ∈Θ0i Pθi (Pi ≤ α) ≤
α for 0 ≤ α ≤ 1.
Besides independency, positive dependency can be a common dependency structure across studies, especially when they share samples. For P1 , . . . , Pn , we assume
that they are either independent or positively dependent (PRDS) (Benjamini and
Yekutieli, 2001) Under any parameter θ ∈ Θ. Using the definition of PRDS thus
they have the following property: for ∀(p1 , · · · , pn ) ∈ [0, 1]n and ∀θ ∈ Θ,
Y
Pθ (P1 ≤ p1 , · · · , Pn ≤ pn ) ≥
Pθi (Pi ≤ pi )
(1)
i

For a given r ≤ n, the PC null hypothesis and alternative hypothesis are defined
as
r/n

: {at most r − 1 hypotheses are non-null},

r/n

: {at least r hypotheses are non-null}.

H0
H1

and

r/n

r/n

The null space is defined as Θ0 = {θ ∈ Θ : H0 is true}.
We use 1:r to denote {1, 2, . . . , r} and similarly (r + 1):n = {r + 1, r + 2, . . . , n}.
The index set u ⊂ 1:n has cardinality |u| and complement −u = 1:n \ u. Under the
null hypothesis H0u we have θj ∈ Θ0j for all j ∈ u. The null space of H0u is denoted
as Θ0u .
Sometimes we combine points x ∈ Rn and y ∈ Rn into a point z ∈ Rn with
zj = xj for j ∈ u and zj = yj for j 6∈ u. Such a hybrid point is denoted z = xu :y−u .
Let u = {i1 , i2 , . . . , ik }, then θu is defined as the combination (θi1 , θi2 , . . . , θik ).
We can extend the definition of validity to meta-analysis and PC p-values. The
combination of k p-values (k may differ from n later) produces the combined p-value
r/k
pr/k = fr,k (P1 , . . . , Pk ) which is a valid p-value for testing H0 if
sup Pθ (Pr/k ≤ α) ≤ α,

∀ 0 ≤ α ≤ 1.

r/n

θ∈Θ0

r/n

Definition 2 (Sensitivity). A sensitive p-value Pr/k = fr,k (P1 , . . . , Pk ) for H0
isfies
lim inf Pr/k = 0
Pu →0

for ∀u ⊂ 1:k and |u| = r.
5

sat-

Sensitivity requires that PC p-value drops to 0 when we are certain to reject any
of r individual hypotheses. For meta-analysis p-value P1/k , it means that we reject
the global null when we are certain at any of the individual hypothesis. We think
r/k
that it is a practically reasonable requirement for a p-value for testing H0 .
Here are some examples of valid and sensitive meta-analysis p-values given valid
p-values p1 , . . . , pk . The combination for a method M is defined in terms of a function
fM,k which may incorporate sorting of its arguments.
Example 1. Simes’ method:

pS,k = fS,k (p1 , . . . , pk ) ≡ min

i=1,··· ,k


kp(i)
.
i

Example 2. Fisher’s method:


pF,k = fF,k (p1 , . . . , pk ) ≡ P

χ2(2k)

≥ −2

k
X


log pi .

i=1

√
Example 3. Weighted Stouffer test: Consider test statistics Ti ∼ N ( ni θi /σi , 1),
with sample sizes ni for i = 1, . . . , k and known σi > 0. The p-value for the null that
θi = 0 versus the alternative that θi > 0 is pi = 1 − Φ(Ti ) = Φ(−Ti ). A weighted
1/k
Stouffer p-value for H0 takes the form
!
Pk √
−1
n
Φ
(1
−
p
)/σ
i
i
i
i=1
qP
pWS,k = pWS,k (p1 , . . . , pk ) ≡ 1 − Φ
.
k
2
i=1 ni /σi
In fact, pWS,k can be used beyond one-sided tests. For example, for two-sided test
√
Φ−1 (1 − pi ) also has the magnitude roughly proportional to ni . We shall illustrate
the performance under such usage in our simulations in Section 4.
Example 4. Truncated product method (Zaykin et al., 2002): this is a more recently
developed method to gain efficiency in the presence of outliers. The test statistic has
the form
Y 1p ≤γ
wγ =
pi i
i
1/k

where γ is some pre-determined value. The TPM p-value for H0

takes the form

pTPM,k = P(W ≤ wγ ).
where the probability function of W was computed for both independency and dependent scenarios in (Zaykin et al., 2002).
6

For non-symmetric meta-analysis p-values, there are also weighted versions of
Fisher tests. Note that each of the functions f in the previous examples is monotone
according to this definition:
Definition 3. (Monotonicity) the p-value f (p1 , . . . , pk ) is monotone if the function
f is non-decreasing in each argument. The set of such monotone p-value functions is
denoted Fmon . a monotone test is one that rejects its null hypothesis for small values
of a monotone p-value.
A non-monotone test would reject its null hypothesis at some input (p1 , . . . , pk )
but fail to reject at some (p01 , . . . , p0k ) with all p0i ≤ pi . Such a test is typically
unreasonable.
Besides monotonicity, we also clearify the definition of a symmetric combined
p-value:
Definition 4. (Symmetry) The combined p-value f (p1 , . . . , pk ) is symmetric if its
value stays unchanged under any permutation of {p1 , . . . , pk }.
Finally, we state the concept of admissibility using the definition of admissible
tests from Lehmann and Romano (2006, Chapter 6.7). Let a hypothesis test of
H0 versus H1 described by a function ϕ(X) ∈ {0, 1} of the data X. If ϕ(X) = 1
then H0 is rejected and ϕ(X) = 0 otherwise. The test ϕ is valid at level α if
supθ∈Θ0 Eθ (ϕ(X)) ≤ α. In our context, the data are a vector P = (P1 , . . . , Pn ) of
p-values and ϕ(P1 , . . . , Pn ) = 1f (P1 ,...,Pn )≤α where f is a p-value combination function.
Definition 5 (Ψ,α-admissibility). The level-α test ϕ ∈ Ψ is α-admissible for testing
H0 : θ ∈ Θ0 against H1 : θ ∈ Θ1 if for any other level-α test ϕ0 ∈ Ψ
Eθ (ϕ0 ) ≥ Eθ (ϕ),

for all θ ∈ Θ1

implies Eθ (ϕ0 ) = Eθ (ϕ) for all θ ∈ Θ1 .
The definition of admissibility depends on the alternatives in Θ1 as well as the
space Ψ of test functions. The constraints on Θ1 are important. for the ordinary
meta-analysis, Birnbaum (1954) shows that every monotone p-value is admissible
when the component p-values are independent and the null hypothesis is simple,
because there is then some alternative at which that p-value gives optimal power.
However, those optimizing alternatives may not all be reasonable. Birnbaum (1955)
and Stein (1956) (generalized later by Matthes and Truax (1967) to include nuisance
parameters) also showed that for the ordinary meta-analysis, when the test statistic
distribution is an exponential family with θ as canonical parameter, a necessary and
7

sufficient condition for admissibility is to have a closed convex acceptance region of
underlying test statistics.
For the space of test functions Ψ, traditionally it contains all possible functions
when considering admissibility. However, for the partial conjunction null hypothesis,
we restrict Ψ to only include tests using monotone p-values to avoid unreasonble
more powerful tests (see Section 3.2 for details).

2.2

BHPC p-values

Now we restate Theorem 1 of Benjamini and Heller (2008).
Theorem 2.1. Let P1 , . . . , Pn be independent valid p-values, and for k = n−r +1 let
fM,k (P1 , . . . , Pk ) be a valid and symmetric meta-analysis p-value where fM,k ∈ Fmon .
r/n
Then Pr/n = fM,n−r+1 (P(r) , P(r+1) , . . . , P(n) ) is a valid p-value for H0 .
As mentioned, we call the combined p-value Pr/n described in 2.1 a BHPC pvalue for short. In practice it makes sense to require that the p-value combination
1/k
function fM,k (·), for k = n−r+1, be a sensitive one for H0 . Notice that if fM,k were
s/k
a partial conjunction test of H0 for s > 1, then fM,k is still a valid meta-analysis
p-value but is not sensitive any more. The BHPC p-values satisfy the chain rule in
(r+s−1)/n
. Though
the sense that Pr/n in Theorem 2.1 now becomes a valid test for H0
r/n
Pr/n would still be a valid test of H0 , it is less efficient.
Based on the relationship between hypotheses testing and building confidence
r/n
set, if we have for each r = 1, 2, . . . , n a valid combined p-value pr/n for H0 , then
the 1 − α confidence set for r is
i = {r : Pr/n ≤ α}
Benjamini and Heller (2008) showed that if each Pr/n is a BHPC p-value with fM,k =
fS,k in 1, then we have p1/n ≤ P2/n ≤ · · · ≤ Pn/n and i = [r̂, n] becomes a 1 − α
confidence interval with r̂ = max{r : Pr/n ≤ α}.

3

GBHPC p-values

Motivated by the BHPC p-value, we discuss a more general class of combined p-values
with good power properties. These are defined as GBHPC p-values:

8

Definition 6 (GBHPC p-value). For each u ⊂ 1:n with |u| = k = n − r + 1 let
gu be a function from [0, 1]k to [0, 1] such that gu is non-decreasing and is a valid
meta-analysis p-value for H0u . Then
f ? (p) = f ? (p1 , · · · , pn ) =

max gu (pu )

(2)

u⊂1:n
|u|=n−r+1

is a generalized BHPC (GBHPC) p-value.
r/n

The alternative hypothesis H1 that at least r out of n hypotheses are false is
equivalent to the statement that for every n−r+1 of the n hypotheses, there is at least
one of them that is false. This is the reason that the GBHPC p-value is the maximum
of all the meta-analysis p-values of size n − r + 1. Using above explanation, the next
proposition states validity of GBHPC p-values under any dependency structure of
the individual p-values.
r/n

Proposition 3.1. Any GBHPC p-value is a valid p-value for H0 .
r/n

Proof. Consider a GBHPC p-value of the form (2). From the definition of H0 , for
r/n
all θ ∈ Θ0 , there exists u with |u| = n − r + 1 such that θj ∈ Θ0j for all j ∈ u. then
for any α ∈ [0, 1],
Pθ (f ? (P ) ≤ α) ≤ Pθ (gu (Pu ) ≤ α) = Pθu (gu (Pu ) ≤ α) ≤ α.
r/n

Thus f ? (P1 , · · · , Pn ) is valid for H0 .
The BHPC p-value is a special case of GBHPC p-values. If all gu ≡ g in (2)
and g a monotone and symmetric combined p-value, then f ? (p) = g(p(r) , . . . , p(n) )
becomes a BHPC p-value. Some meta-analysis methods, such as the weighted Stouffer test in Example 3, treat their component p-values differently depending on the
relative sample sizes on which they are based. The GBHPC framework includes such
methods.
Next we show that a sensitive GBHPC p-value has a unique representation in the
form of (2).
Proposition 3.2. If a GBHPC p-value f ? is sensitive, then each gu is also sensitive
and the representation of f ? in (2) is unique with
gu (Pu ) =

inf

P−u

∈(0,1]r−1

9

f ? (P1 , · · · , Pn )

(3)

Proof. Consider any given u ∈ 1:n with |u| = n − r + 1. Then for ∀i ∈ u, let
ui = {−u, i}, then |ui | = r. As f ? is sensitive, we have
lim inf gu (Pu ) = lim inf gu (Pu ) ≤ lim inf f ? (P ) = 0
Pi →0

Pui →0

Pui →0

Thus, gu is sensitive for every u with |u| = n − r + 1. On the other hand, for a given
u0 with |u0 | = n − r + 1, using definition (2), we have
h
i
inf r−1 f ? (P ) =
inf r−1 max gu (Pu ) = gu0 (Pu0 )
P−u0 ∈(0,1]

P−u0 ∈(0,1]

u

this also proves the uniqueness of the representation.
Remark 3.1. Equation (3) also shows that any symmetric GBHPC p-value is a BHPC
p-value. If f ? is symmetric, then gu constructed in (3) is also valid, monotone and
symmetric. Also, gu ≡ g is the same for all u. Thus f ? = g(p(n) , · · · , p(r) ) is a BHPC
p-value.
Remark 3.2.
 Equation (2) involves taking the maximum of meta-analysis p-values
n
over r−1
number of subsets. Thus, the computational cost of non-symmetric
GBHPC p-values can be high for large r and n. One situation to use a non-symmetric
n
GBHPC p-value is when r = 2, then r−1
= n which is typically acceptable. Besides, it is sometimes possible to make use of the special structure of the problem
to construct easy-to-compute non-symmetric GBHPC p-values. For instance in Section 5, the GBHPC p-values we constructed for the pharmaceutical data has almost
the same computational cost as BHPC p-values, but can give much smaller p-values.

3.1

Monotone α-admissiblity

Now we discuss the sufficient and necessary conditions for admissible combined pvalues of a PC hypothesis. Each of our results uses some combination of the following
three assumptions on the individual p-values.
Assumption 1 (Strong alternatives). ∀α > 0 and i = 1, . . . , n, supθi ∈Θ1i Pθi (Pi ≤
α) = 1.
Assumption 2 (Continuity). For ∀θi ∈ Θ1i and i = 1, . . . , n, Pθi (Pi = 0) = 0.
Assumption 3 (Completeness). The family {Pθu : θu ∈
/ Θ0u } for any subset u with
|u| = n − r + 1 is complete.

10

Assumption 1 states that for each individual hypothesis there are strong enough
alternatives that we can almost certainly reject the null. Assumption 2 is a technical
assumption assuming that the probability that the p-value is exactly 0 is zero under
any alternative. The completeness in Assumption 3 is to guarantee that if two level
α meta-analysis tests for H0u has the same power at every point in the alternative
space then they are the same test. Roughly speaking, both Assumptions 1 and 3
require that the alternative space of each individual hypothesis is large enough to
include various possibilities. The three assumptions can be satisfied in common tests.
For example, tests satisfying Assumption 1 include testing the parameters of
exponential families and location families. Lehmann and Romano (2006, Theorem
4.3.1) show that completeness is satisfied for testing the natural parameter of a kdimensional exponential family if the alternative space Θ1i contains a k-dimensional
rectangle. If the individual p-values are independent, then completeness of the alternative space for each individual hypothesis implies Assumption 3. Thus, we believe
that assumptions 1 to 3 can cover a large class of problems.
Theorem 3.3 shows that GBHPC p-values form a complete class of monotone
r/n
α-admissible p-values for H0 . Theorem 3.4 states that a sufficient condition for a
sensitive GBHPC p-value to be monotone α-admissible is that each gu is admissible
for H0u .
Theorem 3.3. Let P1 , . . . , Pn be independent or positively dependent (PRDS) Pvalues satisfying assumptions 1 and 2. Let Pr/n be a valid monotone p-value for
r/n
H0 . Then there exists a valid GBHPC p-value p?r/n that is uniformly at least as
powerful as Pr/n .
Theorem 3.4. Let P1 , . . . , Pn be PRDS P-values satisfying assumptions 1 to 3. For
?
a sensitive GBHPC p-value Pr/n
= f ? (P ) of the form (2), a sufficient condition
?
for Pr/n to be monotone α-admissible is that each gu is an admissible meta-analysis
p-value for H0u .
We introduce the following lemma, which is the key reason that 3.3 and 3.4
hold. It shows that given a valid monotone p-value that is not of the GBHPC form,
we can expand its rejection region while retaining its validity.
Lemma 3.5. Let P1 , . . . , Pn be PRDS P-values satisfying assumption 1. Let f (P1 , · · · , Pn )
r/n
be a valid monotone p-value for H0 and for u ⊂ 1:n with |u| = n − r + 1, define
gu (Pu ) =

inf

P−u ∈(0,1]r−1

f (P1 , · · · , Pn )

Then gu is a valid monotone meta-analysis p-value for H0u .
11

(4)

Proof. Monotonicity of f implies monotonicity and measurability of gu . Next, suppose that gu is not valid for H0u . Then there is an α ∈ [0, 1] and a θu? with
 θj ∈ Θ0j
for all j ∈ u such that Pθu? (gu (Pu ) ≤ α) = Pθu? inf P−u ∈(0,1]r−1 f (P ) ≤ α > α + 
for some  > 0. From the monotonity of f , there is some fixed pe ∈ (0, 1] with
Pθu? (f (Pu :p−u ) ≤ α) > α +  for any p−u ∈ [0, pe]r−1 . Since the p-values are PRDS
and {p : f (pu , p−u ) ≤ α} is a decreasing set for any fixed p−u , we have

Pθ f (P ) ≤ α | P−u ∈ [0, pe]r−1 ≥ α + /2
?
with θj? ∈ Θ1j
for any θ satisfying θu = θu? . Using Assumption 1, there also exists θ−u
1/(r−1)
for ∀j ∈ −u such that Pθj? (Pj ≤ pe) ≥ (α + /2)/(α + )
. Since the p-values
are PRDS, using (1) we have


? ) f (P ) ≤ α ≥ P(θ ? :θ ? ) f (P ) ≤ α, Pj ≤ p
P(θu? :θ−u
e
,
∀j
∈
−u
> α + /2
u −u

contradicting the validity of f (P ).
Now we are ready to prove Theorems 3.3 and 3.4.
?
Proof of 3.3. Let gu (Pu ) be defined in (4). Then Pr/n ≥ Pr/n
when P ∈ (0, 1]n .
?
Using Assumption 2, Pr/n is then uniformly at least as powerful as Pr/n . It then
follows directly from Lemmas 3.1 and 3.5 that p?r/n is a valid GBHPC p-value.

Using Lemma 3.1, to prove 3.4, we only need to prove the monotone α-admissibility
?
.
of Pr/n
Proof of 3.4. To prove the monotone α-admissibility of f ? (P1 , · · · , Pn ), suppose
that there is a valid monotone test f ?? satisfying Pθ (f ?? (P ) ≤ α) ≥ Pθ (f ? (P ) ≤ α)
r/n
for all θ ∈ Θ1 . By 3.3 we can assume that f ?? is a GBHPC p-value:
f ?? (P ) =

max
gu0 (Pu ),
u⊂1:n
|u|=n−r+1

where gu0 is a valid monotone meta-analysis p-value. Notice that since f ? is sensitive,
equation (3) holds. We now show that for each u ⊂ 1:n with |u| = n − r + 1, and
any θu 6∈ Θ0u ,


Pθu
inf r−1 f ? (P ) ≤ α ≤ Pθu (gu0 (Pu ) ≤ α) ≡ β 0
(5)
P−u ∈(0,1]

using a similar strategy as in the proof of 3.5. If (5) does not hold for some set
u and a corresponding θu , then there exist some  > 0 and pe ∈ (0, 1] such that
12

Pθu (f ? (Pu :p−u ) ≤ α) > β 0 +  for any p−u ∈ (0, pe]r−1 . Using Assumption 1, there
1/(r−1)
.
exists θ? with θj? ∈ Θ1j for j ∈ −u such that Pθj? (Pj ≤ pe) ≥ (β 0 +/2)/(β 0 +)
Thus,
?
?
? ) (f (P ) ≤ α) ≥ P(θ :θ ? ) (f (P ) ≤ α, Pj ≤ p
P(θu :θ−u
e, ∀j ∈ −u) > β 0 + /2
u −u
??
? ) (f
> Pθu (gu0 (Pu ) ≤ α) ≥ P(θu :θ−u
(P ) ≤ α)

which violates the assumption that f ?? is uniformly at least as powerful as f ? . Thus,
(5) holds. equation (3) and (5) implies that Pθu (gu0 ≤ α) ≥ Pθu (gu (Pu ) ≤ α) for any
θu 6∈ Θ0u and any α ∈ [0, 1]. As gu (Pu ) is α-admissible for H0u , we have Pθu (gu0 (Pu ) ≤
α) = Pθu (gu (Pu ) ≤ α). Further, using Assumption 3 we have gu0 (Pu ) = gu (Pu ) a.e..
r/n
Thus, for all θ ∈ Θ1 , Pθ (f ?? (P ) ≤ α) = Pθ (f ? (P ) ≤ α) which shows that f ? is
r/n
monotone α-admissible for H0 .
Remark 3.3. As mentioned in 3.1, the BHPC p-values are symmetric GBHPC pvalues As a consequence, the BHPC p-values characterize the form of symmetric
monotone admissible combined p-values.
Combining 3.4 with results of Birnbaum (1955) and Lehmann and Romano (2006,
Theorem 6.7.1) who characterized admissible tests for the global null in exponential
families, we can give more specific conditions when 3.4 is applied to exponential
families. here is the result for a simple senario.
Example 5. Suppose that independent test statistics Ti for i = 1, . . . , n are available
on hypotheses H0i : θi = ai ∈ Rki against H0i : θi ∈ Rki /{ai }. Here we assume that
every Ti is the sufficient statistics for an exponential family with natural parameter
θi . For a sensitive GBHPC p-value f ? (p), suppose that Pθ0u (gu (Pu ) ≤ α) = α.
Also, for ∀α ∈ [0, 1] the set of Tu for which gu (pu ) > α (the acceptance region) is
a closed and convex set, except for a subset of measure 0. Then f ? (p) is monotone
r/n
α-admissible for H0 .
Related work on convexity and admissibility also appears in Matthes and Truax
(1967) for testing parameters of exponential families with presence of nuisance parameters, Marden et al. (1982) and Brown and Marden (1989) for generalization to
distribution families beyond exponential families, and Owen (2009) for tests powerful against alternatives with concordant signs. Notice that the n-dimensional set of
test statistics T itself for which f ? (p) > α is not convex. For partial conjunctions,
the null hypothesis for the parameter usually includes all of the coordinate axes and
the smallest convex set containing the axes is all of Euclidean space. As a result
convexity of the acceptance region is not appropriate to partial conjunction testing.
13

3.2

Inadmissibility
r/n

In Section 3.1 we constructed monotone α-admissible p-values for H0 , we we show
that they fail to be admissible if we allow non-monotone tests. For the case n = r = 2,
the construction of such counter-examples dates back to Lehmann (1952) and Iwasa
(1991).
Here we demonstrate that if we don’t require monotone tests then a BPHC test is
inadmissible. Let n = r = 2. If both P1 and P2 are α-admissible, then using 3.3 and
3.4, the constructed combined p-value is just P(2) , which is monotone admissible. At
a given α, the critical function is ϕ = 1p(2) ≤α (p1 , p2 ).
Now we can easily construct a more powerful α-level test, by adding to the original
rejection region a square around the top-right corner in the p-value space (Solid
shaded regions in Figure 1). Define the set
(
{(p1 , p2 ) | p(1) ≥ 1 − α}, if α < 21
S=
{(p1 , p2 ) | p(1) ≥ α},
if α ≥ 21 .
Then the test ϕ0 with critical function ϕ0 (P ) = ϕ(P ) + 1(P1 ,P2 )∈S is uniformly and
strictly more powerful than ϕ. To prove that ϕ0 is an α-level test, we note that
S ∩ {p(2) ≤ α} = ∅. Therefore E(ϕ0 (P ) | P1 = p0 ) ≤ α holds for any p0 ∈ [0, 1].
Similarly, E(ϕ0 (P ) | P2 = p0 ) ≤ α. Since p0 is arbitrary we conclude that ϕ0 is an αlevel test. Actually, as shown in Figure 1, we can further expand the rejection region
of ϕ0 to include also the dotted shaded regions and to get an even more powerful but
still valid test ϕ.
e The rejection region of ϕ
e in the p-values space consists of small
squares along the diagonal line.
If the test statistics are Z1 ∼ N (µ1 , 1) and Z2 ∼ N (µ2 , 1), and H1 and H2 are
two-sided tests for the mean µ1 and µ2 respectively, then the top two plots of Figure 1
show the rejection region of ϕ0 and ϕ
e at level α = 0.1 in the p-value space and in
the test statistic space. The bottom two plots compare the power of ϕ and ϕ
e as a
function of (µ1 , µ2 ). They show that the power gain of the non-monotone ϕ
e only
appears in the low power region where the power is below or near α.
2/2
The more powerful test ϕ0 increases power by strangely rejecting H0 when both
input p-values are large enough. We now use this same approach to show that
without the monotonicity constraint, any GBHPC p-value is inadmissible for any n
r/n
and any r ∈ 2:n. The counter-examples reject H0 when all p-values are large. The
idea is to show that for any GBHPC test, it’s always possible to add a “box”-shaped
rejection region like the square around the origin in the right panel of Figure 1 while
still keeping the test valid. The point is not to advocate for such tests, but rather

14

to reinforce the idea that admissibility is only a useful concept within a well chosen
class of functions.
We need the following mild technical constraint to guarantee that the “box” we
choose can really increase power at least in one alternative hypothesis.
Assumption 4. For each i ∈ 1:n, there exists θi0 ∈ Θ0i that Pθi0 (Pi ≤ α) =
supθi ∈Θ0i (Pi ≤ α) for ∀α ∈ [0, 1]. Let θ0 = (θ10 , θ20 , · · · , θn0 ). Then for any set A,
r/n
if Pθ0 (A) > 0, then there exists θ1 ∈ Θ1 that Pθ1 (A) > 0.
Theorem 3.6. Let P1 , . . . , Pn be independent p-values satisfying assumptions 1, 2
and 4. Let 1 < r ≤ n and α ∈ (0, 1). Then any monotone α-admissible combined
r/n
p-value for testing H0 is not α-admissible without the monotonicity constraint.
Proof. Using 3.3, we only need to consider a GBHPC p-value f ? which is defined in
6. Let θ0 = (θ10 , θ20 , · · · , θn0 ) be the parameter in Assumption 4. define
\
Ru
R = {p ∈ [0, 1]n : f ? (p) ≤ α} =
u⊂1:n
|u|=n−r+1

where Ru = {p ∈ [0, 1]n : gu (pu ) ≤ α} and gu is defined in (2).
First, as Pθ0 (f ? ≤ α) ≤ α < 1 and f ? is non-decreasing, there exists some p0 < 1
such that if pj ≥ p0 for all j ∈ 1:n then f ? (p) > α.
Then, we show that there must exist a set u? with Pθ0 (Ru? ∩Rc ) =  > 0, where Rc
is the complement set of R. If this doesn’t hold, then it means that for any u ⊂ 1:n
with |u| = n − r + 1, the equation 1f ? (p)≤α (p) = 1gu (pu )≤α (p) a.e. Pθ0 holds. This
implies that 1f ? ≤α doesn’t depend on p−u except for a zero probability set under Pθ0 .
As ∪ u⊂1:n −u = 1:n, we get that 1f ? ≤α doesn’t depend on any pj except for a zero
|u|=n−r+1
probability set under Pθ0 , which implies that 1f ? ≤α ≡ 1 or 0 a.e. Pθ0 . It’s obvious
that such a test is either invalid or trivially not admissible, which contradicts our
assumptions.
As a consequence, we have Pθ0 (f ? ≤ α) = Pθ0 (Ru? ) −  ≤ α − . Notice that
0
Pθu0 [f ? ≤ α | P−u ] for any u. using the fact that f ? is nonPθ0 (f ? ≤ α) = Eθ−u
decreasing, Pθu0 [f ? ≤ α | P−u = p−u ] is non-increasing in p−u . Thus there exists
pe < 1, such that for any u, if p−u ∈ [e
p, 1]r−1 , then
Pθu0 [f ? ≤ α | P−u = p−u ] ≤ α − .
Let p? = max(p0 , pe, 1 − 1/(n−r+1) ) and S = ∩i {p ∈ [0, 1]n : pi ≥ p? }. Then we
construct a new test with critical function ϕ: ϕ = 1f ? ≤α + 1S .
15

As {p ∈ [0, 1]n : f ? (p) ≤ α} ∩ S = ∅, we know that ϕ is at least as powerful as
1f ? ≤α . Using Assumption 4, as Pθ0 (S) ≥ (1 − p? )n > 0, there exists θ1 ∈ Θr/n
with
1
1
n
Pθ1 (S) > 0. Thus, ϕ strictly dominates 1f ? ≤α at θ . Finally, for ∀p ∈ [0, 1] and
∀u ⊂ 1:n with |u| = n − r + 1, if θu ∈ H 1/n−r+1 , then

Eθu [ϕ | P−u = p−u ] ≤ Pθu [f ? ≤ α | P−u = p−u ] + 1p−u ∈[p? ,1]r−1
≤ Pθu0 [f ? ≤ α | P−u = p−u ] + 1p−u ∈[p? ,1]r−1 ≤ α.
The second inequality above follows from Assumption 4, independence of the individual p-values and monotonicity of f ? . Thus ϕ is still an α-level test for H r/n . This
shows that f ? is not α-admissible.

4

Simulation

In this simulation example, we compare the power of several GBHPC p-values testing
r/n
for the PC hypothesis H0 with n = 8 studies and r = 2. Compared with other r
2/n
values, the null hypothesis H0 is often of particular interest as it tests whether the
significance of the effect can replicate or not across studies. It is also the case where
the computational cost of non-symmetric GBHPC p-value would typically not be a
concern.
We consider the alternative whose true number of non-null hypotheses is one of
r0 = 2, 4, 6. We assume that all the individual hypotheses are independent.
Each
√
p-value Pi is a two-sided p-value of the corresponding z-value Zi ∼ N ( Ni µi , 1) for
i = 1, 2, . . . , 8. We set three of the sample sizes Ni of the eight individual studies to
100, another three of them to 500 and the last two to 1000. for the effects µi , if H0i
i.i.d
is true, then µi = 0. Otherwise when
p H0i is false, we generate µi ∼ Gamma(α0 , β0 ).
We define µ0 = α0 /β0 and σ0 = α0 /β02 which are the mean and standard deviation
of the non-null effect across studies. We compare the power of each GBHPC p-value
as a function of (µ0 , σ0 ) at the significance level of α = 0.05.
We compare three GBHPC P-values, whose meta-analysis P-values gu are from
examples 1 to 3 respectively. The results are shown in Figure 2. For each r0 and
each form of the GBHPC p-value, we plot a power map against (µ0 , σ0 ). To better
illustrate the difference across methods, for Simes and weighted Stouffer GBHPC
p-values, we plot their powers after they are substracted by the powers of Fisher’s
BHPC p-value at the corresponding location. Here are some observations from the
simulation results. First, Figure 2 shows that when r0 = 2, Simes BHPC p-value
is the most powerful in a large region of the alternative space. The reason is that,
for each subset of hypotheses with size n − r + 1 = n − 1, at the worst case there
16

is only one non-null individual hypothesis, where simes should be most powerful
in detecting extreme p-values. Second, the Weighted Stouffer GBHPC p-value can
have higher power than Fisher’s BHPC p-value when r0 > r = 2 and when the effect
heterogeneity across non-null studies (σ0 ) is not too large to dominate the average
effect (µ0 ). Notice that we are using two-sided p-values to make a fair comparison of
√
the methods, thus taking ni as weights in Stouffer’s method would not be optimal.
However, as we have discussed in Example 3 and shown here, it can still provide
a good test with two-sided p-value. If individual p-values were one-sided, then we
would have seen a even higher power gain. Finally, the three methods would not
have a noticeable difference in power when the power is too low or too high. Most
of the difference apprear when the power is in the range of 0.4 to 0.6.

5

Real data analysis

Compared with BHPC p-values, GBHPC p-values has the flexiblility to make use of
the possibly complex dependency structure across studies, thus may achieve higher
power. We use a real data example to illustrate this benefit.
The dataset (Ruff et al., 2014) is a pooled dataset from four randomized clinical
trials aiming to measure the relative benefit of new anticoagulants (NOAC) compared
with an old drug called warfarin for stroke prevention. One primary goal in the
original paper was to assess and compare the efficiency of these new drugs in different
clinical subgroups of patients. The subgroups and the data are shown in Table 2a.
The data (m/N ) records that among N samples the number of samples who suffer
a stroke or systemic embolic event is m. Thus, to test the different between using
the old and new drug, we estimate the odds ratio and compute p-values for each
subgroup using Fisher’s exact test.
We can use PC tests to assess the consistency of the drug efficiency across different
subgroups, which is one of the major interest of the original paper. One major
difficulty for applying PC tests (and meta-analysis) to theses subgroups is that these
subgroups are correlated. For example, the subgroup of age ≤ 75 clearly overlaps
with the subgroup of female. When there is unknown dependency across individual
hypotheses, the only BHPC p-values available are to have the meta-analysis p-value
g be either the Bonferroni or Simes p-value. The left column of Table 2b are the
r/n
values of the Bonferroni BHPC p-values pBon
with n = 18 and r changing
r/n for H0
Bon
from 2 to 18. As pr/n is increasing in r except when r change from 16 to 17, the
values of Simes BHPC p-values will agree with pBon
r/n except when r = 16.
The question is: is it possible to get smaller valid p-values? The answer is
17

YES, by using a non-symmetric GBHPC p-value. Notice that the groups which
represent different levels of the same grouping factor do not share samples, thus have
independent p-values. For example, the three p-values for the three CHADS2 groups
are independent. Let 1:18 = ∪8i=1 Ii where each Ii is the index set of subgroups using
the ith grouping factor. Then, to build a GBHPC p-value pnew
r/n , we construct gu (pu )
for each u ∈ 1:n with |u| = n − r + 1 as follows: for each Ii ∩ u 6= ∅, a Fisher’s
p-value (Example 2) pu,i is calculated on pu∩Ii , then


gu (pu ) = {i : u ∩ Ii 6= ∅} · min pu,i
i

which is a Bonferroni combination of p-values across the grouping factors. The above
construction obviously provides a valid GBHPC p-value.
The next concern is the computation of pnew
r/n . We claim that for any given r, it
can be quickly computed by checking Table 2c. Table 2c are all pu,i values that can
possibly affect the value of pnew
r/n for any r. Notice that
f ? (p) =

max
u,|u|=n−r+1

min pu,i .
i

Since pu,i is symmetric, it can possibly influence f ? only when pu,i is the Fisher’s
combination on the largest |µ ∩ Ii | p-values in Ii . This explains the values in Table 2c. Then, computing pnew
r/n becomes easy. One nice phenomenon of the p-values in
Table 2c is that p-values on the first row are uniformly smaller than the p-values on
the second row, the latter further uniformly smaller than p-values on the third row.
Such a property can greatly simplify the computation. The calculation of gu (pu ) is by
replacing some pIi ,i (first row p-value in Table 2c) with some larger pu,i (higher row
p-values) or completely removing it. For example when r = 4, there are n − r + 1 = 3
indices that are not in u. Thus for any u, at most 3 of the pIi ,i can be replaced. If
we denote pI(1) ,(1) ≤ . . . pI(8) ,(8) , then it’s easy to see that pnew
4/n = 8pI(4) ,(4) .
It is a bit more complicated when r ≥ 9. First, notice that for the r − 1 indices
that are not in argmaxu gu (pu ), we should have each Ii ∩ −u 6= ∅ to avoid appearance
of p-values on the first row in Table 2c. Then the problem becomes optimizing the
location of the rest r − 1 − 8 indices to maximize gu (pu ). For example when r = 12,
there are 4 indices left and we now just need to examine the p-values on the second
and third row in Table 2c. If we denote the p-values on the second row as pui ,i and
define pu(1) ,(1) ≤ . . . pu(8) ,(8) , using a similar argument as r = 4, one can check that
pnew
12/n = max{7pu(2) ,(2) , 6pu(3) ,(3) , 5pu(4) ,(4) }.
Finally, the pnew
r/n values for all r = 2, . . . , 18 are shown in Table 2b. Compared
with Bonferroni BHPC p-values, the new GBHPC p-values can be much smaller
especially when r is small. Both methods give a 95% confidence interval of the true
proportion of non-null hypotheses r0 /n as r0 /n ∈ [0.72, 1].
18

6

Conclusion and future work

Partial conjunction hypotheses are natural hypotheses to test for measuring repeated
effects across settings/studies. The null is rejected only when at least r hypotheses
are non-null. By testing PC hypotheses at different r values, one can also construct
a confidence interval of r0 /n, the true proportion of non-null hypotheses.
This paper characterizes the admissible p-values for a partial conjunction test
of independent hypotheses or hypotheses with positively dependent P-values, within
r/n
the class of non-decreasing p-values. Any monotone admissible p-value for H0 is the
maximum of the non-decreasing p-values for the global null in each combination of
n−r+1 hypotheses, which we call GBHPC p-values. We have shown that for sensitive
GBHPC p-values, as long as each meta-analysis p-value of the n − r + 1 hypotheses
is admissible, the combined p-value is monotone admissible. A consequence is that
among combined p-values that only depend on the order statistics of individual pvalues, the original BHPC p-values are the only monotone admissible ones. We also
have found inadmissibility of GBHPC p-values without the monotonicity constraint.
However, the dominated tests only have a moderate power gain at low power regions
in the alternative space. Since these counter-examples are not monotone, they are
hard to be explained in practice thus not reasonable choices.
In summary, we illustrated the properties of tests for a PC hypothesis and characterized a class of good tests called GBHPC p-values. Compared with its symmetric
form, the BHPC p-values, GBHPC p-values have more flexibility to adapt to complicated problem structure, thus can have power gain at important regions in the
alternative space, as we showed in our simulations and real data examples. The computational cost of non-symmetric GBHPC p-values can be of a concern, but there are
special cases where GBHPC p-values are computable. One of the future directions
is to expand the applications where computable GBHPC p-values are available.
One other direction is to understand properties of the confidence interval of r
constructed by GBHPC p-values. In Section 5, the result showed that though the
newly proposed GBHPC p-values can be much smaller than Simes or Bonferroni
BHPC p-values at many r values, the confidence interval that the two methods
constructed are still the same. Finally, there are variations of partial conjunctions
that are useful in practice. For example, the count of replicability may vary for
different hypotheses. Replication of effects from two distinct classes can be of more
interest than replication in two similar classes. Another variation is to require that
a null hypothesis is rejected only when there are at least r non-nulls with the same
sign of effect. Such hypotheses can have very complex alternative and null space,
and it can be the future work to understand their properties.
19

References
Benjamini, Y. and R. Heller (2008). Screening for partial conjunction hypotheses.
Biometrics 64 (4), 1215–1222.
Benjamini, Y. and D. Yekutieli (2001). The control of the false discovery rate in
multiple testing under dependency. Annals of statistics, 1165–1188.
Birnbaum, A. (1954). Combining independent tests of significance. Journal of the
American Statistical Association 49 (267), 559–574.
Birnbaum, A. (1955). Characterizations of complete classes of tests of some multiparametric hypotheses, with applications to likelihood ratio tests. The Annals of
Mathematical Statistics 26 (1), 21–36.
Brown, L. D. and J. I. Marden (1989). Complete class results for hypothesis testing
problems with simple null hypotheses. The Annals of Statistics, 209–235.
Flutre, T., X. Wen, J. Pritchard, and M. Stephens (2013). A statistical framework
for joint eqtl analysis in multiple tissues. PLoS Genet 9 (5), e1003486.
Friston, K. J., A. P. Holmes, C. J. Price, C. Büchel, and K. J. Worsley (1999).
Multisubject fMRI studies and conjunction analyses. Neuroimage 10 (4), 385–396.
Friston, K. J., W. D. Penny, and D. E. Glaser (2005). Conjunction revisited. NeuroImage 25 (3), 661–667.
Heller, R., D. Yekutieli, et al. (2014). Replicability analysis for genome-wide association studies. The Annals of Applied Statistics 8 (1), 481–498.
Higgins, J., S. G. Thompson, and D. J. Spiegelhalter (2009). A re-evaluation of
random-effects meta-analysis. Journal of the Royal Statistical Society: Series A
(Statistics in Society) 172 (1), 137–159.
Iwasa, M. (1991). Admissibility of unbiased tests for a composite hypothesis with
a restricted alternative. Annals of the Institute of Statistical Mathematics 43 (4),
657–665.
Lehmann, E. L. (1952). Testing multiparameter hypotheses. The Annals of Mathematical Statistics 23 (4), 541–552.
Lehmann, E. L. and J. P. Romano (2006). Testing statistical hypotheses. Springer
Science & Business Media.
20

Marden, J. I. et al. (1982). Combining independent noncentral chi squared or f tests.
The Annals of Statistics 10 (1), 266–277.
Matthes, T. K. and D. R. Truax (1967). Tests of composite hypotheses for the
multivariate exponential family. The Annals of Mathematical Statistics 38 (3),
681–697.
Nichols, T., M. Brett, J. Andersson, T. Wager, and J.-B. Poline (2005). Valid
conjunction inference with the minimum statistic. Neuroimage 25 (3), 653–660.
Owen, A. B. (2009). Karl Pearson’s meta-analysis revisited. The Annals of Statistics 37 (6B), 3867–3892.
Perlman, M. D. and L. Wu (1999). The emperor’s new tests. Statistical Science 14 (4),
355–369.
Price, C. J. and K. J. Friston (1997). Cognitive conjunction: A new approach to
brain activation experiments. NeuroImage 5 (4), 261–270.
Ruff, C. T., R. P. Giugliano, E. Braunwald, E. B. Hoffman, N. Deenadayalu, M. D.
Ezekowitz, A. J. Camm, J. I. Weitz, B. S. Lewis, A. Parkhomenko, et al. (2014).
Comparison of the efficacy and safety of new oral anticoagulants with warfarin
in patients with atrial fibrillation: a meta-analysis of randomised trials. The
Lancet 383 (9921), 955–962.
Shenhav, L., R. Heller, and Y. Benjamini (2015). Quantifying replicability in systematic reviews: the r-value. arXiv preprint arXiv:1502.00088 .
Stein, C. (1956). The admissibility of Hotelling’s T 2 -test. The Annals of Mathematical Statistics 27 (3), 616–623.
Wang, W., Z. Wei, and W. Sun (2010). Simultaneous set-wise testing under dependence, with applications to genome-wide association studies. Stat. Interface 3,
501–511.
Zaykin, D. V., L. A. Zhivotovsky, P. H. Westfall, and B. S. Weir (2002). Truncated
product method for combining p-values. Genetic epidemiology 22 (2), 170–185.

21

Rejection region of (p1,p2)

Rejection region of (Z1,Z2)

1.0
0.9

2

Z2

1
0.5

0
−2

−1
−1

1

2

−2
0.1
0.0
0.0

0.5

0.9

~
powerZ1of ϕ

(1:10)/10
power
of ϕ

4

4

1.0

2

0.6
0

0

p.seq

2

0.8

−2

−2

0.4

−4

−4

0.2

0.0
−4

−2

0

2

4

−4

−2

0

2

4

Figure 1: The top two plots: rejection regions of ϕ0 and ϕ
e in the p-value space and
the test statistic space, using α = 0.2. The sold shaded region is the rejection region
of ϕ0 , while the rejection region of ϕ
e also includes in the dotted shaded squares. The
Bottom two plots: power comparison of ϕ and ϕ.
e The dashed line is where power is
at 0.15 and the dotted line is where the power is 0.5.

22

0.30

Power of Fisher's BHPC

Power Gain of Simes BHPC

Power Gain of WS GBHPC

1.0

0.10

0.20

0.05

0.6
0.00
0.4

0.10

σ0 (r0 = 2)

0.8

−0.05

0.00

0.2
0.0
0.05

0.15

0.25

−0.10
0.05

0.15

0.25

0.05

0.15

0.25
0.10

0.8

0.05

0.08

0.6
0.00
0.4

0.04

σ0 (r0 = 4)

0.12

1.0

−0.05

0.00

0.2
0.0
0.04

0.08

0.12

−0.10
0.04

0.08

0.12

0.04

0.08

0.12

1.0
0.08

0.8

0.05

0.6
0.00
0.4

0.04

σ0 (r0 = 6)

0.10

−0.05

0.00

0.2
0.0
0.02

0.04

0.06

0.08

0.10

−0.10
0.02

0.04

0.06

µ0

µ0

0.08

0.10

0.02 0.04 0.06 0.08 0.10
µ0

Figure 2: Power comparison of GBHPC p-values: Each row is for one r0 value and
each column is for one form of GBHPC p-value. the significance level is α = 0.05.
The first column are power maps Fisher’s BHPC p-value against (µ0 , σ0 ). the last
two columns are the power difference of Simes and Weighted Stouffer’s GBHPC pvalues against Fisher’s p-value. The green color indicates a power loss compared
with Fisher’s BHPC p-value while the blue color shows a power gain.

23

(a)
Pooled NOAC
(events)

Pooled Warfarin
(events)

Age(years)
≤ 75
496/18073
≥ 75
415/11188
Sex
Female
382/10941
Male
531/18371
Diabetes
No
622/20216
Yes
287/9096
Previous stroke or TIA
No
483/20699
Yes
428/8663
Creatinine clearance (mL/min)
≤ 50
249/5539
50 − 80
405/13055
≥ 80
256/10626
CHADS2 score
0−1
69/5058
2
247/9563
3−6
596/14690
VKA status
Naive
386/13789
Experienced
522/15514
Centre-based TTR
≤ 66
509/16219
≥ 66
313/12742

Estimated
Odds Ratio

p value

578/18004
532/11095

0.85
0.76

9.26E-03
6.61E-05

478/10839
634/18390

0.78
0.83

5.00E-04
2.38E-03

755/20238
356/8990

0.82 2.93E-04
0.79 3.81E-03

615/20637
495/8635

0.78 4.65E-05
0.85 2.14E-02

311/5503
546/13155
255/10533

0.79 6.24E-03
0.74 5.85E-06
1.00 9.64E-01

90/4942
290/9757
733/14528

0.75 7.83E-02
0.87 1.05E-01
0.80 5.21E-05

513/13834
597/15395

0.75
0.86

2.19E-05
1.61E-02

653/16297
392/12904

0.78
0.80

2.49E-05
4.68E-03

(b)
r
pBon
pnew
r/n
r/n
2
3.73E-04 4.49E-05
3
3.98E-04 4.66E-05
4
6.98E-04 7.50E-05
5
7.29E-04 1.18E-04
6
8.59E-04 1.31E-04
7
3.52E-03 1.39E-04
8
5.50E-03 4.23E-04
9
2.38E-02 1.90E-02
10
3.43E-02 2.66E-02
11
3.75E-02 2.81E-02
12
4.37E-02 4.63E-02
13 5.56E-02
6.45E-02
14
8.07E-02 6.45E-02
15
8.56E-02 7.36E-02
16
2.35E-01 7.36E-02
17
2.11E-01
2.11E-01
18
9.64E-01
9.64E-01

(c)
1/k

H0 i
2/k
H0 i
3/k
H0 i

Age
Sex Diabetes
9.37E-06 1.74E-05
1.64E-05
9.26E-03 2.38E-03
3.81E-03
–
–
–

Stroke or TIA
1.47E-05
2.14E-02
–

Creatinine
5.83E-06
3.68E-02
9.64E-01

CHADS2
5.29E-05
4.78E-02
1.05E-01

VKA
5.61E-06
1.61E-02
–

TTR
1.98E-06
4.68E-03
–

Table 2: (a) The original data and individual p-values: the blue color highlights
individual p-values that are not significant at level α = 0.05; (b) the values of
r/n
Bonferroni BHPC p-value and the new GBHPC p-values when r changes in H0 ;
(c) the grouping factor level combined p-values: here ki = |Ii | and each p-value is
r /k
the Fisher’s BHPC p-value on pIi testing for H0 1 i where r1 = 1, 2, 3.

24

