The Control of the False Discovery Rate in Fixed
arXiv:1611.03146v1 [stat.ME] 10 Nov 2016

Sequence Multiple Testing
Gavin Lynch
Catchpoint Systems, Inc., 228 Park Ave S ]28080
New York, NY 10003, U.S.A.
Wenge Guo∗
Department of Mathematical Sciences
New Jersey Institute of Technology
Newark, NJ 07102, U.S.A.
Sanat K. Sarkar†
Department of Statistics, Temple University
Philadelphia, PA 19122, U.S.A.
Helmut Finner
Institute for Biometrics and Epidemiology
German Diabetes Center at the Heinrich-Heine-University
Auf’m Hennekamp 65, D-40225 Düsseldorf, Germany

∗
†

The research of Wenge Guo was supported in part by NSF Grants DMS-1006021 and DMS-1309162.
The research of Sanat Sarkar was supported in part by NSF Grants DMS-1006344 and DMS-1309273.

1

Abstract
Controlling the false discovery rate (FDR) is a powerful approach to multiple
testing. In many applications, the tested hypotheses have an inherent hierarchical
structure. In this paper, we focus on the fixed sequence structure where the testing
order of the hypotheses has been strictly specified in advance. We are motivated to
study such a structure, since it is the most basic of hierarchical structures, yet it is
often seen in real applications such as statistical process control and streaming data
analysis. We first consider a conventional fixed sequence method that stops testing
once an acceptance occurs, and develop such a method controlling the FDR under
both arbitrary and negative dependencies. The method under arbitrary dependency
is shown to be unimprovable without losing control of the FDR and unlike existing
FDR methods; it cannot be improved even by restricting to the usual positive regression dependence on subset (PRDS) condition. To account for any potential mistakes
in the ordering of the tests, we extend the conventional fixed sequence method to one
that allows more but a given number of acceptances. Simulation studies show that
the proposed procedures can be powerful alternatives to existing FDR controlling
procedures. The proposed procedures are illustrated through a real data set from a
microarray experiment.

AMS 2000 subject classifications: Primary 62J15
KEY WORDS: Arbitrary dependence, false discovery rate, fixed sequence, multiple testing, negative association, PRDS property, p-values.

1

Introduction

In many applications of multiple testing, such as genomic research, clinical trials, and
statistical process control, the hypotheses are so structured that they are to be tested in a
particular sequence. This structure may be a natural one, as in Goeman and Mansmann
(2008), where Gene Ontology imposes a directed acyclic graph structure onto the tested
hypotheses, or can be formed by using a data-driven approach for specifying the testing
order of the hypotheses, as in Kropf and Läuter (2002), Kropf et al. (2004), Westfall
et al. (2004), Hommel and Kropf (2005), Finos and Farcomeni (2011), etc. In some
applications, it is not even possible to use the conventional p-value based multiple testing
2

methods, because of some inherent structure among the tested hypotheses. For example,
the hypotheses associated with stream data in sequential change detection problems (Ross
et al., 2011) have a natural temporal structure, but none of conventional methods, such as
the stepwise procedures, which are applicable only when all of p-values are available, can
be used here since the decision concerning a hypothesis has to be made even before the
data associated with the remaining hypotheses are observed.
Some progress has been made in testing structured hypotheses. However, it has been
primarily focused on controlling the familywise error rate (FWER) (Maurer et al., 1995;
Westfall and Krishen, 2001; Wiens, 2003; Wiens and Dmitrienko, 2005; Hommel et al.,
2007; Huque and Alosh, 2008; Li and Mehrotra, 2008; Rosenbaum, 2008; Wiens and
Dmitrienko, 2010; Millen and Dmitrienko, 2011; Dmitrienko et al., 2013). There are
few results towards controlling the false discovery rate (FDR) while accounting for the
structure of the tested hypotheses. Benjamini and Heller (2007), Heller et al. (2009),
and Mehrotra and Heyse (2004) developed methods for testing hypotheses with a specific
hierarchical structure where the structure is limited to only two levels. Yekutieli (2008)
discussed a method that controls the FDR when the tested hypotheses have a general
hierarchical structure. However, that method is shown to control the FDR only under
independence.
The primary objective of this paper is to help advance the theory and methods on controlling the FDR for testing structured hypotheses. We do so by focusing on a structure
where the hypotheses have a fixed pre-defined testing order since this is the simplest of
hierarchical structures, yet it is often seen in real applications such as clinical trials, statistical process control and streaming data analysis. For such a structure, we will refer to
it as a fixed sequence structure throughout this paper. Very recently, several methods have
been introduced for controlling FDR while testing pre-ordered hypotheses. Farcomeni
and Finos (2013) developed a ‘single-step’ FDR controlling method for testing hypotheses with the same critical value α, which tests each hypothesis at level α until a stopping
condition is reached. Barber and Candes (2015), G’Sell et al. (2016) and Li and Barber
(2016) developed several different ‘step-up’ FDR controlling procedures in the context
of high-dimensional regression for testing hypotheses with fixed sequence structure for
which hypotheses are tested from highest-ranked to lowest-ranked, and Lei and Withian
(2016) performed asymptotic power analysis for such ‘step-up’ procedures. In addition,
3

Javanmard and Montanari (2015) developed procedures for controlling the FDR in an
online manner while testing a sequence of possibly infinite pre-ordered hypotheses.
In this paper, we develop ‘step-down’ FDR controlling methods that fully exploit
the fixed sequence structural information, in which hypotheses are tested from lowestranked to highest-ranked. We first consider a conventional fixed sequence multiple testing method that keeps rejecting until an acceptance occurs and develop such a method
controlling the FDR under arbitrary dependence. It is shown to be optimal in the sense
that it cannot be improved by increasing even one of its critical values without losing
control over the FDR, or even by imposing a positive dependence condition on the pvalues, such as the standard PRDS (positive regression dependence on subset) condition
of Benjamini and Yekutieli (2001). This is different from what happens in case of nonfixed sequence multiple testing. For instance, the so-called BY method of Benjamini and
Yekutieli (2001) that controls the FDR under arbitrary dependence can be improved significantly by the BH method of Benjamini and Hochberg (1995) by imposing this PRDS
condition. Since our procedure cannot be improved under positive dependence, we consider the case of negative dependence and develop a more powerful conventional fixed
sequence multiple testing method controlling the FDR under negative dependence which
includes independence as a special case.
There is a potential for loss of power in a conventional fixed sequence multiple testing method if the ordering of the hypotheses, particularly for the earlier ones, does not
match with that of their true effect sizes, potentially leading to some earlier hypothesis
being accepted and the follow-up hypotheses having no chance to be tested. To mitigate
that, we consider generalizing the conventional fixed sequence multiple testing to one that
allows more than one but a pre-specified number of acceptances, and develop such generalized fixed sequence multiple testing methods controlling the FDR under both arbitrary
dependence and independence.
It is not always the case in real data applications that the hypotheses will have a natural fixed sequence structure or information about how to order them will be available
a priori. Nevertheless, the data itself can often provide information on how to order the
hypotheses. In this paper, we discuss such a data-driven ordering strategy which can be
applied to a broad spectrum of multiple testing problems such as one-sample and twosample t-tests, and one-sample and two-sample nonparametric tests. Through simulation
4

studies and a real microarray data analysis, this strategy coupled with our proposed fixed
sequence methods is seen to perform favorably against the corresponding non-fixed sequence methods under certain settings.
The paper is organized as follows. With some concepts and background information
given in Section 2, we present the developments of our conventional and generalized fixed
sequence procedures controlling the FDR under various dependencies in Sections 3 and 4,
respectively. Our fixed sequence procedures coupled with a data-driven ordering strategy
for the hypotheses are applied to a real microarray data in Section 5. The findings from
some simulation studies on the performances of our procedures are given in Section 6.
Some concluding remarks are made in Section 7 and proofs of some results are given in
the Appendix.

2

Preliminaries

Suppose that Hi , i = 1, . . . , m, are the m null hypotheses that are ordered a priori and
are to be simultaneously tested based on their respective p-values Pi , i = 1, . . . , m. Let
m0 and m1 of these null hypotheses be true and false, respectively. For notational convenience, we denote the index of the ith true null hypothesis by ui and the set of indices
of the true null hypotheses by I0 . Let V and S be the numbers of true and false null
hypotheses, respectively, among the R rejected null hypotheses in a multiple testing procedure. Then, the familywise error rate (FWER) and false discovery rate (FDR)
of this

V
procedure are defined respectively as FWER = Pr(V > 0) and FDR = E max(R,1) =

E V V+S 1{V >0} .
Typically, the hypotheses are ordered based on their p-values and multiple testing is
carried out using a stepwise or single-step procedure. However, when these hypotheses
are ordered a prior and not according to their p-values, multiple testing is often performed
using a fixed sequence method. Given a non-decreasing sequence of critical constants
0 < α1 ≤ . . . ≤ αm , a conventional fixed sequence method is defined as follows:
Definition 1 (Conventional fixed sequence method)
1. If P1 ≤ α1 , then reject H1 and continue to test H2 ; otherwise, stop.
2. If Pi ≤ αi then reject Hi and continue to test Hi+1 ; otherwise, stop.
5

Thus, a conventional fixed sequence method continues testing in the pre-determined
order as long as rejections occur. Once an acceptance occurs, it stops testing the remaining hypotheses. In Section 4, we will generalize a conventional fixed sequence method
to allow a given number of acceptances. It should be noted that a conventional fixed
sequence method with common critical constant α, which is often called the fixed sequence procedure in the literature, strongly controls the FWER at level α (Maurer et al.,
1995). We will refer to it as the FWER fixed sequence procedure in this paper in order to
distinguish it from other fixed sequence methods designed to control the FDR.
Regarding assumptions we make about the p-values in this paper, we assume that the
true null p-values, which we denote for notational convenience by Pbi , for i = 1, . . . , m0 ,
are marginally distributed as follows:


Pr Pbi ≤ p ≤ p for any p ∈ (0, 1).

(1)

One of several types of dependence, like arbitrary dependence, positive dependence, negative dependence, and independence, has been assumed to characterize a dependence
structure among the p-values.
By arbitrary dependence, we mean that the p-values do not have any specific form of
dependence. The positive dependence is characterized by the positive regression dependence on subset (PRDS) property (Benjamini and Yekutieli, 2001) as defined below:
Definition 2 (PRDS) The vector of p-values P~ is PRDS on the vector of null p-values
P~0 = (Pb1 , . . . , Pbm0 ) if for every increasing
 set D and for each i = 1, . . . , m0 , the conditional probability Pr P~ ∈ D Pbi = p is non-decreasing in p.
Several multivariate distributions possess this property (see, for instance, Benjamini
and Yekutieli, 2001; Sarkar, 2002). The negative dependence is characterized by the
following property:
Definition 3 (Negative Association) The vector of p-values P~ is negatively associated
with null p-values if for each i = 1, . . . , m0 , the following inequality holds:


Pr Pbi ≤ pui , Pj ≤ pj , j = 1, . . . , m, with j 6= ui


b
≤ Pr Pi ≤ pui Pr (Pj ≤ pj , j = 1, . . . , m, with j 6= ui ) ,
6

(2)

for all fixed pj ’s.
Several multivariate distributions posses the conventional negative association property, including multivariate normal with non-positive correlation, multinomial, dirichlet,
and multivariate hypergeometric (Joag Dev and Proschan, 1983). It is easily seen that
independence is a special case of negative dependence.

3

Conventional Fixed Sequence Procedures

In this section, we present the developments of two simple conventional fixed sequence
procedures controlling the FDR under both arbitrary dependence and negative dependence conditions on the p-values.

3.1

Procedure under arbitrary dependence

Since the FDR is more liberal than the FWER, a conventional fixed sequence method
controlling the FDR under arbitrary dependence is expected to have critical values that
are at least as large as α, the common critical constant for the FWER fixed sequence
method. In the following, we present such a simple conventional fixed sequence FDR
controlling procedure.
Theorem 3.1 Consider a conventional fixed sequence procedure with critical constants
(1)
αi


= min


mα
, 1 , i = 1, . . . , m.
m−i+1

(i) This procedure strongly controls the FDR at level α under arbitrary dependence of
the p-values.
(1)

(ii) One cannot increase even one of the critical constants αi , i = 1, . . . , m, while
keeping the remaining fixed without losing control of the FDR. This is true even
when P~ is assumed to be PRDS on P~0 .
Proof of (i). Since u1 is the index of the first true null hypothesis, the first u1 − 1
null hypotheses are all false. Note that the event {V > 0} implies that S ≥ u1 − 1 and

7

(1)

P̂1 ≤ αu1 , and therefore we have



m0
V
1{V >0} ≤ E
1{V >0}
FDR = E
V +S
m0 + u1 − 1
m0
m0
=
P r(V > 0) ≤
P r(P̂1 ≤ αu(1)
)
1
m0 + u1 − 1
m0 + u1 − 1
m − u1 + 1 (1)
αu1 ≤ α.
≤
m


The first inequality follows from the fact that V /(V + S) is an increasing function of V
and a decreasing function of S. The third inequality follows from the fact that m0 /(m0 +
u1 − 1) is an increasing function of m0 and m0 ≤ m − u1 + 1 since there are at least
u1 − 1 false nulls. This proves part (i).
For a proof of part (ii), see Appendix.
Remark 3.1 Theorem 3.1 shows that when controlling the FDR under arbitrary dependence, the operating characteristic of the proposed fixed sequence method is much different from that of the usual stepwise procedure of Benjamini and Yekutieli (Benjamini
and Yekutieli, 2001) that relys on p-value based ordering of the hypotheses. It cannot be
further improved, even by imposing the PRDS assumption on the p-values, unlike the BY
procedure that is known to be greatly improved by the Benjamini-Hochberg (BH) procedure under such positive dependence. Also, as shown in our proof of Theorem 3.1(ii)
under arbitrary dependence (see Appendix), the least favorable configuration (the configuration which leads to the largest error rate, see Finner and Roters, 2001) of the newly
suggested fixed sequence FDR controlling procedure is when the ordering of the hypotheses is perfect (i.e, when all the false null hypotheses are tested before the true ones), the
false null p-values are all 0 with probability 1, and the true null p-values are the same with
each following U (0, 1) distribution. This least favorable configuration is much different
from that given in Guo and Rao (2008) for the BY procedure under arbitrary dependence.
Although the procedure in Theorem 3.1 cannot be improved under the PRDS condition, we consider in the next subsection the condition of negative dependence which
includes independence as a special case, and under such condition, develop a more powerful conventional fixed sequence method that controls the FDR.

8

3.2

Procedure under negative dependence

When the p-values are negatively associated as defined in Section 2, the critical constants
of the conventional fixed sequence procedure in Theorem 3.1 can be further improved as
in the following:
Theorem 3.2 The conventional fixed sequence method with critical constants
(2)

αi =

iα
, i = 1, . . . , m
1 + (i − 1)α

strongly controls the FDR at level α when the p-values are negatively associated on the
true null p-values.
To prove Theorem 3.2, we use the following lemma, with proof given in Appendix:
Lemma 3.1 Let m0,i and m1,i respectively denote the numbers of true and false null
hypotheses among the first i null hypotheses, and

1{i∈I }
0
di =
(m
1,i−1 1{i∈I0 } − m0,i−1 1{i∈I
/ 0 } )/(i(i − 1))

if i = 1
if i > 1.

Then, the FDR of any fixed sequence procedure can be expressed as
FDR =

m
X

di Pr (R ≥ i) .

i=1

Proof of Theorem 3.2. If u1 = 1, then



(2)
b
FDR ≤ FWER = Pr (V > 0) = Pr P1 ≤ α1 ≤ α.
If u1 > 1, then by Lemma 3.1,
FDR =

m
X

di Pr (R ≥ i) =

i=1

=

m 
X
i=u1

m
X

di Pr (R ≥ i)

i=u1
m
X
m1,i α 
m1,i α
di +
Pr (R ≥ i) −
Pr (R ≥ i) .
i
i
i=u
1

9

(3)

The second equality follows from the fact that di = 0 for i = 1, . . . , u1 − 1.
For each i = u1 , . . . , m, the following inequality holds.

m1,i−1 α
m1,i α 
Pr (R ≥ i) ≤
P r(R ≥ i − 1).
di +
i
i−1

(4)

To see this, we consider, separately, the case when i ∈ I0 and when i ∈
/ I0 . Suppose
i ∈ I0 , then m1,i−1 = m1,i and
m1,i α 
di +
Pr (R ≥ i)
i

 

m1,i−1
m1,i−1 α
(2)
(2)
(2)
=
+
Pr P1 ≤ α1 , . . . , Pi−1 ≤ αi−1 , Pi ≤ αi
i(i − 1)
i

 
m1,i−1 (1 + (i − 1)α) 
(2)
(2)
(2)
≤
Pr P1 ≤ α1 , . . . , Pi−1 ≤ αi−1 Pr Pi ≤ αi
i(i − 1)
m1,i−1 α
≤
P r(R ≥ i − 1).
i−1


The first and second inequalities follow from (2) and (1), respectively.
Now suppose i ∈
/ I0 , then m1,i = m1,i−1 + 1 and


m1,i α 
m0,i−1
(m1,i−1 + 1)α
di +
Pr (R ≥ i) = −
+
Pr (R ≥ i)
i
i(i − 1)
i


m0,i−1 α (m1,i−1 + 1)α
+
Pr (R ≥ i)
≤
−
i(i − 1)
i
m1,i−1 α
m1,i−1 α
=
Pr (R ≥ i) ≤
Pr (R ≥ i − 1) .
i−1
i−1


In the second equality, we use the fact that m0,i−1 + m1,i−1 = i − 1.
Applying (4) to (3), we have
FDR

=
≤
=

m 
X
i=u1
m
X

m
X
m1,i α 
m1,i α
Pr (R ≥ i) −
di +
Pr (R ≥ i)
i
i
i=u
1

m
X

m1,i−1 α
m1,i α
Pr (R ≥ i − 1) −
Pr (R ≥ i)
i−1
i
i=u1
i=u1
m1 α
αPr (R ≥ u1 − 1) −
Pr (R = m)
m

≤ α.

10

The equality follows from that fact that m1,u1 −1 = u1 −1, since the first u1 −1 hypotheses
are false.
Remark 3.2 The conventional fixed sequence procedure in Theorem 3.2 is nearly optimal in the sense that the upper bound of the FDR of this procedure is very close to α
under certain configurations. Consider the following configuration: All the false null hypotheses are tested before the true null hypotheses, the false null p-values are all 0 with
probability 1, and the true null p-values are independently distributed as U (0, 1). Under
this configuration, it is easy to check that (4) becomes an equality. Following the proof
of Theorem 3.2, the FDR of this procedure is exactly α −

m1 α
Pr (R
m

= m). When m1 /m

approaches to π1 as m → ∞ with 0 ≤ π1 < 1, an approximate lower bound of the FDR
is α − π1 αe−(1−π1 )

1−α
α

.

To see why, we first note that
m
m1 α
m1 α Y (2) m1 α (2) m−m1
Pr (R = m) =
(αm )
α <
.
m
m i=m +1 i
m
1

Then, by simple calculation, we have
m1 α (2) m−m1
(αm )
lim FDR ≥ α − lim
m→∞
m→∞ m

−m(1− mm1 )
m1 α
1−α 1
= α − lim
1+
m→∞ m
α m
= α − π1 αe−(1−π1 )

1−α
α

.

This lower bound of the FDR is very close to the pre-specified level α. For example,
for large m, if m1 /m = 0.2, then with α = 0.05, the lower bound under the above
configuration is about 0.04999975.
Remark 3.3 Figure 1 compares the critical constants for the proposed procedures along
with the FWER fixed sequence procedure at level α. It should be noted that the first few
critical constants are the most important ones. If the first few values are too small, then
the procedure might stop too early and the remaining hypotheses will not have a chance
to be tested. With this in mind, it can be seen that the critical constants of the procedure
introduced in Theorem 3.2 are by far the best, and the critical constants of the procedure in
11

10

15

10

20

30

40

50

0.4

0.6

0.8

1.0
0

Index

0.2

0.6

Critical Constant

0.8

1.0
20

0.4

Critical Constant

0.4

5

100 Hypotheses

0.2

Proc. 3.1
Proc. 3.2
FWER Fix Seq. Proc.

0.6

0.8

50 Hypotheses

0.2

Critical Constant

1.0

20 Hypotheses

0

20

Index

40

60

80

100

Index

Figure 1: A plot of the critical constants of the procedures in Theorems 3.1 (solid line),
3.2 (dashed line), and the FWER fixed sequence procedure (dotted line) for m =20, 50,
and 100.

Theorem 3.1 are slightly better than those of the conventional fixed sequence procedure.

4

Fixed Sequence Procedures that Allow More Acceptances

A conventional fixed sequence method might potentially lose power if an early null hypothesis fails to be rejected, with the remaining hypotheses having no chance of being
tested. To remedy this, we generalize a conventional fixed sequence method to one that
allows a certain number of acceptances. The procedure will keep testing hypotheses until
a pre-specified number of acceptance has been reached. The same idea has also been used
by Hommel and Kropf (2005) to develop FWER controlling procedures in fixed-sequence
multiple testing.
Suppose k is a pre-specified positive integer and α1 ≤ · · · ≤ αm is a non-decreasing
sequence of critical constants. A fixed sequence method that allows more acceptances is
defined below.
Definition 4 (Fixed sequence method stopping on the k th acceptance)
1. If P1 ≤ α1 , then reject H1 ; otherwise, accept H1 . If k > 1 or H1 is rejected, then
continue to test H2 ; otherwise stop.
12

2. If Pi ≤ αi , then reject Hi ; otherwise, accept Hi . If the number of accepted hypotheses so far is less than k, then continue to test Hi+1 ; otherwise stop.
It is easy to see that when k = 1, the fixed sequence method stopping on the k th
acceptance reduces to the conventional one.
Theorem 4.1 The fixed sequence method stopping on the k th acceptance with critical
constants
(3)

αi =


α

if

i = 1, . . . , k

 (m−k+1)α

if

i = k + 1, . . . , m

k

(m−i+1)k

strongly controls the FDR at level α under arbitrary dependence of the p-values.
Proof. Let U be the index of the first rejected true null hypothesis. If no true null
hypotheses are rejected, then set U = 0. We will show that for i = 1, . . . , m0 ,

E

V
1{U =ui }
V +S


≤

α
.
k

(5)

If i ≤ k, then

E

V
1{U =ui }
V +S




α α
≤ .
≤ Pr (U = ui ) ≤ Pr P̂i ≤
k
k

Now, assume i > k. Note that the event {U = ui } implies V ≤ m − ui + 1 and
S ≥ ui − k, because the first ui − 1 hypotheses were either false nulls or not rejected true
nulls and among the first ui − 1 hypotheses tested, there can be at most k − 1 acceptances.
Thus,





V
m − ui + 1
E
1{U =ui } ≤ E
1{U =ui }
V +S
(m − ui + 1) + (ui − k)
 α
m − ui + 1 
≤ .
≤
Pr P̂i ≤ αu(3)
i
m−k+1
k
The first inequality follows from the fact that V /(V + S) is an increasing function of V
and a decreasing function of S.

13

From (5), we have
min(m0 ,k)

FDR =

X


E

i=1

V
1{U =ui }
S+V


≤

k
X
α
i=1

k

= α,

where the first equality follows from the fact that if none of the first k true null hypotheses
are rejected, then V = 0.
We should point out that the result in Theorem 4.1 is weaker than that in Theorem
3.1, although the method in Theorem 4.1 reduces to that in Theorem 3.1 when k = 1.
More specifically, we cannot prove that the procedure in Theorem 4.1 is optimal in the
sense that its critical constants cannot be further improved without losing control of the
FDR under arbitrary dependence of the p-values. However, under certain distributional
assumptions on the p-values, the critical constants of this procedure can potentially be
improved. In particular, we have the following result.
Theorem 4.2 Consider a fixed sequence method stopping on the k th acceptance with
critical constants
(4)

αi =

(ri−1 + 1)α
, i = 1, . . . , m,
k + (i − k)α

where ri is the number of rejections among the first i tested hypotheses, with r0 = 0, for
i = 1, . . . , m. This procedure strongly controls the FDR at level α if the true null p-values
are mutually independent and are independent of the false null p-values.
Before presenting a proof of the above theorem, let us introduce a few more notations.
For i = 1, . . . , m, let Vi and Si be the numbers of false rejections and true rejections
among the first i rejections and Ji be the index of the ith rejected hypothesis. If there are
less than i rejections, we define Vi = Vi−1 , Si = Si−1 and Ji = m + 1. In addition, for
notational convenience, define V0 = S0 = J0 = 0, V0 /0 = 0, and S0 /0 = 1.
We use the following two lemmas, with proofs given in Appendix, to prove the theorem.
Lemma 4.1 The FDR of any fixed sequence method stopping on the k th acceptance can
be expressed as
FDR = E

m 
X
Vi
i=1

Vi−1
−
i
i−1
14



!
1{Ji <m+1} .

Lemma 4.2 Consider the procedure defined in Theorem 4.2, the following inequality
holds for i = 1, . . . , m,


Vi−1
Vi
−
1{Ji <m+1}
E
i
i−1


(k − Ji−1 + i − 1)α Si−1
(k − Ji + i)α Si
≤ E
1{J <m+1} −
1{Ji <m+1} .
k
i − 1 i−1
k
i


Proof of Theorem 4.2. By Lemmas 4.1 and 4.2, we have
FDR = E

m 
X
Vi
i=1

Vi−1
−
i
i−1

!


1{Ji <m+1}



(k − J1 + 1)α
S1 1{Ji <m+1}
≤ E α−
k
m 
X
(k − Ji−1 + i − 1)α Si−1
+E
1{J <m+1}
k
i − 1 i−1
i=2

(k − Ji + i)α Si
−
1{Ji <m+1}
k
i


(k − Jm + m)α Sm
1{Jm <m+1} ≤ α.
= E α−
k
m
Remark 4.1 When k = 1, the generalized fixed sequence method in the above theorem
reduces to the conventional fixed sequence method in Theorem 3.2, since in this case
ri−1 = i − 1, and thus continues to control the FDR when the p-values are negatively
associated. However, when k > 1, it can only control the FDR under the independence
assumption made in the theorem. It can be shown that this method, when k > 1, may
no longer control the FDR when the p-values are negatively associated. Consider, for
example, the problem of simultaneously testing two hypotheses H1 and H2 for which both
of them are true, the associated p-values Pb1 and Pb2 are both U (0, 1), and Pb2 = 1 − Pb1 .
It is easy to see that under such configuration, when k = 2, the FDR of this procedure is
equal to

α
2−α

+

α
2

> α.

15

5

Data Driven Ordering

The applicability of the aforementioned fixed sequence methods depends on availability of natural ordering structure among the hypotheses. When the hypotheses cannot
be pre-ordered, one can use pilot data available to establish a good ordering among the
hypotheses in some cases. For example, in replicated studies, the hypotheses for the
follow-up study can be ordered using the data from the primary study. However, when
prior information is unavailable ordering information can usually be assessed from the
data itself. Such data-driven ordering has been used by several authors in fixed sequence
methods controlling the FWER and generalized FWER (Kropf and Läuter, 2002; Kropf
et al., 2004; Westfall et al., 2004; Hommel and Kropf, 2005; Finos and Farcomeni, 2011).
Assume that the variables of interest are Xi , i = 1, . . . , m, with n independent observations Xi1 , . . . , Xin on each Xi . An ordering statistic, Yi , and a test statistic, Ti ,
are determined for each i = 1, . . . , m. The Yi ’s are used to order all of the hypotheses
H1 , . . . , Hm , Ti is used to test the hypothesis Hi , i = 1, . . . , m, and Pi is the corresponding p-value. In addition, Yi is chosen such that it is independent of the Ti ’s under Hi and
tends to be larger as the effect size increases. The approach is outlined below.
Definition 5 Data-Driven Ordering Procedure
1. The hypotheses are ordered based on Y1 , . . . , Ym where the hypothesis corresponding to the largest Yi is placed first, the hypothesis corresponding to the second
largest is placed second, and so on.
2. The hypotheses are tested using a fixed sequence procedure based on the the pvalues P1 , . . . , Pm and the testing order established in Step 1.
We give a few examples to further illustrate the approach.
Example 1: One sample T-test. Consider testing Hi : µi = 0 against Hi0 : µi 6= 0
P
simultaneously where Xi follows a N (µi , σ 2 ) distribution. Let X̄i = nj=1 Xij /n and
P
s2i = nj=1 (Xij − X̄i )2 /(n − 1) be the sample mean and variance, respectively, based
P
on the observations Xi1 , . . . , Xin . Let Yi = nj=1 Xij2 be the ordering statistics, that is,
the hypotheses are ordered according to the values of the corresponding sums of squares,
√
and Ti = nX̄i /si is the usual t-test statistic for testing Hi . Then, Pi = 2 (1 − F (|Ti |)),
16

i = 1, . . . , m, where F (·) is the cdf of the t-distribution with n − 1 degrees of freedom,
are the p-values. When µi = 0, Ti and Yi are independent (see, for instance, Lehmann
and Romano, 2005; p. 156). Furthermore,
E(Yi ) = n(µ2i + σ 2 ),
which suggests that |µi | tends to increase as Yi increases.
(1)

Example 2: Two sample T-test. Consider testing Hi : µi
(1)

(2)

(2)

= µi

against Hi0 :

(l)

µi 6= µi simultaneously using n = n1 + n2 data vectors. Suppose Xij , j = 1, . . . , nl ,
(l)

follows a N (µi , σ 2 ) distribution, for l = 1, 2. Then, the hypotheses can be tested using
the two-sample t-test statistics Ti and are ordered through the values of the ‘total sum
P2 Pnl
P2 Pnl
(l)
(l)
2
of squares,’ which is Yi =
l=1
j=1 Xij /n,
l=1
j=1 (Xij − X̄i ) , where X̄i =
for i = 1, . . . , m. The rationale behind this is independence between the Yi ’s and Ti
under Hi (see, for instance, Westfall et al., 2004), and the following result: E[Yi ] =
(1)

(2)

(n − 1)σ 2 + n1 n2 (µi − µi )2 /n.
Example 3: Nonparametric test. Kropf et al. (2004) describe a data-driven ordering
strategy for nonparametric tests. In the one sample case, we are interested in testing
Hi : µi = 0 against Hi0 : µi 6= 0 where Xij , j = 1, . . . , n are assumed to be symmetric
about µi . The hypotheses are tested using the one-sample Wilcoxon test and ordered
based on Yi = med(|Xi1 |, . . . , |Xin |). In the two sample case, we are interested in testing
(1)

Hi : µi
(l)
Xij , j

(2)

= µi

(1)

against Hi0 : µi

(2)

6= µi

using n = n1 + n2 data vectors, where
(l)

= 1, . . . , nl are assumed to be symmetric about µi for l = 1, 2. The hypotheses

are tested using the two-sample Wilcoxon test and ordered based on the interquartile range
Yi = q3i − q1i , where q1i and q3i are respectively the 1st and 3rd quartile of the mixture of
(1)

(2)

Xij ’s and Xij ’s.
When our proposed fixed sequence procedures are used in applications coupled with
the aforementioned data-driven ordering strategy, the FDR controls are still maintained
under the independence assumption, if the ordering statistics are chosen to be independent
of the test statistics in the data-driven ordering strategy, even though the same data is
repeatedly used for ordering and testing the hypotheses. We have the following result.

17

Theorem 5.1 Suppose X1 , . . . , Xm are mutually independent. If the hypotheses H1 , . . . , Hm
are ordered based on the ordering statistics Yi , i = 1, . . . , m, tested using the test statistics Ti , i = 1, . . . , m, and Yi is independent of Ti under Hi , then the fixed sequence
multiple testing procedures introduced in Theorems 3.1-4.2 can still strongly control the
FDR at level α.
Proof. Assume without any loss of generality that Y1 ≥ · · · ≥ Ym , so that conditional
on the Yi ’s, Hi is the ith hypotheses to be tested in our fixed sequence multiple testing
methods. When Hi is true, Pi is independent of both Yi and Xj , j = 1, . . . , m with
j 6= i. This follows from independence of the Xi ’s and that of Yi and Ti under Hi . Thus,
conditional on the Yi ’s, each true null p-value Pi still satisfies (1) and is independent of all
other p-values Pj with j 6= i. Therefore, we have for each of the procedures in Theorems
3.1, 3.2, 4.1, and 4.2,

E

V
Y1 , . . . , Y m
max(R, 1)


≤ α.

(6)

This proves the desired result.
We applied our proposed methods to the HIV microarray data (vant Wout et al., 2003)
used by Efron (2008). These data consist of m = 7680 gene expression levels across
eight subjects, four HIV infected and four uninfected. The data were log-transformed and
normalized. Our goal is to determine which genes are differentially expressed by testing
(1)

Hi : µi

(2)

(1)

= µi versus Hi0 : µi

(1)

(2)

6= µi simultaneously for i = 1, . . . , 7680, where µi

(2)

and µi are the gene specific mean expressions for HIV infected and uninfected subjects,
respectively.
We applied our proposed procedures with the p-values generated from two sample
t-tests for the genes. Since there is no natural ordering among the genes, we used the
ordering statistics for two sample t-tests in Example 2 to order these tested hypotheses.
We compared the procedure in Theorem 4.2 with the BH procedure. The results are
summarized in Table 1 for different values of k where k/m = 0.05, 0.1, and 0.15. As seen
from Table 1, for all values of k except k = 1, the procedure in Theorem 4.2 generally
has more rejections than the BH procedure. When α is small, k/m = 0.05 tends to
have the most rejections, but for large α, k/m = 0.1 has the most rejections. Also, we
compared the procedure in Theorem 4.1 with the BY procedure. The results are displayed
in Table 2. As seen from Table 2, for most values of k, our procedure outperforms the
18

Table 1: The Number of Discoveries out of m = 7680 Genes in the HIV Data from vant
Wout et al. (2003) by the procedure from Theorem 4.2 and the BH Procedure

α = 0.001
α = 0.01
α = 0.025
α = 0.05
α = 0.1

k=1
11
11
11
11
20

Procedure from Theorem 4.2
k/m = 0.05 k/m = 0.1 k/m = 0.15
13
9
8
18
17
16
18
18
18
20
19
19
21
24
20

BH Procedure
8
13
13
18
22

Table 2: The Number of Discoveries out of m = 7680 Genes in the HIV Data from vant
Wout et al. (2003) by the procedure from Theorem 4.1 and the BY Procedure

α = 0.001
α = 0.01
α = 0.025
α = 0.05
α = 0.1

k=1
11
11
11
11
11

Procedure from Theorem 4.1
k/m = 0.05 k/m = 0.1 k/m = 0.15
10
8
8
13
13
11
15
13
13
16
15
13
18
16
16

BY Procedure
0
8
10
10
13

BY procedure in terms of the number of rejections. When α = 0.001, the BY procedure
cannot reject any hypotheses, but the procedure in Theorem 4.1 has at least 8 rejections
for all the values of k considered.

6

Simulation Study

A simulation study was conducted to address the performances of the proposed procedures. We will refer to the procedures in Theorems 3.1, 3.2, 4.1, and 4.2 as Procedures
1-4, respectively. Specifically, we addressed the following two questions:
1. How do Procedures 1, 2, 3, and 4 compare against the BH and BY procedures in
terms of FDR and power?
2. For Procedures 3 and 4, how should k be chosen so that the power is large?

19

40

0.06

30

0.04

2

0

0.00

10

20

30

40

50

60

70

80

10

20

30

40

50

60

70

80

50

60

70

80

50

60

70

80

40

1

30

0.04

5

6

0

0.00

10

0.02

4

20

ρ = 0.3

0.06

1

10

20

30

40

50

60

70

80

1

10

20

30

40

30

0.04

40

0.06

1

8

9

0

0.00

10

0.02

7

20

ρ = 0.7

3

10

0.02

1

20

ρ=0

Average Power (%), m0 = 90

FDR, m0 = 90

1

10

20

30

40

50

60

70

80

1

10

20

k

10

11

13

Proc. 3

30

40
12

Proc. 4

14

BH Proc.

k

BY Proc.

Figure 2: The FDR (left) and average power (right) of Procedure 3 (solid), Procedure
4 (dotted), the BH procedure (dashed), and the BY procedure (dotted dash) for 100 hypotheses with m0 = 90 and n = 10 under common correlation.

In each simulation, n independent m dimensional random normal vectors with covariance matrix Σ and components Zi ∼ N (µi , 1), i = 1, . . . , m, were generated. The
p-value for testing Hi : µi = 0 vs. Hi0 : µi > 0 was calculated using a one-sided,
one-sample t-test for each i. The µi corresponding to each false null hypothesis is set to
the value at which the power of one-sample t-test at level 0.05 is 0.75. As for the joint

20

dependence, we considered a common correlation structure where Σ had off-diagonal
components equal to ρ and diagonal components equal to 1.
We set α = 0.05 and m = 100. The hypotheses were ordered using the ‘sum of
squares ordering’ used in Example 1 from Section 5. We had 5,000 runs of simulation
for each of the procedures considered. We noted the false discovery proportion and the
proportion of correctly rejected false null hypotheses for each procedure in each of these
runs. The simulated FDR and average power (the expected proportion of correctly rejected false null hypotheses) were obtained by averaging out the corresponding 5,000
values.
We first compared Procedures 1-4 with the BH and BY procedures when m0 = 90 and
n = 10. Figure 2 displays these comparisons in terms of the simulated FDR and average
power, respectively, under common correlation with ρ = 0 (independence), ρ = 0.3,
and ρ = 0.7 while k varies from 1 to 80. We did not explicitly label Procedures 1 and
2 as these are both special cases of Procedures 3 and 4 when k = 1, respectively. As
seen from Figure 2, Procedure 3 controls the FDR at level 0.05 under every dependence
configuration. Procedure 4 controls the FDR under independence (ρ = 0), generally
maintains controls of the FDR under mild correlation (ρ = 0.3), but loses control of the
FDR for small values of k under strong correlation (ρ = 0.7). From Figure 2, one can
also see that Procedure 3 tends to have its largest power when k is between about 3 and
10, and Procedure 4 tends to have its largest power when k is between about 5 and 15. In
addition, Figure 2 shows that for a well chosen k, Procedures 3 and 4 outperform the BH
procedure, and for all values of k, Procedures 3 and 4 outperform the BY procedure.
Next, we compared the FDR and average power of these procedures when m0 = 80
and n = 10. Figure 3 displays these comparisons for ρ = 0, ρ = 0.3, and ρ = 0.7 while
k varies from 1 to 80. In terms of FDR, the results shown in Figure 3 are similar to the
results shown in Figure 2. In terms of power, Figure 3 shows that the powers of all the
procedures tend to show an increase over the powers shown in Figure 2. Figure 3 also
shows that for a well chosen k, Procedures 4 can outperform the BH procedure, but, in
general, Procedure 3 cannot. Again, for all values of k, Procedures 3 and 4 outperform
the BY procedure.

21

40

0.06

30

0.04

2

0

0.00

10

20

30

40

50

60

70

80

10

20

30

40

50

60

70

80

50

60

70

80

50

60

70

80

40

1

30

0.04

5

6

0

0.00

10

0.02

4

20

ρ = 0.3

0.06

1

10

20

30

40

50

60

70

80

1

10

20

30

40

30

0.04

40

0.06

1

8

9

0

0.00

10

0.02

7

20

ρ = 0.7

3

10

0.02

1

20

ρ=0

Average Power (%), m0 = 80

FDR, m0 = 80

1

10

20

30

40

50

60

70

80

1

10

20

k

10

11

13

Proc. 3

30

40
12

Proc. 4

14

BH Proc.

k

BY Proc.

Figure 3: The FDR (left) and average power (right) of Procedure 3 (solid), Procedure
4 (dotted), the BH procedure (dashed), and the BY procedure (dotted dash) for 100 hypotheses with m0 = 80 and n = 10 under common correlation.

7

Concluding Remarks

In this paper, we have developed ‘step-down’ procedures which control the FDR and exploit the structure of pre-ordered hypotheses. We have been able to produce the desired
methods in the most simple as well as a general setting covering different dependence scenarios. Our simulation study and real data analysis show that in some cases, the proposed
22

procedures can be powerful alternatives to existing FDR controlling procedures.
Using some of the techniques developed in this paper, it is possible to develop other
types of fixed sequence procedures controlling the FDR, such as a fallback-type procedure. Unlike the conventional and generalized fixed sequence procedures developed
in this paper, the fallback-type procedure tests the remaining hypotheses no matter how
many earlier hypotheses are accepted, which is needed for analyzing stream data in sequential change detection problems.
Although we have only considered the simplest hierarchical structure – fixed sequence structure – by using the similar techniques presented in this paper, we were able
to develop simple and powerful procedures that control the FDR under various dependencies when testing multiple hypotheses with a more complex hierarchical structure. We
plan to present these procedures in a future communication.

Appendix
7.1

Proof of Theorem 3.1 (ii)

For any u1 , 1 ≤ u1 ≤ m, consider a joint distribution of the p-values such that the first
u1 − 1 hypotheses are false null hypotheses whose corresponding p-values are 0 with
probability one. The remaining m − u1 + 1 hypotheses are true null hypotheses such
that P̂1 ∼ U (0, 1) and P̂i = P̂1 for i = 2, . . . , m0 . Under such joint distribution of
(1)

the p-values, the FDR of the conventional fixed sequence procedure is exactly αu1 (m −
(1)

u1 + 1)/m. If αu1 = 1 then the critical constant is already at its largest and cannot be
(1)

(1)

improved. Otherwise, if αu1 < 1, then FDR = α and thus αu1 cannot be made any larger.
In the above construction P~ is PRDS on P~0 . We only need to note that for i =
1, . . . , m0 , Pbi = p implies P~ = (0, . . . , 0, p, . . . , p). Thus, for any increasing set D,




Pr P~ ∈ D|Pbi = p =


1

if (0, . . . , 0, p, . . . , p) ∈ D

0

otherwise,

which is an increasing function in p.

23

7.2

Proof of Lemma 3.1

Lemma 3.1 can be regarded as a special case of Lemma 4.1 with k = 1. Note that for
i = 1, . . . , m, m1,i = m1,i−1 and m0,i = m0,i−1 + 1 when i ∈ I0 , m0,i = m0,i−1 when
i ∈
/ I0 , and m0,i−1 + m1,i−1 = i − 1. Thus, when k = 1, the event {R ≥ i} implies
Vi = m0,i and hence

1{1∈I

for i = 1

Vi
Vi−1
0}
−
= m
/ 0}
i
i − 1  1,i−1 1{i∈I0 } −m0,i−1 1{i∈I

(7)

for i = 2, . . . , m.

i(i−1)

By (7) and Lemma 4.1, the desired result follows.

7.3

Proof of Lemma 4.1

It is easy to see that
FDR = E
= E

m
X
Vi

i
i=1
m 
X
i=1

= E

!
1{R=i}

i=1

Vi
Vi−1
−
i
i−1

m 
X
Vi
i=1

=E

m 
X
Vi

Vi−1
−
i
i−1

Vi
1{R≥i} − 1{R≥i+1}
i
i

!

!


1{R≥i}

!


1{Ji <m+1}

,

the desired result.

7.4

Proof of Lemma 4.2

It is easy to see that for i = 1, . . . , m, if there are at least i rejections , then i ≤ Ji ≤
min(i + k − 1, m). For ease of notation, let ji∗ = min(i + k − 1, m). For i, j = 1, . . . , m,
define fi (j) =

(k−j+i)α Si
k
i

and Wi (j) = 1{Ji−1 ≤j,Ji >j} . Regarding the relationship between

Ji and Wi (j), there are the following two equalities available:
1{Ji =j} = Wi (j − 1)1{Pj ≤α(4) }
j

24

(8)

and
Wi (j) − Wi (j − 1) = 1{Ji−1 =j} − 1{Ji =j} .

(9)

The first equality follows from the fact that for i = 1, . . . , m and j = i, . . . , ji∗ , when
Ji = j, there are i − 1 rejections among the first j − 1 tested hypotheses and the ith
rejection is exactly the j th tested hypothesis, thus
1{Ji =j} = 1{Ji−1 ≤j−1,Ji >j−1,Pj ≤α(4) } = Wi (j − 1)1{Pj ≤α(4) } .
j

j

The second equality follows from the fact that the event {Wi (j) = 1} implies that there
∗
are exactly i−1 rejections among the first j tested hypotheses, thus for j = i−1, . . . , ji−1
,

Wi (j) − Wi (j − 1) = 1{Ji−1 ≤j,Ji >j} − Wi (j − 1)
= 1{Ji−1 =j} + 1{Ji−1 ≤j−1,Ji >j−1,Pj >α(4) } − Wi (j − 1)
j

= 1{Ji−1 =j} − Wi (j − 1)1{Pj ≤α(4) }
j

= 1{Ji−1 =j} − 1{Ji =j} ,
where, the third equality follows from the fact that 1{Pj >α(4) } = 1 − 1{Pj ≤α(4) } and the
j

j

fourth follows from (8).
By using the above two equalities, we can prove two inequalities below, which are
needed in the proof of this lemma. Firstly, we show by using (8) that the following
inequality holds:

E





Vi−1
α Si−1
Vi
−
+ fi (j) − fi−1 (j) 1{Ji =j} ≤ E
Wi (j − 1) .
i
i−1
ki−1

(10)

Proof of (10). To see this, we consider, separately, the case when j ∈ I0 and when
j∈
/ I0 .
Suppose j ∈ I0 , then Si = Si−1 and Vi = Vi−1 + 1 when Ji = j. Using the fact that

25

Vi−1 + Si−1 = i − 1, the left hand side of (10), after some algebra, becomes



k + (j − k)α Si−1
E
Wi (j − 1)1{Pj ≤α(4) }
j
ki
i−1
 


k + (j − k)α Si−1
(4)
Wi (j − 1) Pr Pj ≤ αj
= E
ki
i−1


k + (j − k)α Si−1
iα
≤ E
Wi (j − 1)
ki
i−1
k + (j − k)α


α Si−1
Wi (j − 1) .
= E
ki−1
The first equality follows from these two facts: (i) When Ji = j, i.e. the ith rejection is
Hj , Si−1 is only dependent on the first j − 1 p-values, since Si−1 is the number of false
null hypotheses among the first i − 1 rejections; (ii) Wi (j − 1) is also only dependent on
the first j − 1 p-values, since Wi (j − 1) is 1 if and only if there are exactly i − 1 rejections
among the first j − 1 hypotheses tested. The inequality follows from (1).
Now suppose j ∈
/ I0 , then Si = Si−1 + 1 and Vi = Vi−1 . Similarly, using the fact that
Vi−1 + Si−1 = i − 1, the left hand side of (10), after some algebra, becomes




−Vi−1
((j − k)Si−1 + (i − 1)(k − j + i))α
E
+
Wi (j − 1)1{Pj ≤α(4) }
j
i(i − 1)
ki(i − 1)



−Vi−1 (k − j + i)α ((j − k)Si−1 + (i − 1)(k − j + i))α
+
Wi (j − 1)
≤ E
i(i − 1)
k
ki(i − 1)


α Si−1
Wi (j − 1) .
= E
ki−1
The inequality follows by the fact that j ≥ i so that k − j + i ≤ k. In addition, in the last
line we use the fact that Vi−1 + Si−1 = i − 1.
Next, we show by using (9) that the following inequality holds:



ji∗
X

α Si−1
E fi−1 (Ji−1 )1{Ji−1 <m+1} − fi−1 (Ji )1{Ji <m+1} ≥ E 
Wi (j − 1) . (11)
k
i
−
1
j=i

26

Proof of (11). By using (9), we have
E fi−1 (Ji−1 )1{Ji−1 <m+1} − fi−1 (Ji )1{Ji <m+1}
 ∗

ji−1
ji∗
X
X
= E
fi−1 (j)1{Ji−1 =j} −
fi−1 (j)1{Ji =j} 

= E

j=i−1

j=i

∗
ji−1

∗
ji−1

X

fi−1 (j)1{Ji−1 =j} −

j=i−1


= E

X




fi−1 (j)1{Ji =j} 

j=i

∗
ji−1



X


fi−1 (j) 1{Ji−1 =j} − 1{Ji =j} + fi−1 (i − 1)1{Ji−1 =i−1} 

j=i


= E

∗

ji−1
X


fi−1 (j)(Wi (j) − Wi (j − 1)) + fi−1 (i − 1)Wi (i − 1)

j=i


= E

∗

ji−1
X


∗
∗
(fi−1 (j − 1) − fi−1 (j)) Wi (j − 1) + fi−1 (ji−1
)Wi (ji−1
)

j=i


≥ E

ji∗



X α Si−1
Wi (j − 1) ,
k
i
−
1
j=i

∗
the desired result. Here, the second equality follows from the fact that if ji−1
= m, then
∗
= i + k − 2 and ji∗ = i + k − 1 so that fi−1 (ji∗ ) = 0. The
ji∗ = m; otherwise, ji−1

fourth equality follows from (9) and the fact that Wi (i − 1) = 1{Ji−1 =i−1} . The inequality
follows from the definition of fi−1 (j).
Proof of Lemma 4.2. Finally, by combining these two inequalities, we can get the

27

desired result as follows.


Vi−1
Vi
−
1{Ji <m+1}
E
i
i−1



Vi
Vi−1
−
+ fi (Ji ) − fi−1 (Ji ) 1{Ji <m+1}
E
i
i−1

−E fi−1 (Ji−1 )1{Ji−1 <m+1} − fi−1 (Ji )1{Ji <m+1}

+E fi−1 (Ji−1 )1{Ji−1 <m+1} − fi (Ji )1{Ji <m+1}

 ∗

ji 
X
Vi−1
Vi
−
+ fi (j) − fi−1 (j) 1{Ji =j} 
E
i
i
−
1
j=i

−E fi−1 (Ji−1 )1{Ji−1 <m+1} − fi−1 (Ji )1{Ji <m+1}

+E fi−1 (Ji−1 )1{Ji−1 <m+1} − fi (Ji )1{Ji <m+1}
 ∗

ji
ji∗
X α Si−1
X α Si−1
E
Wi (j − 1) −
Wi (j − 1)
k
i
−
1
k
i
−
1
j=i
j=i

+E fi−1 (Ji−1 )1{Ji−1 <m+1} − fi (Ji )1{Ji <m+1}

E fi−1 (Ji−1 )1{Ji−1 <m+1} − fi (Ji )1{Ji <m1 } ,


=

=

≤

=

The inequality follows from (10) and (11).

References
[1] A HARONI , E. and ROSSET, S. (2014). Generalized a-investing: definitions, optimality results and application to public databases. Journal of the Royal Statistical
Society: Series B 76 771–794.
[2] BARBER , R. and C ANDES , E. (2015). Controlling the false discovery rate via
knockoffs. The Annals of Statistics 43 2055–2085.
[3] B ENJAMINI , Y. and H ELLER , R. (2007). False discovery rates for spatial signals. J.
Amer. Satist. Assoc. 102 1272–1281.

28

[4] B ENJAMINI , Y. and H OCHBERG , Y. (1995). Controlling the false discovery rate: A
practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57
289–300.
[5] B ENJAMINI , Y. and Y EKUTIELI , D. (2001). The control of the false discovery rate
in multiple testing under dependency. Ann. Statist. 29 1165–1188.
[6] D MITRIENKO , A., D’AGOSTINO , R., and H UQUE , M. (2013). Key multiplicity
issues in clinical drug development. Statistics in Medicine 32 1079–1111.
[7] E FRON , B. (2008). Microarrays, empirical Bayes and the two-groups model. Statistical Science 23 1–22.
[8] FARCOMENI , A. and F INOS , L. (2013). FDR control with pseudo-gatekeeping
based on a possibly data driven order of the hypotheses. Biometrics 69 606–613.
[9] F INNER , H. and ROTERS , M. (2001). On the false discovery rate and expected type
I errors. Biometrical Journal 43 985–1005
[10] F INOS , L. and FARCOMENI , A. (2011). k-FWER control without multiplicity correction, with application to detection of genetic determinants of multiple sclerosis
in Italian twins. Biometrics 67 174–181.
[11] F ITHIAN , W., TAYLOR , J., T IBSHIRANI , R., and T IBSHIRANI , R. (2015). Selective Sequential Model Selection. arXiv preprint arXiv:1512.02565.
[12] G’S ELL , M. G., WAGER , S., C HOULDECHOVA , A., and T IBSHIRANI , R. (2016).
Sequential selection procedures and false discovery rate control. Journal of the
Royal Statistical Society: Series B 78 423–444.
[13] G OEMAN , J. and F INOS , L. (2012). The inheritance procedure: Multiple testing
of tree-structured hypotheses. Statistical Applications in Genetics and Molecular
Biology 11 1–18.
[14] G OEMAN , J. and M ANSMANN , U. (2008). Multiple testing on the directed acyclic
graph of gene ontology. Bioinformatics 24 537–544.

29

[15] G OEMAN , J. and S OLARI , A. (2010). The sequential rejection principle of familywise error control. Ann. Statist. 38 3782–3810.
[16] G UO , W. and R AO , M. (2008). On control of the false discovery rate under no
assumption of dependency. Journal of Statistical Planning and Inference 28 3176–
3188.
[17] H ELLER , R., M ANDUCHI , E., G RANT, G., and E WENS , W. (2009). A flexible twostage procedure for identifying gene sets that are differentially expressed. Bioinformatics 25 929–942.
[18] H OMMEL , G., B RETZ , F., and M AURER , W. (2007). Powerful short-cuts for multiple testing procedures with special reference to gatekeeping strategies. Statistics in
Medicine 26 4063–4074.
[19] H OMMEL , G. and K ROPF, S. (2005). Testing for differentiation in gene expression
using a data driven order or weights for hypotheses. Biometrical Journal 47 554–
562.
[20] H UQUE , M. and A LOSH , M. (2008). A flexible fixed-sequence testing method for
hierarchically ordered correlated multiple endpoints in clinical trials. Journal of Statistical Planning and Inference 138 321–335.
[21] JAVANMARD , A. and M ONTANARI , A. (2015). On online control of false discovery
rate. arXiv preprint arXiv:1502.06197.
[22] J OAG D EV, K. and P ROSCHAN , F. (1983). Negative association of random variables
with applications. Ann. Statist. 11 286–295.
[23] K ROPF, S. and LäUTER , J. (2002). Multiple tests for different sets of variables
using a data-driven ordering of hypotheses, with an application to gene expression
data. Biometrical Journal 44 789–800.
[24] K ROPF, S., LäUTER , J., E SZLINGER , M., K ROHN , K., and PASCHKEB , R. (2004).
Nonparametric multiple test procedures with data-driven order of hypotheses and
with weighted hypotheses. Journal of Statistical Planning and Inference 125 31–47.

30

[25] L EHMANN , E. and ROMANO , J. (2005). Testing statistical hypotheses. Springer,
New York.
[26] L EI , L. and F ITHIAN , W. (2016). Power of ordered hypothesis testing. arXiv
preprint arXiv:1606.01969.
[27] L I , A. and BARBER , R. (2015). Accumulation tests for FDR control in ordered
hypothesis testing. arXiv preprint arXiv:1505.07352 (To appear in JASA.)
[28] L I , J. and M EHROTRA , D. (2008). An efficient method for accommodating potentially underpowered primary endpoints. Statistics in Medicine 27 5377–5391.
[29] M AURER , W., H OTHORN , L., and L EHMACHER , W. (1995). Multiple comparisons
in drug clinical trials and preclinical assays: A-priori ordered hypotheses. Vol. 6,
Fischer-Verlag, Stuttgart, Germany.
[30] M EHROTRA , D. and H EYSE , J. (2004). Use of the false discovery rate for evaluating
clinical safety data. Statistical Methods in Medical Research 13 227–238.
[31] M ILLEN , B. and D MITRIENKO , A. (2011). Chain procedures: A class of flexible
closed testing procedures with clinical trial applications. Statistics in Biopharmaceutical Reseach 3 14–30.
[32] ROSENBAUM , P. (2008). Testing hypotheses in order. Biometrika 95 248–252.
[33] ROSS , G. J., TASOULIS , D., and A DAMS , N. (2011). Nonparametric monitoring
of data streams for changes in location and scale. Technometrics 53 379–389.
[34] S ARKAR , S. K. (2002). Some results on false discovery rate in stepwise multiple
testing procedures. Ann. Statist. 30 239-257.
[35]

VANT

W OUT, A., L EHRMA , G., M IKHEEVA , S., OK EEFFE , G., K ATZE , M.,

B UMGARNER , R., G EISS , G., and M ULLINS , J. (2003). Cellular gene expression
upon human immunodeficiency virus type 1 infection of CD4(+)-T-cell lines. Journal of Virology 77 1392–1402.

31

[36] W ESTFALL , P. and K RISHEN , A. (2001). Optimally weighted, fixed sequence and
gate-keeper multiple testing procedures. Journal of Statistical Planning and Inference 99 25–41.
[37] W ESTFALL , P., K ROPF, S., and F INOS , L. (2004). Weighted FWE-controlling
methods in highdimensional situations. In Recent Developments in Multiple Comparison Procedures, eds. Y. Benjamini, F. Bretz, and S. Sarkar, Vol. 47, Beachwood,
OH: Institute of Mathematical Statistics, pp. 143–154.
[38] W IENS , B. (2003). A fixed sequence Bonferroni procedure for testing multiple endpoints. Pharmaceutical Statistics 2 211–215.
[39] W IENS , B. and D MITRIENKO , A. (2005). The fallback procedure for evaluating a
single family of hypotheses. J. Biopharm. Stat. 15 929–942.
[40] W IENS , B. and D MITRIENKO , A. (2010). On selecting a multiple comparison procedure for analysis of a clinical trial: Fallback, fixed sequence, and related procedures. Statistics in Biopharmaceutical Research 2 22–32.
[41] Y EKUTIELI , D. (2008). Hierarchical false discovery rate-controlling methodology.
J. Amer. Statist. Assoc. 103 309–316.

32

