arXiv:1708.02102v1 [] 7 Aug 2017

Reallocating and Resampling: A Comparison
for Inference
Kari Lock Morgan
Department of Statistics, Pennsylvania State University
August 8, 2017

Abstract
Simulation-based inference plays a major role in modern statistics, and often employs either reallocating (as in a randomization test) or resampling (as in bootstrapping). Reallocating mimics random allocation to treatment groups, while resampling
mimics random sampling from a larger population; does it matter whether the simulation method matches the data collection method? Moreover, do the results differ
for testing versus estimation? Here we answer these questions in a simple setting by
exploring the distribution of a sample difference in means under a basic two group
design and four different scenarios: true random allocation, true random sampling,
reallocating, and resampling. For testing a sharp null hypothesis, reallocating is superior in small samples, but reallocating and resampling are asymptotically equivalent.
For estimation, resampling is generally superior, unless the effect is truly additive.
Moreover, these results hold regardless of whether the data were collected by random
sampling or random allocation.

Keywords: simulation-based inference, randomization-based inference, design-based inference, bootstrapping, randomization test, permutation test

1

1

Introduction

Simulation-based inference commonly takes two different forms: reallocating or resampling.
Reallocating mimics random allocation to treatment groups, while resampling mimics random sampling from a greater population. Moreover, reallocating implicitly assumes a
null hypothesis, while resampling does not. Should the distinction for when to use one
simulation method over the other be based on the original data collection method, the
inferential goal (testing or estimation), or neither? When, and to what extent, does it
matter? Through mathematical derivation, we find that resampling is better for interval
estimation unless the treatment effect is truly additive, reallocating is better for testing
in small samples, and the two are asymptotically equivalent for testing in large samples.
Also, quite surprisingly, and contrary to conventional wisdom, these results hold regardless
of whether the data were obtained by random allocation or random sampling.
Although the phrase “resampling methods” is often used for any simulation-based
method, here we use “resample” to mean take a random sample with replacement from
the original sample, of the same sample size as the original sample, as in a nonparametric bootstrap (Efron, 1979). We use “reallocate” to refer to the reallocation of outcomes
to treatment groups, as is done in the classic permutation or randomization test (Fisher,
1935; Edgington and Onghena, 2007). Reallocating is also commonly referred to as rerandomizing, permuting, shuffling, or scrambling. Reallocating conditions on the fixed sample
and samples without replacement, whereas resampling assumes a greater population and
samples with replacement from the observed sample.
Reallocating originated from the design and analysis of experiments as a method for
testing, dating back to Fisher’s historic text The Design of Experiments (Fisher, 1935).
It was originally proposed as a way to measure significance solely based on the random

2

allocation of an experiment. However, only a year later, Fisher discusses reallocating in
the context of random sampling, while the emphasis on testing remained (Fisher, 1936).
Thus it appears that Fisher viewed reallocating as applicable regardless of the type of
randomness in data collection, but primarily as a method for testing. Others (Tukey, 1993;
Manly, 2007, Section 1.4) have since adapted reallocation for estimation by inverting a
series of tests and creating an interval comprised of values not rejected.
Resampling originated with Efron’s landmark paper on bootstrapping (Efron, 1979),
which addresses only data obtained via random sampling, and focuses almost entirely on
estimation. Efron and others have since gone on to apply resampling for testing (Efron
and Tibshirani, 1994, Section 16.2), but the assumption of random sampling remains.
Advice regarding when either simulation method is appropriate differs. While Fisher
clearly regarded reallocating as applicable under both random allocation and random sampling, Tukey (1988) states that reallocating is applicable only under random allocation.
Tukey (1988) also says that resampling is applicable under either random sampling or allocation, but others advocate for reallocating as a way to bypass the assumption of random
sampling that is often unrealistic for randomized experiments (Cotton, 1973; Bredenkamp,
1980; Ludbrook and Dudley, 1998), suggesting that other methods may not be applicable
without random sampling. LaFleur and Greevy (2009) recommend reallocating for testing and resampling for estimation, and Ludbrook (1995); Ludbrook and Dudley (1998)
recommend reallocating for testing under random allocation, resampling for testing under
random sampling, and resamping for estimation under either data collection method. Other
comparisons prove that resampling and reallocating are asymptotically equivalent for testing (Romano, 1989), reallocating is more powerful than resampling for testing (Donegani,
1991; LaFleur and Greevy, 2009), and resampling performs poorly in small sample sizes
(Donegani, 1991; Hall and Wilson, 1991; Efron and Tibshirani, 1994). A good historical
3

summary comparing the two methods is given in Berry et al. (2014, Section 1.3.5).
While reallocating originated for testing after random allocation and resampling originated for estimation after random sampling, both methods are widely used often irrespective of the data collection method or inferential goal. Moreover, while authors are in
general agreement about the use of reallocating for tests based on experiments and the
use of resampling for estimation on random samples, there is discrepancy as to when (or
whether) it matters, and whether the data collection method or the inferential goal should
take priority if the two don’t align. To the best of our knowledge, this appears to be a
void in the literature that deserves to be addressed. Moreover, most of the existing comparisions are based on conceptual, rather than mathematical, foundations (Romano (1989)
and Donegani (1991) are notable exceptions) and rigorous mathematical justifications are
needed for determining not just whether reallocating or resampling is preferred in a given
situation, but also the extent to which the choice matters.
This paper addresses these questions by comparing the distribution of the common difference in means under different scenarios; true random sampling, true random allocation,
resampling, and reallocating. These are all examined under different contexts: testing a
sharp null hypothesis of no difference (Section 3), estimation (Section 4), and testing a
weaker null hypothesis of equal means (Section 5). Section 6 illustrates with two examples,
one with data generated via random allocation and one with data generated via random
sampling. Section 7 summarizes the findings and concludes.

2

General Framework

This paper examines a quantitative outcome, Y , measured on n units with binary group
membership, W = (W1 , . . . , Wn ) where Wi ∈ {0, 1}. To focus primarily on the distinction
4

between random allocation and random sampling, we restrict our comparison to two simple,
but widely used, settings: a completely randomized experiment and a simple stratified
P
P
sample, both with two groups of fixed sample sizes; n1 ≡ ni=1 Wi and n0 ≡ ni=1 (1 − Wi ).
It is important to note that any results here are applicable only to these two simple designs.
Many note that random allocation and random sampling lead to two fundamentally different modes of inference, given different names by different authors: experimental versus
sampling inference (Kempthorne, 1979), randomization versus population inference (Ludbrook, 1995), finite sample versus super population inference (Imbens and Rubin, 2015),
and permutation versus population model (Berry et al., 2014). The former, stemming from
random allocation and addressing only the sample at hand, originated with Fisher (1935,
1936). The latter, stemming from random sampling and generalizing to a larger population, originated with Neyman and Pearson (1928). Kempthorne (1979) argues that it is
misleading to refer to both using the same single word “inference”. Here we are explicit
about this distinction in our notation.
When random allocation is the source of randomness, we condition on the units in the
sample, and are interested in what might have happened, had they been assigned a different
treatment. Here we follow the potential outcome notation of the Rubin Causal Model
(Rubin, 1974), and let Yi (Wi ) denote the ith unit’s potential outcome under treatment
assignment Wi . For this notation to make sense, we assume the standard Stable Unit
Treatment Value Assumption (SUTVA) (Rubin, 1980); that W has only two well-defined
levels and that potential outcomes for unit i depend only on Wi , and not on the treatment
assignments of other units. We assume here that the potential outcomes are fixed, and only
W is random. Let Y(w) = (Y1 (w), . . . , Yn (w)) denote the vectors of potential outcomes
under treatment w, and Yobs = (Y1obs , . . . , Ynobs ) denote the vector of observed outcomes,
where Yiobs = Yi (1)Wi + Yi (0)(1 − Wi ).
5

When random sampling is the source of randomness, we are now primarily interested in
the underlying distributions for the population from which our sample was drawn. We thus
alter our notation slightly, although for comparison purposes attempt to keep it as similar
as possible. In this context, let Y (w) be a random variable denoting outcomes from group
w, and using the square bracket notation of [mean, variance], we have Yi (w) ∼ [µw , σw2 ], for
w ∈ {0, 1}. As the analog to SUTVA, here we assume independence between units.
Although the notational distinction here is framed in terms of random allocation versus
random sampling, the distinction may be better viewed in terms of causality or generalizability. In the population framework, we are asking a question inherently about generalizability: (how much) do the population distributions differ between the groups? In the
potential outcomes framework, we are asking an inherently causal question: (how much)
would outcomes change under the opposite treatment? The potential outcomes framework
is useful for causal inference in non-experimental settings as well (Rubin, 2007), although
without random allocation confounding variables must be accounted for, for example with
propensity score methods (Rosenbaum and Rubin, 1983). Rosenbaum (1984) incorporated
the propensity score with reallocating for testing causality in observational studies, but
here for simplicity we restrict our causal conclusions to the experimental framework.
While these methods allow for any statistic, throughout we use the difference in means:
Pn
Pn
obs
Y obs (1 − Wi )
obs
obs
i=1 Yi Wi
− i=1 i
.
(1)
τ̂ ≡ Y (1) − Y (0) ≡
n1
n0
Because there are often several different methods for getting from the distribution of τ̂
to a p-value or confidence interval, we restrict our focus here to the distribution of τ̂ . For all
methods, the distribution will be centered at 0 for testing and τ̂ for estimation, assuming
symmetry. Also, although higher moments of τ̂ can certainly be important as discussed in
Section 6.2, under all methods considered here τ̂ will be approximately normally distributed
if n0 and n1 are large enough (Ding, 2017). For these reasons, we focus henceforth on var(τ̂ ).
6

As a crucial point, although the two modes of inference differ in notation and scope of
inference, in both cases we work with the same statistic: τ̂ . Although random allocation or
random sampling may yield different statistics (in part due to the issue of confounding with
the latter), for a given data set τ̂ is calculated agnostic to the mode of inference. Therefore,
any differences in var(τ̂ ) arise solely due to differences in data collection (random allocation
versus random sampling), differences in simulation (reallocating versus resampling), or
differences due to testing versus estimation.

3

Testing a Sharp Null of No Difference

In this section we consider tests of Fisher’s sharp null hypothesis (Fisher, 1935) of no
difference whatsoever. Under the experimental framework this can be interpreted as no
treatment effect for any unit, H0 : Y(1) = Y(0), and under the sampling framework this
can be interpreted as no difference in the distributions for the two groups, H0 : Y (1) ∼ Y (0),
or equivalently, all outcomes come from the same distribution. In either case, the sharp
null implies that the outcomes are independent of W. A weaker null hypothesis of no mean
difference will be considered in Section 5.
In the subsections that follow, we derive var(τ̂ ) under random allocation, random sampling, reallocating, and resampling; under the sharp null hypothesis in this section, and
for estimation in Section 4. The variance of τ̂ is highlighted within each subsection as a
proposition to help the reader distill the key results. The goal of this paper is to synthesize
otherwise disparate results, providing var(τ̂ ) for each of the different scenarios in a cohesive and consistent format, and under as comparable as possible conditions, allowing us
to determine the extent to which the choice of simulation method matters, the extent to
which the simulation method should match the corresponding data collection method, and

7

the extent to which the chosen simulation method should differ for testing and estimation.

3.1

Random Allocation and a Sharp Null Hypothesis

Proposition 1. Under random allocation to two groups of fixed sample sizes n0 and n1 ,
and assuming the sharp null hypothesis H0 : Y(1) = Y(0),


1
1
2
+
,
var (τ̂ ) = s
n1 n0
Pn
obs
1
obs
− Y )2 is the sample variance of Yobs .
where s2 ≡ n−1
i=1 (Yi

(2)

Proof. Define p ≡ P (Wi = 1) = n1 /n, and we derive the covariance of W as follows:
If i = j : cov(Wi , Wj ) = var(Wi ) = p(1 − p)
If i 6= j : cov(Wi , Wj ) = E(Wi Wj ) − E(Wi )E(Wj ) =

−p(1 − p)
,
n−1

so
cov(W) =

p(1 − p)
(nIn×n − 1n×n ) ,
n−1

(3)

where 1n×n is the n × n matrix of all 1s, and In×n is the n × n identity matrix.
Define
s∗w 2

Pn
≡

i=1

2
Yi (w) − Y (w)
,
n−1

(4)

to be the sample variance for Y(w) for all units in the sample (regardless of actual treatment
assignment). Then, recalling Y(w) is fixed and only W is random, for w ∈ {0, 1},
var (Y(w)0 W) = Y(w)0 cov(W)Y(w)
np(1 − p)
=
n−1

n
X

(Yi (w) − Y (w))2

i=1

= s∗w 2 np(1 − p).
8

!

(5)

Likewise, var (Y(w)0 (1n − W)) = s∗w 2 np(1 − p). So for w ∈ {0, 1},


 s∗ 2 np(1 − p)
 obs
s∗w 2 n − nw
w
.
=
var Y (w) =
n2w
nw
n

(6)

Note that (n − nw )/n is almost the familiar finite population correction factor, except
with an n rather than n − 1 in the denominator. This is because we defined s∗w 2 with an
n − 1 in the denominator, despite the calculation being over the entire finite “population”.
Under the sharp null hypothesis, Y(0) = Y(1) = Yobs , so s∗1 2 = s∗0 2 = s2 and



 obs
s 2 n − nw
.
var Y (w) | Y(1) = Y(0) =
nw
n
P
P
P
Also under the sharp null, ni=1 Yiobs Wi + ni=1 Yiobs (1 − Wi ) = ni=1 Yiobs , so

cor Y 1 , Y 0 | Y(1) = Y(0) = −1.

(7)

(8)

Therefore, under random allocation



var (τ̂ | Y(1) = Y(0)) = var Y 1 + var Y 0 − 2cov Y 1 , Y 0

1/2
s2 p
s2 (1 − p)p
s2 (1 − p)
+
+2
=
np
n(1 − p)
n p(1 − p)


1
1
= s2
+
.
n1 n0

Unlike most true variance calculations that rely on unknown quantities, this can actually
be calculated (not just estimated) from the observed data. This differs from the typical
formula found in most introductory statistics textbooks because the sharp null hypothesis
assumes a common variance between treatment groups. This also differs from the usual
pooled variance estimate, as s2 is calculated assuming both equal variances and equal means
between the groups, so considers deviations of each data point from the grand mean, as
opposed to deviations of each data point from its own group mean.
9

3.2

Reallocating for Testing a Sharp Null Hypothesis

Under the sharp null hypothesis, Yobs = Y(1) = Y(0) is the same for every treatment
assignment W. Thus we can empirically create the distribution of τ̂ under the null by
leaving Yobs untouched and simulating many random allocations, W, and calculating τ̂
for each simulated W. Once a null distribution is generated in this fashion, the p-value is
calculated as the proportion of simulated statistics as extreme as, or more extreme than, the
observed difference in means (Fisher, 1935; Pitman, 1937). Note that in a randomization
test the variance of τ̂ is not used or needed, as the p-value is simply computed empirically
as a proportion of simulated statistics, but we give it here for comparison purposes.
If all random allocations were to be enumerated (exact method) the null distribution
achieved through reallocation in this fashion is exactly the true null distribution under
random allocation, and thus the p-value achieved in this fashion is widely recognized as
the gold standard correct p-value (Fisher, 1936; Kempthorne, 1952; Bradley, 1968; Tukey,
1988; Edgington and Onghena, 2007). However, in most realistic cases, enumerating and

calculating the statistic for all nn1 allocations is not feasible, so instead we simulate a large
number of reallocations (Monte Carlo method). In this case, the randomization distribution
will converge to the true null distribution as the number of simulations increases.
Proposition 2. Under reallocation to two groups of fixed sizes n0 and n1 , and assuming
the sharp null hypothesis H0 : Y(1) = Y(0), as the number of simulations increases,


1
1
2
+
.
var (τ̂ ) → s
n1 n0

(9)

Proof. As discussed, under the sharp null hypothesis, the distribution of the statistic under
reallocation will equal the true null distribution under random allocation, either exactly
(exact method) or in the limit (monte carlo method). Therefore, var(τ̂ ) is as in (2).
10

3.3

Random Sampling and a Sharp Null Hypothesis

Proposition 3. Under random sampling of two groups of fixed sample sizes n0 and n1 ,
and assuming the sharp null hypothesis H0 : Y (0) ∼ Y (1),


1
1
2
+
,
var (τ̂ ) = σ
n1 n0

(10)

where σ 2 is the population variance.
Proof. Under fixed sample sizes, simple random sampling, and independence, it is straightforward to derive the familiar formula:
σ12 σ02
var (τ̂ ) =
+ ,
n1 n0

(11)

where σw2 denotes the population variance within group w.
Under the sharp null hypothesis, σ1 = σ0 = σ, yielding


1
1
2
var (τ̂ | Y (1) ∼ Y (0)) = σ
+
.
n1 n0

Therefore, under the sharp null hypothesis, the distributions of τ̂ under random allocation or random sampling are almost equivalent, only differing by s2 versus σ 2 in var(τ̂ ).

3.4

Resampling for Testing a Sharp Null Hypothesis

To resample under the sharp null hypothesis, select samples of size n1 and n0 , each sampled
with replacement from the combined sample of all n units. Repeating this process many
times and calculating τ̂ for each yields a distribution of τ̂ under the sharp null, and as with
reallocating, a p-value can then be calculated as the proportion of the simulated statistics
11

that are as extreme as (or more extreme than) than the observed statistic. There are
other ways of obtaining a p-value from this distribution (Hall and Wilson, 1991), but for
comparison we focus here on var(τ̂ ) under the sharp null.
Proposition 4. Under resampling of two groups of fixed sample sizes n0 and n1 , and assuming the sharp null hypothesis H0 : Y (0) ∼ Y (1), as the number of simulations increases,



1
n−1
1
2
+
.
(12)
var (τ̂ ) → s
n1 n0
n
Proof. As derived by Efron and Tibshirani (1994, pg. 43),


 obs

s2 n − 1
var Y (w) | Y (1) ∼ Y (0) →
.
nw
n

(13)

Because the resampled groups will be independent (due to sampling with replacement),



1
1
n−1
2
var (τ̂ | Y (1) ∼ Y (0)) → s
+
.
n1 n0
n

Thus resampling gives a variance estimate that is biased in small samples. The variance will on average be slightly lower than the variance under random allocation, random
sampling, or reallocating. However, the bias diminishes as n gets large, and the estimate is
consistent under either data collection method. Moreover, the variances under resampling,
(12), or reallocating, (9), differ only by (n − 1)/n, and hence are asymptotically equivalent.
Note that reallocating and resampling are actually quite similar, with only one notable
difference. In either case we take a sample from Yobs , set Wi = 1 for i = 1, . . . , n1 and
Wi = 0 for i = n1 + 1, . . . , n, calculate τ̂ , and repeat many times. The only difference is
whether this sample is taken with (resampling) or without (reallocating) replacement. This
perspective can also help explain the extra factor of (n − n1 )/n in (7) as the adjustment
12

for sampling without replacement. Donegani (1991) shows that asymptotically, sampling
without replacement is better than sampling with replacement from a finite population, in
the sense that means will generally be closer to the overall mean.

4

Interval Estimation

This section mimics Section 3, but in the context of estimation, without a null hypothesis.
We first define the estimands, which differ depending on the mode of inference. Assuming we are interested in the true average difference between groups, the distinction
becomes whether this average difference is over all units in the sample, if both potential
outcomes could be observed, or over all units in the population, if we could collect data on
the entire population. When the only source of randomness is random allocation, we take
our estimand to be the true average treatment effect for the sample:
n

τra =

1X
[Yi (1) − Yi (0)] .
n i=1

(14)

Under random sampling from a greater population, our estimand can be defined as
τrs = E [Y (1) − Y (0)] = µ1 − µ2 .

(15)

Although it does not change the resulting confidence interval, which are based only on
the statistic τ̂ , note that the distinction between τra and τrs is not just finite sample versus
population inference, but again whether the inferences are inherently causal in nature.

4.1

Random Allocation without a Null Hypothesis

Proposition 5. Under random allocation to two groups of fixed sample sizes n0 and n1 ,
s∗1 2 s∗0 2 s∗1−0 2
var (τ̂ ) =
+
−
,
n1
n0
n
13

(16)

where s∗w 2 is defined in (4) and
2

s∗1−0 ≡

Pn

i=1

2
Yi (1) − Yi (0) − (Y (1) − Y (0))
n−1

(17)

is the sample variance of the unit-level treatment effects.
This result is derived in Imbens and Rubin (2015, Theorem 6.2).
We make three important observations regarding (17) before proceeding. The first is
that s∗1−0 2 is always positive, so ignoring this third term in (16), as is typically done, will
provide a conservative (overestimate) of the true randomization variance. The second is
that s∗1−0 2 can never be directly calculated, and usually cannot even be estimated, because
we never observe both potential outcomes for the same unit. Neyman (Splawa-Neyman,
1990, translated from the 1923 Polish version) recognized both facts, and so chose to define
confidence intervals as having at least the specified level of coverage. Lastly, if the treatment
effect is additive, Y(1) = Y(0) + a, then s∗1−0 2 = 0 and




1
1
1
s∗1 2 s∗0 2
1
∗2
∗2
+
= s1
+
+
= s0
.
var (τ̂ | Y(1) = Y(0) + a) =
n1
n0
n1 n0
n1 n0

(18)

Note that in general even if s∗1 2 = s∗0 2 , they will not equal s2 (except in special cases),
because s2 is calculated via taking deviations from the grand mean, whereas each s∗w 2 is
calculated taking deviations from its group mean. The sharp null hypothesis is a special
case of additivity, in which s∗1 2 = s∗0 2 = s2 because the group means are assumed to be the
same, and in this case (18) yields an alternative derivation of (2).

4.2

Reallocation for Interval Estimation

Reallocating requires an assumed relationship between Yi (1) and Yi (0) such that regardless
of which treatment is observed for each unit, both potential outcomes can be assumed
14

known. The most commonly assumed relationship is that of an additive treatment effect:
Y(1) = Y(0) + a. Tukey (1993) acknowledges that with reallocating “we will be pairing
platinum standard significance tests with silver or pewter standard confidence intervals”
due to the necessity of this additivity assumption, stating that “Such rigid displacement is
unlikely to happen in practice. It is a less than completely verifiable assumption.” Moreover,
unlike most of the results from this paper, this idea does not naturally extend to discrete
data, as it would require adding fractional values, potentially resulting in nonsensical values.
However, under additivity, reallocating can be used to generate a confidence interval as
the values of a that would not be rejected under a test of the null hypothesis H0 : Y(1) =
Y(0) + a (Pitman, 1937; Manly, 2007, Section 1.4). This can be quite computationally
intensive; Garthwaite (1996) provides an algorithm for doing this more efficiently.
Proposition 6. Under reallocation to two groups of fixed sample sizes n0 and n1 , and
assuming H0 : Y(1) = Y(0) + a, as the number of simulations increases,



1
1
n0 n1 a
2
(a − 2τ̂ )
+
.
var (τ̂ ) → s +
n(n − 1)
n1 n0

(19)

Proof. Define s2a ≡ s∗1 2 = s∗0 2 to be the common within group variance under H0 : Y(1) =
Y(0) + a, based on the observed data. Because reallocation mimics random allocation, (18)
applies, hence
var (τ̂ ) →

s2a



1
1
+
n1 n0


.

(20)

Under H0 : Y(1) = Y(0) + a,
Yi (1) = Yiobs Wi + (Yiobs + a)(1 − Wi ), and
Yi (0) = (Yiobs − a)Wi + Yiobs (1 − Wi ).
WLOG, assume just for the following derivation that the units are ordered such that Wi = 1
15

for i = 1, . . . , n1 and Wi = 0 for i = n1 + 1, . . . , n. Then
2
Pn
i=1 Yi (1) − Y (1)
2
sa =
n−1
2
 obs
Pn 
n0 a
Y
+
Y
(1)
−
i
i=1
n
=
" n n−1
#
n
1 



X
X
2
2
1
n
a
n
a
obs
obs
0
0
=
Yiobs − Y −
+
Yiobs + a − Y −
n − 1 i=1
n
n
i=n1 +1
"
#
n1
n
X
X
a
(n1 n20 + n0 n21 )a
obs
obs
2
obs
obs
=s +
− 2n0
(Yi − Y ) + 2n1
(Yi − Y )
n(n − 1)
n
i=1
i=n +1
1

n0 n1 a
(a − 2τ̂ ) .
= s2 +
n(n − 1)

We make a few relevant observations regarding (16). Note that (n0 n1 )/(n(n − 1)) does
not go zero as the sample size increases, hence var(τ̂ ) always depends on the ratio of the
group sizes, τ̂ , and the hypothesized additive effect, a. The variance of τ̂ is minimized
when a = τ̂ , and only equals the variance under the sharp null hypothesis of no treatment
effect, (9), when a = 0 or a = 2τ̂ ; var (τ̂ ) is lower than under the sharp null hypothesis for
0 < a < 2τ̂ , and higher otherwise. Figure 1 shows var (τ̂ ) as a function of a.
As an alternative to inverting a series of tests, some use the standard error from reallocating under the sharp null of H0 : Y(1) = Y(0) to create the margin of error around the
sample estimate. However, this will generally not give the correct coverage, because var (τ̂ )
depends on a, and choosing to use the variance when a = 0 is arbitrary and generally
incorrect, even for large samples. If reallocation is to be used for estimation, the method of
inverting many tests should be used, not the variance from one randomization distribution.
Without assuming additivity, estimation via reallocating becomes much more difficult
(Kempthorne, 1955), and as far as we know, there is a not a good solution. Reallocation
16

var(^τ)

^τ

0

2τ^

a

Figure 1: The variance of τ̂ for reallocating under Y(1) = Y(0) + a, as a function of the
assumed additive effect a. The blue dotted line is s2 (1/n1 + 1/n0 ), or var(τ̂ ) under the
sharp null, when a = 0. The scales of the axes are not shown, as they depend on the data.

should not be used for estimation when effects are suspected to vary substantially between
units, or equivalently when within group variances differ substantially between groups.

4.3

Random Sampling without a Null Hypothesis

Proposition 7. Under random sampling of two groups of fixed sample sizes n0 and n1 ,
var (τ̂ ) =

σ12 σ02
+ ,
n1 n0

(21)

where σw2 is the population variance within group w.
This is a commonly known result, and was also derived in (11).

4.4

Resampling for Interval Estimation

The resampling method we consider for estimation involves taking two independent samples
with replacement: one of size n1 taken from units with Wi = 1, and the other of size n0
17

taken from units with Wi = 0. This process is repeated many times, and τ̂ calculated for
each. There are many ways to use this to generate a confidence interval, such as using the
estimated standard error and a t-value to create the margin of error or by generating an
interval via percentiles (DiCiccio and Romano, 1988; DiCiccio and Efron, 1996; Hesterberg,
2015). For comparison purposes, here we again focus only on var(τ̂ ).
Proposition 8. Under resampling of two groups of fixed sample sizes n0 and n1 , as the
number of simulations increases,




s20 n0 − 1
s21 n1 − 1
+
,
var (τ̂ ) →
n1
n1
n0
n0
2

P
obs
1
2
obs
where sw ≡ nw −1 Wi =w Yi − Y (w) is the sample variance within group w.

(22)

Proof. The result follows immediately by the analogous version of (13) and independence.

Again this is slightly biased below the true variance, (21), but a consistent estimate.

5

Testing a Null of Equal Means

Section 3 considers the sharp null hypothesis of no difference, but often we postulate a
weaker null hypothesis, specifying only equality in the means: H0 : µ0 = µ1 for the population framework and H0 : Y (1) = Y (0) for the finite sample framework. How does this
change the resulting standard errors?
The true variances of τ̂ under this null, for both random allocation and random sampling, are equivalent to those derived for estimation, (16) and (21), because both derivations
are based on the variability of the data within groups (and across individual differences in
potential outcomes for random allocation), and these do not change under equal means.
18

As discussed in Section 4.2, the process of reallocating requires a hypothesis that implies
the values for all potential outcomes from the observed data. Although this could be done
without assuming additivity (by postulating a null difference for Yi (1) − Yi (0) that is not
constant across i), this is rarely, if ever, done in practice. Therefore, to avoid the assumption
of equal variances implicit with additivity, reallocating is essentially not a viable option.
Philip Good has been reported as distinguishing testing via resampling and reallocating
by the hypotheses themselves, noting that reallocating tests “hypotheses concerning distributions,” while resampling tests “hypotheses concerning parameters” (Berry et al., 2014,
p. 7). While reallocating is limited to testing the sharp null hypothesis, resampling offers
the flexibility of either null hypothesis. Under the sharp null of no difference, units can be
sampled from the combined sample, as discussed in Section 3.4. Under the null of equal
means, one can shift the groups to have equal means, creating new data, Y∗ , such that


Y obs − τ̂ /2 + b if Wi = 1,
i
∗
(23)
Yi ≡

Y obs + τ̂ /2 + b if Wi = 0,
i
where b can be any value. Common choices for b include τ̂ /2, −τ̂ /2, or 0, but the value of b
is actually irrelevant for a difference in means because it will cancel out. Resampling then
occurs by resampling n1 units from units with Wi = 1 and n0 units from units with Wi = 0,
and using the shifted values, Yi∗ to calculate τ̂ . Because resampled units always come from
their original group, all distributional differences between the groups besides center are
preserved. This is equivalent to resampling from the original values within each group as
described in Section 4.4 and then shifting the bootstrap distribution to be centered at 0,
so the resulting variance is equivalent to that given in (22).
The specification of the null hypothesis makes much more of a difference than the choice
to resample or reallocate, and the two decisions should not be conflated. The resampling
19

variances under a sharp null, (12), and this weaker null, (22), can differ substantially, while
under a sharp null hypothesis it doesn’t make much difference whether you reallocate, (9),
or resample, (12), at least in large samples.

6

Two Examples

We illustrate with two examples. The first is based on data from a randomized experiment
with relatively small sample sizes, approximately equal standard deviations between the
groups, and relatively symmetric distributions. The second is based on data from a random
sample with larger sample sizes, standard deviations differing substantially between the
groups, and positively skewed distributions with high outliers. In each case we simulate
each of the simulation procedures, using 20 million simulations to make the Monte Carlo
variation negligible; in all cases the empirical standard errors are identical to the theoretical
values as derived in Sections 3 to 5. We also calculate empirical p-values or confidence
intervals, which are all stable up to at least three decimal places.

6.1

Sleep versus Caffeine: A Randomized Experiment

Here we analyze data from a randomized experiment comparing the effect of sleep versus
caffeine on memory (Mednick et al., 2008). People were shown a list of words to memorize,
then randomly divided into two equally sized groups, n1 = n0 = 12. One group (Wi = 1)
took a nap for 90 minutes, while the other group (Wi = 0) took a caffeine pill and stayed
awake. The outcome, Yobs , is the number of words recalled after the nap/caffeine break.
The dataset, SleepCaffeine, is available at www.lock5stat.com/datapage.html. The data

20

are shown in Figure 2, and the sample difference in means is
τ̂ = Y

obs

(1) − Y

obs

(0) = 15.25 − 12.25 = 3.

(24)

W=1 (Sleep)

W=0 (Caffeine)

10

15

20

Yobs (Word Recall)

Figure 2: Data from the sleep versus caffeine randomized experiment on memory.

The overall sample standard deviation is s = 3.686, so the true standard error under
random allocation and a sharp null hypothesis can be calculated by (2):

s

2



1
1
+
n1 n0

 12



2



= 3.686

1
1
+
12 12

 12
= 1.505.

By (9), this is also the standard error for reallocating under the sharp null. The standard
error for resampling under the sharp null (from a combined sample), by (12), is

s

2



1
1
+
n1 n0



n−1
n

 12



2



= 3.686

1
1
+
12 12



24 − 1
24

 12
= 1.473.

By (22), the standard error for resampling for intervals or a null of equal means is


s21
n1



n1 − 1
n1



s2
+ 0
n0



n0 − 1
n0

 21

3.3062
=
12




12 − 1
12



3.5452
+
12



12 − 1
12

 12
= 1.340.

The standard errors for resampling are noticeablely smaller because of the small samples.
21

One-sided p-values are calculated as the proportion of simulated samples yielding statistics greater than or equal to τ̂ . We also compute a 95% confidence interval via reallocating
(inverting tests), and resampling using τ̂ ± t∗ SE:
τ̂ ± t∗ SE = 3 ± 2.2(1.34) = (0.05, 5.95).

(25)

Although not recommended, for comparison we also compute the interval using the standard error from the reallocating sharp null distribution. Results are shown in Table 1.
SE
Truth: Random allocation (sharp null) 1.505
Test

-

Reallocating (sharp null)

1.505

0.025

Resampling (sharp null)

1.473

0.022

Resampling (equal means)

1.340

0.013

-

(0.00, 6.00)

Reallocate (sharp null SE)

1.505

(-0.31, 6.31)

Resampling

1.340

(0.05, 5.95)

Reallocating (inverting test)
Interval

p-value or 95% CI

Table 1: Standard errors, p-values, and 95% confidence intervals for each of the different
simulation methods applied to the sleep versus caffeine example.

6.2

Income by Sex: A Random Sample

Here we analyze data obtained via random sampling, looking at how income of employed
American’s differs by sex using a random subsample of employed adults from the American
Community Survey (US Census Bureau, 2010). Let Wi = 1 for males and Wi = 0 for
females; n1 = 214 and n0 = 217. Let Y obs denote annual income in thousands of dollars.
22

The dataset, ACS, is available at www.lock5stat.com/datapage.html. The data are
displayed in Figure 3, and the observed difference in means is
τ̂ = Y

obs

(1) − Y

obs

(0) = 50.96 − 32.16 = 18.8.

(26)

W=1 (Males)

W=0 (Females)

0

100

200

300

400

500

Yobs (Annual Income in $1000)

Figure 3: Data from the American Community Survey on annual income by sex.

The theoretical standard errors are
 12 

 21
 
1
1
1
1
2
2
+
= 52.248
+
= 5.034
s
n1 n0
214 217
under reallocation with a sharp null,

s

2



1
1
+
n1 n0



n−1
n

 21



2



= 52.248

1
1
+
214 217



431 − 1
431

 12
= 5.028

under resampling from a combined population (sharp null), and


s21
n1



n1 − 1
n1



s2
+ 0
n0



n0 − 1
n0

 12

62.8482
=
214




214 − 1
214



36.9202
+
217



217 − 1
217

 12
= 4.962

under resampling within (possibly shifted) groups. Here the sample sizes are larger than
in the previous example, so reallocating and resampling differ less under the sharp null.

23

However, there is more of a difference in the standard deviation between the two groups,
thus resampling within groups differs from the sharp null standard errors.
As with the previous example, we calculated p-values and confidence intervals, with
results given in Table 2. These results are all highly significant, and we have strong evidence
that among employed American adults, males make substantially more than females.

Test

SE

p-value or 95% CI

Reallocating (sharp null)

5.034

2 × 10−5

Resampling (sharp null)

5.028

1.5 × 10−4

Resampling (equal means)

4.962

2.6 × 10−4

-

(9.13, 28.46)

Reallocate (sharp null SE)

5.034

(8.89, 28.72)

Resampling

4.962

(9.03, 28.58)

Reallocating (inverting test)
Interval

Table 2: Standard errors, p-values, and 95% confidence intervals for each of the different
simulation methods applied to the income and sex example.
The testing results across methods are counterintuitive; the standard errors and p-values
are inversely correlated! This is because the simulated distributions are not entirely normal;
as seen in Figure 3, income is highly right-skewed. In either simulation under the sharp
null, the actual sex of participants (W) is irrelevant, and because n1 ≈ n0 , the two groups
sampled are then indistinguishable, guaranteeing symmetry. On the other hand, this does
not hold when resampling within groups; because the males are more highly skewed with
more high outliers, resampling within group yields a positively skewed distribution. This
means there will be more high values in the right tail, explaining how this distribution yields
higher p-values, despite the lower standard error. Here the Pearson Moment Coefficient of
24

Skewness (estimated as m3 /s3 , where m3 is the third central moment) is about 0.15.
Although the standard errors under the sharp null appear almost identical, the p-values
differ by an order of magnitude, again cautioning against the sole use of standard errors
to judge comparability. To get at this difference, we have to go one moment further and
examine kurtosis. Based on our simulations, the sample excess kurtosis (estimated as
m4 /s4 − 3, where m4 is the fourth central moment) is about -0.17 for reallocating and 0.08
for resampling (both sampling from the combined sample and within groups). Because
the distributions based on resampling have fatter tails, in part because resampling allows
outliers to be sampled multiple times while reallocating does not, they can yield larger
p-values despite smaller standard errors. These issues were not relevant to the sleep versus
caffeine example, despite the smaller sample sizes, because the original data is relatively
symmetric and free of outliers (see Figure 2). Further work comparing higher moments for
reallocating and resampling would be worthwhile.

7

Summary

Table 3 summarizes var(τ̂ ) for each of the data collection methods (random allocation
and random sampling) and simulation methods (reallocation and resampling), for testing
a sharp null and generating confidence intervals. The results under a null of equal means
parallel those for intervals, except for reallocating.
When testing a sharp null, all methods yield very similar variances. The variance of τ̂
from random sampling differs from that of random allocation only in that it uses σ 2 rather
than s2 . With enough simulations, reallocation converges to the exact null distribution that
would be observed under random allocation, providing a gold standard test. The variance
of τ̂ under resampling only differs from reallocating by a factor of (n − 1)/n, which is

25

Method
Random Allocation
Reallocating
Random Sampling
Resampling

Testing: Sharp Null


1
1
2
s
+
 n1 n0 
1
1
+
s2
n1 n0 
1
1
σ2
+
 n1 n0  

1
1
n−1
2
s
+
n1 n0
n

Intervals
s∗1 2 s∗0 2 s∗1−0 2
+
−
n0
n


n1
1
1
n
n
a
0 1
2
(a − 2τ̂ )
+
for each a
s +
n(n − 1)
n1 n0
σ12 σ02
+
n1  n0



s21 n1 − 1
s20 n0 − 1
+
n1
n1
n0
n0

Table 3: Theoretical variance of τ̂ under different data collection and simulation methods.
Here s2 and s2w are known sample variances, σ 2 and σw2 are unknown population variances,
and s∗w 2 are unknown sample variances calculated across all units.

negligible for large n, but induces bias for small n. The variance from reallocating gives an
unbiased estimate of the true variance of τ̂ , regardless of whether the data were collected
using random allocation or random sampling, while the variance from resampling gives a
consistent estimate of the true variance, but biased slightly too low. In short, when testing
a sharp null hypothesis reallocation is slightly superior to resampling, regardless of the
method of data collection, although they are asymptotically equivalent.
When generating a confidence interval, the differences in var(τ̂ ) across both data collection methods and simulation methods are more substantial. The true variance of τ̂ under
random allocation involves the variance of the unit-level treatment effects, and thus is impossible to estimate unbiasedly using only the observed data. It will in general be lower
than var(τ̂ ) under random sampling. Reallocating requires an assumed relationship between Y(1) and Y(0) in order to fill in all the missing potential outcomes from Yobs , which
usually takes the form of assuming additivity, Y(1) = Y(0) + a. Inverting many randomization tests for different values of a can yield a valid confidence interval, but only under the
26

assumption of additivity, or equivalently equal variances between treatment groups. Using
the standard error from the sharp null randomization distribution for estimation is not recommended, as the standard error depends on the value of the hypothesized additive effect,
a, and this approach arbitrarily sets a = 0. The variance of τ̂ from resampling is a consistent estimate for the true variance under random sampling, and a conservative estimate
of the true variance under random allocation, which is the best that can be done without
additional assumptions. For intervals, resampling is superior to reallocation, regardless of
the method of data collection, unless the treatment effect is truly additive.
These results are counterintuitive, and go contrary to popular belief. Intuition would
lead one to believe that the randomization in the simulation method should mimic the
randomization in the data collective method (if it matters at all), but this is not the case.
For a difference in means, with either complete randomization to two groups or simple
random sampling of two groups, we find that reallocating is superior for testing a sharp
null hypothesis with small n (while for large n it doesn’t matter which method is used),
that resampling is superior for intervals unless additivity holds, and that these findings
apply regardless of how the data were collected.
Of course, for more complex study designs, it becomes more important that the simulation method mimic the data collection method, and for more complex designs reallocating
will likely be more natural if random allocation was used and resampling more natural if
random sampling was used. Likewise, it is always important that randomness be present
in some form, and hence the data collection method will always be relevant for inference.
Moreover, our results are based solely on var(τ̂ ), but, as evidenced by Section 6.2, further
moments should be considered if τ̂ is not normally distributed. However, for the cases examined here, matching the randomness in simulation to the randomness in data collection
need not be a priority, at least not for mathematical reasons.
27

References
Berry, K. J., Johnston, J. E. and Mielke Jr., P. W. (2014), A Chronicle of Permutation
Statistical Methods, Springer, Switzerland.
Bradley, J. (1968), Distribution-Free Statistical Tests, Prentice-Hall, Englewood Cliffs, NJ.
Bredenkamp, J. (1980), Theorie und Planung psychologischer Experimente, SpringerVerlag, Steinkopff, Darnstadt, Germany.
Cotton, J. W. (1973), ‘Even better than before’, Psyccritiques 18(4), 168–169.
DiCiccio, T. J. and Efron, B. (1996), ‘Bootstrap confidence intervals’, Statistical Science
11(3), 189–228.
DiCiccio, T. J. and Romano, J. P. (1988), ‘A review of bootstrap confidence intervals’,
Journal of the Royal Statistical Sociey. Series B (Methodological). 50(3), 338–354.
Ding, X. L. . P. (2017), ‘General forms of finite population central limit theorems with
applications to causal inference’, Journal of the American Statistical Association .
Donegani, M. (1991), ‘Asymptotic and approximative distribution of a statistic by resampling with or without replacement’, Statistics and Probability Letters 11, 181–183.
Edgington, E. S. and Onghena, P. (2007), Randomization Tests, Chapman & Hall/CRC.
Efron, B. (1979), ‘Bootstrap methods: Another look at the jackknife’, The Annals of
Statistics 7(1), 1–26.
Efron, B. and Tibshirani, R. (1994), An Introduction to the Bootstrap, Chapman &
Hall/CRC.
28

Fisher, R. A. (1935), The Design of Experiments, Oliver and Boyd, Edinburgh.
Fisher, R. A. (1936), ‘’the coefficient of racial likeness’ and the future of craniometry’, The
Journal of the Royal Anthropological Institute of Great Britain and Ireland 66, 57–63.
Garthwaite, P. H. (1996), ‘Confidence intervals from randomization tests.’, Biometrics
52, 1387–1393.
Hall, P. and Wilson, S. R. (1991), ‘Two guidelines for bootstrap hypothesis testing’, Biometrics pp. 757–762.
Hesterberg, T. C. (2015), ‘What teachers should know about the bootstrap: Resampling
in the undergraduate statistics curriculum’, The American Statistician 69(4), 371–386.
Imbens, G. and Rubin, D. (2015), Causal Inference in Statistics, Social, and Biomedical
Sciences: An Introduction, Cambridge Univ Pr.
Kempthorne, O. (1952), The Design and Analysis of Experiments, John Wiley and Sons,
New York.
Kempthorne, O. (1955), ‘The randomization theory of experimental inference’, Journal of
the American Statistical Association 50(271), 946–967.
Kempthorne, O. (1979), ‘Sampling inference, experimental inference and observation inference’, Sankhya: The Indian Journal of Statistics, Series B 40(3/4), 115–145.
LaFleur, B. J. and Greevy, R. A. (2009), ‘Introduction to permutation and resamplingbased hypothesis tests’, Journal of Clinical Child & Adolescent Psychology 38(2), 286–
294.

29

Ludbrook, J. (1995), ‘Issues in biomedical statistics: Comparing means by computerintensive tests’, Australian and New Zealand Journal of Surgery 65(11), 812–819.
Ludbrook, J. and Dudley, H. (1998), ‘Why permutation tests are superior to t and f tests
in biomedical research’, The American Statistician 52(2), 127–132.
Manly, B. F. (2007), Randomization, Bootstrap and Monte Carlo Methods in Biology, 3rd
edn, Chapman & Hall/CRC.
Mednick, S. C., Cai, D. J., Kanady, J. and Drummond, S. P. (2008), ‘Comparing the benefits of caffeine, naps and placebo on verbal, motor and perceptual memory’, Behavioural
Brain Research 193, 79–86.
Neyman, J. and Pearson, E. S. (1928), ‘On the use and interpretation of certain test criteria
for purposes of statistical inference: Part i’, Biometrika pp. 175–240.
Pitman, E. J. (1937), ‘Significance tests which may be applied to samples from any populations’, Supplement to the Journal of the Royal Statistical Society 4(1), 119–130.
Romano, J. P. (1989), ‘Bootstrap and randomization tests of some nonparametric hypotheses’, The Annals of Statistics 17(1), 141–159.
Rosenbaum, P. R. (1984), ‘Conditional permutation tests and the propensity score in observational studies’, Journal of the American Statistical Association 79(387), 565–574.
Rosenbaum, P. R. and Rubin, D. B. (1983), ‘The central role of the propensity score in
observational studies for causal effects’, Biometrika pp. 41–55.
Rubin, D. (1974), ‘Estimating causal effects of treatments in randomized and nonrandomized studies.’, Journal of Educational Psychology 66(5), 688.
30

Rubin, D. B. (1980), ‘Randomization analysis of experimental data: The fisher randomization test comment’, Journal of the American Statistical Association 75(371), 591–593.
Rubin, D. B. (2007), ‘The design versus the analysis of observational studies for causal
effects: Parallels with the design of randomized trials’, Statistics in Medicine 26(1), 20–
36.
Splawa-Neyman, J. (1990), ‘On the application of probability theory to agricultural experiments. essay on principles. section 9.’, Statistical Science 5, 465–472.
Tukey, J. (1988), ‘Randomization and re-randomization: the wave of the past in the future’, Statistics in the Pharmaceutical Industry: Past, Present and Future. Philadelphia
Chapter of the American Statistical Association .
Tukey, J. W. (1993), ‘Tightening the clinical trial’, Controlled clinical trials 14(4), 266–285.
US Census Bureau (2010), ‘American community survey (acs) public use microdata sample
(pums)’.
URL: www.census.gov/programs-surveys/acs/data/pums.html

31

