A Discussion of Three Papers on Variance Estimation 

Phillip S. Kott 

National Agricultural Statistical Service 

Abstract 

This note expands upon my comments at the 2005 FCSM Research Conference on the papers presented by Lenka Mach, 

Katherine (Jenny) Thompson, and Laura Ozcoshun. 

1. Introduction 

The three papers under discussion address variance estimation using some form (or forms) of replication. Lenka Mach 

discusses the bootstrap, Jenny Thompson discusses two versions of the jackknife, and Laura Ozcoshun discusses 

balanced repeated replication or BRR. All three focus on empirical analyses. By contrast, I will concentrate on what 

theory has to say about the methods the presenters employ. I will try to use the notation in the three papers when 

discussing each. 

The estimator for which one needs a measure of precision is often complicated. It may be a nonlinear function of totals 

or incorporate complex adjustments for nonresponse (or undercoverage). Replication is a useful variance-estimation 

technique in such situations. In addition, when users with access to the survey data are not government statisticians, 

replication can be helpful in protecting respondent confidentiality (as alluded to by Mach) or in providing users with a 

simple way to measure the precision of complicated estimators they devise. 

Let me state some well-known theoretical results. If finite population correction (fpc) can be ignored, then the bootstrap, 

BRR, and stratified jackknife variance estimators for an expansion estimator are all exactly unbiased. By contrast, the 

delete-a-group (dag) jackknife need not be. Furthermore, if fpc can be ignored, and the estimator is a smooth but 

nonlinear combination of expansion estimators, then (under mild conditions) all the replication variance estimators are 

only nearly unbiased. 

2. A Few Comments on Mach 

The analysis in this paper is complicated by different randomizations: 

One for the original sample, which has two stages. 

One for the bootstrap samples. 

One for the simulations. 

As Mach herself notes, the sample sizes in her empirical work are too small to draw firm conclusions. 

Mach points out that for a linear estimator t of 2 under a two-stage sample, 

E [(v (t)] = v (t), B BS WR 

where vWR(t) denotes the standard with-replacement-in-the-first-stage variance estimator for Var(t). It is not necessary 

for the actual first-stage (school) sample to have been drawn with replacement for this equality to always hold true. 

Let t be an unbiased linear estimator under an unstratified two-stage sample using without-replacement sampling in the 

first stage of sampling. Then 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

E[v (t)] = E E { 3 (t /B )

2

WR 1 2 i0F i i i ! 3 j

(t / B )(t / B (n)/ !1)} i i j j 

 = E { 3 (2 /B )

2 ! 3 (2 /B )(2 /B )/(n !1)} + E { 3 Var [t ]/B2

)}, 1 i0F i i ij i i j j 1 i0F 2 i i

where t is an estimator for 2 , F is the first-stage sample of schools, and B is the selection probability for school i. Note i i i 

that E { 3 Var [t ]/B

2

)} is the first stage expectation of the second-stage variance. It is captured exactly by the withreplacement variance estimator. 

1 i0F 2 i i 

Mach attempts to adjust for the finite population correction with 1 ! n/N, where n/N is the first-stage sampling fraction. 

I would have used 

2 1 ! 3S k p /m

instead, where S denotes the second-stage sample, p k

the combined selection probability for student k ([n/N][m / M ]), i i 

and m the number of students in the sample. This factor comes from assuming the variable of interest behaves like an 

independent and identically distributed random variable across students. In particular, it ignores the possibility that the 

student attitudes to the question asked are correlated within a school. For the opposite extreme, where all the students 

in a school are assumed to have identical attitudes but that attitude is independent across schools, a reasonable fpc factor 

2 2 would be 1 ! i

 n 3 M /M , where the summations are over all the schools in the population. 

3. A Few Comments on Thompson 

I will concentrate mostly on the dag jackknife since I coined the term. Consider a stratified simple random sample. The 

standard sampling weight for i 0S is h 

w = N / n . i h h

For the dag jackknife, the sample is first systematically divided into K groups so that the numbers of units in each group 

from stratum h are as close to equal as possible. Let S h(k) denote the set of units in stratum h and NOT in group k. I have 

proposed determining jackknife replicate weights for i 0 Sh(k) like so: 

w

Kott = N / n , i(k) h h(k)

where nh(k) is the size of S h(k). . By contrast, Thompson uses 

Thompson wi(k) = (K/[K!1]) N / n . h h 

We both set w i(k) = 0 for i 0Shk, where Shk is the set of sampled unit in stratum h and group k, and then compute

 K 

v( 3 w y ) = 3 ([K!1)/K] ( 3 w y ! 3 w y )2

 i0S k=1 i0S i0S 

dag i i i(k) i i i 

To see how these approaches differ, we investigate the properties of the different approaches under the simple prediction 

model for i 0 U : y ~ (: , F

2

), where the y are independent across the i. A purely randomization analogue to the 

subsequent analysis can be developed, but not as easily. 

h i h h i 

H 2 2 The model variance of t = 3S i i h h h w y is 3 (N / n )F , and 

E [v Kott 

M dag M h h ] = Var (t) + {function of the F for those h where N / K is not an integer}. 

The bias has a slight tendency to be upward and is trivial when nh > 5. By contrast, 

Thompson 2 E [v ] = Var (t) + {function of the M dag M h h : for those h where N / K is not an integer} . 
 

 

 

 

 

 

 

 

 

 

 

 

 

This bias is upward and persists even when the Fh are zero. Nevertheless, v dag

Thompson may not be so bad for Thompson’s 

application because Fh can be large compared to :h when modeling annual capital expenditures. 

To keep things simple, we next consider a count adjustment for nonreponse. For i 0 S , one has the weight adjustment h 

w = (N / n )(n / r ) = N / r , i h h h h h h 

where rh is the number of respondents in h. I would compute the replicate weight 

w

Kott = (N / n )(n / r ) = N / r i(k) h h(k) h(k) h(k) h h(k) 

for i 0 Sh(k) . Thompson computes either the short-cut version: 

Shortcut w = (K/[K i(k) h h !1]) N / r , 

Thompson which is akin to wi(k) and so may not be bad, and a fully-replicated version: 

”Fully” w = (K/[K i(k) h h h(k) h(k) !1])(N / n )(n / r ), 

which to me is neither fish nor fowl and cannot be recommended. 

The theory behind the ratio adjustment for nonresponse used in Thompson’s paper is more difficult, but also unnecessary. 

If ratioing to payroll makes sense in the context of an ACES stratum, then it should be done using the population total 

rather than the sample total. Furthermore, given the weak relationship between the dependent variables (different types 

of capital expenditure) and the auxiliary (payroll), the Census Bureau should consider using a separate-regression rather 

than a separate-ratio estimator with ACES-type data. Kott (2003) describes a method that usually ensures the implicit 

sample weights in a separate regression estimator will all be positive. 

Finally, if there is really a need to incorporate an fpc factor, then a version of the stratified jackknife (with a separate-ratio 

or regression estimator) should be used rather than the dag jackknife, since the dag can not easily be adjusted for high 

sampling fractions in a theoretically defensible manner. Moreover, the added error due to the response/nonresponse 

mechanism is captured by employing (1 ! r / N ) rather than (1 ! n / N ). Notice that this factor can also be used in 

certainty strata that exhibit nonresponse. 

h n h n 

4. A Few Comments on Ozcoshun 

In what follows, I will ignore unit-on-property adjustment and small-cell problems. The latter obviates the need for 

modified half sampling. Modified half sampling is a reasonable thing to do in certain contexts, but is beyond the scope 

of my remarks. 

The “nonresponse adjustment” Ozcoshun describes leads to nearly unbiased estimators under the quasi-random response 

model, where every sampled unit in a cell is equally likely to respond. That is the type of model invoked in the 

Thompson’s paper. 

The “second-stage ratio adjustment” leads to a nearly unbiased estimator under the prediction model, where the variable 

of interest (y ) is i i d within each cell whether or not they respond (the response/nonresponse mechanism is ignorable). 

Unlike with the response model, a separate prediction model is needed for each variable of interest. 

i

If the quasi-random response model is correct, but the prediction model is not, the second-stage ratio adjustment can still 

decrease mean squared errors. Ozcoshun appears to think that the cell definitions for the two adjustment must coincide. 

That is not the case. 

Only one of the models need hold for near unbiasedness is some sense. If the prediction model holds, then partial 

replication is reasonable. If the prediction model fails, but the quasi-random model holds, then only full replication is 

reasonable. If the prediction model holds, there may be an upward bias in the short-cut method. Similarly, for the 
 

 

 

 

response model. 

The added variability in the replicate weights from a replicated ratio adjustment can create an upward bias. The smaller 

the cells, the greater the potential for bias. This is not a problem with the shortcut method, which gets the weights right, 

but the residuals wrong. 

The last several observations parallel findings Thompson’s and my versions of the dag jackknife. Thompson’s version 

of the dag is like Ozcoshun’s shortcut version of BRR, while my version is like her version of full replication. This is 

because my version of the dag jackknife weights treats the sampling strata like model groups but Thompson’s does not. 

5. Concluding Remarks 

Replication does not work by magic. It works because there is asymptotic theory supporting its use. I have simplified 

the analysis by stripping away some of the complexity from the applications and by assuming simple models. My results

nonetheless provide useful insights into what replication does in certain contexts. Unfortunately, we live in a finite 

world, where asymptotic theory is not always the last word. That is why we need empirical studies like those described 

in these three useful papers. 

Additional Reference 

Kott, P.S. (2003). A Practical Use for Instrumental-Variable Calibration, Journal of Official Statistics 19, 265-272.