1

Building a Better Delete-a-Group 

Jackknife for a Calibration Estimator 

(Like That Based on Data from the ARMS III) 

Phillip S. Kott 



This note summarizes much of the theory behind the use of the Delete-aGroup (DAG) jackknife with calibrated survey data like that coming from 

the third phase of the Agricultural Resources and Management Survey 

(ARMS III). A DAG jackknife employing 15 sets of replicate weights is 

used for variance estimation with ARMS-III data. Many analyses of this 

data are conducted by the Economics Research Service and by other 

agricultural economists outside of the US Department of Agriculture, 

making a portable variance-estimation technique like the DAG jackknife 

especially desirable. Some suggestions are offered to improve the 

present methodology. 

KEY WORDS: Sampling Weight; Replicate Weight; Variance; Mean 

Squared Error. 



Phillip S. Kott is Chief Research Statistician, Research and Development Division, 

National Agricultural Statistics Service, 3251 Old Lee Highway, Room 305, Fairfax, 

Virginia, 22030. An earlier version of this paper was prepared for the Workshop on 

Calibration and Estimation in Surveys, October/November 2007, Ottawa, Ontario. The 

final section (“The ARMS III Application”) and Appendix B were not in the original. 
2

Introduction 

 The National Agricultural Statistics Service (NASS) has increasingly been using a 

delete-a-group (DAG) jackknife for variance estimation. In surveys where this technique 

is used, each sampled element k (usually a farm) is given 16 weights: the element’s 

actual sampling weight after incorporating all nonresponse and calibration adjustments, 

wk, and 15 jackknife replicate weights, wk(g), with g = 1, ..., 15. 

 The estimator, t, for a univariate parameter of interest, T, is computed using the 

actual (post-adjustment) sampling weights. Fifteen replicate estimators, tg, each 

computed with its respective set of replicate weights are then calculated. The DAG 

jackknife variance estimator for t is 

vDAG = (14/15)∑

15 (t − t(g))2

. (1) 

NASS recommends that when computing coverage intervals or testing hypotheses, one 

treat the associated z statistic, z = (t − T)/√vJ, as if it had a Student’s t distribution with 

14 degrees of freedom. 

 This note summarizes much of the theory behind the use of the DAG jackknife 

with data from the third phase of the Agricultural Resources and Management Survey 

(ARMS III). The survey is based on a multi-phase composite sample of farms. There is 

also an area-frame component to pick up farms not on the NASS list. The ARMS III 

provides an annual assessment of the economic conditions of US farms. 

The respondent sample for the ARMS III is calibrated to targets determined from 

other NASS surveys. A delete-a-group jackknife employing 15 sets of replicate weights 

is used for variance estimation. Much of the analysis of ARMS III data is conducted by 

the Economics Research Service and by other agricultural economists outside of the 

US Department of Agriculture, making a portable variance-estimation technique like the 

DAG jackknife especially desirable. 

In general, the theory underpinning the use of the DAG jackknife (and all 

jackknives for that matter) is asymptotic. See Kott (1998). 

The focus here will be on the properties of the DAG jackknife for a calibration 

estimator, with a particular emphasis on model-based properties. If a variance 
3

estimator does not behave well under a reasonable model, then it is unlikely to behave 

well when that model fails. 

 

Some Calibration Theory

Suppose our goal is to estimate a population total, Ty = ∑U yk, where U denotes 

the population of N members. The expansion estimator for Ty is 

ty_E = ∑S dkyk, 

where S is the sample of n members, and dk is the original sampling weight of element k 

(usually the inverse of the unit’s selection probability). 

Suppose there is a row vector of auxiliary variables, xk = (x1k, ..., xPk), for which 

the population total(s), Tx = ∑U xk, is known. Deville and Särndal (1992) coined the term 

“calibration estimator” to describe an estimator of the form 

ty_CAL = ∑S wkyk, 

where the wk are close to the dk, and the calibration equation, 

∑S wk xk = ∑U xk, 

holds. 

As with the expansion estimator, the same set of calibration weights can be used 

no matter what the y-variable. Usually, ty_CAL is nearly randomization unbiased for a 

sufficiently large sample under mild conditions. See, for example, Kott (2005a). 

 Observe that ty_CAL estimates Ty perfectly when yk equals xkβ exactly for all k. 

Thus, it is reasonable to expect ty_CAL to be a good estimator when yk and xkβ are 

almost always close. This is formalized by assuming the yk are random variables 

satisfying the linear prediction model: 

 yk = xkβ + εk, (2) 

 

where E(εk |xg, Ig ) = 0 for all k ∈ U; Ig = 1 when g ∈ S, 0 otherwise. Under this model, 

it is easy to see that ty_CAL is an unbiased estimator for Ty in the sense that 
4

Eε(ty_CAL ! Ty) = 0. 

Most of the calibration weighting in practice involve a variant of least squares, 

where the calibration weights have linear the form: 

 wk = dk(1 + ckxkg) = dk(1 + ckg'xk') 

for some set of constants {ck}, and a vector: g = (∑S djcjxj'xj)

-1 (Tx −∑S djxj)'. 

 This calibration estimator is also a generalized regression (GREG) estimator: 

ty_CAL = ∑S dk (1 + ck g'xk') yk 

 = ∑S dkyk + g' ∑S dkckxk'yk 

 = ty_E + ( Tx − ∑S djxj)( ∑S djcjxj

'xj)

 -1∑S dkckxk'yk 

 = ty_E + ( Tx − ∑S djxj) b, 

 

 

 

where b = ( ∑S djcjxj'xj)

 -1∑S dkckxk'yk. It is easy to see the b is an unbiased estimator for 

β under the model. 

If the εk in the model are uncorrelated, each with a variance of σk

2

, then the 

model variance of t_CAL is 

Eε[( ∑S wkyk − ∑U yk )

2

] = Eε[( ∑S wkεk − ∑U εk )

2

] 

 = ∑S wk

2

σk

2

− 2 ∑S wkσk

2

 + ∑U σk

2

 

 ≈ ∑S wk

2

σk

2

 

when most wk >> 1, which we will assume to be the case here. 
5

The Delete-a-Group Jackknife

Suppose the (respondent) sample was randomly divided into 15 mutually 

exclusive and nearly equal groups denoted Sg, where g = 1, ..., 15. We call the 

complement of each group, the jackknife replicate group and denote it by S(g). 

Observe that 

 

∑S wkεk − ∑ 15 14 S(g)( / )wkεk = ∑ ε Sgw k k − ∑ 1 14 S(g)( / )wkεk. 

A little work reveals that 

 

 EM[∑

15 (14/15)( ∑S wkεk − ∑ 15 14 S(g)

( / )wkεk)

2

] = ∑S wk

2

σk

2

, 

the approximate model variance of t_CAL. 

Unfortunately, the εk are unknown. We could replace them by the yk in the above 

expression if the calibration equation 

 ∑ 15 14 S(g)

( / )wkxk = ∑U xk 

held for all g. It generally does not. We can, however, replace each (15/14)wk within 

S(g) by the corresponding (nonstandard) g’th replicate weight: 

wk(g) = (15/14)wk + (∑U xi − ∑ 15 14 S(g)[ / ]wixi)[ ∑ x x S(g) j j j j c d ' ]

-1ckdkxk' . 

(Most of the results discussed here hold equally well for a more conventionally defined 

replicate weight, but the one above is more useful for our purposes.) 

Each replicate weight in S(g) is close to its corresponding (15/14)wk. Moreover, 

as a group, they satisfy the calibration equation since 

∑ x S(g)

w k(g) k = ∑ 15 14 S(g)( / )wkxk + 

 (∑U xi − ∑ 15 14 S(g)[ / ]wixi)[ ∑ x x S(g) j j j j c d ' ]

-1 ∑ x x S(g) k k k k c d '

 = ∑U xi. 

Defining wk(g) to be 0 when k ∈Sg, the delete-a-group jackknife variance estimator 

can be expressed as 
6

 vDAG = 

15 2

1

14 15

= ∈ ∈

∑ ∑ ∑ k k k(g) k −

g k S k S

( / )( w y w y ) 

 

 = ∑

15 (14/15)( ∑S wkεk − ∑ S wk(g)εk)2 

 (3) 

 ≈ ∑

15 (14/15)( ∑S wkεk − ∑ 15 14 S(g)

[ / ]wkεk)

2 

.

 

 

Thus, vDAG is a nearly unbiased estimator for the model variance of ty_CAL. It is not hard 

to see that the model bias, caused by the replication weights not exactly equaling 

(15/14)wk for k ∈S(g), is usually upward. There is no theoretical guarantee of that, 

however. 

Let us now turn to the randomization-based properties of the DAG jackknife 

under two particular sampling designs. Kott (2006a) shows that under Poisson 

sampling vDAG is almost unbiased for the randomization mean squared error of ty_CAL

when the population residual, ek = yk − xk(∑U cjxj

'xj)

 -1∑U cjxj

'yj, is such that ∑U ek = 0. 

The key is that both terms of 

∑S wkek − ∑ 15 14 S(g)( / )wkek 

need to estimate ∑U ek = 0. For that to happen ck must be expressible as xkλ for some 

column vector λ. This will often rule out the randomization-optimal estimator where 

ck = dk − 1. Although not always a good estimator for the randomization mean squared 

error of ty_CAL under Poisson sampling, the DAG jackknife remains a nearly unbiased 

estimator for the model variance so long as the wk are all large. 

Let us now look at the expansion estimator under stratified simple random 

sampling before any calibration or nonresponse adjustment. Ignoring finite population 

correction, it has a randomization variance of 

 H 

Vrand = ∑ (Nh

2

/nh) ∑ ek

2

/(Nh − 1) ≈ ∑ wkek

2

, 

 

 h=1 

 

 

 k∈Uh k∈U 

where ek = yk − ∑ ∈ h

i U j h y /N , and wk = Nh &nk for k 0 Uh. 

 
7

Although incalculable (because the ek are unknown), 

v = ∑S w

2

ek

2

 

is an unbiased estimator for ∑U wkek

2

, while v itself is approximated by the DAG 

jackknife in equation (1) with wk(g) = Nh &nh(g) for k ∈ Sh so long as Nh &nh(g) is 

approximately (15/14)Nh &nh. This last approximation requires large-enough nh since 

each (14/15)nh must approximate nh(g). A good rule of thumb is that all nh should exceed 

5. Kott (2001) proposes an extended DAG jackknife for designs with stratum sample 

sizes smaller than that. 

When the calibration occurs after the latest phase of a multi-phase sample, one 

can construct the following replicate weights: 

wk(g) = (dk(g)/dk)wk + ∑U xi − ∑S(g) i(g) i [d /d ] wixi)[ ∑ x x S(g) j j j j c d ' ]

-1ckdkxk', 

where dk is the sampling weight before calibration, while dk(g) is the replicate weight 

before calibration. When dk(g) &dk ≈ 15/14, the approximate model unbiasedness of the 

DAG jackknife holds, but there is an additional tendency for the bias to be upward. 

If the DAG jackknife is almost randomization unbiased before calibration, it remains so 

after. 

Nonresponse and Coverage Adjustment 

We can use the same prediction model to justify both calibration weighting and 

the DAG jackknife when the sample is subject to nonresponse or the frame to coverage 

errors. For handling coverage errors, the true (without coverage-error) Tx is assumed 

known. 

An alternative justification for calibration treats unit response or frame coverage 

as another phase of Poisson sampling. This is called quasi-randomization or quasirandom response (coverage) modeling. The theory supporting the use of the DAG 

jackknife is analogous to the theory in Kott (2006b) for the conventional stratified 

jackknife. 
8

Can We Build a Better DAG Jackknife (for the ARMS III)? 

Should NASS increase the number of replicates? 

Having more than 15 replicates would be helpful for multivariate testing and for 

increasing the precision of variance estimates. On the other hand, when there are 

fewer replicates, there are more units within each, which is more compatible with the 

invocation of asymptotic normality. Setting G = 30 seems a reasonable compromise. 

Can we remove the upward bias in the DAG jackknife? 

It is not hard to see that replacing 

 

 vDAG = ∑

G

 [(G −1)/G](∑S wkyk − ∑ S wk(g)yk)

2

 

 

 with 

 vDAG* = ∑

 G

2

2

∑

∑ −

Sg k

S k k(g)

w

(w w )

 (∑S wkyk − ∑ S wk(g)yk)

2

 (4) 

will remove the bias of the DAG under the model in equation (2) when the εk are 

uncorrelated and have a common variance. Note that the asymptotic randomizationbased properties of the DAG jackknife are unaffected by this substitution. 

We can similarly remove the bias under an alternative working model − say, that 

the element variances are proportional to fk − by replacing 

 

2

2

∑

∑ −

Sg k

S k k(g)

w

(w w )

 with 

2

2

∑

∑ −

Sg

k k

S k k(g) k

w f

(w w ) f

 

in equation (4). So long as the εk are uncorrelated, the (bias-adjusted) DAG jackknife

remains nearly model unbiased even when the working-model speculation about the 

relative sizes of the σk

2

 is wrong. 

Although we have so far only discussed totals, the DAG jackknife is also effective 

when t is component of a regression coefficient of the form: 

 b = (∑S wjhjzj

') -1 ∑S wkhkyk 
9

for column vectors hj and zj (in a common example both are equal to xk'). To compute 

the DAG jackknife we replace wk and wk(g) with the analogous components of 

ωk = (∑S wjhjzj

') -1 wkhk and 

ωk(g) = (∑S wj(g)hjzj

') -1 wk(g)hk . 

For the special case of a domain mean, zj is the scalar 1, while hj is 1 when j is in the 

domain and 0 otherwise. 

In Appendix A, a working model is invoked to compute a theoretically better 

measure for the effective degrees of freedom of the DAG jackknife than G – 1. It 

remains an open question how relevant removing the model bias and better measuring 

the model variance of the DAG jackknife are in the face of inevitable model failure. 

What can we do about replicate-weight outliers? 

Ideally, a replicate weight should be close to G/[G−1]wk. What if it were not? 

We could truncate troublesome replicate weights or even remove troublesome target 

variables from the replicate calibration. Surprisingly, this may tend to cause an upward 

bias in variance estimation. 

To see why, look at the model expectation of DAG jackknife expressed in 

equation (3). The term 

15

1

14 15

=

∑

g

( / )(∑S wkxkβ − ∑ S wk(g)xkβ)

2

 = ∑ (14/15)[(∑S wkxk − ∑ S wk(g)xk)β]

2

is missing from the right hand side of the equation because the wk(g), like the wk, are 

calibrated on all the components of xk. When that calibration is not enforced (due, for 

example, to truncating the replicate weights), the term above adds a positive model bias 

to the DAG jackknife. This size of the bias is directly related to the effectiveness of 

calibration in reducing the variance/mse of the estimator itself (through the absolute 

sizes of the components of β). 

Since the DAG jackknife both before and after truncating replicate weights − or 

after removing calibration targets when computing the replicate weights − has a 
10

tendency to be biased upward, computing the DAG both ways and taking the lesser 

value is a reasonable policy. 

The ARMS III Application 

The actual survey design used for the ARMS surveys varies from year to year. 

In most years, the ARMS begins with a stratified simple random sample drawn from the 

NASS list frame. This screening sample is selected in late summer to determine which 

potential farms on the list frame are in business. The sample is also often used to 

determine whether which farms are engaged in particular enterprises of interest. Those 

enterprises could be corn and hogs in one year, wheat in another, and nothing in a third. 

A subsample of the farms engaged in each enterprise of interest serves as a 

component of the ARMS III. 

Although the stratum sample sizes almost always exceed five in the screening 

survey described above, the same cannot always be said about the accompanying June 

area sample used to identify farms missing from the NASS list frame. Rather than 

using an extended DAG jackknife to handle such situations, NASS treats the larger 

area-frame land-use strata as the design strata in DAG variance estimation (the actual 

design strata for the survey are the geographically-determined substrata). When one or 

more land-use strata contain five of fewer sampled segments, the resulting DAG 

jackknife has, if anything, a tendency to overestimate variances. Since such sampled 

segments usually contain no farms, this tendency is more theoretical that real. 

In years where there are one or more enterprise samples, they are combined 

with a general subsample of the screening sample to form the list-component of the 

ARMS-III sample. The factor(s) used in weighting the composite list sample is (are) 

used in constructing the replicate weights. 

The list sample is combined with farms identified by the area-frame sample as 

not-on-the-list (NOL). Weights for this final ARMS-III sample are calibrated at the region 

level (with many large states serving as regions) to meet outside targets determined 

from other NASS surveys. These targets are treated as known constants by NASS for 
11

variance-estimation purposes. Kott (2005b) describes how to incorporate targets with 

their own variances into a DAG jackknife. 

 NASS uses a truncated version of linear calibration to create ARMS-III sampling 

weights. First, the ck are set at 1 − 1/dk, where dk is the weight before calibration. After 

an initial round of calibration, weights that would otherwise be less than 1 are set to 1, 

and weights deemed too high are set back to dk. Farms having these truncated weights 

are removed before a second round of linear calibration is conducted. The process of 

truncating too-small and too-large weights and then removing the associated farms from 

the calibration is repeated as necessary. 

 In recent years, replicate weights have been truncated at zero (to aid users 

employing software packages that cannot handle negative weights). In addition, some 

targets used in calibration have been dropped when constructing replicate weights on a 

region-by-region basis. 

Appendix B describes a method for constructing a “stratified” jackknife from 2006 

ARMS-III replicate weights. This jackknife allows the sophisticated user to conduct 

multivariate tests in certain situations where the DAG jackknife cannot be used. 

Appendix A: Effective Degrees of Freedom 

 

 For simplicity, let G = 15 and assume wk(g) ≈ (15/14)wk for k 0 S(g) (which holds in 

the limit as S becomes asymptotically large), Consequently, 

t ! t(g) = ∑ wkεk + ∑ (wk − wk(r)) εk, 

 

 

 k∈Sg k∈S(g)

 ≈ ∑ wkεk − ∑ wkεk /14, 

 k∈Sg k∈S(g)

 = (15/14)( ∑ wkεk − ∑ wkεk /15), 

 k∈Sg k∈S 

which is dominated by the first summation. This lead to the following ad hoc

approximation: 

 G 

 t ! t(g) ≈ (15/14)( ∑ wkεk − 3 ∑ wkεk /15), 

 

 k∈Sg r=1 k∈Sr
12

where the dominant term is correct, but the other term relies on wk(g) ≈ (15/14)wk. 

 From this last expression, we have the following approximation for vDAG: 

vapprox = (15/14)2 { ∑

15 ug 2

 − ( ∑

15 ug)2

/15 }, 

where ug = ∑ ε Sg

w . k k

If we assume the ug are approximately normal random variables, then following 

the logic in Kott (1994) eq. (6), we can see that vapprox has roughly the multiple of a chisquared distribution with 

 15 15 

∆ = ( ∑

 qg)2 / { ∑ qg 2

 + ∑ qgqg’ /196}. 

 g=1 g=1  g’ ... g 

degrees of freedom, where qg is a multiple of the model variance of ug. If we conjecture 

that σk 

2 ∝ fk, then qg = ∑

g

2

S w f k k . Consequently, the z statistic, z = (t − T)/√v, has 

roughly a Student’s t distribution with ∆ degrees of freedom. 

 Even if the model holds, σk 

2 ∝ fk, and wk(g) ≈ (15/14)wk, when the sample size is 

small (less than, say, 15 x 15 = 225) assuming the ur are roughly normal is tantamount 

to assuming that the εk are roughly normal or, at least, each has a fourth moment 

approximately equal to 3σk 

4

. This may be unreasonable in many applications. 

Nevertheless, assuming the z statistic has ∆ degrees of freedom is more reasonable 

than assuming it has 14 degrees of freedom. 

Appendix B: Creating a “Stratified” Jackknife for the 2006 ARMS III 

Estimating variances for a vector of coefficients in a large model can be 

problematic when a DAG jackknife has only 15 replicates. To see why, let t be a 

d-vector of estimated coefficients. Its DAG jackknife variance estimator, vDAG(t), is the 

d × d matrix computed by replacing (t − t(g))

2

 in equation (1) with (t − t(g))(t − t(g))' ) (itself 

a d × d matrix). 
13

A test statistic for the null hypothesis t = 0 is F = [ ]−t v t t 1

DJ ' ( ) d, which has 

approximately an F(d, 15 − d) distribution under the hypothesis. This statistic only 

exists when d is less than 15. In fact, when d ≥15, vDJ(t) will not be invertible. 

In some circumstances, a user will be able to construct a “stratified” jackknife 

variance estimator for a vector like t that is invertible when the DAG is not. This 

appendix sketches how. 

The 2006 ARMS III (respondent) sample can be divided into H = 20 mutually 

exclusive calibration regions or “strata.” These strata consist of 15 large states and 5 

“rump” collections of smaller states. We let Th denote that part of the 2006 sample from 

stratum h. 

For the stratified jackknife, we can create 300 (20 × 15) stratified replicate 

weights for each sample farm with the formula: 

 

  ∈  

=      

k(g) h

k(hg)

k

w when k T

w

w otherwise 

 (B1) 

For any estimated parameter vector t computed with the calibration weights, a user can 

also compute 300 stratified replicate estimates, each denoted t(hg), using the appropriate 

set of stratified replicate weights, and then estimate the variance of t with 

= =

= − − ∑ ∑

H 15

SJ (hg) (hg)

h 1 g 1

v t t t t (t) (14 /15) ( )( )'. (B2) 

Observe that if t is computed from data deriving from a subset S′ of the sample, 

and the subset only contains farms in H′ < 20 strata, then a user could repl ace H in the 

first summation of the right-hand side of equation (B2) by H′ without changing the result. 

In fact, if the data was wholly contained within a single stratum, then the stratified 

jackknife in equation (B2) would be identical to the DAG jackknife. 

When evaluating a large model involving data from a number of strata, the 

stratified jackknife variance estimator for t in equation (B2) may be invertible when that 
14

analogous DAG jackknife is not. The test statistic χ

2

 = [ ]−t v t t 1

SJ ' ( ) will then be 

asymptotically chi-squared with 14 degrees of freedom under the null hypothesis that 

t = 0. The alternative ad hoc statistic F = χ

2

 /d will be roughly F(d, 14H′ + 1 − d ) (note: 

replace H′ with H when the sample contains farms from every stratum). This statistic is 

ad hoc because, among other things, the strata are not necessarily of nearly equal size. 

It is nonetheless preferable to use F in place of χ

2

 when 14H′ + 1 − d is not large. 

When the sample contains data from all 20 strata, some users may find 

computing the 300 replicate estimates necessary for determining vSJ(t) in equation (B2) 

burdensome. A simplified version of the stratified jackknife can be constructed that 

collapses the 20 strata into a more manageable number of variance strata. Stratified 

replicate weights, replicate estimates, and test statistics can be computed accordingly. 

For example, one could collapse the 15 state strata into five mutually exclusive variance 

strata, each containing three states, while placing the states in the five rump strata into 

a sixth variance strata. Each farm would then have 90 (6 × 15) replicate weights 

computed using equation (B1) with h redefined as a variance stratum and Th as the 

subsample containing farms in h. The new replicate weights could then be used to 

compute 90 replicates of t. Armed with these replicate estimates, one could them 

employ equation (B2) to estimate a variance matrix with h redefined as above and H 

replaced by 6. 

Finally, note that in the spirit of the “better” DAG jackknife of equation (4), we 

could replace the stratified jackknife in equation (B2) with 

∩

= = ∩

∑

= − − ∑ ∑

∑ −

g h

h

2

H 15 S T k

SJ 2 (hg) (hg)

h 1 g 1 S T k k(hg)

w

(t) ( )( )'.

(w w )

v t t t t

 

 
15

References 

Kott, P.S. (1994). "Hypothesis Testing of Linear Regression Coefficients with Survey 

Data," Survey Methodology, 159-164. 

Kott, P.S. (1998). Using the Delete-a-Group Jackknife Variance Estimator in NASS 

Surveys, NASS Research Report 98-01 (revised 2001). Reproduced at 

http://www.nass.usda.gov/research/reports/RRGJ7.pdf. 

Kott, P.S. (2001). “The Delete-a-Group Jackknife, Journal of Official Statistics, 

 521-526. 

Kott, P.S. (2005a). “Randomization-Assisted Model-Based Survey Sampling,” Journal 

 of Statistical Planning and Inference, 263-277. 

Kott, P.S. (2005b). “Using the Delete-a-Group Jackknife in an Economic Survey of US 

Farms,“ Presented at the 55th Session of the International Statistical Institute, 

Sydney, Australia (CD-Rom). 

Kott, P.S. (2006a). “Delete-a-Group Variance Estimation for the General Regression 

 Estimator under Poisson Sampling,” Journal of Official Statistics, 759-767. 

Kott, P.S. (2006b). “Using Calibration Weighting to Adjust for Nonresponse and 

 Coverage Errors,” Survey Methodology, 133-142.