RTI International is a registered trademark and a trade name of Research Triangle Institute. www.rti.org

Contrafactual Degrees of Freedom

Phillip S. Kott

pkott@rti.org 

1 FCSM 2022

Fcontrafactual = 

∑h=1

HH ∑jj=1

nnh ∑kk∈sshjj wwkk

2

2

∑h=1

HH ∑jj=1

nnh ∑kk∈sshjj

wwkk

2

2

+ 1

nnh−1

2 ∑jj=1

nnh ∑kk∈sshjj

wwkk

2

2

−∑jj=1

nnh ∑kk∈sshjj

wwkk

2

2
Introduction 

The nominal degrees of freedom of a variance estimator under 

design-based survey sampling theory

The Contrafactual degrees of freedom 

called the effective degrees of freedom in 

Kott (Survey Methodology, 1994) and

Korn and Graubard (Analysis of Health Surveys, 1999) 

Variants

A regression coefficient

An empirical example

Some comments

2 FCSM 2022
Background

When constructing coverage (confidence) intervals for

an estimated mean or regression coefficient (m), design-based

practice is to use a stratified sandwich variance estimator (vv

�)

with (nominal) degrees of freedom equal to 

The number of primary sampling units minus the number of 

strata (minus the number of regressors plus 1).

Replication produces the same value. 

This practice has little theoretical justification except under a 

very unlikely outcome model. 

3 FCSM 2022

t

ˆ
Background (2)

From a model-based point of view, the stratified 

sandwich variance estimator allows

Nonnormal and heteroscedastic errors, and

almost arbitrary clustering within PSUs.

When estimating means, using nominal degrees of 

freedom implicitly assumes:

PSU-level aggregates are normal and identically 

distributed, and

Strata are nuisances. 

4 FCSM 2022
The Contrafactual Degrees of Freedom 

5 FCSM 2022

assumes (contra factually) that the elements are normal and iid,

and that the relative variance of vv

� is 2/its degrees of freedom 

(as in a chi-squared distribution – a Satterthwaite approximation). 

So

Fcontrafactual = 

∑h=1

HH ∑jj=1

nnh ∑kk∈sshjj wwkk

2

2

∑h=1

HH ∑jj=1

nnh ∑kk∈sshjj

wwkk

2

2

+ 1

nnh−1

2 ∑jj=1

nnh ∑kk∈sshjj

wwkk

2

2

−∑jj=1

nnh ∑kk∈sshjj

wwkk

2

2 ,

where wk is the weight of element k, and

shj is the (respondent) sample in PSU j in stratum h (of H).

There are nh PSUs in stratum h. 
Variants

6 FCSM 2022

The weight can be a design weight or a calibrated weight.

Each weight can be multiplied by a domain-inclusion indicator 

for a domain mean.

Element-level weights can be replaced by PSU-level sums of

weights assuming complete correlation within each PSU. 

Each weight can be multiplied by an element-level standard error. 
Alternative Formula and Another 

7 FCSM 2022 JSM 2016

Let wwhjj

(2) = ∑kk∈sshjj wwkk

2, then

Fcccccccc. =

∑h=1

HH ∑jj=1

nnh wwhjj(2) 2

∑h=1

HH ∑jj=1

nnh wwhjj(2)2

+ 1

h − 1 2 ∑jj=1

nnh wwhjj(2) 2

− ∑jj=1

nnh wwhjj(2)2

Contrafactual Degrees of Freedom for a Domain Mean 

treating the wwhjj

(2) as 0 or 1 (i.e., the PSU is empty or not empty)

number of nonempty PSUs in h

↓

Fcccccccccccccccccccccccccc =

∑h=1

HH ddh

1+

∑h=1

HH dd ⁄ h(ddh−1) nnh−1

2

∑h=1

HH ddh
A Regression Coefficient 

8 JSM 2016

Define z2k* = z2k − z1k (∑S wjfj'z1j z1j

T)-1 ∑S wj

fj'z1j z2j . 

FCSM 2022

,

 b2 in b = �

bb1

bb2

� the solution to

 ( ) T

kk k

k

k

S

w yf

∈

  − ∑   z z b = 0 

Replace wk with wk z2k*.

The difference between domain means can be estimated as a 

regression coefficient. 
An Empirical Example

9 FCSM 2022 JSM 2016

In Kott (2020), I reorganized the MU281 population of 281 

Swedish administrative municipalities (from the big Yellow 

Book) into 20 strata containing 108 clusters. 

Using simple random sampling with replacement, 

I drew 2500 fresh simulated samples containing four clusters 

per stratum.

I looked at estimated variances (vv�) for estimates (denoted mm�)

among municipalities where at least 50% of the municipal 

council in 1982 were Social Democrats, that is, Group 1: 

A. The mean population size in 1985. 

B. The fraction of municipalities in 1985 with tax/person 

rates at or above 8,000 Kronor. 
Results for the Group-1 Means 

My computations of the contrafactual degrees of freedom are 

(implied t-value at two-sided 95% confidence level)

Elements independent PSUs independent

Mean 20.6 (2.08) 14.2 (2.14)

Min 14.4 (2.14) 8.1 (2.30)

Max 27.3 (2.05) 20.2 (2.08)

Nominal: 60 (2.00) 

Empirical F = 2/emp. relVar(vv�): A = 27.4 (2.10), B = 8.2 (2.37) 

↑ ↑

(where the cumulative distribution of | mm� −mm

vv� | is at 95%)

10 FCSM 2022
Results for Differences in Group Means 

My computations of the contrafactual degrees of freedom are 

(implied t-value at two-sided 95% confidence level)

Elements independent PSUs independent

Mean 31.0 (2.04) 16.0 (2.14)

Min 24.3 (2.06) 8.3 (2.29)

Max 36.5 (2.03) 23.3 (2.07)

Nominal: 59 (2.00) 

Empirical F = 2/emp. relVar(vv�): A = 34.0 (2.20), B = 14.5 (2.21) 

↑ ↑

(where the cumulative distribution of | mm� −mm

vv� | is at 95%)

11 FCSM 2022
Some Comments 

Although better than the nominal, the contrafactual degrees of 

freedom is not perfect (hence not “effective”) because:

The variance estimators aren’t chi-squared.

The estimators and their estimated variances are asymptotically 

unbiased, but the world is finite. 

The data may not be normally distributed.

Weights may not be ignorable. 

Strata may not be ignorable.

Within-cluster correlations may be complex. 

Selection of clusters was with replacement. 

12 FCSM 2022
Some Comments 

When estimating a regression model, it may be reasonable to 

assume a model. 

In particular a model, where

Strata are ignorable (which is easier when there are a lot more 

PSUs than strata). 

Within-cluster correlations only occur at a level below the PSU 

(e.g., within households or blocks as opposed to counties).

When testing groups of coefficients, consider using a series of 

Bonferroni-adjusted t tests in place of an adjusted Wald F test. 

13 FCSM 2022
References

Korn E. and Graubard, B. (1999). Analysis of Health Surveys. 

New York: John Wiley (especially Chapter 5)

Kott, P. (2020). The degrees of freedom of a variance estimator 

in a probability sample. Research Triangle Park, NC: RTI 

Press

Kott, P. (1994). Hypothesis testing of linear regression 

coefficients with survey data. Survey Methodology, 159–164 

14 FCSM 2022