Evaluation and Selection of Models 

for Attrition Nonresponse Adjustment 

Eric V. Slud1,2 and Leroy Bailey1

1Census Bureau, SRD, and 2Univ. of Maryland College Park

Eric.V.Slud@census.gov, Leroy.Bailey@census.gov

Abstract. The setting of this paper is a longitudinal survey like SIPP, with successive “waves” of data collection from 

sampled individuals,in which nonresponse attrition occurs and is treated by weighting adjustment, either through 

adjustment cells or a model like logistic regression in terms of auxiliary covariates. Following Bailey (2004) and 

Slud and Bailey (2006), we measure the discrepancy in estimated initial-wave (“Wave 1”) attribute totals between 

the survey-weighted estimator in the frst wave and for the corresponding weight-adjusted estimator for the same 

Wave-1 item total based on later-wave respondents. The present research defnes a composite metric of quality of a 

model used for nonresponse adjustment of a longitudinal survey. The metric combines the magnitudes of estimated 

between-wave adjustment biases based on subsets of the sample, relative to the estimated total, for various survey 

items. The maximum of the adjustment biases for estimated totals of a survey item are calculated from the frst 

j sample units, as j ranges from 1 to the size of the entire (Wave-1) sample, after each of a number of random reorderings either of the whole sample or of the units within specifed cells (which are then also randomly re-ordered); 

and the average over re-orderings of the maximal adjustment bias is divided by the estimated wave-1 attribute total 

to give the metric value. Confdence bands for the metric are estimated, and the metric is applied to judge the 

quality of and to select among a collection of logistic-regression models for nonresponse adjustment in SIPP 96. 

Keywords: Adjustment cells, Logistic regression, Model Selection, Nonresponse Weighting Adjustment, Random 

Re-ordering, Subdomains. 

This report is released to inform interested parties of ongoing research and to encourage discussion of work in progress.

The views expressed on statistical and methodological issues are those of the authors and not necessarily those of the

Census Bureau.

1 Introduction 

Slud and Bailey (2006) studied the estimated biases between Wave 1 totals of various Survey of Income and Program 

Participation (SIPP) 1996 cross-sectional survey items and their estimates based on nonresponse-adjusted totals 

of the same Wave 1 items using only response data from a later Wave (4 or 12). The nonresponse adjustments 

studied were derived either by an adjustment cell method (using the 149 standard SIPP adjustment cells described 

by Tupek 2002) or a parsimonious logistic regression model for the later-wave response probabilities. One of the 

tentative conclusions of that work was that the relative and standardized magnitudes of the estimated biases varied 

considerably and somewhat erratically from one adjustment model to another. Many competing adjustment models 

could be defned, depending on which attribute variables were used in constructing adjustment cells or as logistic 

regression predictors. Slud and Bailey (2006) noted that using Poverty as a logistic regression predictor did have 

the artifcial eect, akin to raking, of making the sample-wide estimated Wave 1 total of Poverty particularly small. 

However, since that eect directly stems from the sample-wide estimating equation defning the logistic regression 

coeÿcients, it was conjectured that this artifciality could be removed by considering estimated Wave 1 bias within 

a number of dierent subdomains. 

In Slud and Bailey (2006) the possibility was also considered of customizing the adjustment model in order to remove 

between-wave adjustment biases as far as possible. This suggests creating a composite metric defned by combining 

the magnitudes of estimated between-wave adjustment biases for various SIPP items. The considerations of the 

previous paragraph suggest also including in the metric the estimated biases on multiple subdomains of the SIPP 

target population. We present below two constructions of such a composite metric, and we exhibit its values for 

the models studied in Slud and Bailey (2006). The ultimate objective of this research is then to use the metrics 
defned to choose an optimal model – which might be either of adjustment-cell or logistic-regression form, although 

we restrict attention in this paper to the latter – based on SIPP 96 data. 

2 Background & Previous Work 

To measure the quality of nonresponse adjustment in a longitudinal survey, one would certainly try to evaluate the 

biases of adjustments using external data on the sample frame and the same variables whenever such external data 

are available. But that will seldom be the case. 

There seems not to have been much published methodological work on how to measure the biases of adjustment, from 

the internal evidence of a longitudinal survey. One important paper on this topic is that of Dufour et al. (2001). That 

paper starts from the perspective that large longitudinal national studies will almost always be raked or calibrated 

to population totals derived from a past or current census or survey of high quality. The paper specifcally considers 

calibration, and proposes to measure magnitudes of adjustment through a metric the authors defne for tracking 

weight change through several stages of a weight-adjusted longitudinal study. By conducting a large simulation 

study within which they randomly subsample from a large longitudinal survey dataset (SLID, the Canadian Survey 

of Labor and Income Dynamics), the authors compare the weight-changes experienced from nonresponse weighting 

adjustment done by two main model-based adjustment approaches (Logistic regression with stepwise variable selection 

and Response Homogeneity Groups — what we call below the adjustment-cell method — with cells defned using a 

CHAID-based Segmentation Model). Calibration optimally adjusts weights according to a model (of adjustment-cell 

or logistic-regression type), in order that estimated population totals in designated subsets perfectly match the totals 

from an external study. Then the estimated adjustment biases (as in Bailey 2004 and Slud and Bailey 2006) for 

population totals of other early-stage variables could be used to judge the overall success of the modelling approach 

used in adjustment. This could have been, but was not, done in Dufour et al. 2001, nor were eects of weighting 

adjustment on population subdomains examined. 

By contrast, we propose to adjust simultaneously, using models for later-stage response (also of adjustment-cell or 

logistic-regression type), the later-stage estimates of totals of population and other survey variables. We will then 

measure the biases of later-stage subtotals for population and the early-stage survey variables, for an array of dierent 

population subdomains including the cells to which calibration would have been done. The metric for eectiveness 

will combine the magnitudes of relative biases of specifc survey variables over an interesting array of population 

subdomains. The results will then be assembled for a set of dierent survey variables along with population count 

into a weighted loss function. Unlike the calibration-frst approach, this method provides the possibility of giving 

heavy but not overwhelming weight to population adjustment biases as opposed to biases in totals of other survey 

variables. 

Although raking or calibration to updated-census population totals in defned cells will ultimately be done in practice 

whenever a weighting adjustment is applied to a large national longitudinal study, it may not be best to do all 

comparisons of adjustment methods with respect to bias in the presence of such raking/calibration adjustments, 

which often function as a black-box method superimposed on otherwise simple and understandable adjustment 

models. Therefore, in this paper as in Bailey (2004) and Slud and Bailey (2006), we restrict attention to weighting 

adjustments based on SIPP weights before adjustment-cell raking. 

There has been a great deal of work on calibrating, reconciling, and benchmarking time series of diering reporting 

periods and accuracies, c.f. Dagum and Cholette (2006). But our literature search has yielded few papers (Dufour 

et al. 2001 in particular) explicitly recognizing survey and weighting-calibration aspects in this regard. 

There has also been previous theoretical work on the large-sample behavior of model-based nonresponse weight 

adjustment methods. One recent example is Kim and Kim (2007); and these same authors, in an unpublished 2007 

preprint, have considered the problem of choosing between alternative parametric models for survey nonresponse 

using the same data on which the estimated parameters will be applied to adjust the weights. 

The goal of this research is to devise metrics to aid in the comparison of dierent model-based methods of adjustment 

for nonresponse due to attrition, which will provide a basis for choosing among adjustment methods. Several 

earlier comparative investigations related to adjustment methods have been conducted, even within the SIPP survey 

structure, but they seem not to have resulted in clear advantage for any adjustment method over others. (See Rizzo 

et al. 1994 for example.) 
3 Formal Development: Metrics and Bounds 

Let Sdenote the sample of n= |S|persons drawn from sampling frame U, with known single inclusion 

probabilities {πi}i2U , and responding in Wave 1. For a series of cross-sectional survey measurements indexed by 

k= 1,. .. K, such as the K= 11 items studied by Bailey (2004) and Slud and Bailey (2006), denote by y

(k)

i

the Wave 1 item values and xia vector of auxiliary variable values for all i∈ U. Let ridenote individual 

response indicators (observed for all i∈ S) in a specifed later Wave of the same survey, and let pi

= P(ri= 1 | S) 

denote the (unknown) conditional probabilities of later-wave resp ˆ onse. Let pˆi

= g(xi, θ) denote estimators of these 

unknown probabilities derived (using a known function g) from a parametric model using auxiliary data xi

, within 

which parameter-estimators θ

ˆ are obtained via estimating equations (Kim and Kim, 2007). For any p

P 

opulation 

attribute zi

, i∈ U, the frame-population total is denoted tz= i2U zi, and the corresponding Horvitz-Thompson 

estimator is tˆz

= P zi/πi.

i2S 

For each survey item y

(k)

, i∈ U,with respect to the specifc strategy of adjustment embodied in the estimated i

response probabilities pˆi, and for each domainsubset D ⊂ Uof the population, defne the estimated nonresponse 

bias 

X   ri Bˆ

k(D) = − 1 y

(k)

/ πi(1) i

pˆi

i2D\S 

In Slud and Bailey (2006) and earlier papers of Bailey, the domain Dwas all of U, and the quantity Bˆ

k(U) was 

interpreted as the dierence between an adjusted estimator of t using only the data (y

(k)

, xi (k) , i∈ S) and the y i

ordinary Horvitz-Thompson estimator tˆ (k) , and was regarded as an estimator of attrition nonresponse bias due to y

the method of adjustment. 

3.1 Relative Subdomain Bias 

We now propose a measure of the typical relative bias in estimating item totals over subdomains. The idea is to 

consider the largest value of absolute relative bias Bˆ

k(D)/tˆ (k) over a collection of dierent subsets D ⊂ U. Suppose y

that we re-order the elements of U, inducing a re-ordering τ= (τ (1), τ (2),. .., τ (n)) of the nelements of S. 

The largest absolute bias in survey variable kover consecutively τ -indexed subdomains of Sis 

max |Bˆ

k({τ (i) : a

≤i≤b})| ≤2 ·max |Bˆ

k({τ (1),

.. ., τ (a)})|

1abn

1an

To measure the overall relative bias in estimating item ktotals over subdomains, we defne 

  mk

= Eτmax |Bˆ

k({τ (1),

. .., τ (a)})| / tˆ

y(k) (2)

1an

where the expectation is taken, for a fxed sample, over random permutations τchosen equiprobably from the n! 

permutations of the elements of S. The quantity mk

is smaller than the largest relative bias |Bˆ

k(D)|/tˆ (k) over all y

subsets C ⊂ U— which is too large an estimate of error, and also too expensive to calculate — but does represent 

the typical magnitude of the worst relative bias in a random scanning order of the sampled population. 

In settings where the relative estimated bias 

X   (k) . ri

y

δ

(k) ≡

Bˆ

k(U)/tˆ

y(k) =

ˆ

− 1 itˆ

y(k) (3) 

pi

πi

i2S 

is large, we will see below that mkand its estimator mˆ kare not much dierent from |δ

(k)

|. However, if 

δ

(k)

is small — which may be true for artifcial reasons if the model used to defne pˆiprominently features the 

(k)

attributes {y , j∈S}), then mˆ kwill often be meaningfully large, refecting the fact that the model-ftting j

does not simultaneously adjust for weighted y

(k)

totals over arbitrary subsets of the sample. This is an attempt to i

penalize models which directly adjust the population-wide total of an attribute. 

The bias measure mk

cannot be calculated directly from the sample data, but can be estimated by evaluating 

its defning expectation over random permutations τusing a Monte Carlo simulation strategy. For each of a set 

1,. .. ,Rof indices cdenoting Monte Carlo replicates, we defne independent random permutations τc

of the 
indices i

∈ S. For each b, (τc(j),1 ≤j≤n) is equiprobably chosen from the n! possible re-orderings of S, 

which is easily implemented in a Monte Carlo simulation by defning a sample of independent Uniform(0, 1) variates 

Vc

= (Vci, i∈ S) and to let τc(j) be the sequence of indices iof the Vciobservations written in increasing order. 

Then the estimator mˆ k

defned in (2) is 

R

.

= max 

1 X |Bˆ mˆ k({τc(1),

. .., τc(j)})|tˆ k

y(k)

R

1jn

c=1





 X 



 (4) 

R

= max 

R tˆ

y(k)

X

(k)

1 I[Vcix] (

ri

pˆi

− 1) yi

0<x1 πi

c=1 i2S 

As a metric for nonresponse bias combined over all survey variables indexed by k= 1,. .. ,K, we propose a simple 

weighted average and estimator 

K K

 . 

M= wkmk= wk·sup X X |Bˆ

k(D)|

tˆ (k) (5) y

DˆS k=1 k=1

X

K

M

ˆ = wk

mˆ k(6) 

k=1

where w = {wk}

K

is a fxed vector of positive weights summing to 1. If all survey variables are considered k=1

equally important then, as below, we would use wk= 1/K. 

The quality of estimation of mk

in terms of mˆ k, and relationships between these and |δ

(k)

|, are addressed 

in Section 3.3 below. We turn frst to the modifcation of (2) and (4) to allow expected and estimated maximum 

absolute relative discrepancies with respect only to those random re-orderings which preserve specifc cells of the

population, such as the cells to which population totals would be raked or calibrated. 

3.2 Metric for Subdomain Bias with Distinguished Cells 

Most random permutations of the sample completely shatter any meaningful sample subdomains. Yet the idea behind 

raking or calibration is precisely that certain estimated subdomain totals — usually, the estimated population totals 

over the cells Aj

of a specifed geographic-demographic partition U= ∪j

J

=1 Aj

of the frame population — must be 

constrained equal to those of a current (updated) census. For that reason, it makes sense to measure bias estimates 

Bˆ

k(Aj ) over these cells, where we assume from now on that a partition A

of Uinto cells Aj , j= 1,. .. , J, 

has been fxed. The idea is to modify (2) so that the allowed permutations must retain the consecutive indexing of 

elements in each cell Aj . 

One approach would be to aggregate these biases into an relative accumulated absolute bias

X

J

= ω

. (k)

|Bˆ

k(Aj )|

tˆ

j

Cum

m (k) (7) k

y

j=1

(k) where ωj

(S) are a set of cell- and item-specifc weights. A related approach would be to replace each term

|Bˆ

k(Aj )| in (7) by the expectation of maxl2Aj

|Bˆ

k({i

∈Aj: τj (i) ≤τj (l)}|with respect to a random permutation 

τj

of the elements of S ∩Aj. However, in either of these two forms, the relative accumulated absolute bias is likely 

too conservative to be very useful, because it aggregates across cells the worst sub-cell biases, as though all domain 

totals in all cells could be simultaneously badly biased. 

A less extreme modifcation of (2), which we adopt below, would combine cellwise biases within a partition Aso 

that the permutations τ— now denoted σ— leave the cells Aj

∩ Sinvariant. To explain this invariance, we 

assume that the sample is indexed in such a way that the nj

≡ |Aj ∩S|sampled elements in the j’th cell Ajappear 

consecutively in the enumerated sample S. The invariance of the cells under σmeans that for all 1 ≤j ≤J, the 

elements {σ(i) : i∈Aj

∩ S}also form a consecutively indexed block in the indexed sample S. Now the allowed 

QJ

random permutations σof the sample elements are chosen equiprobably from the J!

j=1 nj ! permutations which 

frst permute the Jcomplete blocks Aj

∩ Sof njelements each and then permute the elements within the 
re-ordered blocks. Finally, we defne the expectation over σ

of the maximum absolute cumulative weighted sum of 

cellwise biases relative to tˆ (k) , as follows: y

 

m 

k

≡Eσmax |Bˆ

k({σ(1),

.. ., σ(q)})| / tˆ (k) = (8) y

1qn

 X

j−1

Eσ

max 

1jJ, b2Aj







ωσ(l) Bˆ ˆ k(Aσ(l)) + ωσ(j) Bk({τ (a) : a

∈Aσ(j), a≤b}) 

 



 

l=1

(k) Here ωj

= ωj(S) are again a set of cell- and item-specifc weights (usually taken to be 1) which may depend on 

PJ

(k)

the sample, but which satisfy the relation J

−1 ω = 1. j=1 j

An estimator for the modifed quantity (8) can be implemented in terms of a collection of random batches Vcof n

independent Uniform(0, 1) random variates, along with independent batches Uc

of Jindependent Uniform(0, 1) 

variates, for 1 ≤c≤R. For each fxed batch-index c, we use the ordering of the variates Uc1, Uc2,

. .., UcJto 

determine the c’th random ordering of the blocks Aj

∩ S,1 ≤j ≤J. Next, the c’th reordering of the elements i

within the re-ordered block Aj

∩ S, is given by the order of the variates (Vci, i∈Aj∩ S). For each q= 1,. .. ,n

indexing an element of the sample S, denote by j(q) the index jfor which q∈Aj . With these notations in 

mind, we express the estimator for (8) as 





 

R

1qn

c=1 l:Uc,l<Uc,j(q)

X X Bˆ ≡

(R tˆ k(Al) y

(k) mˆ

 

k

(k) )

−1 max ωl







(k) Bˆ

k({i

: i∈Aj(q), Vci≤Vcq})

j(q)

X

R

+ ω (9)

or equivalently, 





 







(k)

riy  X i

k

≡(R tˆ (k) )

−1 mˆ y max ( − 1) (10)

1 pˆi

πi

qn

c=1 i: Uc,j(i)Uc,j(q), VciVcq

We next show how to place confdence bounds on the dierences bet  ween the quantities mk, m and their estimates k

m  ˆ k, mˆ and on the dierences between these quantities and |δ

(k)

|. k

3.3 Confdence Intervals and Bounds for mk

In the frst part of this Section, we provide a theoretical development of confdence intervals by bounding ˆmk− mk

and mk− |δ

(k)

| probabilistically. However, all of these quantities are functions of the sampled survey data, and the 

probability statements made at this stage concern only the chance element introduced by the random permutations 

τ,τc

used in defning (2) and (4). At the end of the Section, we interpret the meaning of sample-based metricestimators mˆ k

for the survey population and adjustment model. 

We begin with the simplest and clearest confdence statement. Since mˆ k

is calculated as the empirical average over 

quantities calculated from a series of Rrandom permutations of the sample, its sampling variability due to those 

permutations can be assessed by empirical standard errors 

R

yi













h i1/2

(k)

1 1 X ri

−tˆ (k) mˆ k)

2

se y( ˆmk) ( = max I[Vcix] ( − 1)

|tˆ (k) |

R(R − 1)

y c=1

0<x1 pˆi

πi

Thus, with approximate 99% confdence when Ris large, 

|mk

−mˆ k| ≤2.576 ·se( ˆmk) (11) 

and similar confdence statements with respect to the randomness of the permutations σc

can be given bounding 

m − mˆ

 

k k. 

The dierence between the metric value mk

and the overall relative bias |δ

(k)

|is due to the fuctuations with 

varying xof the quantities 

(k)

riy X i

Zk(x) = I[Vix] ( − 1) 

pi πi

i2S 
being maximized in (4), where Vi

= Vcidenote independent identically distributed Uniform(0, 1) variates. If these 

quantities were replaced by their expectations (i.e., if I[Vix] were replaced by x), then the expression (4) would 

become |δ

(k)

|. Thus, the discrepancy mˆ k− |δ

(k)

| can be bounded by the maximum absolute value of the random 

weighted empirical process indexed by a continuous argument x∈[0, 1], 

1 X   (k)

(

ri

pˆi

i − 1) y

βk(x) = √ I[V (12) ix] −x

n

πi

i2S 

(k)

n o

conditionally with all sample data i,ri

, xi,(y ,1 ≤k≤K) : i∈ Sfxed and with only the variates i

Vi, i∈ S, regarded as random. The process βk(·) has mean 0 , and according to a slight extension of the Donsker 

Theorem (Pollard 1980), which can be proved as Corollary of the Martingale Central limit Theorem (Hall and Heyde 

1980), has approximate distribution for large nthe same as 

h 1 (k)

ri(y )

2 i1/2 X i

p

γ

(k) W

(x) ≡W − 1) (x) (13) 2(

π

2

i n

pˆi

i2S 

as a random continuous function of x∈[0, 1], where W(x) denotes a tied-down Wiener process or Gaussian 

process with mean 0 and 

Cov(W

(v),W(u)) = min(v,u) −v·u

The scaling constants governing the amplitude of fuctuations of βk(·), 

1 X ri

(k)

γ

(k) = − 1)2

(y )

2

/π2

i

( (14) i

npˆi

i2S 

can readily be computed from the sample data, and under general assumptions remain bounded for large n. 

By defnition of mk

and the remark that (4) would become |δ

(k)

| if I[Vix] were replaced by x, 







mk

− |δ

(k)

|











 





  1

 X 

(k)

(I[Vix] − x) ( ri

pˆi

− 1) yi

πi

≤EV max 

tˆ (k) 0<x1

y i2S 

p p

tˆ tˆ

since 1.2286 is the expectation of sup

 n γ(k) n γ(k)

EV( max |βk(x)| ≈1.2286

0<x1

= (15)

y(k) y(k)

x2[0,1] |W

(x)| which arises in calculating percentage points of the one-sample 

Kolmogorov-Smirno statistic, readily calculated using the density of this random variable given by Kolmogorov and 

reproduced by Feller (1948). 

For specifc items k, we fnd when nis large that for moderate numbers Rof random permutations, the dierence 

   mˆ (or k

− mk mˆ −m ) is generally very small compared to mk(respectively m ). Then, by calculating the k k k

right-hand side of (15), we fnd roughly how small the value mˆ kmust be in order that the sample data be compatible 

with a zero relative bias δ

(k)

. The objective of this kind of analysis is frst of all to fag as ‘inadequately adjusted’ 

those items for which model-based attrition nonresponse adjustment has resulted in estimated metric values mˆ k

 greater than the sum of the right-hand sides of (11) and of (15). Since we will fnd generally that the values of mˆ k

 and mˆ k

are roughly the same, we will use the same threshold for metric values mˆ k. 

Next, we compare the estimated metric values m  ˆ kand mˆ k, individually or in their aggregated form (6), across 

dierent adjustment models with a view to choosing a ‘best’ model in a specifc survey application. 

4 Adjustment Metric Values in SIPP 96 

For the case of SIPP 96, with K= 11 cross-sectional items, response probabilities pˆiwere estimated by the specifc 

adjustment-cell and logistic-regression models mentioned above, all as described in detail by Slud and Bailey (2006). 

Briefy, the cross-sectional survey items y

(k)

studied are: indicators that the individual lives in a Household which i
Table 1: Logistic regression models used to adjust Wave 4 or Wave 12 nonresponse in SIPP 96. Df is the number 

of independent coeÿcients in each model, including Intercept, and Dev the deviance for the 94444-record SIPP 96 

sample data. 

Model Df Variables Dev 

A 7 

8 

Wnotsp Renter College RefPer 

Black Renter*College Black*College 

76558.0 

B same as A, plus Pov 76544.6 

C 13 same as B, plus Foodst Mdcd 

Heins UnEmp Div 

76299.3 

D 13 same as B, minus Black*College 

plus Mdcd Heins UnEmp Pov*Heins 

Mdcd*Heins Heins*College 

76242.4 

E 17 same as D, plus hisp + Famtyp 76017.1 

F 18 same as C, plus Afdc SocSec Emp Mar 76279.9 

receives (i) Food Stamps (Foodst), or (ii) Aid to Families with Dependent Children (AFDC); or indicators that the 

individual receives (iii) Medicaid (Mdcd), or (iv) Social Security (SocSec); and indicators that the individual (v) 

has health insurance (Heins), (vi) is in poverty (Pov), (vii) is employed (Emp), (viii) is unemployed (UnEmp), (ix) 

is not in the labor force (NILF), (x) is married (MAR), or (xi) is divorced (DIV). 

In this data example, nonresponse is adjusted in one of two ways: either using an adjustment-cell model based on 

149 standard cells (Tupek 2002) defned in terms of variables including race, hispanicity, and family-type; or using 

one of a series of logistic regression models A–F summarized in Table 1. (Of these models Model A and B were the 

ones used in Slud and Bailey 2006.) The models C–E were selected to have progressively better ft, using an indicator 

of Wave 4 response as response-variable within the 94444 SIPP Wave-1 sample records with positive base-weights. 

The variables used in these regression models include race, hispanic origin, Renter versus Owner of housing unit,

indicator that individual is the Household Reference Person, indicator of College education, a 4-category variable of 

Family type, plus some or all of the 11 SIPP survey items listed above. 

The method followed in this data analysis, as described and justifed in the previous Section, is based on searching for 

metric values m  ˆ k, mˆ which are large compared to the bounds obtained by adding the right-hand sides of (11) and k

(15). This contrasts with the approach of Slud and Bailey (2006) which, in the present notation compared estimated 

population-wide adjustment biases δ

(k) with their design-based standard errors as found by a Balanced Repeated 

Replication method. A summary of the results of Slud and Bailey (2006) in the present notation is given in Tables 6 

and 7 of the Appendix. Despite the dierences in method, the two sets of results from the two dierent methods

seem quite consistent. 

Calculations of mˆ k

have been made with R= 100 random-permutation Monte Carlo replications, with the results 

for Model B presented in Table 2 below. (Because n=94444 is so large, the between-replication dierences are small 

and this choice of Ris ample.) The fnal columns of Table 2 respectively display the bounds b4,k, b12,k

on the righthand sides of (15) (which turn out to be virtually identical for the adjustment-cell and logistic-regression adjustment 

methods) for adjustments of Wave 4 and 12 nonresponse. It also turns out that for all items and combinations 

4C, 4L, 12C, and 12L, the bounds on the right-hand side of (11) are much smaller, ranging from 1–5% of the 

corresponding bounds (15). The analogous Table with logistic models A and D and F, also calculated with R= 100 

iterations, are displayed as Table 3. However, the columns of bounds b4,k, b12,k

are included in the latter Table 

only for model D, because the bounds for the other models are virtually identical with these, and again the bounds 

from (11) are only a few percent of the bounds (15). 

Inspection of Tables 2 and 3 reveals that the metric mˆ k

with very few exceptions in Wave 12 clearly exceeds the 

corresponding bounds bk

for the adjustment-cell model and all of the logistic regression models A and B. One notable 

exception is Pov, where as seen by Slud and Bailey (2006), model B includes Pov as a predictor and does adjust 

eectively both in Waves 4 and 12. Similarly, we see that Model D which includes variables Pov, Mdcd, Heins, and 

UnEmp as predictors, does a particularly good job of adjusting the totals of these same variables as measured by the 

metric ˆmk. Indeed, the most striking preliminary conclusion from examining the tables of metric values under these 

various logistic regression models is that including a variable as a predictor generally results in very good adjustment 
Table 2: Quantities mˆ k

in (4) estimated from SIPP96 data, for later-wave nonresponse adjustment either to wave 4 

or 12, and by either the Adjustment-Cell (C) or Logistic-Regression (L) method (Model B) and based on R= 100 

replications. The last two columns are the bounds in (15), with α= .01. 

Item 4C mˆ

4L mˆ12C mˆ12L m b ˆ 4,k

b12,k

Foodst .0052 .0186 .0442 .0130 .0056 .0123 

AFDC .0067 .0248 .1040 .0350 .0078 .0173 

Mdcd .0066 .0279 .0163 .0426 .0053 .0119 

SocSec .0191 .0116 .1118 .1038 .0041 .0086 

Heins .0085 .0065 .0197 .0133 .0019 .0040 

Pov .0187 .0033 .0372 .0091 .0047 .0097 

Emp .0016 .0017 .0082 .0122 .0020 .0041 

UnEmp .0534 .0594 .1176 .1280 .0131 .0250 

NILF .0032 .0034 .0333 .0462 .0033 .0069 

MAR .0111 .0018 .0508 .0226 .0025 .0051 

DIV .0124 .0201 .0235 .0390 .0067 .0133 

Table 3: Quantities mˆ k

estimated from SIPP96 data based on R= 100 random permutations, for wave 4 or 12 

nonresponse adjustment by logistic regression model A (frst two columns) or model D (next two columns). The last 

two columns are the bounds b4,k, b12,k

from (15) using model D. 

Item 4,A mˆ

12,A mˆ4,D mˆ12,D mˆ4,F mˆ12,F m b ˆD

4,k

b

D

12,k

Foodst 

AFDC 

Mdcd 

SocSec 

Heins 

Pov 

Emp 

UnEmp 

NILF 

MAR 

DIV 

.0120 

.0175 

.0219 

.0117 

.0076 

.0123 

.0021 

.0626 

.0026 

.0023 

.0201 

.0086 

.0446 

.0346 

.1040 

.0148 

.0127 

.0116 

.1322 

.0447 

.0236 

.0390 

.0076 

.0067 

.0035 

.0125 

.0013 

.0032 

.0015 

.0095 

.0029 

.0018 

.0168 

.0110 

.0624 

.0078 

.1066 

.0027 

.0074 

.0161 

.0207 

.0456 

.0213 

.0334 

.0039 

.0053 

.0037 

.0027 

.0012 

.0032 

.0014 

.0098 

.0023 

.0017 

.0011 

.0093 

.0134 

.0084 

.0073 

.0028 

.0085 

.0034 

.0184 

.0063 

.0037 

.0026 

.0056 

.0077 

.0052 

.0041 

.0019 

.0047 

.0020 

.0139 

.0033 

.0025 

.0068 

.0123 

.0170 

.0114 

.0086 

.0039 

.0098 

.0041 

.0288 

.0069 

.0051 

.0139 

as measured either by metric mˆ k

or mˆ

 

k

. This is true even under Model F, where we can see from Table 1 that 

the last batch of variables entered between model D and F are not very signifcant as measured by an increase in 

maximized loglikelihood, or equivalently in decreased Deviance. 

Recall that we devised the metrics mˆ k, mˆ

 

k

in part to penalize model-based adjustment which, like raking, removes 

bias directly in terms of population totals. Recall also that mˆ

 

k

diered only by fnding maximum absolute 

discrepancies over consecutive sequences of re-ordered indices which keep adjustment-cells consecutively indexed. In 

fact, the metric values mˆ

 

k

turn out to be only slightly larger than mˆ k, and they follow a very similar pattern across 

the dierent models. Consider Table 4 charting the progression of averaged mˆ

 

k

metrics (over k= 1,. .. ,11 and 

Population Count) as the adjustment model varies over the Adjustment Cell model and the six Logistic Regression 

models described in Table 1, and for brevity let denote the average of Mˆ  ˆ these metric values analogous to Min 

(6), with equal weights wk

= 1/12. The logistic regression models are all clearly better than the cell-based model in 

adjusting at Wave 12, but at Wave 4, models A and B actually seem a little worse than the cell-based method. Since 

the models A–E are listed in order of decreasing Deviance or AIC, there is no strict relationship between decreasing 

AIC and decreasing Mˆ 

. Model C looks to be the best among A–E, and might have been chosen also for parsimony 

Mˆ 

from examination of deviances; but the metric rewards model F for including essentially all of the SIPP items 

as predictors. 

Although we would not have chosen model F from likelihood considerations, it may well be that this model is 

a good choice from the vantage point of nonresponse adjustment. The SIPP dataset is large enough (n=94444) 
Table 4: Metric values mˆ calculated on SIPP 96 data for Adjustment-cell model and for logistic regression models 

k

A–F and averaged over k= 1,. .. ,12, where ‘item’ 12 is Population Count (y

(12) ≡

1). i

Model 

Adj.Cell 

LReg, A 

LReg, B 

LReg, C 

LReg, D 

LReg, E 

LReg, F 

Wave-4 Wave-12 

0.01228 0.04741 

0.01451 0.03942 

0.01504 0.03893 

0.00426 0.02475 

0.00571 0.02812 

0.00481 0.02654 

0.00342 0.00782 

Table 5: Metric (8) values for Wave 4 adjustment, based on the Adjustment cell and logistic regression models, using 

SIPP 96 data with adjustment cells as partition elements Aj . 

item ModA ModB ModC ModD ModE ModF Adj.Cell 

Fdst .0667 .0700 .0615 .0627 .0608 .0609 .0594 

AFDC .0800 .0840 .0728 .0734 .0705 .0722 .0697 

Mdcd .0588 .0620 .0517 .0516 .0514 .0511 .0527 

SocS .0300 .0300 .0305 .0310 .0315 .0287 .0332 

Hins .0249 .0247 .0238 .0241 .0226 .0237 .0224 

Pov .0627 .0632 .0574 .0576 .0556 .0566 .0572 

Emp .0276 .0281 .0263 .0261 .0234 .0259 .0227 

UnEmp .1005 .0988 .0868 .0866 .0878 .0867 .0946 

NILF .0315 .0315 .0312 .0315 .0313 .0304 .0327 

MAR .0207 .0206 .0206 .0207 .0198 .0216 .0225 

DIV .0498 .0498 .0459 .0476 .0443 .0451 .0480 

POP .0260 .0264 .0243 .0244 .0222 .0238 .0221 

that all of the SIPP survey items except AFDC and Emp have highly signifcant coeÿcients. Moreover, the highly

parametrized adjustment models C–F are accomplishing something that simple raking cannot: they are generating 

response probabilities with good behavior over adjustment cells considered as subdomains. To see this more clearly, 

consider the unforgiving metric (8): for each item k, we sum the absolute estimated biases |Bˆ

k(Aj )|

over all 

adjustment cells and form the ratio of the total to tˆ (k) . The result on the SIPP 96 data is given in Table 5. 

Although the Adjustment Cell model is given an advantage by evaluating adjustment eectiveness over exactly the 

same adjustment cells used to form ratio weighting-adjustments, a few of the logistic regression models (especially 

models D–F) do at least as well, item by item, with far fewer parameters than the 149 adjustment-cell response 

fractions. Nevertheless, with respect to this metric none of these models except possibly Model F shows much 

advantage over the others. 

y

5 Conclusions 

This paper has developed metrics for nonresponse-adjustment eectiveness, calculated after randomly re-indexing 

the survey sample and calculating maximum discrepancies over consecutively indexed subdomains. The objective 

was to discount any advantage which an adjustment regression model might achieve toward eliminating whole-sample 

nonresponse biases by including survey attributes as predictors. However, when applied to SIPP 96 data, the metrics 

developed did not have the expected eect. Those regression models which incorporated most or all of the interesting 

survey attributes did exceptionally well with respect to the new metrics, even though some of those models would

not have been preferred from examination of likelihood ratios or deviance. While the same adjustment strategy 

could not be tried if the selected set of ‘interesting’ survey attributes were very large, the strategy may actually be a 

good one in the setting chosen, where the selected set of attributes was still small enough to contain variables which 

were almost all highly predictive of response and yet not redundant (except for the triple Emp, UnEmp, NILF which 
partitions the population by defnition.) 

One important check on the usefulness of the adjustment eectiveness metrics developed here remains to be pursued 

in depth: namely, an examination of the sampling variability of the metrics through calculation of their design-based 

variances. The variances could be calculated by a BRR method, although that computationally intensive calculation 

involves a design crossing many replicate weight-factors with many iterated random permutations of the sample. 

This kind of investigation of sampling variability of the metrics considered as statistics, may show that the present 

adjustment eectiveness study was statistically stable enough to justify a reliable adjustment strategy for SIPP. 

6 References 

Bailey, L. (2004), Weighting alternatives to compensate for longitudinal nonresponse in the Survey of Income and 

Program Participation. Census Bureau internal report, Nov. 16, 2004. 

Billingsley, P. (1968) Convergence of Probability Measures. Wiley: New York. 

Dagum, E. and Cholette, P. (2006) Benchmarking, Temporal Distribution, and Reconciliation Methods

for Time Series. Lect. Notes in Statist. 186, Springer: New York. 

Dufour, J., Gagnon, F., Morin, Y., Renaud, M. and S ̈arndal, C.-E. (2001), A better understanding og weight 

transformation through a measure of change. Survey Methodology, 27, 97-108. 

Kim, Jae-Kwang and Kim, Jay (2007), Nonresponse weighting adjustment using estimated response probability, 

Canadian Jour. of Statist., to appear. 

Feller, W. (1948) On th Kolmogorov-Smirnov limit theorem for empirical distributions. Ann. Math. Statist. 19, 

177-189. 

Pollard, D. (1980) Convergence of Stochastic Processes. Springer-Verlag: New York. 

Rizzo, L., Kalton, G., Brick, M. and Petroni, R. (1994), Adjusting for panel nonresponse in the Survey of Income 

and Program Participation, ASA Surv. Res. Methodology Proceedings paper, JSM 1994. 

S ̈arndal, C.-E., Swensson, B., and Wretman, J. (1992) Model Assisted Survey Sampling. Springer-Verlag: New 

York. 

Slud, E. and Bailey, L. (2006), Estimation of attrition biases in SIPP. ASA Surv. Res. Methodology Proceedings 

paper, JSM 2006, Seattle, WA. 

Tupek, A. (2002) SIPP 96: specifcations for the longitudinal weighting of sample people. Internal Census Bureau 

memorandum. 
7 Appendix: Results from Slud & Bailey 2006 

Table 6: Relative biases δ

(k) between waves 4 versus 1 and 12 versus 1 in SIPP 1996 survey-weighted estimated 

(adjusted) population total of 11 Wave 1 survey items, as found by Slud and Bailey (1996). The entries are indexed 

by wave 4 or 12 and C (cell-based adjustment model) or L (logistic regression model B). Entries are the biases given

in Tables 1 and 2 of Slud and Bailey divided by Wav1 item totals. Totals given here in 1000’s. 

Item Total μ4C

μ4Lμ12Cμ12L

Foodst 

AFDC 

Mdcd 

SocSec 

Heins 

Pov 

Emp 

UnEmp 

NILF 

MAR 

DIV 

27268 

14030 

28173 

37087 

194591 

41796 

191201 

6406 

66647 

114367 

18463 

-0.00315 

-0.00394 

0.00544 

0.01885 

0.00837 

-0.01843 

0.00099 

-0.05257 

0.00221 

0.01096 

-0.01116 

0.01814 

0.02412 

0.02764 

0.01115 

0.00634 

0.00071 

0.00113 

-0.05868 

0.00245 

0.00083 

-0.01936 

-0.04325 

-0.10352 

-0.01410 

0.11168 

0.01948 

-0.03655 

-0.00758 

-0.11609 

-0.03290 

0.05060 

-0.02063 

0.00956 

-0.03274 

0.04149 

0.10364 

0.01299 

0.00587 

-0.01172 

-0.12657 

0.04588 

0.02231 

-0.03732 

Table 7: Relative standard errors (rse) of the Wave 4 versus Wave 1 and Wave 12 versus Wave 1 dierences between 

estimated totals of 11 SIPP96 Wave 1 survey items. Variances were calculated by Slud and Bailey (2006) using Fay’s 

BRR method, and rsetable entries are the respective SE columns in Tables 3 and 4 of Slud amd Bailey (2006) 

divided by the Wave 1 totals. The rse’s given here should be multiplied by 2.576 to give 99% confdence interval 

half-widths comparable to the bounds bk

in Table 2. 

Item Total rse4C

rse4Lrse12Crse12L

Foodst 

AFDC 

Mdcd 

SocSec 

Heins 

Pov 

Emp 

UnEmp 

NILF 

MAR 

DIV 

27268 

14030 

28173 

37087 

194591 

41796 

191201 

6406 

66647 

114367 

18463 

0.00615 

0.01107 

0.00481 

0.00335 

0.00117 

0.00429 

0.00069 

0.00874 

0.00187 

0.00170 

0.00551 

0.00655 

0.01150 

0.00523 

0.00387 

0.00121 

0.00020 

0.00086 

0.00902 

0.00244 

0.00159 

0.00523 

0.01251 

0.02078 

0.01050 

0.00682 

0.00234 

0.00901 

0.00149 

0.02051 

0.00415 

0.00365 

0.01134 

0.01509 

0.02375 

0.01274 

0.00766 

0.00258 

0.00187 

0.00191 

0.02015 

0.00549 

0.00337 

0.01072