On checking whether response is ignorable or not

Michail Sverchkov, Bureau of Labor Statistics

The opinions expressed in this paper are those of the author and do not 

necessarily represent the policies of the Bureau of Labor Statistics

www.bls.gov

1 
 

 

 

 

 

 

Introduction and Notation:

{ , ; } Y X i U i i  - finite population from unknown pdf ( ) i i f Y X

(“pdf” - probability density function when Yiis continuous or

the probability function when Yi

is discrete)

{ , ; } Y X i S i i  - sample drawn from finite population U with

known inclusion probabilities Pr( ) i   i S

Yi

- target variable, 1

( ,..., ) K X X X i i i

 - covariates (observed 

for entire sample). R - sample of respondents (the sample 

with observed outcome values)

2 
 

 

 

 

Let ( , ) i i p Y X  Pr( | , , ) i i i R Y X i S   . If ( , ) i i p Y X were known 

then the sample of respondents could be considered as a

sample from the finite population with known selection 

probabilities ( , ) population model parameters i i i i    p Y X 

(or finite population parameters) could be estimated as if 

there was no non-response.

3 
 

 

 

 

 

Also, if known, the response probabilities could be used to 

impute the missing sample data via the relationship between

the sample and sample-complement distributions

(Sverchkov & Pfeffermann 2004);

( | , , ) i i f Y y X x i R i S     

1

1

[ ( , ) 1] ( | , )

{[ ( , ) 1]| , }

i i

i

p y x f Y y X x i R

E p y x X x i R





   

  

(1)

Note that ( | , ) i i f Y y X x i R    refers to the observed data

and therefore can be estimated using classical statistical 

inference procedures.

4 
 

 

 

 

 

Most methods of estimating the response probabilities 

assume (explicitly or implicitly) that the missing data are

‘missing at random’, Pr( | , , ) i i i R Y X i S   Pr( | , ) i

i R X i S   .

In this case, if auxiliary data is not missing, Pr( | , ) i

i R X i S  

refers to fully observed data and can be estimated by use of

classical regression techniques.

5 
 

 

In many practical situations, MAR assumption is not valid:

the probability of responding often depends directly (or 

indirectly) on the outcome value. In this case the use of

methods that assume MAR can lead to large bias of

population parameter estimators and large imputation bias. 

6 
 

 

 

 

 

 

 

 

The case where the missing data are NMAR can be treated 

by postulating a parametric model for the distribution of the 

outcomes before non-response, , and a 

model for the response mechanism, 

[ | , ; ] i i f Y X i S  

( , ; ) i i p Y X  ,

 two models define a parametric model for the joint 

distribution of the outcomes and the response indicators

 the parameters of these models can be estimated by

maximization of the likelihood based on this joint distribution.

7 
 

 

 

 

 

Problems: i) Modeling the distribution of the outcomes

before non-response can be problematic since it refers to the 

partly unobserved data.

ii) The same problem with the response mechanism.

iii) Estimators assuming NMAR are usually much less stable

than estimators assuming MAR. (Moreover, often NMAR 

estimator does not exist - too many unknown parameters).

8 
 

 

 

 

Sverchkov JSM 2008 suggested an approach that allows

estimation of the parameters of the response model without 

modeling the distribution of the outcomes before nonresponse:

For simplicity assume that auxiliary variables are not

missing. Let ( , ; ) i i p Y X  Pr( | , , ; ) i i i R Y X i S    and suppose

that p is differentiable with respect to the (vector) parameter

 . 

9 
 

 

 

 

 

If the missing data were later observed,  could be 

estimated by solving: 

log ( , ; ) 0

i i

i R

p Y X 

 



 





log[1 ( , ; )]

c

i i

i R

p Y X 

 

 



 (2)

Denote the observed data by { , ; , } O Y i R X k S    i k . 

Missing Information Principle: since the outcome values are

missing for i R i S   , , we propose to solve instead, 

log ( , ; ) 0 {[ i i

i R

p Y X E



 



 





log[1 ( , ; )]]| }

c

i i

i R

p Y X O

 

 







log ( , ; ) i i

i R

p Y X 

 









log[1 ( , ; )] { | , }

c

i i c

i R

p Y X E O i R 

 

 

 





Eq.1

10 
log ( , ; ) i i

i R

p Y X 

 









1

1

log[1 ( , ; )] {[ ( , ; ) 1] | , }

0

c {[ ( , ; ) 1]| , }

i i

i i i

i R i i i

p Y X E p Y X X i R

E p Y X X i R















 

 



 

 

 

 

 

 

(3)

Parameter  can be estimated by solving (3).

Note that the second sum in (3) predicts the unobserved 

second sum in (2).

11 
 

 

 

 

 

Note also that if ( , ; ) j j p Y X  is a function of X j and  only

(missing data is MAR) then (3) reduces to the common loglikelihood equations,

log ( ; ) 0

i

i R

p X 

 



 





log[1 ( ; )]

c

i

i R

p X 

 

 



 . (4)

The proposed approach can be generalized to the case

when auxiliary variables are partly missing. See also

Sverchkov JSM 2010 for similar approaches.

12 
 

 

 

 

 

 

 

 

 

The proposed approach requires knowledge of the

parametric form of the response model which refers to the

unobserved data in the case of NMAR.

 On the other hand, if the response is MAR, the

propensity score, ( ; ) i

p X  Pr( | , ; ) ii R X i S    , can be 

estimated from the observed data, for example, by solving 

the log-likelihood equations (4). 

The latter estimators are much more stable than the 

estimators assuming NMAR. 

Can we check whether the response is MAR or NMAR?

13 
 

 

 

 

 

Testing whether the response is MAR or NMAR

Step 1. Fit the model for propensity score,

( ; ) ip X   Pr( | , ; ) ii R X i S    , 

and estimate the parameter  from the observed data

assuming MAR.

14 
 

 

 

 

 

 

 

Step 2. Define a class of models for

( , ; ) i i p Y X  Pr( | , , ; ), i i i R Y X i S     

some  , ( , ; ) i i p Y X   ( ; ) i

p X  . It is recommended to use 

, in such way that for 

models that include the Y-component in a simple form, 

EXAMPLE: if logit[ ( ; )] ( ; ) i i p X g X    then one can consider 

logit[ ( , ; )] i i p Y X  g(X ; ) , ( , ) i i      cY c , so in this case for 

   ( ,0), ( , ; ) i i p Y X   ( ; ) i

p X  .

Step 3. Obtain estimating equations (3) based on the class

of models defined in Step 2.

15 
 

 

 

 

Step 4.1. Solve them and check whether Y-component is

significant (in which case the response is NMAR ) or not (the 

response is MAR or “not very informative”). 

The latter can be done by a bootstrap procedure: one can 

take B simple random samples with replacement from the

original sample and repeat steps 1 – 4 above in order to get 

a variance estimate for the Y-component. 

16 
 

 

 

 

 

 

 

Remark. Since the parametric family defined in Step 2 does

not necessarily include the true response probability 

Pr( | , , ) i i i R Y X i S   , we cannot conclude for sure that

response is MAR even if the Y-component is insignificant.

We recommend assuming MAR in this case. If response is

very informative then one can expect that the Y-component

will be significant even when fitting a simplified model. 

17 
 

 

 

Instead of Step 4.1 one can do 

Step 4.2. Substitute  from Step 2 (which corresponds to

MAR assumption) into (3) obtained in Steps 1 - 3 and check 

whether the result of this substitution is significantly non-zero 

(response is NMAR) or not (response “seems to be” MAR 

since  corresponds to the propensity score). The latter can

also be done by use of a bootstrap.

18 
 

 

 

 

 

assume 

U S 

Empirical illustration.

For simplicity that the finite population and the 

sample coincide, . The simulation study consists of the

following steps.

Step A: Generate independently 100 finite populations, each

of size 1000, where Xi ~ Uniform (-1,1),

( 1| ) P Y X i i 

1

(exp{ 0.1 } 1) Xi



   ,

( 0 | ) 1 ( 1| ) P Y X P Y X i i i i    . 

19 
 

 

 

Step B: For each population the response indicators were

generated as:

1 1

0 1 2 ( 1| , ) [exp( ) 1] [ 2] P R Y X X Y i i i i i   

        .

We repeat the study for different values of parameter

0 1 2      ( , , ).

20 
 

 

 

 

Step C: For each sample of respondents estimate the

response probabilities assuming the response is MAR and 

the response model is logistic, i.e. 0 1 ( , )   ˆ ˆ is a solution of the 

likelihood equations 

( ) 1

0 1 log{[exp( ) 1] } m

i

i R d

  X







  







( ) 1

0 1 log{1 [exp( ) 1] } 0,

c

m

i

i R d

  X







   





 d  0,1

These estimates were derived using Proc Logistics of SAS.

21 
 

 

 

 

Step D: Define estimating equations (3) assuming that

response follows the logistic model,

1

0 1 2 ( 1| , ) [exp( ) 1] P R Y X X Y i i i i i   



    

and substitute 0 1 ( , )   ˆ ˆ into the estimating equations. 

Test:

If the result is significantly non-zero then response is 

NMAR

22 
 

 

 

 

 

# of rejections MAR hypothesis, 0

  1, 1

  1

2

 95% 99%

-1 99 99

-0.9 99 99

-0.6 99 95

-0.3 50 15

0 4 0

0.3 32 5

0.6 68 32

0.9 91 71

23 
 

 

 

 

 

# of rejections MAR hypothesis, 0

 1, 1  1

2

 95% 99%

-1 99 99

-0.9 99 93

-0.6 73 35

-0.3 19 4

0 8 0

0.3 7 1

0.6 28 6

0.9 56 10

24 
THANKS !!! (Sverchkov.Michael@bls.gov)

25