Quantifying CHI doorstep concerns as risk factors of 

Wave 1 nonresponse for the CE Interview Survey 

Brett McBride and Lucilla Tan 

Bureau of Labor Statistics 

Submitted: March 12, 2014 

Revised: April 11, 2014 

Contents 

Summary ....................................................................................................................................................... 1

I. Introduction ............................................................................................................................................... 1

II. Methods .................................................................................................................................................... 2

a. Study sample & outside-sample data ................................................................................................... 2

b. Formation of doorstep concern themes. .............................................................................................. 3

c. Risk factors ............................................................................................................................................ 4

d. Bivariate analysis ................................................................................................................................... 6

e. Multivariate logistic regression ............................................................................................................. 8

f. Assessment of the final model............................................................................................................... 8

g. Use of outside-sample data ................................................................................................................10

h. “Approximate” population attributable risk .......................................................................................10

III. Findings ..................................................................................................................................................11

Estimated effects of doorstep concern themes ......................................................................................13

IV. Limitations .............................................................................................................................................14

References ..................................................................................................................................................15

APPENDIX A. Study sample ........................................................................................................................16

APPENDIX B. Initial models attempted ......................................................................................................17

APPENDIX C. Final Model Diagnostics ........................................................................................................23

Multicollinearity check ............................................................................................................................23

Diagnostic plots .......................................................................................................................................24

Outlier covariate patterns ....................................................................................................................... 26

Outside-sample estimation .....................................................................................................................28

APPENDIX D. Index to SAS Code ................................................................................................................30

Acknowledgement: we thank Steve Miller from the Office of Survey Methods Research for his instructive 

comments. 
1 

Summary 

Objective. The objective of this study is to quantify the association of sample units’ expressions of initial reluctance 

at the survey request, as represented by “doorstep concerns” recorded in the Contact History Instrument (CHI), 

with Wave 1 survey (or unit) nonresponse in the CE Interview Survey (CE). 

Application. If these doorstep concerns on Wave 1 unit nonresponse vary in a quantifiable way, we can potentially 

differentiate among sample units who are at varying degrees of risk of nonresponse: sample units with concerns 

that are associated with higher nonresponse propensity labelled as “high risk of nonresponse”, and sample units 

with concerns that are associated with lowest nonresponse propensity as “low risk of nonresponse,” and others in 

between as “potential risk of nonresponse.” Following survey management and adaptive design principles of 

focusing resources on cases for which data collection interventions can most effectively solicit response, this 

research can help identify cases that are more likely to benefit from such interventions and deploy resources more 

efficiently. Accordingly, we would propose targeting interventions on sample units identified as “potential risk of 

nonresponse” as a priority over sample units in a “low risk” or “high risk” categories. In addition, knowledge of 

which of these doorstep concern risk factors affect subsets of sample units can inform how the interventions can 

be tailored for them. 

Methodology. We used multivariate logistic regression with Wave 1 survey nonresponse modelled as the event of 

interest, and doorstep concerns (DS) “themes” from the CHI as the primary risk factors of interest. In addition to 

the usual diagnostics and goodness-of-fit assessments of the final model, we used data from outside the study 

sample (i.e. data from a different time period) to re-estimate the final model, and computed “approximate” 

population adjusted risk statistics to take account of the difference in proportions of doorstep concerns among 

members of the study sample. However, we did not take account of the complex survey design of the CE in our 

analysis.1

 Findings. 

1. The odds of Wave 1 nonresponse among sample units observed with the “Not interested/Hostility” DS theme 

was at least two times higher compared to sample units without this DS theme (Table 6). We propose these 

sample units be classified as “high risk” of unit nonresponse. 

 

1 Our study sample was created as a subset of the production sample because we subset the production sample to sample units 

to have at least 1 contact record in the Contact History Instrument for their doorstep concerns to be recorded. Thus, the final 

weights and replicate weights in the production data files would not be applicable. 
2 

2. The odds of Wave 1 nonresponse among sample units with “Time”, or “Gatekeeping,” or “Survey 

content/Privacy” DS themes, respectively, were not consistently higher or lower compared to sample units 

without each of these DS themes. We propose sample units with any of these DS themes be classified as 

“potential risk” of nonresponse.

3. The odds of Wave 1 nonresponse among sample units without any doorstep concerns is about 4 times lower 

compared to sample units with concerns. We propose they be classified as “low risk” of nonresponse.
1 

I. Introduction 

One of the main areas of research using Contact History Instrument (CHI) data on interviewer perceptions of 

contacted sample person’s reactions at the time of the survey request (doorstep concerns) has been to predict 

survey cooperation (unit response). Unlike socio-demographic information that largely depend on sample units’ 

response to the survey, CHI information have the advantage of being available for a larger proportion of the 

eligible sample since these data are (theoretically) available for all sample units who have been contactd at least 

once regardless of the final outcome of the survey request. CHI data have been found to improve the ability to 

predict unit response over socio-demographic variables alone (e.g. Groves and Couper (1996), Bates, Dahlhamer, 

and Singer (2008), Dahlhamer and Simile (2009)). In addition to predicting final disposition, because doorstep 

concerns and other CHI information about the sample unit are available even before a case is resolved, these data 

can be used to identify and prioritize potential non-responding households who could benefit from further 

recruitment efforts over others who may not. 

 

Among the findings in their exploratory study on the associations between contacted sample units’ doorstep 

concerns from the CHI and key survey performance measures for the Consumer Expenditure Interview Survey 

(CEQ), Kopp, McBride, and Tan (2013) found that the propensity to complete none or all 5 waves of the CEQ survey 

panel differed significantly among doorstep concern “themes.” This suggests that different themes affect survey 

response differently. 

0

10

20

30

40

50

60

70

80

90

Not

interested/hostile

(N=433)

Survey content /

Privacy (697)

Time (904) Gatekeeping (201) Prior wave (307 )* No concerns (1561)

Percent reporting in Wave 1 *

Sample grouped by doorstep concern themes

Association of doorstep concern themes in Wave 1 with prevalence of completed interviews 

in the CE Interview Survey panel (Kopp, McBride, and Tan 2013, Figure 4b) 

No. of completed interviews: 0 1 2 3 4 5

* Sample size for Wave 

2; all other themes 

sample size for Wave 1.
2 

We extend the univariate (single predictor) analyses of Kopp et al. (2013) by using multivariate methods to further 

develop our understanding of how doorstep concerns and other CHI information, particularly from Wave 1, 

associate with response propensity in the CEQ survey panel. The first question we attempt towards this endeavor 

and we which is the topic of this study is to quantify the association of doorstep concerns with Wave 1 survey 

nonresponse, and examine how these effects may differ by type of doorstep concerns. 

We focus on Wave 1 because that is the first encounter that sample units have with the CEQ, and thus an 

understanding of the effects of Wave 1 doorstep concerns on survey cooperation can serve as a baseline for the 

effects of the doorstep concerns in subsequent waves of the survey panel. In addition, we have anecdotal evidence 

that sample units who are Wave 1 respondents are more likely to also be respondents in Wave 2, compared to 

Wave 1 nonrespondents. For example, 69 percent of Wave 1 respondents were also Wave 2 respondents 

compared to only 6 percent of Wave 1 nonrespondents in the study sample of Tseng & Tan (2011; N=7,773) ; in the 

study sample of in Kopp et al (2013; N=3,000), it was 79 percent of Wave 1 respondents to 4 percent Wave 1 

nonrespondents. This suggests that minimizing Wave 1 nonresponse may have positive association with 

minimizing survey nonresponse in Wave 2. 

Application. If the effects of these doorstep concerns on Wave 1 survey nonresponse vary in a quantifiable way, 

then we can potentially differentiate among sample units who are at risk of survey nonresponse: labelling sample 

units with concerns that are associated with higher nonresponse propensity as “high risk of nonresponse”, and 

sample units with concerns that are associated with lower nonresponse propensity as “potential risk of 

nonresponse,” or “low risk of nonresponse.” Following survey management and responsive design principles of 

focusing resources on cases for which data collection interventions can most effectively solicit response, this 

research can help identify cases more likely to benefit from such interventions. Accordingly, we would advise 

targeting interventions on sample units with “potential risk of nonresponse,” over sample units identified to be 

“low risk” or “high risk.” In addition, knowledge of how these doorstep concern risk factors affect subsets of 

sample units can help in the tailoring of survey solicitation strategies to mitigate those risks. 

II. Methods 

a. Study sample & outside-sample data 

Our study sample is the same as that used in the study by Kopp et al. (2013) - these were sample units whose 

Wave 1 eligibility occurred on or after October 2011 and their Wave 5 eligibility occurred by the end of March 

2013, and whose final outcome to each survey request was known for all 5 waves of their survey panel during this 

period. Excluded from the study sample were replacement sample units and sample units who were never 

contacted. These criteria yielded a study sample of 3,000 unique sample units who belonged to 6 CEQ survey 
3 

panels.2 For this study, we use only the Wave 1 data of the study sample. Among the 3,000 sample units in the 

study sample, 13.6 percent were nonrespondents in Wave 1. 

We also extracted additional Wave 1 data of eligible sample units from a different time period, April 2013 through 

November 2013, to form our “outside-study sample.” Excluded sample units were those whose CHI records 

indicated they were never contacted. This outside-study sample was used to re-estimate the final multivariate 

logistic model. 

All data analyses in this study was conducted using SAS version 9.3. 

b. Formation of doorstep concern themes. 

We followed the strategy of Kopp et al. (2013, Table 7) in grouping the CHI doorstep concern (DS) items to form 

“themes”, as shown in Table 1 below. They adopted this pre-defined, “fixed items” approach of assigning items to 

DS themes so that the themes could be easily understood and be based on a consistent set of items at every wave 

of the CEQ survey panel. Table 1 also shows the indicator variable name of each DS theme used in our analysis. 

Since this study focuses on Wave 1, the theme “Prior Wave” is excluded from our analyses; however, there were 

sample units for which these DS items were reported in the study sample. For this study, we created another DS 

theme “Other” and its indicator variable, “ioth” to indicate if any of the “Prior Wave” items or the DS item “23-

Other: specify” were observed for the sample unit. 

If a sample unit’s contact attempt history in Wave 1 showed that a DS item in Table 1 was recorded at least once, 

the corresponding DS theme indicator variable was flagged to indicate that DS theme was observed at least once in 

Wave 1 for the sample unit. For example, if a sample unit’s contact attempt history indicated at least one Wave 1 

occurrence of the DS item “2-too busy”, then the indicator variable “itime1” was assigned the value 1 for that 

sample unit. There was also an indicator variable, “inoconc1” set to value 1 to identify a sample unit without any 

DS concerns noted in its contact attempt history. 

 

2

For a more detailed description of the study sample, see Appendix A Table A1. 
4 

Table 1. Grouping of CHI doorstep concern items to form doorstep concern themes 

Doorstep concern 

theme

Prior Wave

[ not relevant for 

Wave 1]

Time Not interested/ 

Hostilty

Survey Content/ 

Privacy 

Gatekeeping

Indicator variable 

( suffix j is Wave #j)

ipwavej itimej inhj icpj igatej

CHI doorstep 

concern items

16 - R. requests 

same FR

17-info previously 

given

18 - too many 

questions 

previously

19- too many 

interviews

20 - intv too long 

previously

21-intends to quit 

survey

2-too busy

3-intv too time 

consuming

4-brk appt-puts 

off FR indefinitely

5-scheduling 

difficulty

1-not interested

11-hangs 

up/slams door

12-hostile 

/threatening

6-survey 

voluntary

7-privacy 

concerns

8-anti-govt

9-does not 

understand 

survey

10-survey content 

not applicable

13-othr hh 

members say 

don’t do survey

14-talk to specific 

hh member

15-family issues

No. items: 6 4 3 5 3

Note: Since this study focuses on Wave 1, the theme “Prior Wave” is excluded from our analyses; however, there were sample units 

for which these DS items were reported in the study sample. For this study, we created another DS theme indicator, “ioth” to indicate 

if any of the “Prior Wave” items, or the DS item “23-Other: specify” were observed for the sample unit.

c. Risk factors 

In this study, the event of interest to be modelled is unit nonresponse in Wave 1, with the risk factors of interest 

being the DS themes in Wave 1. The frequency distribution of the number of DS themes reported, the 

combinations of themes reported, and the rate of Wave 1 survey nonresponse by these theme combinations for 

the study sample are shown in Table 2. More than half of the sample units were observed to have no doorstep 

concerns (52.0 percent). The most frequently reported number of DS themes for sample units was one (22.7 

percent). Among sample units with only one DS theme, “Time” (n=412) was the most frequent theme observed 

and “Not interested/Hostility” (n=54) was the least frequent; however, the prevalence of unit nonresponse was 

10.7 percent compared to 50.0 percent, respectively, for these 2 DS themes, suggesting “Time” and “Not 

interested/Hostility” have differential effects on response propensity . In addition, regardless of the number of 

themes observed, the rates of Wave 1 nonresponse were consistently highest among sample units whose DS 

theme combinations included “Not interested/Hostility.”
5 

Table 2. Combination of DS themes observed and final disposition of nonresponse in Wave 1

No. of

DS themes reported

Combination of

DS themes*

No. of sample units 

with combination

(n)

Percent distribution 

(n/N=3,000)

% nonresponse

for theme 

combination **

0 00000 1,561 52.0 2.1

1 812 27.1

10000 54 1.8 50.0

00010 38 1.3 21.1

01000 178 5.9 11.2

00100 412 13.7 10.7

00001 130 4.3 9.2

2 401 13.4

11000 96 3.2 72.9

10100 52 1.7 46.2

10010 5 0.2 20.0

00110 41 1.4 19.5

01100 153 5.1 14.4

01110 33 1.1 9.1

01010 21 0.7 4.8

3 186 6.2

11010 13 0.4 61.5

10110 10 0.3 60.0

11100 163 5.4 58.9

4 11110 40 1.3 62.5

Total 3000 100.0 407

* Note on theme combination pattern: the 1st position represents the theme of inh, the 2nd icp, the 3rd itime, the 

4

th igate, and the 5th ioth; the value ‘0’ implies the theme was not observed, ‘1’ that the theme was observed.

** For e.g., of the 54 sample units reported have the DS theme combination ‘10000’, 50 percent resolved as 

nonresponse.

In addition to the DS themes described in Table 1, other CHI information were included in our multivariate 

analyses to serve as control variables. We considered the following factors to be indicative of a “difficult” case and 

positively associated with the likelihood of survey nonresponse: 

1. If a sample unit’s contact attempt history indicated there was at least 1 change in interviewer 

(ichgfrwv1=1)

2. A lower proportion (compared to sample median) of contact attempts resulting in contact with the 

sample unit (ipctatmp1=0) 

3. A higher (compared to sample median) number of days between the first and final contact attempt. The 

longer a case stays unresolved may be an indication of a case at risk (indaysatm1l1=1)

4. A higher (compared to sample median) proportion of soft (interim) refusals (ipctsftr=1) 

5. One or more contact attempt strategies were used (istrg1=1)
6 

Among socio-demographic variables for controls, we considered factors for which information may be obtained 

without an interview: 

6. If the sample unit lived in a dwelling structure with only a single housing unit (isglhunit1=1)

7. If the housing unit was within a MSA (imsa1=1)

8. If 20% or more of the population in the tract lived in poverty (iareapov1=1)

9. If the size of consumer unit was larger than the sample median (igpsize=1)

Race (whether White or not), language spoken upon contact (whether English or not, from CHI) were also 

considered, but dropped from consideration for the multivariate analyses because of their high incidence of 

missing values among nonrespondents (394 and 379 sample units missing race and language, respectively, out of 

407 Wave 1 nonrespondents, see Table 3). 

The descriptive statistics for continuous variables are displayed in Appendix A Table A2. Median values of these 

continuous variables from the “overall” group were used to create corresponding indicator variables shown in 

Table 3. The counts and distribution of the categorical risk and control factors considered for the multivariate 

analyses of DS themes and survey nonresponse in Wave 1 are shown in columns 4 through 7 in Table3. 

d. Bivariate analysis 

Logistic regressions with Wave 1 nonresponse as the event modelled (dependent variable) were conducted with 

each of the risk factors as a single predictor. This provided the unadjusted odds ratios and Chi-square statistic. 

These results appear in columns 8 through 11 of Table 3. 

Table 3. Distribution statistics on potential predictors and unadjusted odds ratios from bivariate logistic regression modelling 

survey nonresponse in Wave 1 as the event

Variable Description Wave 1 final disposition Unadjusted 

OR

95LCI 95UCI WaldChi p

Nonreponse

(N=407)

Response

(N=2,593)

N % N %

inoconc1 No DS 

concerns 

0=No 375 12.5 1064 35.5 0.06 0.04 0.09 <.0001

1=Yes 32 1.1 1529 51.0

inh1 DS: not 

interested / 

hostility

0 150 5.0 2417 80.6 23.53 18.27 30.30 <.0001

1 257 8.6 176 5.9

icp1 DS: survey 

content / 

privacy

0 162 5.4 2141 71.4 7.16 5.73 8.95 <.0001

1 245 8.2 452 15.1
7 

Table 3. Distribution statistics on potential predictors and unadjusted odds ratios from bivariate logistic regression modelling 

survey nonresponse in Wave 1 as the event

Variable Description Wave 1 final disposition Unadjusted 

OR

95LCI 95UCI WaldChi p

Nonreponse

(N=407)

Response

(N=2,593)

N % N %

itime1 DS: time 0 179 6.0 1917 63.9 3.61 2.92 4.48 <.0001

1 228 7.6 676 22.5

igate1 DS: 

gatekeeping

0 347 11.6 2452 81.7 3.01 2.18 4.15 <.0001

1 60 2.0 141 4.7

ichgfrwv1 At least 1 

interviewer 

change 

0=No 342 11.4 1535 51.2 11.23 8.90 14.16 <.0001

1=Yes 65 2.2 1058 35.3

ipctatmp1 More than 1⁄2 

attempts 

resulted in 

contact

0 342 11.4 1535 51.2 0.28 0.21 0.36 <.0001

1 65 2.2 1058 35.3

indaysatm1l1 No. days 

between 1st

and last 

attempt >10

0 55 1.8 1492 49.7 8.67 6.46 11.64 <.0001

1 352 11.7 1101 36.7

ipctsftr1 Any soft 

refusal

0 160 5.3 2413 80.4 20.70 16.12 26.57 <.0001

1 247 8.2 180 6

istrg1 At least 1 

strategy 

used 

0 217 7.2 1458 48.6 1.13 0.91 1.39 0.2717

1 190 6.3 1135 37.8

isglhunit1 Single 

housing unit 

in structure

missing 15 0.5 10 0.3 1.32 1.02 1.70 0.0325

0 88 2.9 713 23.8

1 304 10.1 1870 62.3

imsa1 In MSA 0 41 1.4 311 10.4 1.22 0.86 1.72 0.2639

1 366 12.2 2282 76.1

iareapov1 >=20% in 

poverty 

0 352 11.7 2214 73.8 0.91 0.67 1.24 0.5567

1 55 1.8 379 12.6

igpsize1 CU size > 2 0 276 9.2 1525 50.8 0.68 0.54 0.85 0.0006

1 131 4.4 1068 35.6

ilang1 English missing 394 13.1 10 0.3 0.31 0.07 1.42 0.1329

0 2 0.1 139 4.6

1 11 0.4 2444 81.5

ircwhite1 White missing 379 12.6 . . 0.45 0.21 0.98 0.0429

0 11 0.4 589 19.6

1 17 0.6 2004 66.8

ihmowner1 Homeowner 0 103 3.4 742 24.7 1.18 0.93 1.50 0.1681

1 304 10.1 1851 61.7

Note: For each categorical variable, the “%” columns (columns 5 and 7) show the cell percent distribution for each combination of level and 

final disposition, and sum to 100%.
8 

e. Multivariate logistic regression 

We first considered a model without any CHI information (non-CHI model). A factor from Table 3 was added to this 

non-CHI model if its p-value from the bivariate Chi-square test was less than 0.25 (Hosmer & Lemeshow 1989, p. 

86), or it was of substantive interest. Subsequently, nested models were compared and factors were retained if 

they were significant at p<0.05. Interactions were considered significant at p<0.01. Since the association of each 

DS theme with survey nonresponse was the primary issue of interest, all DS theme indicators were retained in the 

final model, regardless of their statistical significance. 

The addition of the DS themes and other CHI factors to the no-CHI model improved the area under the Receiver 

Operating Curve from 0.57 to 0.92 (see Appendix B, Table B1, Model NoCHI and Model 3). The Likelihood ratio test 

of this pair of nested models confirmed the improvement in the overall fit of the model when the CHI information 

was included (Likelihood Ratio statistic Chi-square with 10 degrees of freedom had p-value<0.01; AIC of 1,348 

compared to 2,309). 

Given the high rate of nonresponse for the combinations of themes that included “Not interested/Hostility”

reflected in Table 2, we also examined the interaction between “Not interested/Hostility” and other themes (see 

Appendix B, Table B2). Only the pair “Not interested/Hostility” and “Survey content/Privacy”, inh1* icp1, was 

significant (p<0.001). The estimated parameters of the final model with and without this interaction term appear 

in Appendix B, Table B3. However, we chose the final model to exclude this interaction term based on reestimating the model with this interaction using 3 separate monthly data from the outside-study sample; this 

interaction effect was not consistently significant(p>0.3) (see Appendix B, Table B4).3

 In addition, we observed 

greater fluctuation in the magnitudes of estimated logistic coefficients using monthly data for the model including 

the interaction effect (e.g. the coefficients of icp1, ipcatmp1, and istrg1 in Table B4) compared to the model 

without the interaction term. For these reasons, and the principle of parsimony, we selected the final model to 

exclude the interaction term. 

f. Assessment of the final model. 

Multicollinearity. 

We looked for indications of multicollinearity (when two or more predictors are near perfect linear combinations 

of one another) among the predictors. The variance inflation factor of all predictors in the final model was less 

 

3 We used monthly data for estimation because that is the regular duration of the CE Interview Survey’s field 

period for data collection, and thus would provide some idea of the impact of having monthly data for estimating 

such models in “real time”.
9 

than 1.5, and the largest condition index value was 8.8, suggesting that multicollinearity was not a significant 

problem (see Appendix C, Table C1). 

Model selection. 

We compared pairs of nested models using the Log Likelihood ratio test, as well as giving consideration to their 

relative AIC values. We examined the estimated overdispersion parameter for each model, as well as the overall 

measures of goodness-of-fit, significance of the Hosmer-Lemeshow statistic, and the area of the Receiver 

Operating Curve (ROC). Although not significant (p>0.4), the variable ISTRG1 variable was left in the model to 

correct for overdispersion (estimated overdispersion parameter = 1.22, p<0.01, when ISTRG1 was excluded); its 

inclusion reduced the estimated overdispersion parameter to close to 1. The other models we considered are 

documented in Appendix B.

Outliers. 

The diagnostic statistics, graphs, and influential covariate patterns for the final model are shown in Appendix C. We 

identified covariate patterns that were poorly fitted by large changes in their Pearson Chi-square statistic or 

Deviance statistic (DELTAChi or DELTAd values greater than 4), shown in Appendix C, Table C2. From this subset of 

covariate patterns, we identified those with a large influence on estimated coefficients identified by causing large 

changes in all the model coefficients when all subjects with a specific covariate pattern were removed from the 

model estimation (large DELTAbeta values; Hosmer & Lemeshow, equations 5.10-5.11, page 156). In a separate 

analyses, we re-estimated the final model after excluding two of these poorly fitted and influential covariate 

patterns one at a time (see Appendix C Table C3), and another final model estimation excluding 7 of these 

covariate patterns simultaneously (see Table C4). We concluded that the estimated final model parameters for the 

risk factors of interest did not change substantively with the exclusion of the covariate patterns, so the final model 

estimation was done without excluding any covariate pattern. 

Classification accuracy. 

Even if the predictors in the final model had no relationship to the dependent variable, the model can still correctly 

classify group membership by chance – referred to as “by chance accuracy”. Using the marginal frequencies for the 

number of events and nonevents in the study sample, the proportional-by chance accuracy rate was computed as 

[(407/3000)^2 + (2593/3000)^2]=0.765 (for example, see page 203 in Petrucci 2009). For the generally acceptable 

prediction improvement criteria of 25 percent over the proportional-by chance accuracy rate, the accuracy criteria 

would be (1.25*76.5%)= 95.7 percent for our application. However, our final model’s accuracy rate is 90.7 percent 

(from the final model’s classification table at the predicted probability cutoff of 0.5), which is only about an 18 

percent prediction improvement. 
10

g. Use of outside-sample data 

We re-estimated the final model using data collected from a time period different from that for the study sample: 

using April 2013, July 2013, and September 2013 separately, as well as using 8 months, April 2013 through 

November 2013, collectively (see Appendix C Table C5). We chose to examine estimated parameters using only 

one month of Wave 1 sample units because in practice, a new sample of Wave 1 sample units is in the field each 

month for the CE Interview Survey. We wanted to see how the final model’s estimated coefficients for the DS 

themes fluctuated with typical one-month sample sizes. We also compared the observed number of events (i.e. 

Wave 1 unit nonresponse) each month with the final model’s predicted number of events each month. We 

summed up the predicted probabilities from the final model estimated using the 8 months of outside-sample 

collectively to get the count of predicted events, and compared that to the number of observed events in each 

month as shown in Figure 1 (see Appendix C, Table C8 for estimated parameters). 

h. “Approximate” population attributable risk 

Since the prevalence of the different risk factors among the study sample was highly variable, the overall effect of 

a significant risk factor with a large estimated coefficient might be small if occurs very infrequently among the 

population. In public health research, the population attributable risk statistic (PAR) describes the proportion of a 

disease that could be prevented if exposure to a specific risk factor could be eliminated from the target population. 

209

167

194 180 171

189 192

217 225

204 218 219

190

206

101

157

2013m04 (n=757)

2013m05 (745)2013m06 (758)2013m07 (783)2013m08 (741)2013m09 (718)2013m10 (646)2013m11 (764)

Month of data collection and sample size

Figure 1. Observed and predicted counts of Wave 1 survey nonresponse from 

final model covariates using outside-study sample

# observed # predicted
11

For our application, the PAR measure would quantify the proportional reduction of Wave 1 unit nonresponse that 

would occur if exposure to a specific risk factor could be eliminated. The PAR was computed as: 

rj*(ORj-1) / [1+ rj*(ORj-1)], 

where j is the index for the risk factor, and rj the proportion of the sample exposed to factor j (e.g. Taylor 1977). 

Although PAR computations have typically been applied to case-control (prospective) studies, the prevalence of 

exposures are estimated from the target population, and relative risk instead of odds ratios are used in the PAR 

formula, we use this statistic “approximately” to provide a perspective on the relative importance of the risk

factors associated with Wave 1 unit nonresponse by taking account of their prevalence; we refer to it as the 

“approximate PAR” statistic for our study. In our study sample, eliminating the positive factor of “no concerns” 

would ‘approximately’ increase the proportion of Wave 1 nonresponse by 64%, and eliminating the negative factor 

of “not interested/hostility” would reduce the proportion of Wave 1 nonresponse by 38 percent (see Table 4).

III. Findings 

The estimated parameters from the final multivariate logistic regression model for Wave 1 unit nonresponse is 

summarized in Table 4. 

Table 4. Estimated parameters from final logistic regression model.

Dependent variable (event): Wave 1 unit nonresponse.

Risk factors 

(j)

Proportion

sample with

exposure

(rj)

Logistic 

coefficient

SE P 

value

Odds

Ratio

(ORj)

95LCI 95UCI Approximate 

population 

attributable 

risk

intercept -3.16 0.27 <.001

inoconc1 0.520 -1.40 0.25 <.001 0.25 0.15 0.40 -0.64

DS themes

 inh1 0.144 1.65 0.18 <.001 5.18 3.67 7.35 0.38

 icp1 0.232 0.11 0.17 0.522 1.11 0.80 1.55 0.02

 itime1 0.301 -0.23 0.16 0.162 0.8 0.58 1.10 -0.06

 igate1 0.067 0.13 0.22 0.558 1.14 0.74 1.75 0.01

ichgfrwv1 0.374 0.83 0.16 <.001 2.29 1.69 3.11 0.33

ipctatmp1 0.374 -1.26 0.19 <.0001 0.28 0.20 0.41 -0.37

indaysatm1l1 0.484 1.04 0.19 <.001 2.82 1.97 4.08 0.47

ipctsftr1 0.142 1.16 0.18 <.001 3.18 2.25 4.50 0.24

istrg1 0.442 -0.10 0.15 0.504 0.91 0.68 1.21 -0.04

isglhunit1 0.725 0.43 0.17 0.012 1.54 1.11 2.16 0.28

N=2,583; No. of events=392; No. of unique covariate patterns: 424

Overdispersion parameter: 1.08 (p>0.1)

-2 Log Likelihood: 1,325; AIC: 1,348

Hosmer & Lemeshow Chisq: 12.12, df=8, p=0.15

Classification: %Concordant=92.1, % Discordant=7.6, ROC area=0.92

Prediction improvement over “by-chance accuracy”: 18%
12

Our research interest was in how the DS themes associated with Wave 1 nonresponse. The sample distribution 

statistics in Table 2 had suggested differential effects among the DS themes on Wave 1 nonresponse, and the 

multivariate analysis confirmed this. In the study sample, sample units identified with the DS theme “Not 

interested/Hostility” had odds of unit nonresponse in Wave 1 that were 5 (3.7, 7.4) times higher than sample units 

not identified with this theme, holding other factors constant.4

 Other DS themes were not statistically significant. 

In contrast, the odds of Wave 1 unit nonresponse among sample units without any DS concerns was 4 (0.2, 0.4) 

times lower compared to sample units with concerns. 

Other non-DS concerns risk factors from the sample units’ contact history that were significantly associated with 

higher odds of Wave 1 nonresponse were: the need for more than one interviewer to resolve the case (1.7, 3.1), a 

longer duration between the first and final contact attempt (2.0, 4.1), and a higher rate of interim “soft refusal”

(2.3, 4.5). However, the odds associated with “Not interested/hostility” remained higher than each of these 3 risk 

factors related to characteristics of contact attempt. 

When the differential proportions of the sample exposed to each DS theme risk factor was taken into account, the 

effect of the DS theme “Not interested/hostility” measured by the “approximate PAR” statistic, 0.38, moved closer 

to the three significant non-DS concerns risk factors, but it remained larger than the PAR values of the other DS 

themes “Survey content/Privacy” “Time”, and “Gatekeeping” (0.02, -0.06, and 0.01, respectively). 

An alternative presentation of the final model’s estimated parameters is in the form of the predicted probability of 

Wave 1 unit nonresponse. The observed probability and the predicted probability by the presence of DS themes is 

presented in Table 5. 

Table 5. Predicted probabilities of Wave 1 nonresponse from final model

Wave 1 unit nonresponse

Presence of doorstep concern 

themes # profiles

# 

sample 

units # events

Observed 

probability

Predicted 

probability 95LCI 95UCI

No doorstep themes observed 39 1,561 32 0.02 0.04 0.02 0.06

Yes - excluding"Not interested /

 Hostility"

218 1,006 118 0.12 0.15 0.13 0.17

Yes – including “Not interested /

 Hostility"

184 433 257 0.59 0.49 0.46 0.53

Note:

Observed probability = # events / # sample units, where event is Wave 1 nonresponse.

Predicted probability = exp (constant + ∑ b̂

j jxj

j

) /[1 + exp (constant + ∑ b̂

j jxj

j

)]

 

4

 Notation: the 95% confidence interval for the reported odds ratio is shown by the interval in parenthesis, e.g. 

(3.7, 7.4). 
13

Estimated effects of doorstep concern themes 

We used outside-sample data to re-estimate the final model as an informal indication of the generalizability of the 

magnitudes of the DS theme risk factors. The odds ratios for the doorstep concerns on Wave 1 unit nonresponse 

estimated from the final multivariate logistic regression model using the outside-study sample are summarized in 

Table 6. This summary table highlights the following: 

1. Not interested / hostility: the point estimate of the odds ratio continued to be greater than 1 and 

significant, but its effect weakened to about 3 (2.9, 4.6) times higher odds of Wave 1 unit nonresponse 

with final model estimation using combined 8 months of outside-sample data. 

2. No doorstep concerns: the point estimate of the odds ratio continued to be less than 1, but this effect 

weakened (0.63, 1.04) with final model estimation using the combined 8 months of outside-sample data. 

3. Time: the point estimate of the odds ratio fluctuated between greater and less than 1 for monthly 

outside-sample estimation. However, it is close to the final model point estimate and it does reach 

significance when the combined 8 months of outside-sample data were used (0.61, 0.95). 

4. Gatekeeping and Survey content/Privacy: the point estimate of the odds ratio fluctuated between 

greater and less than 1, and persists in being not significant. 

From estimating the final model using the study sample and outside-sample data, we conclude the following: 

 The odds of Wave 1 nonresponse among sample units with the “Not interested/hostility” DS theme was at 

least two times higher compared to sample units without this DS theme. We would thus classify these sample 

units as “high risk” of unit nonresponse in Wave 1; 

 The odds of Wave 1 unit nonresponse among sample units with “Time”, “Gatekeeping,” “Survey 

content/Privacy” DS themes was not consistently significantly different from sample units without these DS 

themes. We would classify these sample units as “potential risk of unit nonresponse in Wave 1; 

 The odds of Wave 1 nonresponse among sample units without any doorstep concerns was about 4 times lower

compared to sample units with concerns. We propose they be classified as “low risk” of nonresponse.
14

Table 6. Summary of estimated odds ratios of doorstep concerns from final multivariate logistic model using outsidestudy sample

OR 95LCI 95UCI OR 95LCI 95UCI

No DS concerns (inoconc1) Time (itime1)

Final model 0.25 0.15 0.40 Final model 0.80 0.58 1.10

Outside sample estimation Outside sample estimation

 2013 m4 0.26 0.12 0.53 2013 m4 0.71 0.43 1.19

 2013 m7 0.22 0.10 0.48 2013 m7 0.74 0.44 1.27

 2013 m9 0.45 0.20 0.99 2013 m9 1.48 0.85 2.59

 2013 m4 through m11 0.81 0.63 1.04 2013 m4 through m11 0.76 0.61 0.95

Not interested/hostility (inh1) Gatekeeping (igate1)

Final model 5.18 3.67 7.35 Final model 1.14 0.74 1.75

Outside sample estimation Outside sample estimation

 2013 m4 4.76 2.80 8.20 2013 m4 1.16 0.60 2.25

 2013 m7 3.53 2.03 6.14 2013 m7 0.65 0.30 1.37

 2013 m9 4.14 2.25 7.67 2013 m9 0.68 0.30 1.49

 2013 m4 through m11 3.62 2.87 4.56 2013 m4 through m11 1.04 0.76 1.40

Survey content/Privacy (Icp1)

Final model 1.11 0.80 1.55

Outside sample estimation

 2013 m4 0.90 0.53 1.50

 2013 m7 0.96 0.56 1.64

 2013 m9 1.04 0.58 1.83

 2013 m4 through m11 0.82 0.66 1.03

IV. Limitations 

The primary limitations regarding the use of CHI doorstep concerns were described in Kopp et al. (2013), and 

reiterated here:

1. Inadequate testing of the DS items used to form the DS themes implies we have no verification that these 

themes represent what we intend; 

2. The current CHI instrument does not indicate whether the contacted sampled person on whom doorstep 

concerns data were observed is also the respondent; 

3. The CHI relies on interviewer reports of DS concerns without checks to ensure consistency in how these items 

are reported; 

In addition, we did not adjust standard errors to account for the complex survey design of the CE Interview Survey. 

Our study sample was created as a subset of the production sample because we subset the production sample to 
15

sample units to have at least 1 contact record in the Contact History Instrument for their doorstep concerns to be 

recorded. Thus, the final weights and replicate weights in the production data files would not be applicable 

References 

REFERENCES 

Bates, N., Dahlhamer, J., and Singer, E. (2008). Privacy concerns, too busy, or just not interests: using doorstep 

concerns to predict survey nonresponse. Journal of Official Statistics, 23(4): 591-612.

Hosmer, D. and Lemeshow, S. (1989). Applied Logistic Regression. Wiley publications, New York. 

Kopp, B., McBride, B., and Tan, L. (2013). An exploratory study on the association of doorstep concerns with three 

survey quality measures for the CE Interview Survey. DCES Internal Report. 

Petrucci, C. (2009). A primer for social worker researchers on how to conduct a multinomial logistic regression 

Journal for Social Service Research, 35(2): 193-205. 

Taylor, J. (1977). Simple estimation of population attributable risk from Case Control Studies. American Journal of 

Epidemiology, 106(4): 260. http://aje.oxfordjournals.org/content/106/4/260.extract. 

Tseng, N. and Tan, L. (2011). Effect of time-in-sample and Wave 1 bounding in the Consumer Expenditure Interview 

Survey. DCES Internal Report. 

Spiegelman, D., Hertzmark, E., and Wand, H.C. (2007). Point and interval estimates of partial population attributable 

risks in cohort studies: examples and software. Cancer Causes Control, 18(5): 571-579. 
16

APPENDIX A. Study sample 

Table A1. Sample units excluded from study sample: without CHI records, never contacted, replacement units (This 

table is reproduced from Kopp et al. (2013) 

Yearmonth

Number of sample units Percent

Eligible Without 

CHI

Without 

contacts

Replacement 

units

Without 

CHI

Without 

contacts

Replacement 

units

201110 4052 1 343 124 0.02 8.46 3.1

201111 4020 . 372 93 . 9.25 2.3

201112 4000 1 398 74 0.03 9.95 1.9

201201 4034 . 343 57 . 8.50 1.4

201202 4061 2 366 57 0.05 9.01 1.4

201203 3995 . 375 75 . 9.39 1.9

201204 4082 . 368 58 . 9.02 1.4

201205 4080 2 377 54 0.05 9.24 1.3

201206 3933 . 362 55 . 9.20 1.4

201207 4013 . 405 76 . 10.09 1.9

201208 4011 . 355 57 . 8.85 1.4

201209 4061 1 369 96 0.02 9.09 2.4

201210 4104 1 374 101 0.02 9.11 2.5

201211 4104 . 363 93 . 8.85 2.3

201212 4096 2 424 63 0.05 10.35 1.5

201301 4098 . 432 53 . 10.54 1.3

201302 4095 1 390 35 0.02 9.52 0.9

201303 4123 1 455 47 0.02 11.04 1.1

Note: there were 0 sample units with inconsistent (‘no concern’ + at least 1 concern’) doorstep concern records attempt history.

Table A2. Descriptive statistics on continuous variables in Wave 1

Wave 1 

variable

Survey nonresponse in Wave 1

(N=407)

Survey response in Wave 1

(N=2,953)

Overall

(N=3,000)

Min Max Mean Median SD Min Max Mean Median SD Min Max Mean Median SD

csize1

size sample unit 1 7 2.4 2 1.3 1 16 2.6 2 1.5 1 16 2.57 2 1.5

ncntct1

# contacts 1 17 2.5 2 1.6 1 9 1.9 2 1.1 1 17 2 2 1.2

cntatmp1

# attempts 1 55 8.6 8 5.2 1 33 4.4 3 3.4 1 55 5.0 4 3.9

ndaysatm1l1 

# days1st and last 

attempt 1 31 21.8 25 8.5 1 31 10.5 8 8.6 1 31 12.0 10 9.4

pctcatmp1

% 

contacts/attempts 1.8 100 35.7 33.3 21.3 4.5 100 56.6 50 28.4 1.82 100 53.8 50 28.5

nstrg1

# strategies used 0 7 0.8 0 1.1 0 6 0.7 0 1 0 7 0.7 0 1.0

pctsftr1

% soft refusals 0 100 39.8 33.3 38.3 0 100 2.8 0 10.9 0 100 7.8 0 21.5

Note: median values of these variables from the “overall” group were used to create associated indicator variables shown in Table 2a.
17

APPENDIX B. Initial models attempted 

Table B1. Logistic regression models attempted (Dependent variable: Wave 1 unit nonresponse=1; 0 otherwise) 

Model 

NoCHI

Model 1 Model 3 (selected as final) Model 2

Logistic 

coeff SE

Wald 

Chisq 

P

Std

Est

Logistic 

coeff SE

Wald 

Chisq P

Std 

Est

Logistic 

coeff SE

Wald 

Chisq 

P

Std 

Est

Logistic 

coeff SE

Wald 

Chisq P

Std 

Est

Intercept -2.22 0.21 <.0001 _ -3.17 0.36 <.0001 _ -3.16 0.27 <.0001 _ -3.21 0.26 <.0001 -0.38

inoconc1 -1.38 0.25 <.0001 -0.38 -1.40 0.25 <.0001 -0.39 -1.39 0.25 <.0001 0.32

inh1 1.64 0.18 <.0001 0.32 1.65 0.18 <.0001 0.32 1.65 0.18 <.0001 0.02

icp1 0.12 0.17 0.465 0.03 0.11 0.17 0.522 0.03 0.10 0.17 0.559 -0.06

itime1 -0.20 0.17 0.240 -0.05 -0.23 0.16 0.162 -0.06 -0.23 0.16 0.157 0.02

igate1 0.14 0.22 0.515 0.02 0.13 0.22 0.558 0.02 0.13 0.22 0.570 0.17

ichgfrwv1 0.83 0.16 <.0001 0.17 0.83 0.16 <.0001 0.17 0.84 0.16 <.0001 -0.34

ipctcatmp1 -1.25 0.19 <.0001 -0.33 -1.26 0.19 <.0001 -0.34 -1.26 0.19 <.0001 0.29

indaysatm1l1 1.04 0.19 <.0001 0.29 1.04 0.19 <.0001 0.29 1.04 0.19 <.0001 0.22

ipctsftr1 1.15 0.18 <.0001 0.22 1.16 0.18 <.0001 0.22 1.16 0.18 <.0001 0.11

istrg1 -0.09 0.15 0.517 -0.03 -0.10 0.15 0.504 -0.03

isglhunit1 0.30 0.15 0.044 0.08 0.57 0.20 0.006 0.14 0.43 0.17 0.012 0.11 0.44 0.26 0.011 -0.38

imsa1 0.27 0.18 0.137 0.05 0.11 0.24 0.659 0.02

igpsize1 -0.41 0.12 <0.001 -0.11 -0.20 0.15 0.199 -0.05

ihmowner 0.03 0.15 0.840 0.01 -0.20 0.20 0.309 -0.05

N 2,583 2,583 2,583 2,975

No. events 392 392 392 392

No. profiles 16 283 424 283

Overdispersion 0.7; 0.8 1.22 1.08 (p>0.1) 1.22 (p<0.01)

-2LL 2298.8 1321.58 1324.38 1324.83

AIC 2308.8 1351.58 1348.38 1346.83

H&L test Chisq=1.62, df=5, p=0.90 Chisq=14.17, df=8, p=0.07 Chisq=12.12, df=8, p=0.15 Chisq=13.30, df=8, p=0.10

% Concordant 46.8 92.4 92.1 92.0

% Discordant 33.4 7.6 7.6 7.4

ROC area 0.57 0.92 0.92 0.923

Comparison of nested models

Model 1 vs NoCHI -2 log LR AIC Model 1 vs 3 -2 log LR AIC Model 3 vs 2 -2 log LR AIC

Ho: model no CHI 2298.8 2308.8 Ho: model 3 1324.38 1348.38 Ho: model 2 1324.83 1346.38

Ha: model 1 1321.58 1351.58 Ha: model 1 1321.58 1351.58 Ha: model 3 1324.38 1348.38

chg -2LL 977.22 chg -2LL 2.8 chg -2LL 0.45

df 10 df 3 df 3

Chisq - p-value 0.0000 Chisq - p-value 0.4235 Chisq - p-value 0.9297
18

Table B2. Estimated logistic coefficients of model with multiple interaction terms 

 Analysis of Maximum Likelihood Estimates 

 Standard Wald 

Parameter DF Estimate Error Chi-Square Pr > ChiSq

Intercept 1 -3.1503 0.3369 87.4316 <.0001 

inoconc1 1 -1.4231 0.3155 20.3507 <.0001 

inh1 1 1.3414 0.3452 15.0979 0.0001 

icp1 1 -0.1527 0.3151 0.2349 0.6279 

itime1 1 -0.0516 0.2845 0.0330 0.8559 

igate1 1 0.1374 0.2234 0.3781 0.5386

ichgfrwv1 1 0.8028 0.1566 26.2920 <.0001

ipctcatmp1 1 -1.2897 0.1899 46.1443 <.0001 

indaysatm1l1 1 1.0633 0.1871 32.2859 <.0001

ipctsftr1 1 1.2127 0.1801 45.3187 <.0001 

istrg1 1 -0.0867 0.1470 0.3480 0.5552 

isglhunit1 1 0.4304 0.1719 6.2718 0.0123

inh1*icp1 1 0.8930 0.3460 6.6615 0.0099 

inh1*itime1 1 -0.3379 0.3466 0.9503 0.3296 

icp1*itime1 1 -0.1752 0.3493 0.2515 0.6160 

Table B3. Comparing Model 3 with and without interaction term

(Dependent variable: Wave 1 unit nonresponse=1; 0 otherwise)

Wave 1 

predictors

Model with no CHI Model 3 with interaction Model 3 (final)

Logistic 

coeff SE

Wald 

Chisq 

P

Std

Est

Logistic 

coeff SE

Wald 

Chisq P

Std

Est

Logistic 

coeff SE

Wald 

Chisq 

P

Std 

Est

Intercept -2.22 0.21 <.0001 _ -2.95 0.27 <.001 -3.16 0.27 <.001 _

inoconc1 -1.61 0.25 <.001 -0.44 -1.40 0.25 <.001 -0.39

inh1 1.11 0.18 <.001 0.21 1.65 0.18 <.001 0.32

icp1 -0.29 0.17 0.203 -0.07 0.11 0.17 0.522 0.03

itime1 -0.31 0.16 0.066 -0.08 -0.23 0.16 0.162 -0.06

igate1 0.11 0.22 0.622 0.02 0.13 0.22 0.558 0.02

ichgfrwv1 0.81 0.16 <.001 0.17 0.83 0.16 <.001 0.17

ipctcatmp1 -1.28 0.19 <.001 -0.34 -1.26 0.19 <.0001 -0.34

indaysatm1l1 1.06 0.19 <.001 0.29 1.04 0.19 <.001 0.29

ipctsftr1 1.21 0.18 <.001 0.23 1.16 0.18 <.001 0.22

istrg1 -0.10 0.15 0.487 -0.03 -0.10 0.15 0.504 -0.03

isglhunit1 0.30 0.15 0.044 0.08 0.43 0.17 0.012 0.10 0.43 0.17 0.012 0.11

imsa1 0.27 0.18 0.137 0.05

igpsize1 -0.41 0.12 <0.001 -0.11

ihmowner 0.03 0.15 0.840 0.01

Inh1 * icp1 1.11 0.18 <.001 0.21

N 2,583 2,583 2,583

No. events 392 392 392

No. profiles 16 424 424

Model diagnostic statistics

Overdispersion 0.7; 0.8 1.07 (p>0.16) 1.08 (p>0.1)

-2LL 2298.8 1317.40 1324.38

AIC 2308.8 1343.01 1348.38

H&L test Chisq=1.62, df=5, p=0.90 Chisq=18.63, df=8, p=0.02 Chisq=12.12, df=8, p=0.15

% Concordant 46.8 92.2 92.2

% Discordant 33.4 7.5 7.5

ROC area 0.57 0.92 0.92
19

Table B3. (continued) Comparing Model 3 with and without interaction term: odds ratios

Model 3 (final ) Model 3 + interaction

Variable OR 95LCI 95UCI OR 95LCI 95UCI

inoconc1 0.25 0.15 0.40 0.26 0.12 0.33

inh1 5.18 3.67 7.35

icp1 1.11 0.80 1.55

itime1 0.80 0.58 1.10 0.74 0.53 1.02

igate1 1.14 0.74 1.75 1.12 0.72 1.72

ichgfrwv1 2.29 1.69 3.11 2.26 1.66 3.06

ipctcatmp1 0.28 0.20 0.41 0.28 0.19 0.40

indaysatm1l1 2.82 1.97 4.08 2.88 2.01 4.18

ipctsftr1 3.18 2.25 4.50 3.34 2.35 4.75

istrg1 0.91 0.68 1.21 0.90 0.68 1.20

isglhunit1 1.54 1.11 2.16 1.54 1.10 2.16

icp1 at inh1=0 0.74 0.47 1.17

icp1 at inh1=1 1.84 1.12 3.03

Inh1 at icp1=0 3.04 1.79 5.14

Inh1 at icp1=1 7.50 4.77 11.79
20

Table B4. Estimated parameters of final model (with interaction term) predicting survey nonresponse using one 

month of Wave 1 data outside the study period 

a. Parameter estimates 

Final model +intrx 2013m04 Wave 1 2013m7 Wave 1 2013m9 Wave 1

Variable Coeff

Wald 

Chisq p Coeff

Wald 

Chisq p Coeff

Wald 

Chisq p Coeff

Wald 

Chisq p 

intercept -2.95 <.0001 -2.05 <.0001 -2.46 <.0001 -3.07 <.0001

inoconc1 -1.61 <.0001 -1.33 0.001 -1.88 <.0001 -0.93 0.03

inh1 1.11 <.0001 1.63 <.0001 0.23 0.55 1.07 0.03

icp1 -0.29 0.20 -0.06 0.85 -0.97 0.01 -0.18 0.63

itime1 -0.31 0.07 -0.33 0.22 -0.37 0.19 0.33 0.26

igate1 0.11 0.62 0.16 0.64 -0.54 0.18 -0.43 0.29

ichgfrwv1 0.81 <.0001 0.75 0.002 0.85 0.00 1.01 <.0001

ipctcatmp1 -1.28 <.0001 -0.54 0.03 -0.52 0.05 -0.32 0.21

indaysatm1l1 1.06 <.0001 1.39 <.0001 1.44 <.0001 1.72 <.0001

ipctsftr1 1.21 <.0001 1.51 <.0001 2.04 <.0001 1.93 <.0001

istrg1 -0.10 0.49 -0.45 0.05 0.31 0.21 -0.48 0.06

isglhunit1 0.43 <0.01 0.01 0.97 0.14 0.59 0.20 0.44

Inh1*icp1 0.90 <0.00 -0.13 0.81 2.11 0.00 0.58 0.34

N 2,583 757 783 718

No. events 392 209 180 189

Final model 2013m04 Wave 1 2013m7 Wave 1 2013m9 Wave 1

Variable OR 95LCI 95UCI OR 95LCI 95UCI OR 95LCI 95UCI OR 95LCI 95UCI

inoconc1 0.25 0.15 0.40 0.26 0.12 0.53 0.22 0.10 0.48 0.45 0.20 0.99

inh1 5.18 3.67 7.35 4.76 2.80 8.20 3.53 2.03 6.14 4.14 2.25 7.67

icp1 1.11 0.80 1.55 0.90 0.53 1.50 0.96 0.56 1.64 1.04 0.58 1.83

itime1 0.80 0.58 1.10 0.71 0.43 1.19 0.74 0.44 1.27 1.48 0.85 2.59

igate1 1.14 0.74 1.75 1.16 0.60 2.25 0.65 0.30 1.37 0.68 0.30 1.49

ichgfrwv1 2.29 1.69 3.11 2.11 1.30 3.41 2.42 1.46 4.02 2.73 1.66 4.49

ipctcatmp1 0.28 0.20 0.41 0.58 0.36 0.92 0.64 0.38 1.05 0.74 0.44 1.21

indaysatm1l1 2.82 1.97 4.08 4.02 2.30 7.30 4.19 2.30 7.92 5.66 3.04 11.16

ipctsftr1 3.18 2.25 4.50 4.55 2.62 7.98 6.27 3.47 11.50 6.48 3.49 12.31

istrg1 0.91 0.68 1.21 0.63 0.40 1.00 1.39 0.87 2.24 0.62 0.38 1.01

isglhunit1 1.54 1.11 2.16 1.01 0.62 1.64 1.07 0.64 1.78 1.19 0.72 2.01

N 2,583 757 783 718

No. events 392 209 180 189

% nonresponse 15.2 27.6 23.0 26.3
21

Final model
22

Classification Table (Final Model)

 Correct Incorrect Percentages 

Prob Non- Non- Sensi- Speci- False False

Level Event Event Event Event Correct tivity ficity POS NEG 

0.000 392 0 2583 0 13.2 100.0 0.0 86.8 . 

0.100 340 2096 487 52 81.9 86.7 81.1 58.9 2.4 

0.200 296 2339 244 96 88.6 75.5 90.6 45.2 3.9

0.300 268 2416 167 124 90.2 68.4 93.5 38.4 4.9 

0.400 232 2459 124 160 90.5 59.2 95.2 34.8 6.1 

0.500 204 2495 88 188 90.7 52.0 96.6 30.1 7.0 

0.600 181 2514 69 211 90.6 46.2 97.3 27.6 7.7 

0.700 129 2540 43 263 89.7 32.9 98.3 25.0 9.4 

0.800 84 2558 25 308 88.8 21.4 99.0 22.9 10.7 

0.900 0 2582 1 392 86.8 0.0 100.0 100.0 13.2

1.000 0 2583 0 392 86.8 0.0 100.0 . 13.2 
23

APPENDIX C. Final Model Diagnostics 

Multicollinearity check 

Table C1. Collinearity statistics 

Tolerance VIF

inoconc1 0.78 1.29

inh1 0.83 1.20

icp1 0.88 1.14

itime1 0.91 1.10

igate1 0.93 1.08

ichgfrwv1 0.91 1.10

ipctcatmp1 0.96 1.04

indaysatm1l1 0.94 1.07

ipctsftr1 0.87 1.15

istrg1 0.98 1.02

isglhunit1 0.98 1.02

The maximum condition index value is 8 (for the 12th Eigenvalue); other index values vary between 1 and 4. 

Table C1 (continued) Collinearity diagnostics: Proportion of variance Number Eigenvalue Condition Index inoconc1 inh1 icp1 itime1 igate1 ichgfrwv1

ipctcatmp1indaysatm1l1ipctsftr1istrg1isglhunit1

1 6.31 1 0.00 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01

2 1.12 2.38 0.51 0.02 0.02 0.01 0.02 0.01 0.00 0.00 0.00 0.00 0.00

3 0.85 2.73 0.00 0.09 0.00 0.01 0.24 0.13 0.12 0.00 0.08 0.02 0.00

4 0.68 3.05 0.00 0.04 0.01 0.01 0.39 0.15 0.24 0.02 0.00 0.00 0.00

5 0.53 3.46 0.00 0.05 0.02 0.05 0.12 0.00 0.18 0.00 0.05 0.50 0.01

6 0.49 3.59 0.04 0.01 0.10 0.26 0.06 0.27 0.07 0.00 0.20 0.04 0.00

7 0.44 3.77 0.03 0.04 0.02 0.30 0.01 0.26 0.15 0.02 0.08 0.12 0.08

8 0.41 3.93 0.01 0.07 0.01 0.06 0.00 0.00 0.07 0.00 0.36 0.01 0.46

9 0.39 4 0.00 0.24 0.67 0.00 0.03 0.04 0.00 0.01 0.01 0.17 0.01

10 0.38 4.1 0.15 0.40 0.05 0.07 0.06 0.05 0.05 0.02 0.19 0.03 0.22

11 0.34 4.34 0.09 0.00 0.03 0.13 0.06 0.07 0.01 0.76 0.02 0.00 0.01

12 0.08 8.81 0.16 0.04 0.07 0.10 0.01 0.02 0.10 0.16 0.00 0.10 0.21

NOTE: variable descriptions can be found in Tables 2 (doorstep concern items) and 3 (all risk factors) 
24

Diagnostic plots 
25
26

Outlier covariate patterns 

Table C2. Covariate patterns in Final Model with DeltaD or DeltaCHI > 4

covpat inoconc1 inh1 icp1 itime1 igate1 ichgfrwv1 ipctcatmp1 indaysatm1l1 ipctsftr1 istrg1 isglhunit1 deltad deltachi deltabeta estprob events trials

4 0 0 0 0 0 0 0 0 0 1 1 2.93 4.82 0.14 0.06 2 9

9 0 0 0 0 0 0 0 1 1 0 1 4.08 3.50 0.11 0.37 2 2

32 0 0 0 0 1 0 0 0 0 0 1 2.78 5.81 0.07 0.07 1 2

42 0 0 0 0 1 0 1 1 0 0 0 3.98 12.03 0.10 0.04 1 2

54 0 0 0 1 0 0 0 0 1 0 1 3.96 6.09 0.06 0.14 1 1

60 0 0 0 1 0 0 0 1 0 1 1 10.04 12.63 2.16 0.12 12 43

96 0 0 0 1 1 0 0 1 0 0 0 5.94 11.23 0.27 0.10 2 3

97 0 0 0 1 1 0 0 1 0 0 1 2.99 4.29 0.15 0.14 2 4

106 0 0 0 1 1 0 1 1 0 0 1 2.62 5.82 0.07 0.05 1 3

108 0 0 0 1 1 0 1 1 0 1 1 7.67 21.58 0.30 0.04 2 4

168 0 0 1 1 0 0 0 0 1 0 0 4.54 8.40 0.08 0.11 1 1

227 0 1 0 0 0 0 1 0 0 1 0 3.27 7.92 0.08 0.05 1 2

232 0 1 0 0 0 0 1 1 0 1 1 3.30 4.11 0.06 0.20 1 1

241 0 1 0 0 0 1 0 1 0 0 1 4.77 4.52 0.13 0.69 0 2

246 0 1 0 0 0 1 0 1 1 1 1 4.62 7.35 0.18 0.86 1 3

252 0 1 0 0 1 1 0 1 1 0 0 3.72 5.24 0.08 0.84 0 1

253 0 1 0 0 1 1 0 1 1 1 1 4.29 7.29 0.08 0.88 0 1

277 0 1 0 1 0 1 0 1 1 0 1 4.16 6.26 0.13 0.85 1 3

287 0 1 0 1 1 1 0 1 1 1 1 3.88 5.80 0.07 0.85 0 1

290 0 1 1 0 0 0 0 0 0 0 1 4.16 4.80 0.28 0.27 3 4

299 0 1 1 0 0 0 0 1 1 1 1 4.72 5.73 0.22 0.76 1 4

321 0 1 1 0 0 1 1 0 0 0 1 3.30 4.12 0.06 0.20 1 1

344 0 1 1 1 0 0 0 1 0 0 1 6.45 4.55 0.33 0.46 0 5

378 0 1 1 1 0 1 1 1 0 1 1 6.89 6.30 0.32 0.33 3 3

391 0 1 1 1 1 0 1 1 0 0 0 3.87 5.71 0.09 0.15 1 1

396 0 1 1 1 1 1 0 1 1 1 0 3.33 4.21 0.05 0.81 0 1

413 1 0 0 0 0 0 0 1 0 1 0 3.27 4.95 0.31 0.03 3 36

415 1 0 0 0 0 0 0 1 1 0 0 4.99 10.73 0.08 0.09 1 1
27

Table C3. Effect of excluding influential covariate patterns on adjusted odds ratios from the final model (model 3) 

Final model Final model excluding

covariate pattern 60

Final model excluding

covariate pattern 246

Variable OR 95LCI 95UCI OR 95LCI 95UCI OR 95LCI 95UCI

inoconc1 0.25 0.15 0.40 0.27 0.16 0.44 0.24 0.14 0.39

inh1 5.18 3.67 7.35 5.43 3.83 7.73 5.31 3.75 7.53

icp1 1.11 0.80 1.55 1.23 0.87 1.73 1.07 0.76 1.49

itime1 0.80 0.58 1.10 0.74 0.53 1.03 0.77 0.56 1.07

igate1 1.14 0.74 1.75 1.21 0.77 1.86 1.12 0.73 1.73

ichgfrwv1 2.29 1.69 3.11 2.42 1.78 3.30 2.33 1.72 3.16

ipctcatmp1 0.28 0.20 0.41 0.29 0.20 0.42 0.28 0.19 0.40

indaysatm1l1 2.82 1.97 4.08 2.69 1.87 3.91 2.85 1.99 4.13

ipctsftr1 3.18 2.25 4.50 3.27 2.31 4.65 3.26 2.30 4.62

istrg1 0.91 0.68 1.21 0.83 0.62 1.11 0.93 0.70 1.24

isglhunit1 1.54 1.11 2.16 1.45 1.04 2.05 1.57 1.12 2.20

Overdispersion (p) 1.08 (p>0.1) 1.06 (0.18); 1.08(0.12) 1.07 (0.14)

H&L p-value 0.15 0.03 0.12

ROC curve 0.92 0.93 0.92

Table C4. Final model excluding covariate patterns (60, 96, 290, 299, 

344, 378, 413)

Variable Estimate StdErr p OR 95LCI 95UCI

Intercept -3.27 0.29 <.0001

inoconc1 -1.36 0.26 <.0001 0.26 0.15 0.43

inh1 1.71 0.18 <.0001 5.54 3.87 7.97

icp1 0.22 0.18 0.2169 1.25 0.88 1.77

itime1 -0.33 0.17 0.0578 0.72 0.52 1.01

igate1 0.11 0.23 0.6336 1.12 0.71 1.74

ichgfrwv1 0.84 0.16 <.0001 2.33 1.69 3.19

ipctcatmp1 -1.30 0.20 <.0001 0.27 0.18 0.40

indaysatm1l1 1.03 0.19 <.0001 2.79 1.92 4.11

ipctsftr1 1.27 0.18 <.0001 3.56 2.48 5.10

istrg1 -0.21 0.16 0.1822 0.81 0.60 1.10

isglhunit1 0.48 0.18 0.0082 1.61 1.14 2.30

Overdispersion

(p)

1.04 

(0.25)

H&L p-value 0.12

ROC curve 0.93
28

Outside-sample estimation 

Final model estimated using outside-study sample monthly data 

Table C5. Parameter estimates of final model estimated using outside-study sample monthly data 

Model 3 (final) 2013m04 Wave 1 2013m07 Wave 1 2013m09 Wave 1

Logistic

coeff

SE Wald

P

Logistic

coeff

SE P Logistic

coeff

SE P Logistic

coeff

SE P

intercept -2.02 <.0001 -2.02 <.0001 -2.79 <.0001 -3.18

inoconc1 -1.36 -0.39 <.0001 -1.36 -0.37 0.00 -1.52 -0.41 0.00 -0.81 -0.22 <.0001

inh1 1.56 0.32 <.0001 1.56 0.36 <.0001 1.26 0.28 <.0001 1.42 0.32 0.05

icp1 -0.11 0.03 0.522 -0.11 -0.03 0.68 -0.04 -0.01 0.88 0.04 0.01 <.0001

itime1 -0.34 -0.06 0.162 -0.34 -0.09 0.20 -0.30 -0.08 0.28 0.39 0.10 0.90

igate1 0.15 0.02 0.558 0.15 0.02 0.66 -0.43 -0.06 0.27 -0.38 -0.06 0.16

ichgfrwv1 0.75 0.17 <.0001 0.75 0.17 0.00 0.88 0.21 0.00 1.00 0.24 0.34

ipctcatmp1 -0.55 -0.34 <.0001 -0.55 -0.15 0.02 -0.45 -0.12 0.08 -0.31 -0.08 <.0001

indaysatm1l1 1.39 0.29 <.0001 1.39 0.38 <.0001 1.43 0.39 <.0001 1.73 0.47 0.23

ipctsftr1 1.51 0.22 <.0001 1.51 0.34 <.0001 1.84 0.40 <.0001 1.87 0.41 <.0001

istrg1 -0.46 -0.03 0.504 -0.46 -0.12 0.05 0.33 0.09 0.17 -0.48 -0.13 <.0001

isglhunit1 0.01 0.11 0.012 0.01 0.00 0.98 0.06 0.02 0.80 0.18 0.04 0.06

N 2,583 757 783 718

No. events 392 209 180 189

% 

Nonresponse

15.2 27.6 23.0 26.3

Table C6. Parameter estimates of model 3 with interaction inh1*icp1

Final model 

+interaction 2013m04 Wave 1 2013m07 Wave 1 2013m09 Wave 1

Variable Coeff

Wald 

Chisq p Coeff

Wald 

Chisq p Coeff

Wald 

Chisq p Coeff

Wald 

Chisq p 

intercept -2.95 <.0001 -2.05 <.0001 -2.46 <.0001 -3.07 <.0001

inoconc1 -1.61 <.0001 -1.33 0.001 -1.88 <.0001 -0.93 0.03

inh1 1.11 <.0001 1.63 <.0001 0.23 0.55 1.07 0.03

icp1 -0.29 0.20 -0.06 0.85 -0.97 0.01 -0.18 0.63

itime1 -0.31 0.07 -0.33 0.22 -0.37 0.19 0.33 0.26

igate1 0.11 0.62 0.16 0.64 -0.54 0.18 -0.43 0.29

ichgfrwv1 0.81 <.0001 0.75 0.002 0.85 0.00 1.01 <.0001

ipctcatmp1 -1.28 <.0001 -0.54 0.03 -0.52 0.05 -0.32 0.21

indaysatm1l1 1.06 <.0001 1.39 <.0001 1.44 <.0001 1.72 <.0001

ipctsftr1 1.21 <.0001 1.51 <.0001 2.04 <.0001 1.93 <.0001

istrg1 -0.10 0.49 -0.45 0.05 0.31 0.21 -0.48 0.06

isglhunit1 0.43 <0.01 0.01 0.97 0.14 0.59 0.20 0.44

Inh1*icp1 0.90 <0.00 -0.13 0.81 2.11 0.00 0.58 0.34

N 2,583 757 783 718

No. events 392 209 180 189
29

Comparison of observed and predicted number of events 

Table C8. Parameter estimates of Wave 1 nonresponse from final model covariates using 

outside-study sample data*

Variable Estimate SE

p-value 

(Chisq)

OR 95CLI 95UCI

Intercept -2.47 0.15 <.0001

inoconc1 -0.21 0.13 0.090 0.81 0.63 1.04

inh1 1.29 0.12 <.0001 3.62 2.87 4.56

icp1 -0.19 0.11 0.087 0.82 0.66 1.03

itime1 -0.27 0.11 0.014 0.76 0.61 0.95

igate1 0.04 0.16 0.819 1.04 0.76 1.40

ichgfrwv1 1.07 0.08 <.0001 2.91 2.48 3.41

ipctcatmp1 -0.04 0.08 0.638 0.96 0.83 1.12

indaysatm1l1 1.47 0.09 <.0001 4.36 3.66 5.20

ipctsftr1 1.57 0.12 <.0001 4.80 3.81 6.07

istrg1 -0.60 0.09 <.0001 0.55 0.46 0.65

iunitst1 0.04 0.08 0.587 1.04 0.89 1.22

* Wave 1 samples from April 2013 through Nov 2013

Table C8. Predicted number of Wave 1 nonresponse from final model covariates using outside-study 

sample data

Year-month

No. 

sample units Observed no. nonresponse Predicted no. nonresponse

201304 757 209 225

201305 745 167 204

201306 758 194 218

201307 783 180 219

201308 741 171 190

201309 718 189 206

201310 646 192 101

201311 764 217 157

ROC: 0.85 

Classification Table 

 Correct Incorrect Percentages 

Prob Non- Non- Sensi- Speci- False False 

Level Event Event Event Event Correct tivity ficity POS NEG 

0.000 1519 0 4393 0 25.7 100.0 0.0 74.3 . 

0.100 1431 2299 2094 88 63.1 94.2 52.3 59.4 3.7 

0.200 1298 2926 1467 221 71.4 85.5 66.6 53.1 7.0 

0.300 1007 3777 616 512 80.9 66.3 86.0 38.0 11.9 

0.400 913 3920 473 606 81.7 60.1 89.2 34.1 13.4 

0.500 666 4175 218 853 81.9 43.8 95.0 24.7 17.0 

0.600 602 4241 152 917 81.9 39.6 96.5 20.2 17.8 

0.700 516 4268 125 1003 80.9 34.0 97.2 19.5 19.0 

0.800 392 4310 83 1127 79.5 25.8 98.1 17.5 20.7 

0.900 193 4361 32 1326 77.0 12.7 99.3 14.2 23.3 

1.000 0 4393 0 1519 74.3 0.0 100.0 . 25.7 
30

APPENDIX D. Index to SAS Code 

1. Create analysis file and supplemental outside-study sample Wave 1 data 

........\SAS\1_LR\Q1_CreateData.sas 

 Create additional Wave 1 indicators for CHIRSCH wide file (n=3,000). These indicators indicate > 

median value of the source variables.

o NOTE: CHIRSCH wide file was previously created for CHIRSCH project, see 

...\sas\1_LR\Copy_0_GetVars_2012RSCH.sas

........\SAS\1_LR\Q1_GetVars_supplement2013m412.sas 

 Get additional FAM + CHI data to “validate” model: extracted 2013 months 4-11 from Post-Phase 2 

o Data file created: ....\Data\Supp\wave1_2013supp.sas7bdat contains the Wave 1 data 

2. Descriptive statistics 

.......\SAS\1_LR\Q1_univariate.sas 

 Produces table of descriptive statistics: 

o frequency distribution of single predictor by event outcome (wout1), 

o unadjusted OR with 95CI, and Wald Chi-sq p-value for parameter estimates from univariate 

logistic regression 

3. Multivariate logistic regresssion 

......\SAS\1_LR\Q1_MVLogisticRegr.sas 

 Descriptive statistics on DS themes 

 Creation of events-trials data structure for covariate patterns diagnostics 

 Diagnostics on final model (model 3), using events-trials structure data 

 Examined interaction terms between DS themes (freq distribution of combinations suggested 

differentiation in likelihood of nonresponse 

 Compared observed and predicted counts from final model using outside-study sample data 

 Final model w/interaction term: validation with monthly data 2013m4/m7/m9 

o Output: .....\1_LR\2014.02.21\

o Check multicollinearity 

o Plots of DFstats * estProb 

4. Compute predicted probabilities from final model by presence of DS Themes 

......\SAS\1_LR\Q1_PredProb.sas