1 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

How Different are “Not Eligible” Cases and the 

ATUS Sample: A Consideration of Demographic 

Factors and Data Quality Indicators

David Biagas

Office of Survey Methods Research

Survey Methods Symposium

July 16th, 2019
2 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

A Loss of Cases

 ATUS response rates steadily declining

49.9% (2013) to 45.6% (2017)

 Cases of low quality on ATUS are currently 

flagged as not eligible (NE) for estimation 

purposes

Between 3.46% in 2014 to 4.31% in 2017

Fewer than five activities

More than three hours of time coded as “gap/can’t 

remember” or “refusal/none of your business”
3 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

The Present Study

 Compare the characteristics of eligible and NE 

respondents:

Demographic Characteristics 

– (e.g., labor force status, race, age, sex, income)

Data Quality and Collection Strategies

– (e.g., % income imputed, week of response, weekday/weekend assignment, 

number of call attempts)

Activities Reported 

– (comparison of reported activities)
4 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

The Present Study

 Compare the characteristics of eligible and NE 

respondents:

Demographic Characteristics 

– (e.g., labor force status, race, age, sex, income)

Data Quality and Collection Strategies

– (e.g., % income imputed, week of response, weekday/weekend assignment, 

number of call attempts)

Activities Reported 

– (comparison of reported activities)
5 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Sample and Analytic Strategy

 ATUS from 2013-2017

 n=56,809

 2,211 flagged for exclusion (3.89%) 

 Parametric Tests:

Compare NE Cases to Eligible Cases

 Descriptive Statistics:

Compare subgroups 
6 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Evaluation of NE Criteria

 Operationalized criteria:

Inactive Subgroup (<5 activities)

Forgetful Subgroup (3+ hours “don’t know”)

Reluctant Subgroup (3+ hours “refusal”)

 Some (minimal) overlap between subgroups
7 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Table 1. Demographic Characteristics of NE and Eligible ATUS Cases

Demographic Factor NE Cases 

(n=2,211)

ATUS Eligible 

(n=54,598) t-statistic or χ2

Age (years) 58.07 48.88 23.51***

Female (%) 59.16 55.51 11.43***

Race/Ethnicity (%) 98.83***

White 57.26 64.95

Black 21.71 14.35

Hispanic, Any Race 14.97 14.97

Asian 4.07 3.92

American Indian 0.63 0.54

Hawaiian/Pacific Islander 0.18 0.20

Bi or Multiracial 1.18 1.06

Marital Status (%) 326.05***

Married 39.94 48.62

Divorced/Separated/Never 39.30 42.11

Widowed 20.76 9.27

Presence of Minor in HH (%) 20.17 32.46 147.44***

Household Size 2.20 2.62 -12.95***
8 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Results: Demographic Comparison Table 1. Demographic Characteristics of NE and Eligible ATUS Cases

Demographic Factor NE Cases 

(n=2,211)

ATUS Eligible 

(n=54,598) t-statistic or χ2

Age (years) 58.07 48.88 23.51***

Female (%) 59.16 55.51 11.43***

Race/Ethnicity (%) 98.83***

White 57.26 64.95

Black 21.71 14.35

Hispanic, Any Race 14.97 14.97

Asian 4.07 3.92

American Indian 0.63 0.54

Hawaiian/Pacific Islander 0.18 0.20

Bi or Multiracial 1.18 1.06

Marital Status (%) 326.05***

Married 39.94 48.62

Divorced/Separated/Never 39.30 42.11

Widowed 20.76 9.27

Presence of Minor in HH (%) 20.17 32.46 147.44***

Household Size 2.20 2.62 -12.95***
9 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Table 2. Demographic Characteristics by Subgroup

Demographic Factor

Inactive 

Group

(n=365)

Forgetful 

Group

(n=1,375)

Reluctant 

Group

(n=544)

ATUS 

Eligible

Age (years) 52.99 61.24 53.39 48.88

Female (%) 47.12 62.33 57.54 55.51

Race/Ethnicity (%)

White 50.41 60.51 50.92 64.95

Black 30.14 19.85 21.51 14.35

Hispanic, Any Race 13.97 13.75 20.96 14.97

Asian 1.64 4.22 4.78 3.92

American Indian 1.10 0.51 0.55 0.54

Hawaiian/Pac Isl 0.55 0.00 0.55 0.20

Bi or Multiracial 2.19 1.16 0.74 1.06

Marital Status (%)

Married 36.16 38.84 45.59 48.62

Div/Sep/Nev 52.05 35.85 39.71 42.11

Widowed 11.78 25.31 14.71 9.27

Presence of Minor in HH (%) 18.90 17.82 26.10 32.46
10 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Table 2. Demographic Characteristics by Subgroup

Demographic Factor

Inactive 

Group

(n=365)

Forgetful 

Group

(n=1,375)

Reluctant 

Group

(n=544)

ATUS 

Eligible

Age (years) 52.99 61.24 53.39 48.88

Female (%) 47.12 62.33 57.54 55.51

Race/Ethnicity (%)

White 50.41 60.51 50.92 64.95

Black 30.14 19.85 21.51 14.35

Hispanic, Any Race 13.97 13.75 20.96 14.97

Asian 1.64 4.22 4.78 3.92

American Indian 1.10 0.51 0.55 0.54

Hawaiian/Pac Isl 0.55 0.00 0.55 0.20

Bi or Multiracial 2.19 1.16 0.74 1.06

Marital Status (%)

Married 36.16 38.84 45.59 48.62

Div/Sep/Nev 52.05 35.85 39.71 42.11

Widowed 11.78 25.31 14.71 9.27

Presence of Minor in HH (%) 18.90 17.82 26.10 32.46
11 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Table 3. Socioeconomic Status of NE Cases and the ATUS Sample 

Socioeconomic Factor NE Cases 

(n=2,211)

ATUS Eligible 

(n=54,598) t-statistic or χ2

Income (midpoint in $) $47,968 $63,467 -15.93***

Employment Status (%) 680.93***

Employed 34.64 60.56

Unemployed 2.67 3.99

Not in Labor Force 62.69 35.45

Educational Attainment (%) 232.51***

HS Dropout 20.99 13.15

HS Graduate 31.25 24.54

Some College 24.97 27.18

College Graduate 22.80 35.12
12 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Results: SES Comparison Table 3. Socioeconomic Status of NE Cases and the ATUS Sample 

Socioeconomic Factor NE Cases 

(n=2,211)

ATUS Eligible 

(n=54,598) t-statistic or χ2

Income (midpoint in $) $47,968 $63,467 -15.93***

Employment Status (%) 680.93***

Employed 34.64 60.56

Unemployed 2.67 3.99

Not in Labor Force 62.69 35.45

Educational Attainment (%) 232.51***

HS Dropout 20.99 13.15

HS Graduate 31.25 24.54

Some College 24.97 27.18

College Graduate 22.80 35.12
13 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Results: Subgroup SES

Table 4. Socioeconomic Status of Cases by Subgroup

Socioeconomic Factors

Inactive 

Group

(n=365)

Forgetful 

Group

(n=1,375)

Reluctant 

Group

(n=544)

ATUS 

Eligible

Income (midpoint in $)

$47,174 $46,667 $52,010 $63,467 

Employment Status (%)

Employed 43.56 28.29 44.49 60.56

Unemployed 2.74 3.35 0.92 3.99

Not in Labor Force 53.70 68.36 54.60 35.45

Educational Attainment (%)

HS Dropout 19.45 22.25 19.85 13.15

HS Graduate 32.88 31.78 30.15 24.54

Some College 25.75 23.42 26.29 27.18

College Graduate 21.92 22.55 23.71 35.12
14 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Results: Subgroup SES

Table 4. Socioeconomic Status of Cases by Subgroup

Socioeconomic Factors

Inactive 

Group

(n=365)

Forgetful 

Group

(n=1,375)

Reluctant 

Group

(n=544)

ATUS 

Eligible

Income (midpoint in $)

$47,174 $46,667 $52,010 $63,467 

Employment Status (%)

Employed 43.56 28.29 44.49 60.56

Unemployed 2.74 3.35 0.92 3.99

Not in Labor Force 53.70 68.36 54.60 35.45

Educational Attainment (%)

HS Dropout 19.45 22.25 19.85 13.15

HS Graduate 32.88 31.78 30.15 24.54

Some College 25.75 23.42 26.29 27.18

College Graduate 21.92 22.55 23.71 35.12
15 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Results: Activity Counts

Inactive

Forgetful Reluctant

ATUS

Median = 18
16 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Results: Activity Counts

Inactive

Forgetful Reluctant

ATUS

Median = 18 Median = 4
17 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Results: Activity Counts

Inactive

Forgetful Reluctant

ATUS

Median = 18 Median = 4

Median = 9
18 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Results: Activity Counts

Inactive

Forgetful Reluctant

ATUS

Median = 18 Median = 4

Median = 15 Median = 9
19 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Results: Top 10 Activities

Table 5. Comparison of Most Frequently Reported Activities among NE Cases and 

Eligible ATUS Cases

NE Cases % Eligible ATUS Cases % 

Sleeping 13.49 Sleeping 10.94

Eating and drinking 10.19 Eating and drinking 10.14

Gap/can't remember 7.56 Television and movies (not religious) 7.99

Television and movies (not religious) 7.54 Washing, dressing and grooming oneself 7.11

Washing, dressing and grooming oneself 5.76 Food and drink preparation 4.83

Food and drink preparation 4.74 Work, main job 4.03

Interior cleaning 2.16 Socializing and communicating with others 2.86

Refused/"none of your business" 2.15 Travel related to working 2.78

Reading for personal interest 2.06 Physical care for hh children 2.37

Socializing and communicating with others 2.01 Travel related to shopping, ex groceries, food, 

and gas 2.18

Note: Activities in bold differ between groups
20 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Results: Subgroup Activities Table 6. Comparison of Most Frequently Reported Activities among NE Subgroups

"Inactive" Group % "Forgetful" Group % "Reluctant" Group %

Sleeping 42.95 Sleeping 12.54 Sleeping 11.37

Television and movies 

(not religious) 15.94 Eating and drinking 10.38 Refused/"none of your business" 10.25

Eating and drinking 10.32 Gap/can't remember 9.68 Eating and drinking 9.33

Refused/"none of your 

business" 3.61 Television and movies (not religious) 7.70 Washing, dressing and grooming oneself 6.09

Health-related self 

care

3.44 Washing, dressing and grooming 

oneself 5.84 Television and movies (not religious) 5.10

Work, main job 3.36 Food and drink preparation 5.00 Food and drink 

preparation 4.32

Relaxing, thinking 2.77 Reading for personal interest 2.37 Work, main job 3.24

Gap/can't remember 2.43 Interior cleaning 2.24 Travel related to 

working 2.70

Washing, dressing and 

grooming oneself 2.01 Socializing and communicating with others 2.14 Interior cleaning 2.09

Travel related to working 1.26 Kitchen and food clean-up 1.87

Travel related to 

shopping (except 

grocery shopping)

1.90

Note: Activities in bold were not among the ten most frequently reported activities among eligible ATUS respondents
21 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Summary

 NE cases differ systematically from eligible 

cases (e.g., labor force status, age, ses, sex, race)

Modeled the probability of being flagged for ineligibility 

and found consistent results

Activities reported seem to differ and track closely with 

observed demographic differences (e.g., labor force status 

and minor present in hh)
22 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Discussion

 “Gap/Can’t Remember” criteria is most 

promising

Forgetful respondents report better data than reluctant 

and inactive respondents

– Not actively refusing to report activities and report a higher 

number of activities

– Also, tend to report earlier in the data collection period and 

require fewer contact attempts (not shown here)

Including them (subset) might reduce demographic 

differences between NE cases and eligible ATUS cases
23 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Recommendations

1. Analyses to develop revised criteria for 

exclusion

2. Analyses to investigate the potential bias (if 

any) resulting from the exclusion of NE cases

3. Develop, revise, or implement interviewer 

strategies to aid recall
Contact Information

24 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

David Biagas, PhD

Research Statistician

Office of Survey Methods Research

https://www.bls.gov/osmr/contact.htm

202-691-5558

biagas.david@bls.gov
25 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Table #. Distribution of Number of Activities Reported by Subgroup

Percentile NE Cases Inactive Group Forgetful Group Reluctant 

Group

ATUS 

Eligible

100% Max 90 4 89 90 90

75% Q3 17 4 20 13 24

50% Median 11 4 15 9 18

25% Q1 7 3 10 7 14

0% Min 1 1 1 1 5

Note: Activity counts include don't know and refusal/gap
26 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Results: Subgroup Response 

Period

0

10

20

30

40

50

60

70

80

 1-2 3-4 4-6 7-8

Inactive Group Forgetful Group Reluctant Group ATUS Eligible

Week of Response
27 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

 Demographic: labor force status, sex, race, 

age, educational attainment, income, 

household size, etc.

 Data Quality: CPS income imputed, labor force 

proxy, reference person 

 Data Collection Strategy: reporting for 

weekend day, # of call attempts, reporting 

period (quarter)

Results: Predicting NE Status
28 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov