1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Evaluating Quality in CE's 

Proof of Concept Test

Laura Erhard, Safia Abdirizak, 

Yezzi Angi Lee, and Brett McBride

Bureau of Labor Statistics

Joint Statistical Meetings

Baltimore, MD

August 1, 2017
2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Outline

 Brief Background – CE Redesign (Gemini Project)

 Test Design

 High Level Results and Sample Analysis

 Data Quality

Recall and Records Interviews

Diaries

 The Good, the Bad, the Unknown...

 Next Steps
3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

What is the Gemini Project?

“...aims to redesign the Consumer Expenditure (CE) survey to 

improve data quality through a verifiable reduction in 

measurement error—particularly error caused by underreporting.”
4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

What is the Gemini Project?

“...aims to redesign the Consumer Expenditure (CE) survey to 

improve data quality through a verifiable reduction in 

measurement error—particularly error caused by underreporting.”
5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Test Design
6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Proof-of-Concept (POC) Test

 Fielded July 2015 – Sep 2015

 Atlanta, Chicago, Denver, and New York 

Census Regional Offices

 Ensure that the basic underlying structure 

and components of the new design are 

feasible

 Designed to mirror the proposed design to 

the fullest extent possible
7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Gemini Redesign Plan

Post Wave

Non -monetary Incentive:

CE Experience Package

Respondent 

engagement 

mailings

Diary Week

& Records Collection

Advance Mailing

Wave 1

Visit 1: Recall Interview

Visit 2: Records Interview

Wave 2

12 months 

between waves

Advance Mailing

Visit 1: Recall Interview

Diary Week

& Records Collection

Visit 2: Records Interview
9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

High Level Results

and Sample Analysis
10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Final Response

N - complete Response 

Rate (%)

Test 520 49.8

Control – CEQ (All waves) 1483 62.9

Control – CEQ (Wave 1) 365 64.4

Control – Diary 645 58.6
11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Sample Characteristics 

Test 

(N=520)

Control – CEQ 

(N=1483)

Control – Diary

(N=315)

Race of Respondent

White 82.5 79.2 73.0

Black 11.9 15.0 12.1

Other 5.4 5.8 5.7

Hispanic Origin of Respondent 10.8 12.1 9.8

Gender of Respondent

Female 54.8 54.2 47.6

Male 45.0 45.9 43.2

Age of Respondent

Under 25 years 6.7 5.0 5.4

25-34 years 16.4 17.1 15.6

35-49 years 29.4 25.8 22.9

50-64 years 26.4 18.8 24.8

65 years and older 21.0 23.5 22.2
12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Sample Characteristics 

Test 

(N=520)

Control – CEQ 

(N=1483)

Control – Diary

(N=315)

Education

Less than high 

school 7.5 8.3 7.3

High school grad 21.9 21.0 16.2

Some college 31.9 29.8 23.9

College grad 38.5 40.9 43.5

Household Size

Single person 30.4 30.8 34.9

2-3 46.2 47.9 45.1

4+ 23.5 21.3 20.0

Housing Tenure

Renter 41.0 37.0 36.8

Owner 59.0 63.0 63.2
13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Sample Characteristics 

Test 

(N=520)

Control – CEQ 

(N=1483)

Control – Diary

(N=315)

Education

Less than high 

school 7.5 8.3 7.3

High school grad 21.9 21.0 16.2

Some college 31.9 29.8 23.9

College grad 38.5 40.9 43.5

Household Size

Single person 30.4 30.8 34.9

2-3 46.2 47.9 45.1

4+ 23.5 21.3 20.0

Housing Tenure

Renter 41.0 37.0 36.8

Owner 59.0 63.0 63.2
14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Sample Characteristics

Income

N Average Median

Test 520 $71,526 $45,404

Control-CEQ 1483 $66,666 $43,459

Control-CED 283 $74,057 $50,000

No significant differences.
16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Data Quality 

– Recall and Records Interviews
17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Total Expenditures – Records and Recall

N Average Median

Test 520 $10,784 $7,457

Control-CEQ 1483 $10,157 $7,334
18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Model of log (interview expenditure totals) by sample 

condition and associated covariates (n=1,997)

Coefficient Standard Error

Intercept 8.035 (0.082)

Income 0.391*** (0.018)

Multi-member CU 0.343*** (0.038)

Homeowner 0.133*** (0.036)

POC CU -0.036 (0.040)

Education Level 0.150*** (0.017)

Race of Respondent -0.186*** (0.041)

At Least Some Record Use 0.146*** (0.036)

All Valid Blanks -0.779*** (0.076)

Model R2 0.454
19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Visit 1 - Recall Expenditures

Highlighted labels indicate significant difference between the test and control using a Wilcoxon-Z test statistic

$399 $380

$234 

$653 

$696

$267 $282 

$222 

$707 

$950 

 $-

 $100

 $200

 $300

 $400

 $500

 $600

 $700

 $800

 $900

 $1,000

Appliances Furnishings Subscriptions Trips Miscellaneous

Test Control
20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Visit 2 - Records Expenditures 

$1,083 

$1,664 

$1,258 

$116 

$1,284 

$454 

$587 

$336 

$923 

$1,783 

$1,290 

$156 

$1,404 

$518 

$772 

$334 

 $-

 $200

 $400

 $600

 $800

 $1,000

 $1,200

 $1,400

 $1,600

 $1,800

 $2,000

Rented

Housing

Owned

Housing

Utilities Rented and

Leased

Vehicles

Owned

Vehicles

Non-Health

Insurance

Health

Insurance

Medical

Test Control

Highlighted labels indicate significant difference between the test and control using a Wilcoxon-Z test statistic
21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Distribution of Number of Don’t Knows for Expenditures

69%

14%

13%

4%

79%

11%

9%

2%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

0

1

2-5

6+

Test (N=520) Control (N=1,483)
22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Percent missing income for top five sources

186, 41%

188, 13%

279, 17%

406, 10%

1128, 11%

59, 31%

80, 3%

111, 11%

134, 4%

406, 7%

0% 10% 20% 30% 40% 50%

Self-employment income*

Retirement, survivor, or disability pensions

Interest and dividends income

Social Security or Railroad Retirement income*

Wages and salaries*

Test Control

 The asterix indicates that all members of the CU reported a missing value 

 The data label includes total number reported and percent missing
24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

HH Average Percent of Rounded Expenditure Responses

22%

34%

0%

5%

10%

15%

20%

25%

30%

35%

40%

Test (n=510) Control (n=1,432)
25 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

HH Average Percent of Rounded Expenditure Responses by Question 

19%

13%

46%

11%

23% 22%

29%

3%

16%

25%

44%

23%

58%

17%

23%

26%

34%

7%

26%

35%

0%

10%

20%

30%

40%

50%

60%

70%

Owned

Housing

Utilities Rented and

Leased

Vehicles

Life Insurance Long Term

Care Insurance

Automobile

Insurance

Homeowners'

Insurance

Renter's

Insurance

Health

Insurance

Medical

Test (n=510) Control (n=1,432)
26 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Data Quality - Diaries
27 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Extent of Total Recall

2%

98%

Test (N=520)

Total Recall Not Total Recall

9%

91%

Control (N=291)
28 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Number of HH Diary Entries

18.4

7

1.6

11.2

16.3

4.9

1.4

8.7

0

2

4

6

8

10

12

14

16

18

20

Food (Home) Food (Away) Clothing Other

Test (N=520) Control (N=291)
29 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Diary Expenditures

$110 $81 $43 

$861 

$112 $75 $51 

$798 

 $-

 $100

 $200

 $300

 $400

 $500

 $600

 $700

 $800

 $900

 $1,000

Food (Home) Food (Away) Clothing Other

Test (N=508) Control (N=259)
30 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

The Good, the Bad, the Unknown...
31 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

The Good

 No decrease in expenditures

 Higher number of diary entries

 Less item non-response (income and 

expenditures)

 More accurate reports (less rounding)
32 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

The Bad (or “not so good”)

 Final response rates low – only 50% 

complete all 3 parts of the interview

 Interview length is long, particularly for 

the 2nd visit
33 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

The Unknown

 Would we see statistical differences with a 

larger sample size? (Expenditures, Income)

 Would HHs participate in Wave 2?

 Would finalized question content, tweaked 

protocols, and a new diary instrument 

improve results?

 Did the Incentives bias the sample towards 

renters?
34 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Next Steps

 Large Scale Feasibility Test 

(tentatively 2019)

Larger sample sizes (2,000 

completes)

Two waves tested

Finalized instruments and questions

Increased incentives
Contact Information

35 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Laura Erhard

Branch Chief, BRPD

Division of Consumer Expenditure Survey

www.bls.gov/cex

202-691-5115

Erhard.Laura@bls.gov