1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

IMPACT OF CARI ON DATA QUALITY IN THE 

CE INTERVIEW SURVEY

Ariana Welsh, Brett McBride

Bureau of Labor Statistics

Office of Prices and Living Conditions, Branch of Research and 

Program Development

AAPOR 2024
2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

◼ Recording uses the interviewer’s laptop microphone

◼ Consent obtained from respondent at the start of interview

◼ Pre-selected items are recorded

Not full interview

◼ Recording starts when interviewer navigates to screen 

containing item

Recording stops when interviewer leaves screen

Computer Audio Recorded Interviewing (CARI)
3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Pretest

• Testing period: 4th Quarter, 

2021

• Sample of respondents in 

their final interviews

• Consent question was asked, 

but no actual recordings 

were made

CARI Implementation

• Implementation period: July 

2022

• Analysis period: 3rd & 4th

Quarter, 2022 (six months)

• All respondents asked

• Consent question was asked 

and recordings were made

Computer Audio Recorded Interviewing (CARI)
4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Data Quality Measures

◼ CARI Consent

◼ Subsequent Interview Nonresponse

◼ Interview Timing Duration

◼ Expenditures

◼ Sensitive Question Item Non-Response: Salary

◼ Records Use
5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

“This interview may be recorded for quality purposes. Is that 

okay?”

Computer Audio Recorded Interviewing (CARI)
6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

◼ 64% of respondents 

consented to being recorded 

in Q3 of 2022

4% lower than pretest consent 

rates (68% percent)

◼ Federal surveys with CARI 

programs typically have 80-

90% consent rates

SIPP, with an explicit consent 

question, has a 66% rate

Consent Rates: Pretest vs. 2022

68.1%

63.8%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2021, Pretest 2022, Q3

Consent Rates in 2022 vs. the Pretest
7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Subsequent Interview Nonresponse: 2022

◼ No real difference between 

consenters and refusers in 

subsequent interview 

completion rates (not 

significant)

13.2%

9.3%

78.6%

11.1%

10.2%

77.5%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Refusal

Non-Contact

Complete Interviews

Consenters Refusers
8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Timing Duration: 2022

◼ There is a 9 minute difference 

between consenters and 

refusers

◼ Quarter duration changes 

were not found to be 

significant for consenters and 

refusers

◼ Pretest: consenters had longer 

durations than refusers as well 

(14 minutes)

62

57.5

63.3

53.9

0

10

20

30

40

50

60

70

Consenters Refusers

Minutes

Quarter 2, Pre-CARI Quarter 3, Post-CARI
9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Expenditures: 2022

◼ Expenditure difference between 

consenters and refusers ($1,803)

 This trend is visible in the prior 

interview as well, before CARI 

consent is asked

◼ Consecutive Quarter Trend:

 Consenters: Quarter 3 expenditures 

are higher than the previous quarter 

by $385 -- Significant

 Refusers: Quarter 3 expenditures are 

lower by $69 -- Not significant

$13,608 

$12,259 

$13,993 

$12,190 

$0

$2,000

$4,000

$6,000

$8,000

$10,000

$12,000

$14,000

$16,000

Consenters Refusers

Quarter 2, Pre-CARI Quarter 3, Post-CARI

$385

$69

*

*
10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Pretest, 2021 Quarter 3, 2022

$13,227 

$10,884 

$0

$2,000

$4,000

$6,000

$8,000

$10,000

$12,000

$14,000

Pretest, 2021

Consenters Refusers

$13,885 

$11,727 

$0

$2,000

$4,000

$6,000

$8,000

$10,000

$12,000

$14,000

Quarter 3, 2022

Consenters Refusers

Side-by-Side Expenditures: Pretest vs 2022

◼ To get a side-by-side comparison of expenditures, our sample group of interviews in 2022 was subset 

to those respondents with final interviews only in Quarter 3

◼ Both the pretest and Q3 of 2022 show a self-selection trend between consenters and refusers
11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Salary Item Non-Response: Pretest & 2022

Pretest (2021) 2022

Consented Refused

Salary Refusal 1.3% 5.0%

Salary ‘Don’t 

Know’ 3.4% 6.1%

Total NonResponse

4.7% 11.1%

Consented Refused

Salary Refusal 1.8% 9.2%

Salary ‘Don’t 

Know’ 4.3% 7.9%

Total NonResponse

6.1% 17.1%
12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Record Use: Pretest & 2022

29.7%

33.1%

14.6%

22.9%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Pretest, 2021 Quarter 3, 2022

Consenters Refusers

◼ Consistent with trends, 

consenters were more likely 

to use records in 2022 and in 

the pretest

◼ Record use is higher for both 

groups in 2022
13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

What about respondents who have completed 

two CARI interviews?
14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

12.0%

8.4%

79.6%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Change Unchanged

Consent Change in Consecutive 

Interviews

Consenters to Refusers Refusers to Consenters

Unchanged

◼ From Q3 to Q4 of 2022, 20% 

of respondents changed 

consent response in the next 

interview

12% stopped consenting in 

the following interview

8% started consenting in the 

following interview

Effect on CARI Consent
15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Effect on Interview Duration

◼ Interview duration went 

down in the consecutive 

CARI interview for nearly all 

groups

Are interviewers and 

respondents adapting to the 

CARI process?

62.4

57.6

66.1

53.5 52.5

60.2 60.2

49.6

0

10

20

30

40

50

60

70

Consenters to

Refusers

Refusers to

Consenters

No Change,

Consenters

No Change,

Refusers

Minutes

Quarter 3 Quarter 4
16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Summary

Data Quality Indicators 2022 Data Quality Pretest Trend?

2. Subsequent Interview Nonresponse➢ No real difference between consenters 

and refusers

N/A

3. Timing Duration ➢ Consenters had longer interviews

4. Expenditures ➢ Consenters had higher total 

expenditures

5. Salary Item Non-Response ➢ Consenters had lower item nonresponse

6. Records Use ➢ Consenters had higher use of records
17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Conclusion

◼ Higher data quality among consenters is largely due to selfselection

◼ Weak evidence suggests that CARI may change behavior of 

respondents, most notably consenters:

– Expenditures: consecutive quarter increase for consenters (significant) and 

decrease for refusers

◼ CARI does not have a negative impact on subsequent interview 

completion and data quality
18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

For questions, contact

Ariana Welsh

Welsh.Ariana@bls.gov

Brett McBride

McBride.Brett@bls.gov

Thank you!