1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Development of a CARI Research Plan

Brett E. McBride

Division of Consumer Expenditure Surveys

2023 FedCASIC Workshops

April 12, 2023
2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Outline

A. Overview – CE and CARI

B. Research Process

1. Initial Stages (Identifying Research Questions)

2. Final Selection

3. Item Types

4. Evaluation Methods: Quantitative and Qualitative

C. CARI Interactive Data Access System (CIDA) Overview

1. Quantitative ‘behavior coding’ screens

2. Qualitative ‘case view’ screen

D. Lessons Learned
3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CE and CARI

Consumer Expenditure 

(CE) Survey involves 

Census collecting 

spending data from 

U.S. population, 

sponsored by BLS

Computer Audio 

Recorded Interviewing 

(CARI), an application 

originally developed by 

RTI to record interviews
4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CE and CARI

◼ Initial questions:

1. Can CE staff obtain or listen to recordings (or does everything 

remain ‘off-site’)? 

2. How much can be recorded? 

3. Are there storage limitations? 

4. What burdens does this place on interview process?
5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CE and CARI

◼ Initial questions:

1. Can CE staff obtain or listen to recordings (or does everything 

remain ‘off-site’)? No and yes [listen not obtain, within Census 

Virtual Desktop Infrastructure (VDI)]

2. How much can be recorded? [limited to recording by survey item]

3. Are there storage limitations? [practically, no]

4. What burdens does this place on interview process? [few: 

recordings triggered automatically, invisibly by interviewers’ 

computers; does involve respondent consent]
6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Initial Stages

◼ CE solicited ideas for survey Research Questions (RQs) from 

survey experts:

Staff in charge of data processing

Staff monitoring changes in interview content

◼ Requirements:

RQs can be addressed via spoken questions/responses

RQs involve specific survey items (not sections nor entire interviews)
7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Final Research Question (RQ) Selection

◼ Solicitation and prior research informed RQs:

Items flagged by data editing 

[homeowners insurance amounts, new vehicle purchase components, private 

pension deductions] & record use

New/revised survey items 

[new items about health insurance coverage, ‘extended recall’ purchases, ‘outlet’ 

location of purchases]

Representative items 

[vehicle services item with ‘laundry list’ style of response options]
8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Item Types

◼ Seven (Initial) RQs with different characteristics

RQ1 Health Insurance Coverage

What [other] kind of health insurance or health care coverage [do/does] [you/your household] have or pay for? INCLUDE 

those that pay for only one type of service (nursing home care, vision care, or dental care). EXCLUDE private plans that 

only provide extra cash while hospitalized. 

1. Private health insurance, including employer and healthcare exchange plans?

2. Medicare

3. Medicare Supplemental Insurance (Medi-Gap)?

4. Medicaid

5. SCHIP

6. Military

7. HIS

8. Single service plan (e.g., dental, vision, long-term care)?

9. No coverage of any type

[if #2 not selected and respondent age 65+] 

Are you [is anyone in this household] covered by Medicare? 
9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Item Types (cont.)

RQ8 Private Pension Deductions

Does the respondent have a paper or electronic pay check record present for [Hishernames] last paycheck? [PAYSTUB]

1. Yes

2. No

Was there any money deducted from [your / name’s] pay for – Private pension fund? [PRIVPENS]

1. Yes 

2. No

How much? [PRIVPENX] 
10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Item Types (cont.)

RQ8 Private Pension Deductions

Does the respondent have a paper or electronic pay check record present for [Hishernames] last paycheck? [PAYSTUB]

1. Yes

2. No

Was there any money deducted from [your / name’s] pay for – Private pension fund? [PRIVPENS]

1. Yes 

2. No

How much? [PRIVPENX] 
11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

RQ Evaluation Methods

◼ Quantitative (behavior coding)

Inductive coding

Test data, code refinement

◼ Qualitative 
12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Categorizing Methods

Quantitative

1. Analyst is listening for more general 

problematic issues that might be 

subjective.

2. The analyst must listen and synthesize 

information across multiple survey 

items.

1. Analyst can identify specific, objective 

behaviors/comments that would address 

the research question.

2. Behaviors/comments can be captured 

by a code that is generic and could be 

applicable to survey items for other 

research questions.

Quantitative Qualitative
13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Quantitative Evaluation

Pros

Cons

▪ Easily output into 

reports (via CIDA)

▪ Standardized codes can 

pinpoint specific issues 

affecting response 

process

▪ Allows answering ‘how 

often’ (context for 

issues)

▪ May require doublecoding or coder 

calibration for quality 

control

▪ Significant work needed 

to develop and test 

codes to ensure capture 

of needed information

▪ Does not automatically 

result in illustrative 

examples
14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Qualitative Evaluation

Pros

Cons

▪ Does not require as much 

background information 

into response process for 

the question

▪ Allows answering ‘why’

▪ Produces illustrative 

explanatory output 

▪ May not permit 

quantifying if observed 

issue is widespread

▪ Requires more user 

effort to interpret and 

summarize findings
15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Quantitative: Inductive Coding

◼ Iterative: test data informative

Record use difficult subject for coding

Want to capture rapport-building dialogue (they digress to 

discussion of gas stations and prices)

Need to create codes informed by common behaviors (e.g., code for 

if interviewer represented each laundry list concept and not if they 

read every word of laundry list of responses as scripted)
16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CIDA

CARI Interactive Data Access System

 Pulls in survey item recordings within Census VDI application 

 Accessible by multiple users simultaneously

 Functionality: Pages permit summarizing cases, subsampling cases to output, assignment of codes 

to survey items, display of codes and scripted survey items, filtering to specific cases or cases with 

certain items, managing coding workload, and reporting on code analysis 

 Not accessible directly from non-Census workstations (requires Census credentials, VDI)

 Not geared for sporadic users (account deactivates after 30 days of non-access) 

 Subject to the recording variabilities of the devices used by interviewers (i.e., audibility varies)
17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Initial Audibility Findings

◼ Examined how often both interviewer and respondent could 

be heard (n=212)
18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Selecting a Case to Code
19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Recorded Survey Items and Responses
20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Entering Code Responses
21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Completing Coding
22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Qualitative: Filtering to a Specific Case
23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Lessons Learned

◼ Importance of having buffers (sometimes even before 

question asked) 

◼ Timestamps (shed light onto interviewer progression across 

items)

◼ Listening for certain behaviors not as feasible 

◼ Need to gain expertise in subject matter
Contact Information

24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Brett E. McBride

Division of Consumer Expenditure Surveys

www.bls.gov/cex

mcbride.brett@bls.gov