Testing Global Questions for the Consumer 

Expenditure Gemini Recall Interview 

Final Report

Authors: 

Jennifer Crafts, Ph.D.

Rachel Tesler, Ph.D.

Johnny Blair, M.A.

Jessica Behm, M.A.

Karen Stein, Ph.D.

Mariel Leonard, M.A.

Jasmine Folz, M.A.

September 2015 

Prepared for:

U.S. Department of Labor

Bureau of Labor Statistics

Washington, DC 20212

Kopp.Brandon@bls.gov

Douglas.Pamela@bls.gov

Prepared by:

Westat

An Employee-Owned Research Corporation®

1600 Research Boulevard

Rockville, Maryland 20850-3129

(301) 251-1500
This page is intentionally blank
Cognitive Testing for the CE Recall Interview iii 

 

Table of Contents 

Chapter Page

Executive Summary............................................................................................ ES-1

Introduction........................................................................................... ES-1

Method ................................................................................................... ES-1

Conclusions and Recommendations.................................................. ES-3

1 Introduction and Purpose ................................................................................. 1-1

1.1 Background............................................................................................ 1-1

1.2 Objective of the Task ........................................................................... 1-2

1.3 Research Questions .............................................................................. 1-3

2 Methods ............................................................................................................... 2-1

2.1 Research Design.................................................................................... 2-1

2.1.1 Iterative Testing Rounds ..................................................... 2-1

2.1.2 Testing Locations ................................................................. 2-1

2.2 Development of Question Clusters and Question 

Wording.................................................................................................. 2-2

2.3 Recruitment............................................................................................ 2-2

2.4 Participants............................................................................................. 2-3

2.5 Session Procedure ................................................................................. 2-5

2.6 Key Metrics............................................................................................ 2-5

2.6.1 Expenditure Reports............................................................ 2-5

2.6.2 Comprehension..................................................................... 2-6

2.6.3 Recheck of Expenditures..................................................... 2-7

2.7 Data Analysis......................................................................................... 2-7

3 Results.................................................................................................................. 3-1

3.1 Cluster A: Apparel and Personal Items ............................................. 3-2

3.2 Cluster B: Repairs, Replacements, Upkeep of Home, and 

Home Operations................................................................................. 3-8

3.3 Cluster C: Household Linens and Furnishings................................. 3-14

3.4 Cluster D: Trips and Transportation.................................................. 3-37

3.5 Cluster E: Recreation............................................................................ 3-51
Cognitive Testing for the CE Recall Interview iv 

 

Contents (continued) 

Chapter Page

3.6 Cluster F: Children and Infants .......................................................... 3-68

3.7 Cluster G: Electronics/Technology................................................... 3-73

3.8 Cluster H: Events.................................................................................. 3-85

4 Conclusions and Recommendations................................................................ 4-1

References............................................................................................................ R-1

Appendixes

1 Recruitment Screener......................................................................................... A1-1

2 Consent Form ..................................................................................................... A2-1

3 Session Protocol ................................................................................................. A3-1

4 Lessons Learned and Observations on the Testing ...................................... A4-1

Tables

ES-1 Question testing status at the conclusion of testing...................................... ES-4

2-1 Testing rounds and locations............................................................................ 2-1

2-2 Question clusters and original number of questions..................................... 2-2

2-3 Demographic characteristics of study participants........................................ 2-4

3-1 Cluster A: Summary table of tested question wording and order ............... 3-3

3-2 Cluster B: Summary table of tested question wording and order................ 3-9

3-3 Cluster C: Summary table of tested question wording and order................ 3-15

3-4 Cluster D: Summary table of tested question wording and order............... 3-39

3-5 Cluster E: Summary table of tested question wording and order ............... 3-53
Cognitive Testing for the CE Recall Interview v 

 

Contents (continued) 

Tables Page

3-6 Cluster F: Summary table of tested question wording and order................ 3-69

3-7 Cluster G: Summary table of tested question wording and order............... 3-75

3-8 Cluster H: Summary table of tested question wording and order............... 3-86

4-1 Question testing status at the conclusion of testing...................................... 4-2

4-2 Original wording and recommended wording when testing 

concluded, by cluster.......................................................................................... 4-5

Cognitive Testing for the CE Recall Interview ES-1 

 

Executive Summary 

 Introduction 

The Bureau of Labor Statistics (BLS) is redesigning the Consumer Expenditure Survey (CE), which 

provides data on the buying habits of American consumers. One of three interview components of 

the CE is the in-person interviewer-administered recall interview, in which respondents report their 

household’s expenditures for 38 expenditure categories (Passero et al. 2013) for the preceding three 

month period. BLS is investigating the use “global” questions to collect these expenses as a way to 

potentially decrease burden for both the interviewer and respondent.

The purpose of this study was to conduct iterative testing to identify the appropriate level of 

aggregation and question wording for the expenditure categories to be collected in the recall 

interview. The specific research questions were as follows.

1. For each question, what wording is most effective to aid both recall and reporting of 

applicable expenses and question comprehension?

2. For each question, what is the most effective administration order (relative to other 

questions that will be administered)?

 Method 

Iterative Testing Rounds and Locations. We used cognitive interviewing techniques to gain an 

in-depth understanding of how test participants understood and answered the expenditure 

questions. Each testing round focused on testing a cluster (or subset of the cluster) of questions, 

with approximately 5 participants assigned to test about 20 questions within a one-hour session. We 

completed a total of 85 cognitive interviews over the course of 9 iterative testing rounds. The 

interviews were conducted in Westat’s cognitive interviewing facility in Rockville, MD, as well as in 

Baltimore, MD; Winchester, VA; and Gettysburg, PA.

Question Development. Westat divided the 38 expenditure categories into 8 “clusters” based on 

similar content, then drafted an initial set of 116 questions to cover all of the categories. BLS 

suggested wording the questions based on definitions of the expense items to be covered, where 

possible. In some cases, the wording also provided examples of expenses to help cue recall and 

reporting. BLS reviewed and suggested revisions for the first round of testing.
Cognitive Testing for the CE Recall Interview ES-2 

 

Recruitment. Participants were recruited from Westat’s volunteer database and by placing ads (e.g., 

Craigslist and local newspaper ads, Facebook ads, Google Display ads). Potential participants were 

screened for eligibility by either logging into a web-based survey tool or calling a Westat recruiter to 

answer a series of demographic questions and questions about their household spending over the 

prior three-month period.

Session Procedure and Metrics. Experienced Westat cognitive interviewers conducted all 

85 interviews. In conjunction with BLS, Westat developed a general session protocol (see Appendix 

3) that was used in each testing round. To assess reporting accuracy, participants were first asked 

each expenditure question, and if they responded “Yes,” they were asked for the three-month 

expense total. After answering all questions, the interviewer then followed up by asking, for each 

“Yes” response, for the specific item(s) purchased, with cost and month. Comprehension was 

assessed by asking participants to name examples of expenses for each question, and (for a majority 

of the questions) to categorize a list of items. The comprehension activities also served to cue recall 

of any additional expenses; respondents were asked for any items not already reported, with cost and 

month of purchase. As needed, the interviewer probed to determine the reason(s) for not reporting 

the item(s) earlier in the interview.

Through iterative rounds, we tested the following types of revisions: (1) changing the wording; 

(2) merging two questions to form a higher-level expenditure question; (3) splitting an expenditure 

question into two (or more) questions; and (4) changing the question administration order.

Analysis and Reporting. For each question, items were judged to be accurately reported if they 

were both applicable for the question and purchased within the correct reference period. 

(Telescoping and recall issues were also noted.) We examined the reporting errors in concert with 

the qualitative (comprehension activity-based) data and all participant verbalizations to identify 

comprehension issues and possible revisions to clarify question intent and breadth.

After testing each cluster the first time, Westat analyzed the data and provided a detailed interim 

memo of reporting and comprehension results for each question, with recommendations for 

changing question wording and/or administration order. BLS provided oversight for revisions. For 

the clusters tested a second or third time, Westat provided a summary table with recommendations 

for any further revisions and supporting results.
Cognitive Testing for the CE Recall Interview ES-3 

 

 Conclusions and Recommendations 

As described above, we drafted the 116 original questions as definitions where possible. For each 

question, if comprehension and/or reporting issues (such as reporting items that were not 

applicable, or excluding items that were applicable) surfaced, we considered the following as options

for revisions for a subsequent round of testing.

 Revise the wording of the definition to address specific observed reporting and/or 

interpretation issues that surfaced;

 Use examples instead of, or in addition to, the definition; or

 Add exclusion text (to explicitly address a misinterpretation observed with multiple 

participants or judged likely to be significant for the population, e.g., over- or underreporting of large expenses).

If these options did not seem appropriate (e.g., a question was so broad that it was not possible to 

use a succinct and representative set of examples, adding exclusion text made the question text very 

lengthy) or did not lead to improved performance in subsequent testing, a last resort was to split the 

question into two or more questions, each with a more specific focus.

The wording changed for 65 of the 116 original questions at some point over the course of testing. 

Three original questions were split into 2 or 3 separate questions, and two original questions were 

merged into a single question. When testing concluded, the total number of questions was 119.

The evidence also resulted in changing the administration order for some questions. In these cases, 

participants reported expenses for a question but should have reported them for a subsequent 

question, and/or comprehension results showed that items were considered applicable for a 

question that occurred later in the administration order. We then monitored whether reporting and 

comprehension improved in a subsequent testing round. The main reason for ensuring reporting of 

expenses for the intended questions was the mapping of expenses to Expenditure Categories. If 

expenses are reported for the “wrong” questions, or are reported multiple times (where recall of the 

same expense is cued by the wording of several questions), the mapping of expenses to the

Expenditure Categories will be incorrect and/or an over-estimate of total expenses for the recall 

interview.

Table ES-1 shows, at the conclusion of testing, the status regarding recommendations for further 

testing of the expenditure questions.
Cognitive Testing for the CE Recall Interview ES-4 

 

Table ES-1. Question testing status at the conclusion of testing

Question testing status Number of questions 

Testing considered complete 5

More testing is recommended, to monitor the performance of the question

(due to wording and/or order changes to other questions within the cluster)

72

More testing is recommended, due to revisions to the wording and/or order of 

the question

42

Total 119 

After concluding the last testing round, Westat provided a summary table of recommended wording 

and order for each cluster. That table is included as Table 4-2 in Chapter 4 of this report. Another 

final product of the research was a crosswalk that maps each of the 116 original questions to an 

Expenditure Category number and name, the question’s component Universal Classification Codes 

(UCCs), and the recommended question wording and ordering (within cluster) after testing 

concluded. For applicable individual questions, the crosswalk provides additional notes about 

administration order relative to questions in other clusters, based on the reporting and 

comprehension results.

The large number of questions in the recall interview created two significant limitations for testing. 

First, it was not possible to cover all questions with participants. Assessing both reporting accuracy 

and comprehension limited testing to approximately 15 questions in each one-hour session. (The 

protocol was modified in the later testing rounds to accommodate testing more questions, 

approximately 20 per hour.) Therefore, when testing concluded, we did not offer a definitive 

recommendation for overall order of questions. More testing will be needed to determine an optimal 

order for the full set of questions for the recall interview. To cover more questions with each 

participant will require a longer session time and/or an alternative approach (e.g., cutting elements 

out of the protocol).

A second limitation was the small sample size. A total of 85 participant hours was allocated for 

testing over the 9 rounds. This allowed testing each question in each round with approximately 

5 participants. We recommend future testing with larger numbers per question and round, to collect 

more reporting and comprehension data for judging question performance. Information about an 

appropriate sample size might also come from evaluating the adequacy of samples used for other 

interviews that have similar characteristics to the recall interview. Testing with 20 – 30 participants 

per question would assist in evaluating whether decisions made to date on the basis of these small 

samples are sound, as well as in identifying further issues and possible revisions.
Cognitive Testing for the CE Recall Interview 1-1 

 

1.1 Background 

The Consumer Expenditure Survey (CE) provides data on the buying habits of American 

consumers, including both expenditures and income for the consumer unit (i.e., families and single 

consumers). CE respondents provide a wide range of expenditures, from monthly utility bills to 

irregular entertainment expenses, through the recall interview, the diary, and the records interview.

BLS is currently redesigning the CE in response to evidence of measurement error, changes in 

technology, changes in the types of expenses that people incur, and the need for greater flexibility in 

administration (Gemini Project, 2010). As part of the development process for the redesign, BLS 

began investigating ways to reduce burden for the recall interview. The in-person recall interview

will require respondents to report their household’s expenditures for a selected set of expenditure 

categories for the preceding three month period. To reduce the burden associated with the current 

recall interview, BLS set out to develop from scratch a set of “global” questions to collect data at a 

higher level of aggregation than data collected to date (Beatty, 2010; Goldenberg & Steinberg, 2010;

Peytchev, 2010). Theoretically, asking a global question about expenses for an aggregate or highlevel expenditure category (e.g., appliances) should take less time than asking a longer series of 

questions about the individual items (such as refrigerators, washing machines, ovens) within that 

category, and potentially decrease burden for both the interviewer and respondent. The reduction in 

burden should not be at the expense of increased measurement error, another motivation for the 

redesign. This approach should not significantly reduce the number or accuracy of items reported, 

or have a negative impact on any other measurement error factors. 

Preliminary research (Edgar, 2011; Kopp & Yu, 2013) on several categories of global questions has 

revealed mixed results. While respondents were able to report expenses at a higher level of 

aggregation for some questions, recall and comprehension issues were observed for others. Recall 

issues can lead to under-reporting. Under-reporting of total expenditures can result from reporting 

fewer items, even if the costs are reported accurately. And recall can also produce over-reporting, if 

an item is telescoped into the three-month reference period. Comprehension issues can lead to both 

over-reporting and under-reporting of expenses. For reported items, recall error for the actual costs 

may be either too high or too low, based on either random variability or biased in one direction.

Introduction and Purpose 1 
Cognitive Testing for the CE Recall Interview 1-2 

 

Introduction and Purpose 1 

1.2 Objective of the Task 

The objective of this study was to develop and test global questions for the recall interview within 

the redesigned CE survey. BLS prefers to use fewer questions than currently administered as the 

“detailed questions” in the current quarterly interview survey while not increasing measurement 

error. (Note that while reducing measurement error is desirable, that may be difficult to achieve 

when asking questions at a more global level. A realistic goal may be to not increase measurement 

error, while shortening the interview length.) The objective was to determine an appropriate level of 

aggregation and question wording for the full set of expenditure categories.

BLS identified 38 expenditure categories (provided to Westat as “Appendix A”) that will be covered 

in the CE recall interview1. These 38 categories represent the highest possible level of aggregation 

acceptable for key stakeholders (Passero et al., 2013). Each expenditure category has one or more 

component Universal Classification Codes (UCCs). For each of the 38 categories, BLS staff 

reviewed the breadth of the content and provided a “proposed level of aggregation” – an expert

judgment of how many questions may be required to capture expenditures. If one question would be

developed for each of these aggregations, the number of questions for the recall interview would 

total to approximately 150. BLS expects to use fewer questions for some categories. However,

testing may prove that aggregation for others will lead to over- or under-reporting, so more 

questions may need to be retained. The question wording will need to correspond with respondent 

recall strategies and be at a level of aggregation that resonates across respondents and is easiest to 

answer accurately.

Westat’s task was to conduct cognitive research to develop and evaluate new questions for the recall 

interview. To develop effective question wording, the questions were tested iteratively, with input 

and review from BLS. Westat started with an initial set of 116 questions (as described in 

Section 2.2).

 

1 Note that additional expenditure categories that are not part of the recall interview are covered in the diary and 

records interviews.
Cognitive Testing for the CE Recall Interview 1-3 

 

Introduction and Purpose 1 

1.3 Research Questions 

This study addressed an overarching research question: What is the appropriate level of aggregation 

and question wording for the expenditure categories to be collected in the recall interview?

When the study began, we anticipated addressing this research question with not only qualitative 

data, but also quantitative data. For example, based on collecting data from 10 participants per 

round, we expected to calculate the number and percentage of participants who over- and underreported for each question, and on average, by how much.

However, due to session time constraints, there was a need to reduce the number of questions tested 

within each session. That required reallocating the 116 questions (divided into 8 clusters, as 

explained below) across rounds, and testing with approximately 5 participants per question in each 

round. The reallocation allowed for testing all clusters once over the course of the first 5 rounds, 

and re-testing a subset of the clusters in the last 4 rounds. After discussing the results of the first 

testing round with BLS, the decision was to focus on qualitative data to identify any reporting and 

comprehension evidence that strongly indicated changing question wording and/or order.

The specific research questions were as follows.

1. For each question, what wording is most effective to aid both recall and reporting of 

applicable expenses and question comprehension?

2. For each question, what is the most effective administration order (relative to other 

questions that will be administered)?
This page is intentionally blank
Cognitive Testing for the CE Recall Interview 2-1 

 

2.1 Research Design 

2.1.1 Iterative Testing Rounds 

We used cognitive interviewing techniques to gain an in-depth understanding of how test 

participants understood and answered the expenditure questions. Cognitive testing was conducted 

iteratively, through rounds of roughly 10 interviews per round, for a total of 85 interviews. After 

completing each round, Westat analyzed the data, and with BLS input, revised the questions for the 

next round of testing.

2.1.2 Testing Locations 

The cognitive interviews were conducted in Westat’s cognitive interviewing facility in Rockville, 

MD, and in a hotel conference room in the other locations. The testing locations were chosen for 

operational convenience as well as representation of urban/suburban/rural locations and some 

geographic representation in the states of Maryland, Virginia, and Pennsylvania. A mix of 

demographic characteristics related to purchase behaviors was also a factor, as discussed below. 

Table 2-1 shows the locations where testing occurred and the number of interviews completed for 

each round.

Table 2-1. Testing rounds and locations

Testing round Location(s) 

Round 1 (n = 10) Rockville, MD

Round 2 (n = 10) Baltimore, MD and Rockville, MD

Round 3 (n = 10) Winchester, VA and Rockville, MD

Round 4 (n = 10) Gettysburg, PA and Rockville, MD

Round 5 (n = 10) Rockville, MD

Round 6 (n = 10) Rockville, MD

Round 7 (n = 9) Rockville, MD

Round 8 (n = 10) Gettysburg, PA and Rockville, MD

Round 9 (n = 6) Rockville, MD

Methods 2 
Cognitive Testing for the CE Recall Interview 2-2 

 

Methods 2 

2.2 Development of Question Clusters and Question Wording 

In order to make decisions about which expenditure categories to test in each round, we divided the 

38 recall interview expenditure categories into 8 clusters. The initial development process was a 

collaborative effort, with Westat providing draft questions and BLS staff providing feedback and 

suggestions.

For initial draft question wording, BLS recommended that Westat develop definitions rather than 

use examples (where possible). To develop draft question wording for each question to be tested, we 

reviewed these BLS resources: the UCC wording for each Expenditure Category, the Expenditure 

Category labels, the example items in the InfoBook (which field interviewers use to help cue recall 

of expenses during the current recall interview), and the coding of items from the diary item code 

files (which BLS uses to code open-ended diary items to UCCs).

Table 2-2 shows the clusters and the number of questions developed for each. BLS submitted the 

questions for OMB approval, as well as all participant recruitment materials and protocol for 

conducting the sessions.

Table 2-2. Question clusters and original number of questions

Cluster Number of questions 

A. Apparel and personal items 6

B. Repairs, replacements, upkeep of home; home operations 8

C. Household linens, furnishings, items and supplies, and appliances 34

D. Trips and transportation 21

E. Recreation 26

F. Children and infants 5

G. Electronics/technology 13

H. Events 3

Total 116 

2.3 Recruitment 

The objectives of the recruiting task were to:

 Identify and schedule participants who had expenditures for multiple expenditure 

questions;

 Obtain some representation of participants from urban, suburban, and rural locations; 

and
Cognitive Testing for the CE Recall Interview 2-3 

 

Methods 2 

 Ensure a demographic mix within the participant pool.

We used several methods to recruit participants for each testing round. For the testing in Rockville 

and Baltimore, we sent an email to members of Westat’s research volunteer database that described 

the purpose of the study and invited them to take the next step to determine their eligibility for 

participation. We ran Craigslist ads for each testing location. For the non-Rockville locations, we ran 

newspaper ads in local papers and also placed pay-per-click Google Display Network and Facebook 

ads to reach people who had recently made particular types of purchases.

For all recruiting methods, the potential participants were asked to either complete a web-based 

screener survey tool or call a Westat recruiter to be screened for eligibility. Westat programmed an 

online screener survey to collect demographics and responses about various types of recent 

purchases, using a commercial off-the-shelf (COTS) survey product that ensured secure collection 

and management of the recruitment data. This allowed us to expediently collect and review a large 

pool of screener responses and identify appropriate participants for each round of testing in advance 

of the scheduled dates.

Westat’s IRB reviewed and approved all recruiting and session materials. All recruiting materials 

were also submitted as part of BLS’ OMB package for this study. Appendix 1 provides a copy of the 

questions on the recruitment screener and Appendix 2 provides the BLS consent form.

2.4 Participants 

For each testing round, we reviewed the list of screened survey respondents with the aim of 

scheduling participants representing a balance of demographic categories (gender, education, 

income, race, and household composition). However, in making final decisions about which ones to 

schedule for interviews, we more heavily weighted the number of purchases and breadth of coverage

that we expected the potential participants to provide. Table 2-3 shows the demographic 

characteristics of the 85 study participants.
Cognitive Testing for the CE Recall Interview 2-4 

 

Methods 2 

Table 2-3. Demographic characteristics of study participants

Demographic characteristics Number of participants 

All Participants 85

Age 18 – 29 21

30 – 39 15

40 – 49 17

50 – 59 22

60 – 69 6

70+ 4

Gender Female 56

Male 29

Race White 36

Black 27

Asian 5

Hispanic 12

Pacific Islander 1

American Indian 1

No race indicated 3

Education No diploma 2

High school/GED 11

Some college 28

College degree 27

Post graduate degree 16

Missing 1

Household 

Composition

Lives alone 17

Multiple adults 35

Adult(s) + child (age 2 – 17) 25

Adult(s) + infant 2

Adult(s) + child + infant 5

Missing 1

Marital Status Single 34

Married 29

Separated 7

Divorced 8

Part of a couple 6

Missing 1

We interviewed more women than men; women tended to qualify with more expenses based on the 

screener. The majority of participants were from 18 – 59 years old, and lived in households with 

multiple household members. In terms of education level, the sample was approximately evenly split 

between those with up to some college vs. those with a college degree or post-graduate degree.
Cognitive Testing for the CE Recall Interview 2-5 

 

Methods 2 

2.5 Session Procedure 

Experienced Westat cognitive interviewers conducted all 85 interviews. In conjunction with BLS, 

Westat developed a general session protocol (see Appendix 3) that was used in each testing round. 

The protocol included an overview of the project and the participant’s role, the consent process

(including signature and the opportunity to ask questions), and a series of questions and activities to 

assess reporting accuracy and comprehension. Westat developed a separate interviewer guide for 

each round; each guide contained the specific question wording and order to be tested for the 

cluster assigned to each round.

During each session, we elicited participants’ verbal responses, including: (1) a 3-month expense 

total for each question they said they had spent money for within the reference period, (2) specific 

purchases/expenses, with cost and month, (3) items they named as examples of expenses for each 

question, and (4) responses to various probes. Using a program on a laptop computer, participants 

also categorized a list of items (as applicable or not applicable). (This activity was not covered for a 

subset of questions that were judged to be too “narrow” to provide a reasonable list of items to 

judge.) 

Interviewers used unscripted probes as needed to clarify participants’ responses. For some 

questions, interviewers also used scripted probes (based on the interviewer’s discretion) designed to 

assess comprehension and whether revisions (specific terms and phrases) made between rounds 

improved comprehension. We encouraged participants to comment aloud about the questions, 

specific words, or anything else that they thought was relevant while completing the comprehension 

activities.

All interviews were audio-recorded. Participants received $40 for completing a one-hour interview.

2.6 Key Metrics 

2.6.1 Expenditure Reports 

Each participant was first asked all questions included in the cluster assigned for the testing round. 

The participant answered each question with a Yes or No, and if Yes, was asked to provide an 
Cognitive Testing for the CE Recall Interview 2-6 

 

Methods 2 

estimate of how much money they had spent over the 3-month reporting period. The participant 

completed this for all questions, before moving on to the next part of the protocol.

For each three-month total reported expense, the participant was asked to report each individual 

expense/item, along with the cost and the month of purchase for each. These reported items – both 

correctly reported and over-reported items – provided insight into participants’ interpretations of 

the aggregate question meaning and the range of items considered applicable for the question.

For questions with reported expenses, the participant was asked if there were any other 

expenses/items they had considered, but decided not to report. If the participant said Yes, the 

interviewer asked for the item(s), cost(s) and month of purchase. If needed, the interviewer probed 

to elicit an explanation of why the participant decided not to report it. These explanations provided 

insight into the perceived breadth of the question, as well as areas of uncertainty.

2.6.2 Comprehension 

For most questions, the participant completed one or both of the following activities to assess 

understanding of the question wording and breadth. The activities were also intended to cue recall, 

to aid in rechecking the original expense reporting (as explained below).

 Name 5 Examples. Participants were asked to name 5 examples of expenses that 

would qualify for the question. This activity served the purpose of identifying salient 

examples across participants, as well as examples that came to mind aside from those 

that participants had actual expenses for. Commonly named examples that were not

applicable provided insight into comprehension issues, such as where exclusion text 

might be needed.

 Item Categorization. Participants were shown a list that included both applicable and 

not applicable items (“lures”), for most questions. The participant’s errors of inclusion 

and exclusion helped identify general comprehension issues, as well as specific examples 

of errors — suggesting where question wording changes might reduce both types of 

error.

Some of the questions in each cluster were too “narrow” to provide a reasonable set of 

items for a categorization task. For these questions, participants simply viewed a list of 

items that were applicable.
Cognitive Testing for the CE Recall Interview 2-7 

 

Methods 2 

2.6.3 Recheck of Expenditures 

After completing the comprehension activity(ies) for each question, the interviewer asked if the 

participant had thought of any item(s) for the question that were not already reported, and if so, the 

item, the cost, and the month purchased. As needed, the interviewer probed to determine the 

reason(s) for not reporting the item(s) earlier in the interview. (Note that in some cases, participants 

reported an item because they had seen it listed. They had essentially been “lured” into reporting an 

item and expense that was not applicable for a question. These instances were not treated as underreports during the analysis stage.)

2.7 Data Analysis 

As in BLS’ prior research with global questions, we focused the analysis on level of individual 

expenditure questions. We set up an Access database to merge and track all data for all 8 clusters 

and 116 questions. After each round was completed, we entered data for each participant for all 

questions, including expenditure report data and qualitative comments. The categorization 

judgments collected during the sessions were merged into the database. Then we coded all items as 

(1) applicable or not applicable for the question, and (2) reported within the reference period or 

telescoped. BLS reviewed any outstanding coding decisions.

We analyzed the qualitative data by reviewing all comments to identify instances of reporting and 

comprehension issues. The analysis objective was to inform the revision of questions, increase 

accurate item/expense inclusion, decrease inclusion errors and improve respondent comprehension.

We considered the following points in reviewing the evidence.

 Reporting Accuracy. Participants should have reported all of their relevant items when 

initially asked a question. If they decided to exclude any item(s), those should have only 

been items that are not applicable for the question. Forgotten items should have been 

rare or non-existent, if the wording performed well in cueing all applicable expenses. 

And there should not have been instances where respondents said No to the original 

expenditure question and later realized the answer should have been Yes.

 Comprehension. Comprehension activity results should indicate that participants were 

able to name primarily applicable examples, and that they achieved a high accuracy 

percentage (across participants) in categorizing the item lists.
Cognitive Testing for the CE Recall Interview 2-8 

 

Methods 2 

To address the research questions, we focused the analysis on these points:

 In initially answering each question, to what extent did participants recall and report 

only applicable items?

 What did each of the following indicate about the performance of the question – in 

terms of its wording and order?

– The over-reported (not applicable) items;

– The under-reported (either forgotten or excluded based on the participants’

reasoning) items;

– The named examples; and

– Misjudgments for both the applicable and not applicable items.

After the first testing of each cluster, Westat provided a detailed interim memo of reporting and 

comprehension results for each question, with recommendations for changing the question wording 

and/or order. For the clusters tested a second or third time, Westat provided a summary table with 

recommendations for any further revisions, with supporting results.
Cognitive Testing for the CE Recall Interview 3-1 

 

Sections 3.1 – 3.8 of this chapter present summary-level results for all questions in each cluster. At 

the beginning of each section, a summary table provides all questions in the cluster, in the order 

derived through testing of that cluster. Across all rounds, the questions retained their original 

assigned number. If we split a question into two or more separate questions, a new number was 

assigned to one part, and if two questions were combined, a new number was assigned to the 

combined question. The summary table also provides a column indicating testing status at the 

conclusion of testing, with one of the following three entries for each question:

 Complete: Testing may be considered complete. These questions tended to cover very 

specific expenses; reporting and comprehension evidence indicated good question 

performance. 

 Monitor: These questions would benefit from further testing so as to monitor reporting 

and comprehension evidence with a larger n. In some cases, these questions performed 

well, but performance should be monitored within the context of changes made to the 

wording and/or order of questions around them.

 Test revisions: Due to wording changes and/or order changes that were not retested, 

we recommend further testing with a larger n.

After each testing round, BLS staff and Westat project staff (including consultant Johnny Blair) 

reviewed and discussed the results and recommendations, then decided the wording to test in the 

subsequent round. All sections below describe recommendations, changes, and testing that we 

accomplished as a group.

For each question, we provide:

 The original wording and (in bold font) the wording after testing concluded for that 

cluster;

 A summary statement describing the main changes, if any, over the course of the 

testing;

 Bulleted summary points providing the rationale for recommended wording and/or 

order revisions; and

 [If applicable] Remaining issues, such as what to monitor in additional testing, and why.

Results 3 
Cognitive Testing for the CE Recall Interview 3-2 

 

Results 3 

3.1 Cluster A: Apparel and Personal Items 

This section summarizes the main results and recommendations for the 6 questions in Cluster A. 

The cluster was tested three times, in these rounds:

 Rounds 1 – 4

– In Round 1, partial data (due to session time constraints) were collected from 

5 participants.

– In Rounds 2 – 4, 6 participants completed one or more of the Cluster A questions 

they were eligible for (based on responses to screening questions).

 Round 7 (n = 9).

 Round 9 (n = 6).

Table 3-1 presents the question wording tested in these rounds, and the wording recommended after 

Round 9 concluded. For detailed results and recommendations developed after the first testing, refer 

to the Cluster A Interim Memo. For results and recommendations developed after subsequent 

rounds, refer to the Cluster A Round 7 Interim Summary Table and Cluster A Round 9 Summary 

Table.
Results 3 

Cognitive Testing for the CE Recall Interview 3-3 

 

Table 3-1. Cluster A: Summary table of tested question wording and order

Q # 

Tested wording 

(rounds 1 – 4) Tested wording (round 7) Tested wording (round 9) 

Recommended wording 

(when testing concluded) 

Testing 

status 

A. Apparel and Personal Itemsj

A1 Jewelry, watches, or parts for 

them

Jewelry, watches, or 

accessories for them. 

Include items for men, 

women, or children.

Jewelry, watches, or parts for 

them. Include items for men, 

women, or children.

Jewelry or watches, for men, 

women, or children

Test 

revisions

A2 Watch or jewelry repair, 

including sizing or cleaning

Watch or jewelry care, such 

as repairs, sizing, cleaning, 

or engraving

Care for watches or jewelry, 

such as repairs, sizing, cleaning, 

or engraving

[Same as previous round] Monitor 

A3

Clothing rental or 

preservation, or clothing 

storage outside the home

Clothing rental, clothing 

preservation, or professional 

clothing storage service

Clothing rental, clothing 

preservation, or professional 

services for storing clothes

[Same as previous round] Monitor

A5

Any uniforms or costumes for 

men, women, or children. Do 

not include costs that are 

reimbursed.

[Same as previous round]

Any uniforms or costumes for 

men, women, or children. Do 

NOT include shoes or costs that 

are reimbursed.

[Same as previous round] Monitor

A6 Any type of dresses, for 

women or girls

Any type of dresses, for 

women or girls. Do NOT 

include skirts. 

[Same as previous round] [Same as previous round] Monitor

A4

Any suits, sportcoats, or 

tailored jackets for men, 

women, or children

Any suits, sportcoats, or 

tailored jackets for men, 

women, or children. Do NOT 

include outdoor wear or 

individual tops or pants.

Any suits, sportcoats, or tailored 

jackets for men, women, or 

children. Do NOT include 

outerwear or individual tops or 

pants.

[Same as previous round] Monitor
Cognitive Testing for the CE Recall Interview 3-4 

 

Results 3 

A1 Jewelry, watches, or parts for them

A1 Jewelry or watches, for men, women, or children

Both the wording and order changed over the course of testing.

Rationale for Revising the Wording

 The question wording changed for each testing round. Participants tended to misjudge 

items that males use, such as tie pins and money clips (which are applicable for this 

question), so we added an explicit statement to broaden interpretation to any watches or 

jewelry worn by men, women, or children.

 Although we tested the phrase “or accessories for them” as a substitute for “parts for 

them” after the initial testing, we changed back to “parts for them” for Round 9 testing. 

Participants had interpreted “accessories” very broadly, which resulted in over-reporting 

of a variety of wearable items (e.g., caps, hair accessories, headset, golf glove).

 BLS verified that this question is intended to cover only purchases of jewelry or 

watches. All labor and parts/replacements (such as stones, links, batteries, etc.) are to be 

captured in A2. Therefore, we deleted the text “or parts for them” from the wording.

Rationale for Revising the Order

 We tested this question after A2 in two of the testing rounds, to try to capture expenses 

for repairs and batteries first (and deter duplicate reporting of such items). After the last 

testing round, we moved it back to the position of first in this cluster. BLS clarified that 

this question is not intended to cover any replacement items or parts; those all should 

be reported for A2. 

A2 Watch or jewelry repair, including sizing or cleaning

A2 Care for watches or jewelry, such as repairs, sizing, cleaning, or engraving

Both the wording and order changed over the course of testing.

Rationale for Revising the Wording

 To broaden thinking to any type of adjustment, not just repair, as well as to adjustments 

anyone (other than a trained repair person) might make, we decided to replace the word 

“repair” with “care” and add “engraving” to help convey the question’s breadth.

 A large over-report for a watch purchase indicated the need to ensure respondents 

interpret “care” as applicable to watches as well as jewelry, so we moved that to the start 

of the question.
Cognitive Testing for the CE Recall Interview 3-5 

 

Results 3 

Rationale for Revising the Order

 Comprehension results in the first two testing rounds showed that some participants 

judged watch battery, applicable for this question, as applicable for A1. The decision 

was to move the question to precede A1, to improve comprehension and reporting for 

A1.

Rationale for Revising the Order

 The results of the first testing indicated a need to flip the order of A1 and A2, as 

described above, with the aim of cleaner reporting and comprehension for A2 (watch 

battery applicable for A1, not A2).

Remaining Issues

In any further testing of the question, monitor whether explicit text will be needed to capture battery 

replacement. Over the testing rounds, several participants had issues in reporting watch battery 

replacement (e.g., whether it was applicable for A1 or A2) and the majority had trouble judging it.

A3 Clothing rental or preservation, or clothing storage outside the home

A3 Clothing rental, clothing preservation, or professional services for storing clothes

The question wording changed over the course of testing.

Rationale for Revising the Wording

 There was some evidence of respondent confusion about “preservation” and clothing 

storage outside the home, which led them to name examples of unrelated expenses, e.g., 

public storage units and historic landmark preservation. Interpretation of “clothing 

preservation” also led to reporting of a mending expense, in one instance

 Clothing rental is likely to be the most common type of expense for the question and 

most widely understood, so the decision was to keep that as the first part of the 

question.

 To clarify that clothing storage outside the home is limited to clothing (no other

belongings that might be stored in a storage unit), we changed the vague phrase 

“storage for clothing outside the home” to specify “professional” and “services.”
Cognitive Testing for the CE Recall Interview 3-6 

 

Results 3 

Remaining Issues

In any further testing of the question, it will be important to monitor the need for exclusion text 

such as “Do NOT include cost of storage units,” so as to avoid over-reporting of mini-storage units 

used for household items beyond clothes. Also monitor the need to deter reporting of clothing 

repairs, by narrowing the wording about preservation (e.g., to just dresses or wedding dresses).

A5 Any uniforms or costumes for men, women, or children. Do not include costs that are 

reimbursed.

A5 Any uniforms or costumes for men, women, or children. Do NOT include shoes or 

costs that are reimbursed.

The question wording changed over the course of testing.

Rationale for Revising the Wording

 Round 7 testing yielded some evidence (both over-reporting and comprehension) that 

participants thought shoes or boots – that would be worn with a uniform or costume –

were applicable. Since footwear is not applicable, we decided to add text to the 

exclusion text to reduce the chances of over-reporting.

A6 Any type of dresses, for women or girls

A6 Any type of dresses, for women or girls. Do NOT include skirts.

Both the wording and order changed over the course of testing.

Rationale for Revising the Wording

 The initial testing results showed overly broad interpretation of the original version of 

this question. Based on the comprehension results, several participants considered skirts 

as applicable.

 In terms of reporting, several items were over-reported, including a business suit, baby 

dresses, and other articles of clothing that might be worn with a dress.

 We added and tested the exclusion instruction to focus comprehension on dresses that 

are one piece, rather than on any clothing article (e.g., skirts) that shows legs.
Cognitive Testing for the CE Recall Interview 3-7 

 

Results 3 

Rationale for Revising the Order

 To help determine if reordering improved comprehension and reporting for both this 

question and A4 (the suits and jackets question), we revised the order and tested A6 

before A4.

A4 Any suits, sportcoats, or tailored jackets for men, women, or children

A4 Any suits, sportcoats, or tailored jackets for men, women, or children. Do NOT 

include outerwear or individual tops or pants.

Both the wording and order changed over the course of testing.

Rationale for Revising the Wording

 Reporting and comprehension results showed that participants misjudged some articles 

of clothing that that might be worn with a suit. This led to the decision to add wording 

to exclude clothing that is not part of a suit as well as clothing such as jackets that are 

worn outdoors (e.g., for weather protection).

Rationale for Revising the Order

 We moved this question to follow the dresses question (A6), to assess whether that 

improved both comprehension and reporting (e.g., eliciting costs for dresses before 

asking about suits, which may have multiple pieces).

Remaining Issues

In any further testing of the question, it will be important to monitor how “suit” is interpreted. Since 

the parts of a suit can be purchased separately, the exclusion text “individual tops and pants” might 

cause confusion for respondents who purchase only one piece of a suit. In addition, monitor for 

evidence of over-reporting of items that might be worn with a suit.
Cognitive Testing for the CE Recall Interview 3-8 

 

Results 3 

3.2 Cluster B: Repairs, Replacements, Upkeep of Home, and 

Home Operations 

This section summarizes the main results and recommendations for the 8 questions in Cluster B. 

The cluster was tested once, in Round 5, with 5 participants.

Table 3-2 presents the question wording tested in these rounds, and the wording recommended after 

Round 5 concluded. For detailed results and recommendations developed after the first testing, refer 

to the Interim Memo for Clusters B and F.
Results 3 

Cognitive Testing for the CE Recall Interview 3-9 

 

Table 3-2. Cluster B: Summary table of tested question wording and order

Q # Tested wording (round 5) 

Recommended wording 

(when testing concluded) 

Testing 

status 

B. Repairs, replacements, upkeep of home; home operations 

B1

Indoor flooring installation, repair, or replacement including 

wood, vinyl, tile, and wall-to-wall carpeting. Do NOT include 

expenses that will be reimbursed by someone outside of your 

household.

[Same as previous round] Monitor

B2 Materials for termite or pest control. Do NOT include professional 

service fees. [Same as previous round] Monitor

B3 Any termite or pest control professional service fees [Same as previous round] Monitor

B4 Housekeeping, lawn, or tree care services. Do NOT include 

services covered by management or maintenance fees. [Same as previous round] Monitor

B5 Outdoor equipment or tools, including motorized, electric, or 

manual [Same as previous round] Monitor

B6 Service contracts, maintenance, or repairs for any outdoor 

equipment or tools [Same as previous round] Monitor

B7 Power or non-power tools, their accessories, or hardware [Same as previous round] Monitor

B8 Items used for hanging or storage in the home or yard

[B9] Closet and storage items Test 

revisions

[B10] Shelving or storage in the garage or yard Test 

revisions

[B11] Other hardware not already mentioned Test 

revisions
Cognitive Testing for the CE Recall Interview 3-10 

 

Results 3 

B1 Indoor flooring installation, repair, or replacement including wood, vinyl, tile, and wall-towall carpeting. Do NOT include expenses that will be reimbursed by someone outside of 

your household.

B1 Indoor flooring installation, repair, or replacement including wood, vinyl, tile, and 

wall-to-wall carpeting. Do NOT include expenses that will be reimbursed by 

someone outside of your household.

The question wording and order remained the same after testing.

Comprehension results indicated participants understood the question intent, and there were no 

reporting issues, so we recommended retaining the wording.

Remaining Issues

In further testing of the question, we recommend administering it at some point after Question C2, 

in order to first cover reporting of non-permanent flooring, the more frequently-occurring types of 

items.

B2 Materials for termite or pest control. Do NOT include professional service fees.

B2 Materials for termite or pest control. Do NOT include professional service fees.

The question wording and order remained the same after testing.

Comprehension results were good, and there were no reporting issues, so we recommended 

retaining this wording for any further testing.

Remaining Issues

We suggest continuing to test both B2 and B3, to ensure that survey respondents distinguish 

between expenses for this question (which code to Expenditure Category 69 – Tools, hardware, 

outdoor equipment and supplies), vs. expenses for B3 (which code to Expenditure Category 71 –

Household operations).
Cognitive Testing for the CE Recall Interview 3-11 

 

Results 3 

B3 Any termite or pest control professional service fees

B3 Any termite or pest control professional service fees

The question wording and order remained the same after testing.

We had no reporting evidence as the basis for evaluating question performance. This is likely a lowfrequency type of expense, since some households do not hire a professional for termite and pest 

control services.

Remaining Issues

As noted above, we suggest continuing to test this question to ensure clean reporting of expenses 

for termite and pest control for the appropriate question.

B4 Housekeeping, lawn, or tree care services. Do NOT include services covered by 

management or maintenance fees.

B4 Housekeeping, lawn, or tree care services. Do NOT include services covered by 

management or maintenance fees.

The question wording and order remained the same after testing.

Although participants made reporting errors, these were errors based on memory or telescoping, and 

they seemed to understand the question intent. Applicable expenses for this question may be 

infrequent and/or seasonal in nature, so are perhaps more susceptible to memory issues.

B5 Outdoor equipment or tools, including motorized, electric, or manual

B5 Outdoor equipment or tools, including motorized, electric, or manual

The question wording and order remained the same after testing.

Rationale for Revising the Order

 The limited comprehension evidence indicated that several participants thought that 

plants and outdoor furniture seemed to apply for this question. Therefore, we 

recommended administering this question after Cluster C questions --specifically C19 

(decorations and plants for indoors) and C13 (outdoor furniture), to deter respondents 
Cognitive Testing for the CE Recall Interview 3-12 

 

Results 3 

from thinking that “equipment” in this question includes outdoor items other than 

tools.

Remaining Issues

Note that both this question (outdoor equipment and tools) and Question B7 (power tools) code to 

Expenditure Category 69 (Tools, hardware, outdoor equipment and supplies). There are two 

questions instead of one based on (1) the expectation that two questions are needed to cue reporting 

of all applicable items, and (2) the question following B5 only pertains to maintenance of outdoor 

equipment or tools (rather than to both B5 and B7). As noted above, we recommend ordering this 

question after C13 and C19. 

B6 Service contracts, maintenance, or repairs for any outdoor equipment or tools

B6 Service contracts, maintenance, or repairs for any outdoor equipment or tools

The question wording and order remained the same after testing.

There was very little evidence for judging reporting accuracy and comprehension.

Remaining Issues

This question is likely to have a relatively low base frequency. This question needs further testing 

with a larger n to assess reporting accuracy and to monitor comprehension.

B7 Power or non-power tools, their accessories, or hardware

B7 Power or non-power tools, their accessories, or hardware

The question wording and order remained the same after testing.

Comprehension results were mixed. Participants were able to correctly judge very common types of 

power and non-power tools, but non-applicable items that would be found at a hardware store were 

misjudged as applicable, indicating some confusion about the conceptual boundaries for this 

question.
Cognitive Testing for the CE Recall Interview 3-13 

 

Results 3 

Remaining Issues

This question needs further testing with a larger n to assess reporting accuracy and to monitor 

comprehension.

B8 Items used for hanging or storage in the home or yard

B9 Closet and storage items

B10 Shelving or storage in the garage or yard

B11 Other hardware not already mentioned

The question was split into three separate questions after testing. 

Rationale for Revising the Wording

 This question was originally worded as a definition, and was intended to collect items 

that map to Expenditure Category 69 (Tools, hardware, outdoor equipment and 

supplies), specifically to two UCCs (“Other hardware,” which includes curtain hardware 

– brackets, rods, tie-backs, etc., and “Closet and storage items,” which includes outdoor 

items such as storage sheds).

 Testing revealed significant comprehension issues, which was not surprising given the 

breadth of applicable items for the one question. Several participants seemed to focus 

on the word “hanging” within the question and over-reported tools that would be used 

to hang pictures, etc. Others focused on “storage” and over-reported items such as 

furniture (a dresser) and kitchen storage (a spice rack).

 We considered revision options such as using a definition or providing examples. Both 

of these would make the question longer, and not necessarily resolve the observed 

comprehension and reporting issues. To ensure coverage of all the items the original 

question was intended to capture, we suggested splitting the question into three –

indoor storage, outdoor storage, and a “catch-all” question to capture other hardware 

not already covered by questions for Expenditure Category 69.

Rationale for Revising the Order

 The question was already tested after (and in close proximity to) the questions about 

tools. An additional way to reduce the potential for over-reporting would be to 

administer Questions B9 – B11 after the furniture questions (C8 – C17), so that survey 

respondents will report furniture before they hear this question.
Cognitive Testing for the CE Recall Interview 3-14 

 

Results 3 

Remaining Issues

In further testing, monitor whether respondents consider and report furniture. If so, add exclusion 

text such as: “Do NOT include furniture” to the indoor storage question (revised and renumbered 

to become Question B9). Also, BLS will need to verify that small hardware items are intended to be 

captured with the new question B11.

3.3 Cluster C: Household Linens and Furnishings 

This section summarizes the main results and recommendations for the 35 questions in Cluster C. 

Although there were initially 34 questions, one was split into two questions, which led to the total of 

35. The cluster was tested 3 times, in these rounds: 

 Round 3 (n = 5 for each of two parts of the cluster).

– Partial data collected from 3 participants in Round 1 were also added and 

reported with Round 3 results.

 Round 6 (n = 5 for each of two parts of the cluster).

 Round 8 (n = 5 for each of two parts of the cluster).

Table 3-3 presents the question wording tested in these rounds, and the wording recommended after 

Round 8 concluded. For detailed results and recommendations developed after the first testing, refer 

to the Cluster C Interim Memo. For results and recommendations developed after subsequent 

rounds, refer to the Round 6 Interim Summary Table and Round 8 Summary Table.
Results e 3 

Cognitive Testing for the CE Recall Interview 3-15 

 

Table 3-3. Cluster C: Summary table of tested question wording and order

Q # 

Tested Wording 

(round 3) 

Tested wording 

(round 6) 

Tested wording 

(round 8) 

Recommended wording 

(when testing concluded) 

Testing 

status 

C. Household Linens and Furnishings, Part 1

C1

Any window coverings, 

including curtains, drapes, 

blinds, or shades

[Same as previous round] [Same as previous round]

Any window coverings, 

including curtains, drapes, 

blinds or shades.

Do NOT include cost of

hardware purchased 

separately.

Test 

revisions

C4 Bathroom linens, shower 

curtains, or bath mats [Same as previous round] [Same as previous round]

Any towels or other bathroom 

linens, shower curtains, or 

bath mats

Test 

revisions

C5 Bed linens, blankets, or pillows, 

for either adults or children

Bed linens, blankets, or bed 

pillows, for either adults or 

children

[Same as previous round] [Same as previous round] Monitor

C3 Slipcovers, decorative pillows, 

or cushions [Same as previous round] [Same as previous round] [Same as previous round] Monitor

C6

Kitchen, dining room, or any 

other room’s linens or fabric 

items

Linens or fabric items for the 

kitchen, dining room, or any 

other rooms not already 

covered

[Same as previous round] [Same as previous round] Monitor

C2 Non-permanent floor or stair 

coverings

Non-permanent floor or stair 

coverings, such as mats, rugs, 

treads, or runners 

[Same as previous round] [Same as previous round] Monitor

C28 Electric floor or carpet cleaning 

equipment [Same as previous round] [Same as previous round] Electrical equipment for floor or carpet cleaning

Test 

revisions

C25 Window or wall unit air 

conditioners [Same as previous round] Air conditioners installed into windows or walls [Same as previous round] Monitor

C30 Portable cooling or heating 

equipment

Portable fans, heaters, 

humidifiers, or dehumidifiers [Same as previous round] [Same as previous round] Complete

C29 Indoor lamps, ceiling or wall 

lighting, or ceiling fans

Indoor lamps, ceiling or wall 

lighting, or ceiling fans. Do 

NOT include light bulbs.

[Same as previous round] [Same as previous round] Monitor

C26

Air purifiers, water filters, 

carbon monoxide detectors, or 

smoke alarms

[Same as previous round] [Same as previous round] [Same as previous round] Monitor
Results e 3 

Cognitive Testing for the CE Recall Interview 3-16 

 

Table 3-3. Cluster C: Summary table of tested question wording and order (continued)

Q # 

Tested Wording 

(round 3) 

Tested wording 

(round 6) 

Tested wording 

(round 8) 

Recommended wording 

(when testing concluded) 

Continue 

testing 

C. Household Linens and Furnishings, Part 1

C27

Burglar alarm equipment. Do 

NOT include regular security 

alarm service fees.

[Same as previous round] [Same as previous round] [Same as previous round] Monitor

C35

[C31] Items used to serve, 

consume, or store food and 

drink

Non-disposable containers for 

storing foods or drinks

Containers for storing 

foods and drinks. Do NOT

include items you throw 

away.

[Same as previous round] Monitor

C36

[C31] Items used to serve, 

consume, or store food and 

drink

Non-disposable tableware for 

serving, eating or drinking

Tableware for serving, 

eating or drinking. Do NOT 

include items you throw 

away.

[Same as previous round] Monitor

C32 Nonelectric cookware [Same as previous round] [Same as previous round] [Same as previous round] Monitor

C33

Any other nonelectric kitchen 

gadgets, tools, or cooking 

utensils

[Same as previous round] [Same as previous round] [Same as previous round] Monitor

C. Household Linens and Furnishings, Part 2

C23 Dishwasher that is portable, 

not built-in [Same as previous round] [Same as previous round] [Same as previous round] Complete

C24

Purchase, installation, or 

maintenance of a built-in 

dishwasher, garbage disposal, 

or range hood

[Same as previous round] [Same as previous round] [Same as previous round] Monitor

C20 Small electrical kitchen 

appliances

Small electrical kitchen 

appliances, including 

microwaves

[Same as previous round] [Same as previous round] Monitor

C21

Any larger appliances for the 

kitchen or laundry room, 

excluding dishwashers. Include 

installation and haul-away 

costs. 

Larger kitchen or laundry 

room appliances. Include 

installation and haul-away 

costs. 

Larger kitchen or laundry 

appliances. Include 

installation and haul-away 

costs.

[Same as previous round] Monitor
Results e 3 

Cognitive Testing for the CE Recall Interview 3-17 

 

Table 3-3. Cluster C: Summary table of tested question wording and order (continued)

Q # 

Tested Wording 

(round 3) 

Tested wording 

(round 6) 

Tested wording 

(round 8) 

Recommended wording 

(when testing concluded) 

Continue 

testing 

C. Household Linens and Furnishings, Part 2

C22

Service contracts, maintenance, 

or repairs for those larger 

household appliances

Service contracts, maintenance, 

or repairs for those larger 

household appliances for the 

kitchen or laundry room

Service contracts, 

maintenance, or repairs 

for those larger kitchen or 

laundry appliances

[Same as previous round] Monitor

C19

Potted plants, fresh cut flowers, 

or flowers that can be worn or 

carried

Live potted plants, fresh cut 

flowers, or flowers that can be 

worn or carried 

Houseplants, fresh cut 

flowers, or flowers that 

can be worn or carried

[Same as previous round] Monitor

C18

Household decorative items such 

as clocks, art, mirrors, and 

seasonal decorations

Household decorative items such 

as artificial flowers or plants, 

clocks, art, mirrors, or seasonal 

decorations 

[Same as previous round] [Same as previous round] Monitor

C8 Infants’ furniture [Same as previous round] [Same as previous round] [Same as previous round]

Test 

(based on 

resolution 

of 

category 

mapping)

C15 Any bedroom furniture, 

mattresses, or boxsprings [Same as previous round] [Same as previous round] [Same as previous round] Monitor 

C10 Living, family, or recreation room 

furniture [Same as previous round] [Same as previous round] [Same as previous round] Monitor 

C11 Dining room or kitchen furniture [Same as previous round] [Same as previous round] [Same as previous round] Monitor

C12

Office furniture for home use, 

including filing and storage 

equipment

Filing cabinets, safes, or office 

furniture for home use [Same as previous round] [Same as previous round] Monitor

C16

Other furniture not already 

mentioned, for any other areas in 

the house

Other furniture not already 

mentioned, for any other areas 

inside the house

[Same as previous round] [Same as previous round] Monitor

C7 Rented or leased any furniture Rented or leased any furniture.

Do NOT include TV rental. [Same as previous round] [Same as previous round] Monitor

C17

Repairing, refinishing or 

reupholstering furniture, including 

the cost for fabric

[Same as previous round] [Same as previous round] [Same as previous round] Monitor
Results e 3 

Cognitive Testing for the CE Recall Interview 3-18 

 

Table 3-3. Cluster C: Summary table of tested question wording and order (continued)

Q # 

Tested Wording 

(round 3) 

Tested wording 

(round 6) 

Tested wording 

(round 8) 

Recommended wording 

(when testing concluded) 

Continue 

testing 

C. Household Linens and Furnishings, Part 2

C14 Barbecue grills, outdoor lighting 

or outdoor decorative items

Outdoor grills, lighting or other 

items, including decorations. Do 

NOT include outdoor heaters.

[Same as previous round] [C38] Outdoor grills Test 

revisions

C13 Patio, porch, or outdoor furniture 

and covers [Same as previous round] [Same as previous round] [Same as previous round] Monitor

C14 Barbecue grills, outdoor lighting 

or outdoor decorative items

Outdoor grills, lighting or other 

items, including decorations. Do 

NOT include outdoor heaters.

[Same as previous round]

[C39] Outdoor lighting or 

other items including 

decorations

Test 

revisions

C34 Repair of household equipment 

or decorative items

Repair of any household items 

we’ve covered, except 

appliances or furniture

[Same as previous round] [Same as previous round] Monitor
Cognitive Testing for the CE Recall Interview 3-19 

 

Results 3 

C1 Any window coverings, including curtains, drapes, blinds, or shades

C1 Any window coverings, including curtains, drapes, blinds, or shades. Do NOT 

include cost of hardware purchased separately.

Both the wording and order changed over the course of testing.

Rationale for Revising the Wording

 Both reporting and comprehension evidence indicated that participants thought curtain 

rods and other hardware for hanging/mounting window coverings would apply for the 

question. This prompted addition of the exclusion text.

Rationale for Revising the Order

 In Round 3, we tested this question first in a series of questions related to linens and 

fabric items. In Round 6, we tested it as the first question in Part 2 of the cluster, but 

found two instances of curtains reported when asking about linens in any rooms. To 

assist with clean reporting of curtains (for any room), we moved this question to again 

precede all fabric-related questions.

Remaining Issues

The hardware purchased separately is applicable for the recommended addition of Question B11

(Other hardware not already mentioned).

C4 Bathroom linens, shower curtains, or bath mats

C4 Any towels or other bathroom linens, shower curtains, or bath mats

The question wording changed but the order remained the same over the course of testing.

Rationale for Revising the Wording

 Beach towels were both misjudged and forgotten over the testing rounds. They are 

applicable, but not typically thought of as bathroom linens (since they are used more in 

locations other than a bathroom). That led to the change to include any towels. 
Cognitive Testing for the CE Recall Interview 3-20 

 

Results 3 

Remaining Issues

This question performed well, with the exception of beach towels. In further testing of the 

question, monitor whether there is any additional evidence that beach towels are not considered 

applicable for this question with the revised wording. 

C5 Bed linens, blankets, or pillows, for either adults or children

C5 Bed linens, blankets, or bed pillows, for either adults or children

The question wording changed over the course of testing.

Rationale for Revising the Wording

 This question was one of the best performers in this cluster in terms of both accuracy 

and comprehension. After the initial testing, we added the word “bed” before pillows, 

based on the recommendation for ordering Question C3 (which asks about “decorative 

pillows”) after C5. This recommendation is to ensure reporting of bed pillows with bed 

linens and to minimize confusion about different types/functions of pillows.

Remaining Issues

After testing, BLS questioned whether this question needed to have the “for either adults or 

children” text. In the future, test without that phrase and monitor its performance.

C3 Slipcovers, decorative pillows, or cushions

C3 Slipcovers, decorative pillows, or cushions

The question wording remained the same but the order changed over the course of testing.

Rationale for Revising the Order

 Round 3 evidence indicated some overlap with Question C5 (bed linens, etc.), so in 

Round 6, we tested it after that question to reduce the opportunity for reporting 

bedding-related items for this question. Pillows and cushions are items that could be in 

bedrooms or any other rooms. The question also preceded Question C6, which serves 

as the “catch-all” question (“... or any other room’s linens”) for linen items and is 

intended to be last in the series of four questions about household linens.
Cognitive Testing for the CE Recall Interview 3-21 

 

Results 3 

Remaining Issues

A qualitative comment indicated that a sham may be interpreted as an actual pillow, not just the 

cover. We suggest monitoring whether shams are interpreted as pillows or linens (C5) in further 

testing of the question.

C6 Kitchen, dining room, or any other room’s linens or fabric items

C6 Linens or fabric items for the kitchen, dining room, or any other rooms not already 

covered

Both the question wording and order changed over the course of testing.

Rationale for Revising the Wording

 This question is intended to collect linens that are not for bedrooms and bathrooms 

(covered by Questions C4 and C5). We saw limited evidence that participants thought 

too broadly about any items made from fabric, thought that this question overlapped 

with others (e.g., Question C3, which mentions slipcovers), or reported items (such as 

coffeemaker and crockpot) that would be in a kitchen.

 We decided to re-word the question to mention the types of items first, then the rooms 

where the items are used. This was intended to reduce the chance of survey respondents 

thinking too broadly and reporting non-applicable expenses.

Rationale for Revising the Order

 Due to two reports of curtains for this question in Round 6 testing, we recommended 

testing this question in Round 8 at some point after Question C1.

C2 Non-permanent floor or stair coverings

C2 Non-permanent floor or stair coverings, such as mats, rugs, treads, or runners

Both the question wording and order changed over the course of testing.

Rationale for Revising the Wording

 The original wording was a definition. Based on some confusion in Round 3 testing 

about which types of floors qualify as “non-permanent,” we added examples to help 

clarify the question meaning.
Cognitive Testing for the CE Recall Interview 3-22 

 

Results 3 

Rationale for Revising the Order 

 Comprehension results indicated that bath rugs were considered applicable for this 

question. We moved this question and tested it after Question C4 (bathroom linens, 

which includes bath mats), so that bathmats can be captured in that question instead of 

incorrectly captured or double-reported for this question.

Remaining Issues

In response to probing about “non-permanent,” participants clearly differentiated between movable 

types of floorings (e.g., carpet protectors, area rugs, doormats) and more permanent types of 

coverings (carpet, hardwood, tile). However, the judgment activity still showed some minor 

evidence of confusion about whether any flooring is truly “permanent” and about what applies. This 

should be monitored in further testing.

C28 Electric floor or carpet cleaning equipment

C28 Electrical equipment for floor or carpet cleaning 

The question wording changed but the order remained the same over the course of testing.

Rationale for Revising the Wording

 In addition to several participants forgetting items (such as vacuum cleaner or hand 

vacuum cleaner) at the time of initial reporting, several also reported the cleaning 

solutions used with cleaning equipment. Therefore, we suggested putting “electrical” 

and “equipment” together, which may help to clarify that only the equipment expense is 

applicable, not the supplies that are used in the cleaning process.

C25 Window or wall unit air conditioners

C25 Air conditioners installed into windows or walls

Both the question wording and order changed over the course of testing.
Cognitive Testing for the CE Recall Interview 3-23 

 

Results 3 

Rationale for Revising the Wording

 One participant’s comment that a “wall unit” could refer to central air indicated the 

potential for confusion about the breadth of the question. We changed the wording to 

clarify that units are installed, vs. the more permanent type of heating/cooling.

Rationale for Revising the Order

 Discussion of the potential for overlap with Question C30 led to the decision to 

administer this question just before it, to assist interviewers in real field conditions in 

catching any potential double-reporting issues.

Remaining Issues

In further testing of the question, monitor whether participants correctly interpret “installed into.” If 

they do not, explore “installed through” as an option. Also monitor whether participants who own 

window or wall dual cooling/heating equipment would report those units here or for the next 

question. Such dual units could be reported for either question, as long as they are not doublereported. Expenses for both C25 and C30 would be coded to Expenditure Category 67

(Appliances).

C30 Portable cooling or heating equipment

C30 Portable fans, heaters, humidifiers, or dehumidifiers

The question wording changed over the course of testing.

Rationale for Revising the Wording

 The word “equipment” was intended to be broad enough to cue thinking of humidifiers 

and dehumidifiers, in addition to fans and heaters. All of these need to be coded to 

appliances (Expenditure Category 67). However, based on the comprehension results, 

participants seemed to focus narrowly on just fans and heaters, not items which adjust 

the humidity of the air. Therefore, we added wording that covered the options 

applicable for the question – using explicit text to improve reporting accuracy.

Rationale for Revising the Order

 We placed this question after Question C25, based on the rationale provided above.
Cognitive Testing for the CE Recall Interview 3-24 

 

Results 3 

Remaining Issues

In further testing of the question, monitor whether participants associate “portable” with heaters, 

not just with fans (cooling). If not, change the wording to “Portable fans, portable heaters...”

C29 Indoor lamps, ceiling or wall lighting, or ceiling fans

C29 Indoor lamps, ceiling or wall lighting, or ceiling fans. Do NOT include light bulbs.

Both the question wording and order changed over the course of testing.

Rationale for Revising the Wording

 Due to several instances of reporting light bulbs as well as comprehension evidence that 

some participants thought these were applicable for the question, we added the 

exclusion text “Do NOT include light bulbs.”

Rationale for Revising the Order

 After the initial testing, the decision was to test this question after C30, to avoid 

reporting of portable fans for this question. That change allowed asking about different 

types of fans (ceiling, portable) in close proximity. We also moved the question so that 

it would flow well within a series of questions covering the structure or systems of a 

house – floors, alarms, filters, etc.

C26 Air purifiers, water filters, carbon monoxide detectors, or smoke alarms

C26 Air purifiers, water filters, carbon monoxide detectors, or smoke alarms

The question wording and order remained the same over the course of testing.

Testing yielded all accurate reports over the testing rounds and no comprehension issues.

C27 Burglar alarm equipment. Do NOT include regular security alarm service fees.

C27 Burglar alarm equipment. Do NOT include regular security alarm service fees.

The question wording and order remained the same over the course of testing.
Cognitive Testing for the CE Recall Interview 3-25 

 

Results 3 

There were no reporting issues for the question. However, there were minor comprehension issues; 

several participants considered that other types of security system/devices, such as electric fences or 

medical alerts, might be applicable.

Remaining Issues

In further testing of the question, continue to monitor whether participants understand the breadth 

of the question – only burglar/security alarm equipment, and no other types of security systems.

C31 Items used to serve, consume, or store food and drink

C35 Containers for storing foods and drinks. Do NOT include items you throw away.

The original question was split into two separate questions, but the administration order remained 

the same over the course of testing.

Rationale for Revising the Wording

 Reporting and comprehension results indicated a need to revise the question after the 

initial testing. Participants over-reported disposable items (e.g., plastic utensils, napkins,

aluminum food containers) and exhibited some confusion in judging what qualified as a 

storage item (e.g., a cooler, a bowl without a cover).

 Instead of lengthening the question with examples to indicate the question’s breadth, or 

adding exclusion text, we decided to split the question into two new questions:

– Question 35 – to cover expenses related to storing foods and beverages; and

– Question 36 – to cover expenses for items used in serving and consuming foods 

and beverages.

 Round 6 testing of the revised questions (with the word “non-disposable”) showed 

some evidence of participant confusion. We tested with exclusion text “Do NOT

include items you throw away” in Round 8.

 In Round 8, response to probing (to assess whether the Round 8 wording was clearer 

than the Round 6 wording), participants gave good examples of items that would be 

thrown away, e. g., Styrofoam, paper containers, disposable Ziploc bags. Judgment 

activity results continued to show that participant thinking does not correspond with 

BLS categorization: 4 of 5 participants judged both Gladware and Ziploc containers as 

applicable (these containers may be either disposable or non-disposable). 
Cognitive Testing for the CE Recall Interview 3-26 

 

Results 3 

Remaining Issues

The item categorization scheme does not correspond well with how people think about, and report 

for, this question. Respondents are likely to have differing interpretations of which containers are 

“disposable” vs. more permanent (e.g., Gladware- and Ziploc-type items vs. “Tupperware”). In 

further testing, monitor whether containers intended for a single use are reported.

C31 Items used to serve, consume, or store food and drink

C36 Tableware for serving, eating or drinking. Do NOT include items you throw away.

The original question was split into two separate questions, but the administration order remained 

the same over the course of testing.

Rationale for Revising the Wording

 The rationale for revising this question is explained above for Question C35.

Remaining Issues

For further testing of this question, the same caution noted for C35 applies here.

C32 Nonelectric cookware

C32 Nonelectric cookware

The question wording and order remained the same over the course of testing.

This question performed well in terms of both accuracy and comprehension, so we did not make 

any changes.

C33 Any other nonelectric kitchen gadgets, tools, or cooking utensils

C33 Any other nonelectric kitchen gadgets, tools, or cooking utensils

The question wording and order remained the same over the course of testing.
Cognitive Testing for the CE Recall Interview 3-27 

 

Results 3 

The reporting and comprehension data indicated that the question performed relatively well, so we 

did not make any revisions.

C23 Dishwasher that is portable, not built-in

C23 Dishwasher that is portable, not built-in

The question wording remained the same but the order changed over the course of testing.

None of the participants for Cluster C had any expenses to report (correctly or incorrectly) for this 

question. Due to the narrowness of the question, we also had no comprehension evidence to 

evaluate. Portable dishwashers are not purchased as often as built-in dishwashers.

Rationale for Revising the Order

 After Round 3 testing, the decision was to order the appliance-related questions by 

starting first with C23, the most specific question. This followed a general principle we 

adopted for question ordering: ask about a very specific item first within a series of 

related questions, then move on to more general questions. The intent is to cue a report 

of the specific item once, so that it will not be reported again later in response to a more 

general question.

Remaining Issues

This question is not likely to need further testing. It is at a more specific level than all other 

questions in the cluster, and those who have purchased portable dishwashers are likely to understand 

the question. The text “not built-in” should serve to keep people from reporting the more common 

type of dishwasher until the next question, where it is specifically named. This question was asked 

separately since portable dishwashers are coded to a different Expenditure Category. Built-in 

dishwashers are coded to Expenditure Category 65 (Unpriced maintenance and repair services), 

whereas portable dishwashers are coded to Expenditure Category 67 (Appliances).

C24 Purchase, installation, or maintenance of a built-in dishwasher, garbage disposal, or range 

hood

C24 Purchase, installation, or maintenance of a built-in dishwasher, garbage disposal, or 

range hood

The question wording remained the same but the order changed over the course of testing.
Cognitive Testing for the CE Recall Interview 3-28 

 

Results 3 

Rationale for Revising the Order

 Based on the reasoning noted for C23 above, we moved this question to precede other 

more general appliance questions – C20 (small appliances) and C21 (larger appliances).

Remaining Issues

There was one report during the testing that indicated a potential issue. An over-the-range 

microwave oven may be perceived as serving a similar function to a range hood if it has a fan or 

ventilation system. We retained the administering order for this question (after Question C20), 

which asks about microwave ovens (small electrical kitchen appliances). With explicit text about

microwaves in that question, an interviewer would be aware of how to treat pre-reporting of a 

microwave for this question.

C20 Small electrical kitchen appliances

C20 Small electrical kitchen appliances, including microwaves

Both the question wording and order changed over the course of testing.

Rationale for Revising the Wording

 We added explicit text to include microwaves, to standardize reporting for this question. 

Since these vary widely in size, and have generally increased in capacity (as well as 

function, e.g., combination microwave/convection) over time, there was some 

uncertainty about whether they qualify as small or larger appliances. Using inclusion text 

for this question also avoided the need to add exclusion text to Question C21.

Rationale for Revising the Order

 Based on the reasoning noted for C23 above, we moved this question to follow 

questions that name specific appliances.

Remaining Issues

BLS verified that smaller versions of appliances considered to be “large appliances,” such as minirefrigerators (and wine refrigerators) should be reported as large appliances. All appliances are coded 

to the same Expenditure Category 67, so survey respondents could report smaller versions of 

appliances for this question (as long as they are not double-reported). Odds of that happening are 

reduced by administering these questions in order.
Cognitive Testing for the CE Recall Interview 3-29 

 

Results 3 

C21 Any larger appliances for the kitchen or laundry room, excluding dishwashers. Include

installation and haul-away costs.

C21 Larger kitchen or laundry appliances. Include installation and haul-away costs.

Both the question wording and order changed over the course of testing.

Rationale for Revising the Wording

 Due to confusion about applicability of appliances that may be in a laundry room but 

serve other purposes (e.g., an HVAC system), we decided to delete the word “room.” In 

addition, some laundry appliances are not in an actual room.

Rationale for Revising the Order

 Based on the reasoning noted for C23 above, we moved this question to follow 

questions that name specific appliances. We also retained its order after C20, so as to 

capture smaller appliances (including microwaves) first.

C22 Service contracts, maintenance, or repairs for those larger household appliances

C22 Service contracts, maintenance, or repairs for those larger kitchen or laundry 

appliances

The question wording changed but the order remained the same over the course of testing.

Rationale for Revising the Wording

 This question is intended to collect service and repairs for the kinds of appliances asked 

about in C21. However, the wording “household appliances” in the initially tested 

version is broader than the appliances mentioned in C21. To avoid over-reporting of 

appliance repair expenses that are not applicable (e.g., for water heater, air conditioner, 

heater), we changed to the explicit text “kitchen or laundry appliances.”

C19 Potted plants, fresh cut flowers, or flowers that can be worn or carried

C19 Houseplants, fresh cut flowers, or flowers that can be worn or carried

Both the question wording and order changed over the course of testing.
Cognitive Testing for the CE Recall Interview 3-30 

 

Results 3 

Rationale for Revising the Wording

 In Round 6, we tested an alternative wording “Live potted plants, fresh cut...” to assess 

whether that helped to differentiate living plants/flowers from artificial ones – which 

are covered in Question C18. Reporting and comprehension results showed that some 

participants considered items related to outdoor plants and gardening/landscaping as 

applicable. To capture indoor plants only (not those bought in a pot of any type – and 

then transplanted into the ground from those pots), we tested “Houseplants, fresh cut 

flowers” in Round 8.

Rationale for Revising the Order

 The decision was to test this question before Question C18 so as to capture all living 

flowers and plants first, before artificial flowers or plants.

Remaining Issues

In further testing of the question, continue to monitor whether the change to “houseplants”

eliminates reporting of transplantable plants for outdoors. If outdoor plants continue to be reported, 

consider adding exclusion text such as: “Exclude items for outdoor gardening and landscaping.”

This question should follow (a) tools for outside maintenance (B5) and (b) lawn care services (B4) to 

minimize over-reporting of any outdoor plant-related items. In addition, BLS may need to decide 

whether plants on decks and porches qualify as houseplants (“indoors”) – as long as plant expenses 

are not double-reported.

C18 Household decorative items such as clocks, art, mirrors, and seasonal decorations

C18 Household decorative items such as artificial flowers or plants, clocks, art, mirrors, 

or seasonal decorations

Both the question wording and order changed over the course of testing.

Rationale for Revising the Wording

 In initial testing, this question had significant reporting accuracy issues – both underand over-reporting. One of the reasons is that decorative items may also be functional.

 We also added “artificial plants or flowers” to cue memory for those types of 

decorations.
Cognitive Testing for the CE Recall Interview 3-31 

 

Results 3 

Rationale for Revising the Order

 We re-ordered this question to follow Question C19, as described above, to minimize

the potential for blurring and mis-reporting for these two questions.

Remaining Issues

In further testing of the question, continue to monitor how the phrase “decorative items” is 

interpreted. Decorative items may also be functional, which may affect reporting.

C8 Infants’ furniture

C8 Infants’ furniture

The question wording remained the same but the order changed over the course of testing.

Rationale for Revising the Order

 This question was tested within two different series of questions: with other furniturerelated questions (in Cluster C), and with other questions related to children 

(in Cluster F). The same comprehension issues occurred with both administration 

placements. Participants both named examples and judged furniture items as equipment 

and vice-versa. 

Remaining Issues

The question is problematic due to the conceptual overlap between items that are applicable for this 

question, which code to Expenditure Category 66 (Furniture) vs. Question C9, which code to 

Expenditure Category 51 (Miscellaneous personal goods). Items for infants that serve a similar 

purpose, are of similar size, or are equally mobile code to different category numbers. For example, 

the varieties of items that infants can sit or play in are coded to different categories. Options for 

changing the question – changing the wording to a definition, providing examples of which specific 

furniture items to include, and/or using exclusion text to rule out items that qualify as “infants’

equipment” – would all make the question very lengthy, and may not improve comprehension and 

reporting.

The CPI categorization scheme does not correspond with how people think and report for this 

question. It would be clearer and less burdensome in the interview context to ask respondents to 

report all infant-related items in one question. We strongly suggest asking about all infant-related 
Cognitive Testing for the CE Recall Interview 3-32 

 

Results 3 

expenses in one question, in Cluster C, preceding other questions related to expenses for furniture. 

If that is not possible, test this question with the furniture questions, since they all code to 

Expenditure Category 66 (Furniture). Keeping two questions has the potential for over- or underreporting, whether these questions are administered together or in two separate series of questions.

C15 Any bedroom furniture, mattresses, or boxsprings

C15 Any bedroom furniture, mattresses, or boxsprings

The question wording and order remained the same over the course of testing.

The reporting and comprehension data indicated that the question performed relatively well, so we 

did not make any revisions.

Remaining Issues

In further testing of the question, continue to monitor whether other items that could be in a 

bedroom (but are not applicable) are reported here, such as decorative items, linens, and storage 

items.

C10 Living, family, or recreation room furniture

C10 Living, family, or recreation room furniture

The question wording remained the same but the order changed over the course of testing.

Rationale for Revising the Order

 There was some confusion about applicability of expenses for TVs as well as for items 

(ping-pong or pool table) and entertainment-related activities associated with the rooms 

mentioned in the question wording. Therefore, we tested this question after Question 

C18 (decorative room items) and recommended administering it after several questions

in other clusters (as noted below), so as to minimize over- and under-reporting issues.

Remaining Issues

In further testing of the question, the following administration order recommendations need to be 

taken into account. Administer C10 after:
Cognitive Testing for the CE Recall Interview 3-33 

 

Results 3 

 Question G9 (from the Electronics Cluster ), which specifically addresses expenditures 

for TVs in a home or vehicle;

 Question G8 (from the Electronics Cluster ), which asks for game systems; and

 Question E10 (from the Recreation Cluster), which asks about “Ping-pong tables, pool 

tables, or other similar recreation room items and accessories.”

C11 Dining room or kitchen furniture

C11 Dining room or kitchen furniture

The question wording remained the same but the order changed over the course of testing.

Rationale for Revising the Order

 A reporting error revealed that since a refrigerator is a large item in a kitchen, it might 

be interpreted as furniture, especially if a respondent has not already reported about 

appliances. We moved this question after the large appliances question (Question C21).

C12 Office furniture for home use, including filing and storage equipment

C12 Filing cabinets, safes, or office furniture for home use

Both the question wording and order changed over the course of testing.

Rationale for Revising the Wording

 The comprehension results, as well as the number and type of mis-reports, showed that 

several participants thought more broadly than intended about this question. The words 

“filing and storage” and “equipment” seemed to bleed over to computer-related 

equipment and office supplies. We changed the wording to use examples of categories 

of furniture (safes and cabinets), to emphasize furniture rather than equipment or 

supplies.

Rationale for Revising the Order

 We moved this question based on the reasoning that if respondents have already 

reported computers (Question G5), they may be less likely to consider those expenses 

here.
Cognitive Testing for the CE Recall Interview 3-34 

 

Results 3 

Remaining Issues

In further testing of the question, ensure that this question is administered after Question G5. Also 

continue to monitor whether office supplies are over-reported for this question.

C16 Other furniture not already mentioned, for any other areas in the house

C16 Other furniture not already mentioned, for any other areas inside the house

Both the question wording and order changed over the course of testing.

Rationale for Revising the Wording

 To differentiate more clearly between questions about furniture in the house (Questions 

C7 – C12 and C15) vs. furniture outdoors, we modified this question slightly to “... any 

other areas inside the house.”

Rationale for Revising the Order

 We moved this question to precede the two questions about outdoor items: C13 

(outdoor furniture) and C14 (grills, lighting, décor).

C7 Rented or leased any furniture

C7 Rented or leased any furniture. Do NOT include TV rental.

Both the question wording and order changed over the course of testing.

Rationale for Revising the Wording

 Limited comprehension evidence showed that participants considered TVs to be 

furniture. We added exclusion text to address that, since TV rental may be a common 

type of rental expense.

Rationale for Revising the Order

 BLS noted that in the current recall interview, furniture rental is asked last. The intent of 

that order is to ensure that respondents are cued about all types of furniture (purchases) 
Cognitive Testing for the CE Recall Interview 3-35 

 

Results 3 

before they are asked to report about renting or leasing furniture. The decision was to 

apply that convention here, as well.

Remaining Issues

In further testing of the question, consider administering this question after H1 (catered events) to 

minimize the potential for respondents to report rental of furniture (such as tables for catered 

affairs) for this question.

C17 Repairing, refinishing or reupholstering furniture, including the cost for fabric

C17 Repairing, refinishing or reupholstering furniture, including the cost for fabric

The question wording and order remained the same over the course of testing.

Reporting and comprehension evidence indicated good comprehension. 

C14 Barbecue grills, outdoor lighting, or outdoor decorative items

C38 Outdoor grills

C39 Outdoor lighting or other items including decorations

This question was split into two separate questions; the order also changed over the course of 

testing.

Rationale for Revising the Wording

 In group discussion of this item, the decision was to re-word the question to start with 

the word “outdoor,” so as to clearly indicate what kinds of items to report.

 Across the rounds, grills were both reported and named as examples for outdoor 

furniture when C13 was asked first, and furniture was reported for C14 when it was 

asked first. The decision was to separate this question into two parts: grills (alone) and 

other outdoor items such as lighting and decorations. 

Rationale for Revising the Order

 This question followed C13 in Round 3 testing. Reporting evidence (grills thought of as 

“outdoor furniture” and reported for C13) led to the decision to test C14 first in Round 

6. In Round 6, the opposite occurred – reporting of furniture for this question. 
Cognitive Testing for the CE Recall Interview 3-36 

 

Results 3 

 The decision to break the question into two parts allows for asking first about grills (the 

most specific expense), then furniture, then other outdoor items. 

Remaining Issues

Items collected with this question are to be coded to Expenditure Category 66 (Furniture). Items 

collected with C13 are to be coded to Expenditure Category 69 (Tools, hardware, outdoor 

equipment and supplies). The categorization scheme does not correspond well with how people 

think and report. The potential for reporting errors (e.g., double-reporting) should be minimized by 

breaking the question up and administering in the revised order.

C13 Patio, porch, or outdoor furniture and covers

C13 Patio, porch, or outdoor furniture and covers

The question wording remained the same but the order changed over the course of testing.

Rationale for Revising the Order

 In initial testing, there was evidence of over-reporting of items such as grills that were 

applicable for question C14. Comprehension evidence corresponded with this finding.

Survey respondents may report grills as outdoor furniture due to their size and 

widespread use.

 This question will be administered after the new C38 and before the new C39. 

Remaining Issues

Monitor through further testing whether the revisions to both wording and order for C38, C13, and 

C39 lead to improved comprehension and reporting.

C34 Repair of household equipment or decorative items

C34 Repair of any household equipment or decorative items

The question wording changed but the order remained the same over the course of testing.

Rationale for Revising the Wording

 This question is intended to serve as a catch-all for any repairs not explicitly covered by 

the appliance repair (C22) and furniture repair (C17) questions. We did not have a true 
Cognitive Testing for the CE Recall Interview 3-37 

 

Results 3 

test of the performance of this item, due to the need to administer Cluster C in two 

parts. We tested the question last at the end of Part 1 and Part 2, to see what types of 

repairs were reported and considered to be applicable based on the wording.

 “Equipment” repair was nebulous; participants interpreted it to include appliances, 

major household systems (plumbing and electrical), furniture, and the structure of the 

house itself (drywall, flooring). Based on the participants thinking too broadly about the

types of repair to be included, we revised the question and tested it in a way that applied 

for testing each half of the cluster. There was very little data to evaluate its performance.

Remaining Issues

In further testing, this question will need to be administered at the end of all the Cluster C questions, 

with test participants completing all questions within the cluster.

Under real field conditions, this question will be:

 Repair of household equipment or decorative items.

If there is evidence that participants double-report appliance and/or furniture repairs, add exclusion 

text: “Do NOT include furniture or appliance repairs.”

3.4 Cluster D: Trips and Transportation 

This section summarizes the main results and recommendations for the 5 in-town questions and 16

out-of-town travel questions in Cluster D. 

After answering the in-town questions, participants were asked:

 Did you take any out-of-town trips since the first of [the reference month 3 months 

prior]? 

 During the period from [start of reference month – end of reference period], did you 

pay for anything related to any out-of-town trips? 

All participants had screened in as eligible for some type of out-of-town trip in the three months 

preceding the interview date, so all answered yes to either or both of these questions. Then they 

were told: “For the following questions, please think about money spent since the first of [the 

reference month 3 months prior] for out-of-town trips. Please do not include any costs that:
Cognitive Testing for the CE Recall Interview 3-38 

 

Results 3 

 You will be reimbursed for OR

 Were covered by a package deal.

The CE current interview includes text: “Sometimes when people take a trip they have some sort of 

package deal that covers some or all of the costs. Was all or part of this trip/these trips covered by a 

package deal?” If the respondent replies yes, they are asked a series of yes/no questions about what 

types of expenses (e.g. food and beverages, lodging, transportation) the package deal included. In the 

testing, we focused on testing comprehension of the wording of the out-of-town trips questions; we 

did not collect judgments about what a package deal covered. In further testing of the questions, 

after or in concert with testing comprehension, the interview approach could be extended to ask 

about the allocation of costs to a package deal (if applicable). 

The cluster was tested in the following rounds:

 Rounds 1 – 2

– In Round 1, partial data were collected from 7 participants (the number who 

completed each question varied due to session time constraints).

– In Round 2 (n = 5).

 Round 5 (n = 5).

Table 3-4 presents the question wording tested in these rounds, and the wording recommended after 

Round 5 concluded. For detailed results and recommendations developed after the first testing, refer 

to the Cluster D Interim Memo. For results and recommendations developed after Round 5, refer to 

the Cluster D Summary Table.
Results 3 

Cognitive Testing for the CE Recall Interview 3-39 

 

Table 3-4. Cluster D: Summary table of tested question wording and order

Q # Tested wording (rounds 1-2) Tested wording (round 5) 

Recommended wording 

(when testing concluded) 

Testing 

status 

D. Trips and Transportation: In-town questions (limited Round 1 data)

D1 Automobile service clubs such as AAA, or 

vehicle services such as OnStar and LoJack [Same as previous round] [Same as previous round] Complete

D3 Local tolls or electronic toll passes [Same as previous round] [Same as previous round] Monitor 

D5 Any travel items, luggage, or any type of 

bags for carrying belongings [Same as previous round] [Same as previous round] Monitor

D4 Any membership costs or other expenses 

related to vacation clubs or timeshare dues [Same as previous round] [Same as previous round]

Test 

revisions

D. Trips and Transportation: Out-of-town travel questions Transition text

Did you take any out-of-town trips since the 

first of [the ref month 3 months prior]?

During the period from [start of ref month –

end of ref period], did you pay for anything 

related to any out-of-town trips?

IF YES TO EITHER: For the following 

questions, please think about money spent 

since the first of [the ref month 3 months 

prior] for out-of-town trips. Please do not 

include any costs that:

• You will be reimbursed for OR

• Were covered by a package deal.

D10 Lodging. Do not include expenses previously 

reported for vacation clubs and timeshares.

Cost of hotel or any lodging, including 

taxes. Do NOT include expenses 

previously reported for vacation clubs 

and timeshares.

Cost of hotel or any lodging, 

including taxes. Do NOT include 

optional costs, such as parking, food, 

and Internet service.

Test 

revisions

D6 Airfare, including baggage and change fees

Airfare, including baggage and change 

fees. We’ll ask about other 

transportation costs next.

[Same as previous round] Monitor

D7

D9

Bus or train fare to reach your destination

Local transportation during a trip, such as 

shuttles, taxis, buses, subways or trains, 

and ferries

[D97] Transportation such as shuttles, 

taxis, buses, subways, trains, or ferries, 

at any time during a trip

[Same as previous round] Monitor 
Results 3 

Cognitive Testing for the CE Recall Interview 3-40 

 

Table 3-4. Cluster D: Summary table of tested question wording and order (continued)

Q # Tested wording (rounds 1-2) Tested wording (round 5) 

Recommended wording 

(when testing concluded) 

Testing 

status 

D. Trips and Transportation: In-town (limited Round 1 data)

D8 Fares or tips for cruise ships or ferries Tickets, tips, or service charges for cruise 

ships [Same as previous round] Monitor

D13 Rented vehicles for recreational use. Do not 

include fuel.

Cost to rent an RV, boat, trailer, or any 

kind of vehicle for outdoor recreation

Cost to rent an RV, boat, trailer, or 

any kind of vehicle for outdoor 

recreation. Do NOT include fuel.

Test 

revisions

D12 Rental vehicles for getting around. Do not 

include fuel.

Cost to rent a car, truck, motorcycle, or 

plane

Cost to rent a car, truck, motorcycle, 

or plane. Do NOT include fuel.

Test 

revisions

D11 Gasoline, diesel, or any other vehicle fuels Gasoline, diesel, or any other vehicle 

fuels [Same as previous round] Monitor

D14 Parking fees Parking fees [Same as previous round] Complete

D15 Any tolls Any tolls [Same as previous round] Monitor 

D16 Meals, snacks, or drinks at restaurants, 

bars, or fast food places

Meals, snacks, or drinks at restaurants, 

bars, or fast food places [Same as previous round] Monitor

D17

[Follow-up question to D16]: How much of 

that amount was for alcohol? 

Alcohol at restaurants, bars, or fast food 

places

[Follow-up question to D16]: How much 

of that amount was for alcohol? 

Alcohol at restaurants, bars, or fast food 

places

[Same as previous round] Monitor

D18 Food or beverages at grocery or 

convenience store

Food or beverages from grocery or other 

stores [Same as previous round] Monitor

D19

[Follow-up question to D18]: How much of 

that amount was for alcohol? 

Alcohol at grocery or convenience stores

[Follow-up question to D18]: How much 

of that amount was for alcohol? 

Alcohol from grocery or other stores

[Same as previous round] Monitor 

D21 Admission fees, tickets, tours, or other 

entertainment

Admission fees, tickets, tours, or other 

entertainment

Tours, admission fees, or tickets for 

entertainment

Test 

revisions

D20 To play sports, exercise, or rent any sports 

equipment

To play sports, exercise, or rent any 

sports equipment [Same as previous round] Test revisions

D2

Mass transportation, commuter, or carpool 

services. Do NOT include expenses covered 

by your employer.

[Note: this is an in-town transportation 

question, but was moved here based on 

results and recommendations (below)]

Mass transportation, commuter, or 

carpool services. Do NOT include 

expenses covered by your employer.

[Same as previous round] Test 

revisions
Cognitive Testing for the CE Recall Interview 3-41 

 

Results 3 

D1 Automobile service clubs such as AAA, or vehicle services such as OnStar and LoJack

D1 Automobile service clubs such as AAA, or vehicle services such as OnStar and 

LoJack

The wording and order remained the same over the course of testing.

Limited comprehension results were positive. This question covers very specific expenses and is a 

low priority for further testing. 

D3 Local tolls or electronic toll passes

D3 Local tolls or electronic toll passes

The wording remained the same but the order changed over the course of testing.

Rationale for Revising the Order

 Due to overlaps with comprehension for D2 (described below), the decision was to 

administer this more specific question before D2.

Remaining Issues

In further testing of the question, monitor whether the order change for this question results in 

cleaner reporting (for D2, as well). 

D5 Any travel items, luggage, or any type of bags for carrying belongings

D5 Any travel items, luggage, or any type of bags for carrying belongings

The wording remained the same but the order changed over the course of testing.

Rationale for Revising the Order

 Due to decisions about moving questions that originally preceded and followed this 

question, the question order changed. 
Cognitive Testing for the CE Recall Interview 3-42 

 

Results 3 

Remaining Issues

Both reporting and comprehension provided some evidence of confusion about the breadth of the 

question. “Luggage” seemed to be clear, but both “travel items” and “bags for carrying belongings” 

are vague and led to some differential interpretation. However, the decision was to retain the 

question wording. For further testing of the question, we suggest monitoring evidence of a need to 

split this into 2 questions. If needed, re-test: one question to cover luggage or travel items, and a

separate question to cover the types of items used to carry personal belongings (briefcases, 

backpacks, duffels, totes, gym bags not counted as equipment for a specific sport).

D4 Any membership costs or other expenses related to vacation clubs or timeshare dues

D4 Any membership costs or other expenses related to vacation clubs or timeshare dues

The wording and order remained the same over the course of testing.

D10 Lodging. Do not include expenses previously reported for vacation clubs and timeshares.

D10 Cost of hotel or any lodging, including taxes. Do NOT include optional costs, such 

as parking, food, and Internet service. 

The wording and order changed over the course of testing.

Rationale for Revising the Wording

 The testing showed that it was important to be more directive about what lodging 

includes or excludes. Westat offered several wording alternatives. The wording tested in 

Round 5 was decided in discussion with BLS. We substituted the phrase “hotel or other 

lodging” for “lodging,” based on the judgment that “hotel” is the most common term 

for any type of overnight stay.

 The change in order (described below) allowed deleting the initial exclusion wording. 

Based on Round 5 testing, the decision was to add new exclusion text to help deter 

reporting of costs that have been considered here, such as food at a hotel restaurant, 

room service, parking, WiFI, etc. 

Rationale for Revising the Order

 We first moved the question to follow questions about other types of incidental costs 

(such as meals that might be eaten at a hotel, room service, mini-bar, and parking) with 

the aim of improving reporting for this question.
Cognitive Testing for the CE Recall Interview 3-43 

 

Results 3 

 A later suggestion was to administer D4 right before D10, so that the current exclusion 

text would not be needed. 

Remaining Issues

In further testing of the question, monitor whether the order change for this question has any 

effects on reporting for all other questions in the cluster related to food, parking, WiFi, etc.

D6 Airfare, including baggage and change fees

D6 Airfare, including baggage and change fees. We’ll ask about other transportation 

costs next.

The wording changed but the order remained the same over the course of testing.

Rationale for Revising the Wording

 The issues with this question were due to reporting items too soon, rather than not 

comprehending the wording of the question. (Two of 9 participants who answered this 

question reported taxi or shuttle expenses for getting to an airport when they heard this 

question.) As a result of discussions with BLS, the decision was to add a statement

(“We’ll ask about other transportation costs next”) to alert survey respondents that not 

all trip expenses need to be reported for the first question.

Remaining Issues

Expenses for this question code to an Expenditure Category 107 (Airline fares) – separate from 

other travel-related expenses, so must be captured cleanly, without reporting of other types of travel 

expenses.

D7 Bus or train fare to reach your destination

D9 Local transportation during a trip, such as shuttles, taxis, buses, subways or trains, and ferries

D97 Transportation such as shuttles, taxis, buses, subways, trains, or ferries, at any time 

during a trip

After initial testing, two separate questions were combined into one, and the order was based on the 

administration order of the first question.
Cognitive Testing for the CE Recall Interview 3-44 

 

Results 3 

Rationale for Revising the Wording

 There was evidence of some confusion about the words “rail” vs. “train.” The word 

“train” caused one participant to think more broadly about commuting, not just an outof-town trip, and several others thought of various transportation methods travelers 

might take after leaving home to reach a destination.

 After discussion with BLS, the decision was to combine Questions D7 and D9, for 

broader coverage of any transportation that is not airfare. All expenses for transit via 

intercity and intra-city bus, rail, taxi, etc. code to the same Expenditure Category 108 

(Other public transportation), so can be collected in one question.

Remaining Issues

In future testing, it will be important to monitor the need to add text about including expenses for 

these types of transportation that may be needed to get to a destination as well as expenses incurred 

while at a destination.

D8 Fares or tips for cruise ships or ferries

D8 Tickets, tips, or service charges for cruise ships

The wording changed but the order remained the same over the course of testing.

Rationale for Revising the Wording

 We modified the wording for Round 2 testing based on qualitative comments in the 

first round of partial testing. Several participants thought the original wording (“Ship 

fares or tips”) was asking about the costs (“fares”) for shipping items.

 We decided to use the wording “tickets” or “boarding fares” in place of “fares” to see if 

that might make the wording clearer and alert participants to the breadth of the 

question. However, there may be differences in interpretation of the word “tickets” – as 

physical entities vs. electronic payment or passes.

Remaining Issues

In further testing of the question, monitor whether there are differences in interpretation of the 

word “tickets” – as physical entities vs. electronic payment or passes.
Cognitive Testing for the CE Recall Interview 3-45 

 

Results 3 

D13 Rented vehicles for recreational use. Do not include fuel.

D13 Cost to rent an RV, boat, trailer, or any kind of vehicle for outdoor recreation. Do 

NOT include fuel.

Both the question wording and order changed over the course of testing.

Rationale for Revising the Wording

 Participants did not demonstrate understanding of this question. Westat suggested 

options to clarify and differentiate between D12 and D13. In discussion with BLS, the 

decision was to use examples and the phrase “outdoor recreation.” Another decision 

was to delete mention of gasoline or fuel in the vehicle questions and to ask about gas 

expenses in the following question. The rationale was that survey respondents will 

consider the full range of vehicles for which they might have purchased gas.

 The exclusion text was taken out for testing in Round 5, but the decision was to put it 

back after comprehension results showed that participants considered a gas expense to 

be applicable for a car rental.

Rationale for Revising the Order 

 The decision was to ask about recreational use (D13) first, with the expectation that this 

order will provide cleaner reporting for both questions.

Remaining Issues

In further testing of the question, monitor whether respondents distinguish between D13 and D12 

in their reporting.

D12 Rental vehicles for getting around. Do not include fuel.

D12 Cost to rent a car, truck, motorcycle, or plane. Do NOT include fuel.

Both the question wording and order changed over the course of testing.

Rationale for Revising the Wording

 In Round 2 testing, the decision was to use a definition (instead of examples), with the 

word “vehicle” intended to function as a broader term covering any sort of vehicle. But 

that term did not seem to clearly indicate to participants the breadth of the vehicles to 

think about. Participants seemed to perceive some overlap between the two questions
Cognitive Testing for the CE Recall Interview 3-46 

 

Results 3 

D12 and D13, based on the examples they named. Also in Round 2 testing, there was 

some confusion about what to include as a car rental expense. Despite the instruction to 

exclude fuel, several participants named “gas” as an example of a car rental expense.

 Expenses for the two questions code to different Expenditure Category numbers: 95 

(Leased and rented vehicles) for D12, and 84 (Boats, RVs, and other recreational 

vehicles) for D13. They cannot be combined into one question as a way to make 

reporting easier for interview respondents (e.g., to report for any kind of rented vehicle, 

whether for transportation or recreational purposes).

 One potential revision considered and accepted was to revise the wording to use 

examples rather than a definition in Round 5 testing.

 The exclusion text was taken out for testing in Round 5, but the decision was to put it 

back after comprehension results showed that participants considered a gas expense to 

be applicable for the question. 

Rationale for Revising the Order

 The question was moved to after the recreational vehicles question, so as to avoid prereporting of those types of vehicles for this question.

Remaining Issues

In further testing of the question, it will be important to monitor whether the question is capturing 

only the rental costs, not other expenses as well, such as gas.

D11 Gasoline, diesel, or any other vehicle fuels

D11 Gasoline, diesel, or any other vehicle fuels

The wording remained the same, but the order changed over the course of testing.

Rationale for Revising the Order

 In discussion about the two vehicle questions (D12 and D13), the decision was to move 

the gas/fuel question to follow them instead of precede them. The rationale was that 

gas might be purchased for any type of rented vehicle, so it would be best to cue recall 

of the possible range of rented vehicles first, then ask about fuel for them.
Cognitive Testing for the CE Recall Interview 3-47 

 

Results 3 

Remaining Issues

In further testing of the question, monitor whether expenses for gas are reported for this question, 

rather than for the prior questions.

D14 Parking fees

D14 Parking fees

The wording and order remained the same over the course of testing.

Remaining Issues

In Round 2 testing, there was limited evidence that a parking expense applicable for this question 

was misreported for the lodging question (D10), which preceded it. In further testing, monitor 

whether delivering the lodging question after this question improves the performance of this 

question.

D15 Any tolls

D15 Any tolls

The wording and order did not change over the course of testing.

Reporting and comprehension evidence indicated participants understood the question intent.

D16 Meals, snacks, or drinks at restaurants, bars, or fast food places

D16 Meals, snacks, or drinks at restaurants, bars, or fast food places

The wording and order did not change over the course of testing.

This question was one of the better performing questions in the cluster, based on both reporting 

and comprehension evidence.
Cognitive Testing for the CE Recall Interview 3-48 

 

Results 3 

Remaining Issues

In future testing, BLS might consider asking about all food purchases in one question, if those 

expenses could all be mapped to the same category. This would reduce two questions to one.

D17 was asked as a follow-up question to D16, with this wording: “How much of that amount was 

for alcohol?” 

D17 Alcohol [at restaurants, bars, or fast food places]

D17 Alcohol [at restaurants, bars, or fast food places]

The wording and order did not change over the course of testing.

Although there was evidence of minor under-reporting, this question was also one of the better 

performing questions in the cluster.

Remaining Issues

If this will ever be administered as a standalone question rather than a follow-up to the food 

question (D16), the wording may be changed to “Alcohol at restaurants, bars, or fast food places.”

D18 Food or beverages at grocery or convenience stores

D18 Food or beverages from grocery or other stores

The wording changed but the order remained the same over the course of testing.

Rationale for Revising the Wording

 Participants seemed to have a good understanding of this question, based on both 

reporting and comprehension results. During discussions with BLS, the decision was to 

broaden the type of stores from just convenience stores to “or other stores.”

There is a potential for over-reporting of items bought at gas stations, etc. that might be considered 

fast food instead of “other stores.” However, those two food questions map onto different 

categories. BLS may want to consider having both food and beverage questions map onto the same 

category, to reduce burden of asking for these separately.
Cognitive Testing for the CE Recall Interview 3-49 

 

Results 3 

Remaining Issues

In future testing, BLS might consider asking about all food purchases in one question, if those 

expenses could all be mapped to the same category. This would reduce two questions to one,

which also means that there would only need to be one alcohol follow-up question instead of two.

D19 was asked as a follow-up question to D18, with this wording: “How much of that amount was 

for alcohol?” 

D19 Alcohol [at grocery or convenience stores]

D19 Alcohol [from grocery or other stores]

The wording changed but the order remained the same over the course of testing.

Rationale for Revising the Wording

 Based on the decision for re-wording Question D18, we also made the same minor

wording change for this question for further testing.

Remaining Issues

If this will ever be administered as a standalone question rather than a follow-up to the food 

question (D18), the wording may be changed to “Alcohol at liquor, grocery, or other stores.”

As noted above, in future testing, BLS might consider asking about all alcohol purchases in one 

standalone question, if those expenses could all be mapped to the same category. This would reduce 

two questions to one.

D21 Admission fees, tickets, tours, or other entertainment

D21 Tours, admission fees, or tickets for entertainment

The wording and order changed over the course of testing.
Cognitive Testing for the CE Recall Interview 3-50 

 

Results 3 

Rationale for Revising the Wording

 There were some minor comprehension issues, and one participant’s comment indicated a 

conceptual overlap with D20. The decision was to use the same wording recommended for 

the comparable in-town question (E20): “Tours, admission fees, or tickets for 

entertainment.”

Rationale for Revising the Order

 The decision was to administer D21 before D20, to collect entertainment-related expenses 

before participation expenses.

Remaining Issues

In further testing of the question, monitor whether ordering D21 first “cleans up” reporting. There 

may be some overlap for expenses while on a trip for entertainment and sports (watching vs. 

engaging in), as well as for sports and exercise (which may require paying admission fees).

D20 Play sports, exercise, or rent any sports equipment

D20 Play sports, exercise, or rent any sports equipment

The wording remained the same but the order changed over the course of testing.

Rationale for Revising the Order

 The decision for D21 (above) affected the order for D20 (to collect entertainment-related 

expenses before participation expenses).

Remaining Issues

In further testing of the question, monitor whether the revises ordering improves reporting for both 

D20 and D21. 
Cognitive Testing for the CE Recall Interview 3-51 

 

Results 3 

D2 Mass transportation, commuter, or carpool services. Do NOT include expenses covered by 

your employer.

D2 Mass transportation, commuter, or carpool services. Do NOT include expenses 

covered by your employer.

The wording remained the same, but the order changed over the course of testing.

Rationale for Revising the Order

 Qualitative comments showed some evidence of confusion about both what qualifies as 

mass transportation (planes, subways, and trains carry many people) and what qualifies 

as a “commuter” expense (e.g., an electronic toll pass is a commuter expense but applies 

for Question D3). The decision about resolving the confusion was to administer D2 

both: 

– At some point after D3, the most specific question in this set of in-town 

questions (to fist capture tolls that might be part of commuting costs); and 

– After the out-of-town trip questions (D6 – D21), to help clarify that airline and 

train 

Placing this question last in the series about trips and transportation means that it will 

follow the out-of-town travel questions. Therefore, transition text will be needed before 

this question, to broaden respondents’ thinking to mass transportation at any time, not 

just while on a trip. For example: “For the next question, think about transportation at 

any time.” 

Remaining Issues

In further testing of the question, monitor whether the order change results in clearer 

comprehension and accurate reporting, and whether the transition text broadens respondent 

thinking to transportation while household members are “at home” as well as while on a trip.

 

3.5 Cluster E: Recreation 

This section summarizes the main results and recommendations for the 26 questions in Cluster E. 

The cluster was tested one time, in Round 4, with 5 participants completing one of two parts of the 
Cognitive Testing for the CE Recall Interview 3-52 

 

Results 3 

cluster. Where session time permitted, one or more additional participants completed one or more 

questions from the other part of the cluster.

This text preceded Questions E23, E24, E25, and E26, which were the last four questions 

administered in the cluster:

The next 4 questions ask about membership fees. Please do NOT include contributions to or 

membership in religious, professional, business, or other tax deductible organizations.

Please do NOT include contributions to or membership in religious, professional, business, or other 

tax deductible organizations.

Table 3-5 presents the question wording tested in these rounds, and the wording recommended after 

Round 4 concluded. For detailed results and recommendations developed after the first testing, refer 

to the Cluster E Interim Memo.
Results 3 

Cognitive Testing for the CE Recall Interview 3-53 

 

Table 3-5. Cluster E: Summary table of tested question wording and order

Q # Tested wording (round 4) 

Recommended wording 

(when testing concluded) 

Testing 

status 

E. Recreation

E1 Pets and their supplies, medical care, or other pet services. 

Do NOT include pet food.

Pets, pet medical care, other pet services, or pet supplies. 

Do NOT include pet food. Test revisions

E3 Any equipment used for camping, hunting, or fishing. 

Include guns.

Equipment for camping, hunting, or fishing, including 

guns. Do NOT include vehicles or boats. Test revisions

E4 Winter sports equipment or gear Equipment for snow or ice sports. Do NOT include clothing. Test revisions

E5 Water sports equipment or gear Water sports equipment or gear. Do NOT include 

swimsuits or boats. Test revisions

E2 Protective eyewear or visual goods for magnification Protective goggles, binoculars, microscopes, or 

magnifying glasses Test revisions

E6 Bicycles, parts, equipment, or gear Bicycles, including parts or equipment for them Test revisions

E8 Health and exercise equipment Home exercise or workout equipment or machines Test revisions

E7 Sports and recreation goods or equipment Sports and recreation goods or equipment. Do NOT 

include shoes. Test revisions

E9 Service contracts, maintenance, or repairs for any sport or 

recreational equipment [Same as previous round] Monitor 

E12 Docking or landing fees for boats or planes [Same as previous round] Monitor

E13 Any outboard motors Any outboard boat motors Test revisions

E17 Toys, games, or arts and crafts supplies Toys, games, or arts and crafts materials for adults or 

children. Do NOT include video or computer games. Test revisions

E10 Ping-pong, pool tables, or other similar recreation room 

items and accessories

Ping-pong tables, pool tables, or other similar recreation 

room items or their accessories Test revisions

E11 Playground equipment, sand boxes, or inflatable pools Playground equipment, sand boxes, or portable pools Test revisions

E14

Photography-related equipment, products, or professional 

services? Do NOT include computer software such as 

Photoshop.

Photography equipment, products, or professional 

services. Do NOT include phones, picture frames, or 

computer software. 

Test revisions

E15 Stamp or coin collecting [Same as previous round] Monitor 

E16 Sewing, knitting, or quilting machines, materials, or supplies [Same as previous round] Monitor
Results 3 

Cognitive Testing for the CE Recall Interview 3-54 

 

Table 3-5. Cluster E: Summary table of tested question wording and order (continued)

Q # Tested wording (round 4) 

Recommended wording 

(when testing concluded) 

Continue 

testing 

E. Recreation

E19 Any recreational or instructional classes or lessons Any recreational classes or lessons Yes

E18 Musical instruments, supplies, or accessories, including 

repairs [Same as previous round] No

E20 In-town admission fees, tickets, or tours Tours, admission fees, or tickets for entertainment Yes

E21 Any souvenirs or memorabilia, purchased either in town or 

on a trip [Same as previous round] Yes

For questions E23, E24, E25, E26, this text preceded the 

question wording: 

The next [4] questions ask about membership fees. Please 

do NOT include contributions to or membership in religious, 

professional, business, or other tax deductible organizations.

E23 Membership or entrance fees at golf courses, country clubs, 

health and fitness centers, or pools [Same as previous round] Yes

E22 Fees for participating in leisure, recreational, or organized 

sports [Same as previous round] Yes

E24 Membership fees for online or in-store shopping clubs Membership fees required for online or in-store shopping 

clubs Yes

E26 Any matchmaking, speed dating, or online dating services [Same as previous round] No

E25

Membership or other expenses related to any social, civic, 

fraternal, or service clubs, such as Boy Scouts, Kiwanis, 

AARP, fraternities, and honor societies

For the next question, please do NOT include contributions 

to or membership in religious, professional, business, or 

other tax deductible organizations.

Membership for any social, civic, fraternal, or service clubs

Yes
Cognitive Testing for the CE Recall Interview 3-55 

 

Results 3 

E1 Pets and their supplies, medical care, or other pet services. Do NOT include pet food.

E1 Pets, pet medical care, other pet services, or pet supplies. Do NOT include pet food.

The question wording changed over the course of testing.

Rationale for Revising the Wording

 After discussion with BLS, the decision was that since pet food is one type of 

“supplies,” it should be moved to be closer to the end of the sentence, so that it will be 

the last word participants will hear just before the exclusion text.

Remaining Issues

Future testing should monitor whether costs to purchase or adopt a pet are reported. At least one 

participant in this round had considered that, since they reported an adoption fee, but it also 

happened to be telescoped.

E3 Any equipment used for camping, hunting, or fishing. Include guns.

E3 Equipment for camping, hunting, or fishing, including guns. Do NOT include 

vehicles or boats.

The question wording changed over the course of testing.

Rationale for Revising the Wording

 We removed the word “any” and moved the inclusion text “include guns” into the first 

sentence to make the overall question shorter and easier to follow.

 The comprehension results (all 5 participants misjudged the following not applicable 

items: boat, fishing/hunting license, and RV), and the fact that boats, campers, and RVs 

are large expenses that are commonly used when camping and fishing, led to a wording 

change. Specifically, we added the exclusion text of “Do NOT include vehicles or 

boats.” This makes the question longer, with multiple parts (“including” and “Do NOT 

include”), but the potential for over-reporting and the high cost of such over-reported 

items seemed to necessitate the exclusion text.
Cognitive Testing for the CE Recall Interview 3-56 

 

Results 3 

E4 Winter sports equipment or gear

E4 Equipment for snow or ice sports. Do NOT include clothing.

The question wording changed over the course of testing.

Rationale for Revising the Wording

 There were over-reported costs dealing with school sports that typically take place in the 

winter, such as wrestling. In addition, the interpretation of “winter sports” may vary 

based on geographic location. Avoiding that term should help to prevent 

comprehension errors. Since the sports that apply for this question have to do with 

snow or ice, we suggested using these terms to more clearly establish boundaries of 

applicable expenses.

 Accuracy and comprehension results showed that participants incorrectly considered 

some types of apparel, particularly outerwear such as coats, scarves, and gloves, as 

applicable. Therefore, we added the exclusion text of “Do NOT include clothing.”

E5 Water sports equipment or gear

E5 Water sports equipment or gear. Do NOT include swimsuits or boats.

The question wording changed over the course of testing.

Rationale for Revising the Wording

 Swimsuits may be the most common purchase that people think of when asked about 

water-related sports and are likely purchased yearly since they wear out. All 5

participants in this round judged it as applicable, and several named it as an example, so 

we suggested adding swimsuits to the exclusion text.

 Four participants incorrectly marked boat trips as applicable. Since boats are commonly 

used for water sports and would represent very large over-reported expenses, we also 

suggested adding boats to the exclusion text.

Remaining Issues

Participants seemed to judge this question more broadly than intended, likely due to the vagueness 

of the phrase “equipment and gear.” Other word options that might be substituted – such as 

“items” – are equally vague so might not improve comprehension. In further testing of the question, 
Cognitive Testing for the CE Recall Interview 3-57 

 

Results 3 

larger sample sizes will help determine whether the exclusion text is necessary, and/or if additional 

expenses should be added to the exclusion statement.

E2 Protective eyewear or visual goods for magnification

E2 Protective goggles, binoculars, microscopes, or magnifying glasses

The question wording and order changed over the course of testing.

Rationale for Revising the Wording

 Comprehension and accuracy results were mainly negative, as the question was 

interpreted more broadly than intended. For example, sunglasses may have been 

thought of as offering protection from the sun. One alternative would have been to add 

exclusion text, e.g., “Do NOT include glasses and contacts that are worn to correct 

vision” but that would make the wording longer. Since the question is intended to cover 

four main types of goods (goggles like those used in a lab, magnifying glasses,

binoculars, and microscopes), we suggested using those, to improve clarity and to make 

the wording shorter.

Rationale for Revising the Order

 Reporting for this question might be cleaner if it follows the questions related to sports, 

or at the very least, is administered after the questions about winter sports (since ski 

goggles are applicable for that question) and water sports (since swimming goggles are 

applicable for that question).

E6 Bicycles, parts, equipment, or gear

E6 Bicycles, including parts or equipment for them

The question wording changed over the course of testing.

Rationale for Revising the Wording

 Participants broadened the interpretation of this question to clothing or shoes worn for 

biking, which was likely due to the word “gear.” Leaving off “gear” may help to narrow 

the focus to the bike itself.
Cognitive Testing for the CE Recall Interview 3-58 

 

Results 3 

Remaining Issues

We suggest testing this revised wording, to see: (1) whether test participants still report and consider 

items such as helmets and body reflectors (which are applicable, but may not necessarily be thought 

of as “equipment”) and (2) whether there is still a need to add exclusion text such as “Do NOT 

include clothing or shoes.”

E8 Health and exercise equipment

E8 Home exercise or workout equipment or machines

Both the wording and order changed over the course of testing.

Rationale for Revising the Wording

 Only 50 percent of the named examples were correct. This question is expected to 

focus on expenses such as weights, ellipticals, stationary bikes, treadmills, mats, and 

other equipment that might be in a home gym, but participants named and/or misjudged not applicable examples such as exercise programs/DVDs, water bottles, jogging 

suits, athletic shoes, and jump ropes.

 “Workout” is a common term, and may improve comprehension due to being more 

specific than the word “health.”

Rationale for Revising the Order

 We also suggest testing this before Question E7, to cover reporting of machines and 

equipment before sporting goods.

Remaining Issues

In addition to the wording changes, in future testing we suggest monitoring the need for any 

exclusion text for items such as shoes or clothes.

E7 Sports and recreation goods or equipment

E7 Sports and recreation goods or equipment. Do NOT include shoes.

Both the wording and order changed over the course of testing.
Cognitive Testing for the CE Recall Interview 3-59 

 

Results 3 

Rationale for Revising the Wording

 Participants showed, as for other questions in this cluster, a tendency to think of shoes 

worn for sports activities.

Rationale for Revising the Order

 We suggested placing this question second-to-last in the series about physical recreation 

activities, followed only by E9 (service contracts for sport or recreational equipment). 

That may reduce the opportunity for duplicate reporting of expenses that are more 

specific – such as water sports, winter sports, and biking – but might be judged as goods 

or equipment for this question.

Remaining Issues

“Sport uniforms” is an applicable expense for this question, but shoes are not. Otherwise, if all 

questions about sports and physical activities in this cluster exclude the same expenses of clothing 

and shoes worn for them, we would suggest using introductory text to exclude them, e.g., “For the 

next X questions, do NOT include expenses for shoes or clothing worn during the sport or 

activity.”

E9 Service contracts, maintenance, or repairs for any sport or recreational equipment

E9 Service contracts, maintenance, or repairs for any sport or recreational equipment

After testing concluded, the question wording and order remained the same.

Remaining Issues

There were no expenses reported for this question and comprehension results were sparse. We 

recommend further testing to collect more data for gauging comprehension and reporting accuracy. 

Also, the question should remain last in a series of the questions about equipment for exercise and 

recreation.

E12 Docking or landing fees for boats or planes

E12 Docking or landing fees for boats or planes

After testing concluded, the question wording and order remained the same.
Cognitive Testing for the CE Recall Interview 3-60 

 

Results 3 

Remaining Issues

Since this is a narrow question with a low base rate of occurrence, there are currently no data for 

judging the performance of this question. We recommend retaining the wording and continuing to 

test.

E13 Any outboard motors

E13 Any outboard boat motors

The question wording changed over the course of testing.

Rationale for Revising the Wording

 The 2 over-reported items were moped helmet and moped maintenance. Although the 

participant did not offer any insights into these mis-reported expenses, one possibility is 

that the word “motors” reminded them of any transportation with a motor or wheels, 

such as a motorcycle or moped. Since this is a very narrow question that refers to boat 

motors, we added the word “boat” to avoid misinterpretation.

Remaining Issues

Since this is a narrow question with a low base rate of occurrence, there are currently no reported 

expenses data for judging the performance of this question. We recommend retaining the wording 

and continuing to test.

E17 Toys, games, or arts and crafts supplies

E17 Toys, games, or arts and crafts materials for adults or children. Do NOT include 

video or computer games.

The question wording changed over the course of testing.

Rationale for Revising the Wording

 Since video and computer games were reported by participants as applicable for this 

question even though they are not, we added the exclusion wording above. There was 

some discussion of the potential for respondents to think the question only applies to 
Cognitive Testing for the CE Recall Interview 3-61 

 

Results 3 

children. The final decision was to explicitly mention both adults and children, and to 

use the word “materials” in place of “supplies.”

Remaining Issues

The largest concern with this question is its conceptual overlap with E16 (“Sewing, knitting, or 

quilting machines, materials, or supplies”). Some items could conceivably apply for both questions, 

depending on how those items are used (for example, beads). Reversing the order of this question 

and the one on sewing materials (E16) likely will not be helpful in eliminating the conceptual 

overlap. Furthermore, the arts and crafts portion of the question cannot be combined with the 

sewing question because they map onto different Item Categories: #88 (Sewing) for E16 and #87 

(Other recreational goods) for E17.

Based on the evidence of consistent misjudgments about the applicability of basketballs to this 

question, participants may not clearly distinguish between common recreational sports items and 

toys. We recommend monitoring the need to put this question immediately after the more specific 

question about sporting goods (E7); to date these questions have not been tested together in the 

same interview.

There was some discussion of separating this into two questions, one about toys and games, and one 

about arts and crafts. The final decision was to leave it as one question, but the need to rethink this 

decision should be monitored in future testing.

E10 Ping-pong, pool tables, or other similar recreation room items and accessories.

E10 Ping-pong tables, pool tables, or other similar recreation room items or their 

accessories.

Both the wording and order changed over the course of testing.

Rationale for Revising the Wording

 BLS recommended a slight revision to ensure that “tables” applies to ping-pong as well 

as pool.

Rationale for Revising the Order 

 Testing of this question revealed that participants may include a variety of games that 

are out of scope, such as video games, card games, and board games. This is 
Cognitive Testing for the CE Recall Interview 3-62 

 

Results 3 

understandable if these games are played in a recreation room. To help avoid inaccurate 

reporting, we recommended testing Question E10 as follows:

– After Question E17 (Toys and games), and

– Before the questions related to furniture (C9 through C16), since answering E10 

first should prevent double- or mis-reporting of ping-pong and pool tables as 

furniture (which BLS found in prior testing of furniture questions).

Remaining Issues

The categorization results suggest that participants may think of a broader range of recreational 

equipment than what is found in a recreation room. We suggest continuing to monitor results after 

this change to determine if inaccurate reporting is an issue.

E11 Playground equipment, sand boxes, or inflatable pools

E11 Playground equipment, sand boxes, or portable pools

Both the wording and order changed over the course of testing.

Rationale for Revising the Wording

 Based on discussion that some kiddie pools are hard (not inflatable), BLS and Westat 

decided to substitute “portable” for “inflatable.”

Rationale for Revising the Order

 There were numerous not applicable examples named such as volleyball, roller skates, 

hula hoops, and chalk. This suggests that participants may not distinguish between 

playground equipment and other outdoor toys. Therefore, we recommended moving 

this question after E17 (toys and games) to help avoid misreporting.

Remaining Issues

Two participants named pool supplies such as cleaning products, air pump for an inflatable pool, 

and chlorine. This suggests that participants may not distinguish between pools themselves and 

supplies for pools. Based on the joint BLS and Westat discussion that pool supplies would only 

apply to more permanent pools, the decision was to not use the originally proposed exclusion text of 

“Do NOT include pool supplies.” However, the need for this text should be monitored in future 

testing of the question.
Cognitive Testing for the CE Recall Interview 3-63 

 

Results 3 

E14 Photography-related equipment, products, or professional services. Do NOT include 

computer software such as Photoshop.

E14 Photography equipment, products, or professional services. Do NOT include 

phones, picture frames, or computer software.

The question wording changed over the course of testing.

Rationale for Revising the Wording

 In the judgment task, most participants incorrectly judged digital picture frames as being 

applicable, and a couple also incorrectly judged cell phones as being applicable. We 

extended the exclusion statement within the question to reference picture frames and 

phones. We also took out the example of computer software so as to limit the length of 

the exclusion text and be more inclusive of other photography software.

 We decided to remove the word “related;” it may not add any utility for the question.

Remaining Issues

Remaining potential areas for concern are distinguishing camera products and services from video 

camera products and services, as well as the incorrect inclusion of memory cards. We suggest

administering this question after the Cluster G questions on camcorders (G10) and portable memory

(G3) to deter double-reporting of video cameras and memory cards. In addition, one participant

each incorrectly judged sports photos and class photos to be not applicable. In further testing, it will 

be important to monitor whether these types of expenses are reported.

E15 Stamp or coin collecting

E15 Stamp or coin collecting

The question wording and order remained the same after testing.

Remaining Issues

The item appeared to perform as intended. Although there were no purchases reported for this 

question, participants named 91% applicable examples. Based on the comprehension results, we 

recommend continued monitoring of the question’s performance, especially if participants have 
Cognitive Testing for the CE Recall Interview 3-64 

 

Results 3 

applicable purchases to report. However, due to the narrow scope of this question, this particular 

question may not be of high priority for testing.

E16 Sewing, knitting, or quilting machines, materials, or supplies

E16 Sewing, knitting, or quilting machines, materials, or supplies

After testing concluded, the question wording and order remained the same.

Remaining Issues

Overall the item seemed to perform as intended, although as noted earlier, its overlap with E17 

(Toys, arts and crafts materials) is not easily solved due to their mapping to different categories. In 

addition, a quilting class and knitting lessons were each inaccurately named as examples for this 

question. We recommend continued monitoring of the question’s performance, particularly to 

determine if there are further mentions of lessons or classes and to assess the need to reorder the 

question to E19 (Recreational lessons).

E19 Any recreational or instructional classes or lessons

E19 Any recreational classes or lessons

The question wording changed over the course of testing.

Rationale for Revising the Wording

 Half of the participants named some sort of educational or tuition-based course (e.g., 

night school, trade school, tutoring), indicating a problem with distinguishing between 

“instructional” classes (applicable) and more formal “educational” classes (not 

applicable). We simplified and modified the question to eliminate the “instructional”

wording and instead focus on the word “recreational.”

Remaining Issues

In further testing of the question, we recommend monitoring the need for adding exclusion wording 

such as: “Do NOT include college or trade school classes.” In addition, participants named not 

applicable items such as home study guide, softball team, and soccer team. Participants may have

thought that self-guided lessons belonged here instead of with book purchases, and perhaps sports 

team membership includes some form of instruction. If testing reveals additional evidence that 
Cognitive Testing for the CE Recall Interview 3-65 

 

Results 3 

participants think of sport teams, then this question could be positioned after Question E22 

(organized sports fees).

E18 Musical instruments, supplies, or accessories, including repairs

E18 Musical instruments, supplies, or accessories, including repairs

The question wording remained the same but the order changed over the course of testing.

Rationale for Revising the Order 

 About half of the participants incorrectly judged music lessons as applicable, which is 

understandable since both are highly related to musical instruments. Given that many 

people, especially children, who purchase or rent musical instruments also take lessons, 

we recommended keeping the wording the same, but placing it after Question E19, 

which covers instructional lessons such as music lessons.

E20 In-town admission fees, tickets, or tours

E20 Tours, admission fees, or tickets for entertainment

The question wording changed over the course of testing.

Rationale for Revising the Wording

 There was some confusion over the wording of “in-town.” One participant wasn’t sure 

if “in-town” meant in any town or if it meant local. Two other participants thought that 

“in-town” meant “where I live” and since they live in a rural town, they considered 

everything to be out-of-town. We recommended dropping the phrase “in-town.”

Expenses for similar types of entertainment when away from home on a trip are 

covered in the series of questions about out-of-town travel.

 There was also some uncertainty about the inclusion of movie and theater tickets. We 

recommended revising the question wording to specify that tickets are for 

entertainment.

E21 Any souvenirs or memorabilia, purchased either in town or on a trip

E21 Any souvenirs or memorabilia, purchased either in town or on a trip

After testing concluded, the question wording and order remained the same.
Cognitive Testing for the CE Recall Interview 3-66 

 

Results 3 

Remaining Issues

Souvenirs are intended to be mementos of significant events or trips, so this question presented 

more difficulties for participants. As one participant put it during the Name 5 task, “It could be 

anything, a souvenir is anything.” Classification of an object as a souvenir is somewhat subjective. 

We suggest continuing to test and monitor this question as-is, but guidance from BLS on the 

definition of a souvenir for the purposes of this survey may help in crafting a more focused 

question.

E23 Membership or entrance fees at golf courses, country clubs, health and fitness centers, or 

pools

E23 Membership or entrance fees at golf courses, country clubs, health and fitness 

centers, or pools

The question wording remained the same but the order changed over the course of testing.

Rationale for Revising the Order

 We recommended positioning this question before Question E22 so that gym/fitness 

memberships will be reported before fees for types of organized sports.

Remaining Issues

There were various expenses that were incorrectly named or judged as applicable, such as golf 

lessons, personal trainer fees, and sport equipment rental, which indicated that participants tended to 

lump associated costs in with membership and entrance fees. Future testing should monitor whether 

this is an issue that requires changes to the question wording.

E22 Fees for participating in leisure, recreational, or organized sports

E22 Fees for participating in leisure, recreational, or organized sports

The question wording remained the same but the order changed over the course of testing.

Rationale for Revising the Order

 There was some confusion regarding gym memberships and whether they belong with 

this question. We recommend monitoring the question’s performance with the same 
Cognitive Testing for the CE Recall Interview 3-67 

 

Results 3 

wording, but moving it to follow Question E23 (which includes health and fitness 

center membership) to see if repositioning it results in cleaner reporting.

Remaining Issues

Participants seemed to have difficulty distinguishing between fees for participating in sports and 

expenses related to sports in general. For instance, they all incorrectly judged sport equipment rental 

and team uniforms as applicable. Participants were asked about sporting goods and equipment in the 

interview about the other half of the cluster; this may not be an issue with all Cluster E questions in 

a full interview.

E24 Membership fees for online or in-store shopping clubs

E24 Membership fees required for online or in-store shopping clubs

The question wording changed over the course of testing.

Rationale for Revising the Wording

 The results showed high comprehension for membership fees for in-store shopping 

clubs. However, 2 participants out of 5 incorrectly suggested buying items from either 

the internet or television shopping networks that do not require a membership fee. We 

added the word “required” to clarify that only clubs that require membership fees are 

applicable for this question.

E26 Any matchmaking, speed dating, or online dating services

E26 Any matchmaking, speed dating, or online dating services

After testing concluded, the question wording and order remained the same.

Participants demonstrated strong comprehension of this question.

E25 Membership or other expenses related to any social, civic, fraternal, or service clubs, such as 

Boy Scouts, Kiwanis, AARP, fraternities, and honor societies

E25 Membership for any social, civic, fraternal, or service clubs

The question wording changed over the course of testing.
Cognitive Testing for the CE Recall Interview 3-68 

 

Results 3 

Rationale for Revising the Wording

 While participants showed good comprehension of fees for membership, the wording 

“other expenses related to” led them to include additional related expenses that are 

covered in other clusters, such as Boy/Girl Scouts uniform, trip fees, and a beer keg for 

a fraternity party. Therefore, we removed that wording.

 BLS and Westat decided to remove the list of examples to shorten the question 

wording, and to instead include the examples in the Infobook.

Remaining Issues

Despite the instructions preceding the group of questions on membership fees, two participants 

were not certain whether to include donations and one mentioned professional association dues. 

With the instructions now immediately preceding this question and targeted specifically for this 

question, we recommend monitoring future testing results to see if donations and professional 

association dues are still incorrectly reported.

3.6 Cluster F: Children and Infants 

This section summarizes the main results and recommendations for the 5 questions administered in 

Cluster F (plus C9, also tested in Cluster C ). The cluster was tested once, in Round 5, with 

5 participants.

Table 3-F presents the question wording tested and the wording recommended after BLS staff and 

Westat project staff (including consultant Johnny Blair) reviewed and discussed the results. For 

detailed results and recommendations developed after the first testing, refer to the Interim Memo 

for Clusters B and F.
Results 3 

Cognitive Testing for the CE Recall Interview 3-69 

 

Table 3-6. Cluster F: Summary table of tested question wording and order

Q # Tested wording (round 5) 

Recommended wording 

(when testing concluded) 

Testing 

status 

F. Children and infants

F4 Any child care services inside or outside the home [Same as previous round] Test 

revisions 

F3 Overnight camps, including transportation to and 

from the camps Overnight camps Test revisions

F1 Instructional day camps, such as basketball camp 

or acting camp

Day camps that teach specific skills, such as basketball camp 

or acting camp

Test 

revisions

F2 General, non-instructional day camps Any other day camps not already mentioned Test 

revisions

C9 Infants’ equipment [Same as previous round]

Test (based 

on resolution 

of category 

mapping)

F5 Any meals at school for preschool through high 

school age children [Same as previous round] Monitor

C8* Infants’ furniture Infants’ furniture

Test (based 

on resolution 

of category 

mapping)

*The recommendation is to test Question C8 with Cluster C questions, in the series of other furniture-related questions.
Cognitive Testing for the CE Recall Interview 3-70 

 

Results 3 

F4 Any child care services inside or outside the home

F4 Any child care services inside or outside the home

The question wording remained the same but the order changed after testing.

Rationale for Revising the Order

 After reviewing the other questions in this cluster, we recommended putting this 

question first, since it is likely to be the most common type of expense for households 

with children of any age that require supervision.

Remaining Issues

In further testing of the question, it will be important to monitor whether respondents consider this 

question to be asking about after-school activities or camps and report those types of expenses. If 

so, the question could be moved after the questions about camps.

The question would benefit from testing at points during the year where children are in school, and 

at points when they are on summer vacation. This would reveal whether there are seasonal variations 

in how the question is interpreted and in what expenses are reported.

F3 Overnight camps, including transportation to and from the camps

F3 Overnight camps

Both the question wording and the order changed after testing.

Rationale for Revising the Wording

 Expenses for this question code to out-of-town trips, so must be collected separately 

from other types of camps. There was limited evidence of over-reporting, specifically 

money spent for clothes to wear while attending camp. 

 In discussion with BLS, the decision was to simplify the question to just “overnight 

camps.” 

Rationale for Revising the Order

 Overnight camps are likely the most salient type of camp for parents to report about; it 

is a memorable experience when children are away from home. We recommended 
Cognitive Testing for the CE Recall Interview 3-71 

 

Results 3 

ordering this question first in the series of three camp questions, to get those expenses 

reported before moving on to expenses for day camps.

Remaining Issues

Consider administering this question with the out-of-town travel questions (Cluster D). The exact 

placement within that cluster would need to be assessed during testing.

If this question remains in Cluster F, administer it first in the series of camp questions (before F1 

and F2). In further testing, we recommend administering the camp-related questions after Cluster E 

(Recreation) questions, which collect expenses for recreational activities such as dance classes. This 

may minimize over-reporting of recreational expenses for children’s after-school activities.

F1 Instructional day camps, such as basketball camp or acting camp

F1 Day camps that teach specific skills, such as basketball camp or acting camp

Both the question wording and the order changed after testing.

Rationale for Revising the Wording

 Reporting evidence indicated a comprehension issue: one participant over-reported 

expenses for several after-school activities (e.g., gymnastics, soccer). These expenses are 

applicable for Cluster E (Recreation).

Rationale for Revising the Order

 We recommend moving this to follow the question about overnight camps. Overnight 

camp is likely the easiest type of camp to ask parents about first. With camp fees for this 

question coded to Expenditure Category 89 (Recreation services), the concern is to 

guard against double-reporting of camps in two questions. Therefore, we also 

recommend asking these camp questions after Cluster E.

Remaining Issues

In further testing of the question, monitor the need to add exclusion text, e.g., “Do NOT include 

after-school activities” in order to minimize double-reporting.
Cognitive Testing for the CE Recall Interview 3-72 

 

Results 3 

F2 General, non-instructional day camps

F2 Any other day camps not already mentioned

Both the question wording and the order changed after testing.

Rationale for Revising the Wording

 The intent of this question is to be a catch-all to collect general camp-related expenses 

that are not overnight camps (because those qualify as out-of-town trips) and are not 

for a specific recreational activity (because those qualify as Recreation services). We 

suggested revising the wording to accomplish this purpose.

Rationale for Revising the Order

 Based on the limited comprehension and reporting data, we recommended asking this 

question: after F3 (overnight camp expenses code to out-of-town trips) and after F1 

(day camps for skills instruction, which code to Expenditure Category 89 – Recreation 

services). The expectation is that remaining camps could be more cleanly coded to their 

appropriate Expenditure Category 11 (Child care).

Remaining Issues

In further testing of the camp questions, monitor respondents’ comprehension of three separate 

questions. If needed, revise the wording and order. Another option would be to cover all childrens’

camp expenses – except for those related to out-of-town trips – in one question and coding to one 

category. Although this would require changing the categorization, it would likely reduce interviewer 

and respondent burden and minimize the potential for double-reporting.

C9 Infants’ equipment

C9 Infants’ equipment

The question wording remained the same but the order changed after testing.

Rationale for Revising the Order

 We provided specific recommendations for ordering the other questions (which all 

relate to children’s care or activities) in this cluster. This question and F5 (meals at 

school) differ from the others, so we suggested placing them at the end.
Cognitive Testing for the CE Recall Interview 3-73 

 

Results 3 

Remaining Issues

We did not recommend a wording change, based on the assumption that the Expenditure Category

51 (Miscellaneous personal goods) cannot be changed. However, if the categorization can be 

changed, we recommend revising the question to cover both furniture and equipment for infants in 

one question: “Any furniture or equipment for infants” and administering it first within the furntiure 

question series.

In further testing of the question as currently worded, monitor how well this placement works 

within the flow of questions about children.

F5 Any meals at school for preschool through high school age children

F5 Any meals at school for preschool through high school age children

The question wording and remained the same after testing.

The limited evidence we collected in testing did not indicate a need for any changes.

Remaining Issues

In further testing of the Cluster F questions, continue to monitor performance of this wording and 

order.

3.7 Cluster G: Electronics/Technology 

This section summarizes the main results and recommendations for the 13 questions in Cluster G. 

The cluster was tested 3 times, in these rounds:

 Rounds 1 – 2

– In Round 1, partial data (due to session time constraints) were collected from 

5 participants.

– Round 2 (n = 5).
Cognitive Testing for the CE Recall Interview 3-74 

 

Results 3 

 Round 7 (n = 9).

 Round 9 (n = 6).

Table 3-7 presents the question wording tested in these rounds, and the wording recommended after 

Round 9 concluded. For detailed results and recommendations developed after the first testing, refer 

to the Cluster G Interim Memo. For results and recommendations developed after subsequent 

rounds, refer to the Round 7 Interim Summary Table.
Results 3 

Cognitive Testing for the CE Recall Interview 3-75 

 

Table 3-7. Cluster G: Summary table of tested question wording and order

Q # 

Tested wording 

(rounds 1 and 2) Tested wording (round 7) Tested wording (round 9) 

Recommended wording 

(when testing concluded) 

Testing 

status 

G. Electronics/Technology

G1

Cell or landline phones or 

accessories. Do NOT include 

phone service fees.

Cell or landline phones or their 

accessories. Do NOT include 

phone service fees.

[Same as previous round] [Same as previous round] Monitor 

G4 Digital book readers or 

tablets

Digital book readers or tablets. 

Do NOT include download or 

subscription fees.

Tablets or digital book readers. 

Do NOT include download or 

subscription fees.

[Same as previous round] Monitor 

G2

Internet services away from 

home, such as internet 

cafes, and WiFi access fees 

in hotels, airports, or planes

[Same as previous round] [Same as previous round] [Same as previous round] Monitor

G3

Portable memory storage for 

computers and other 

electronic devices

Portable memory, such as 

flash drives, memory cards, or 

blank CDs or DVDs

[Same as previous round]

Portable memory, such as 

flash drives, external hard 

drives, memory cards, or 

blank CDs or DVDs

Test 

revisions

G5

Laptops or desktop 

computers, or equipment

that you use with the 

computer. Do NOT include 

expenses for business use.

[Same as previous round] [Same as previous round]

Computers or equipment 

that you use with the 

computer. Do NOT include 

expenses for software or 

expenses for business use.

Test 

revisions

G13

Office devices for nonbusiness use, such as 

calculators, fax machines, or 

paper shredders

[Same as previous round] [Same as previous round]

Fax machines, paper 

shredders, calculators or 

other office devices for nonbusiness use

Test 

revisions

G6

Any installations, repairs, or 

service contracts for 

personal use computers or 

equipment

Any installations, repairs, or 

service contracts for 

computers, phones, digital 

book readers, or tablets

Setup, service, repairs, or 

warranty for computers, 

phones, tablets, or digital book 

readers

Setup, service, repairs, or 

warranties for computers, 

phones, tablets, or digital 

book readers

Monitor

G7

GPS devices, aside from 

those on cell phones or built 

into cars

GPS devices or their 

accessories, aside from those 

on cell phones or built into 

cars

GPS devices or their 

accessories. Do NOT include 

GPSs built into cell phones or 

cars.

[Same as previous round] Complete

G10 Video cameras or small 

personal music players

Camcorders or small personal 

music players [Same as previous round] [Same as previous round] Monitor
Results 3 

Cognitive Testing for the CE Recall Interview 3-76 

 

Table 3-7. Cluster G: Summary table of tested question wording and order (continued)

Q # 

Tested wording 

(rounds 1 and 2) Tested wording (round 7) Tested wording (round 9) 

Recommended wording 

(when testing concluded) 

Testing 

status 

G. Electronics/Technology

G11

Stereos, radios, speakers, or 

sound components, 

including those installed in 

vehicles. Include charges for 

installation or setup.

[Same as previous round] [Same as previous round] [Same as previous round] Monitor 

G9

Any televisions for a home or 

vehicle, including equipment 

that you can use with them. 

Include charges for 

installation or setup.

Any televisions for a home or 

vehicle, including equipment 

that you can use with them. 

Include charges for installation 

or setup. Do NOT include 

subscription fees for cable or 

satellite.

[Same as previous round] [Same as previous round] Monitor 

G8

Any video game systems or 

accessories. Do NOT include 

actual video games.

[Same as previous round]

Any video game systems or 

their accessories. Do NOT 

include actual video games.

[Same as previous round] Monitor

G12

Service contracts, 

maintenance, or repairs for 

any audio or video electronic 

equipment

[Same as previous round] [Same as previous round] [Same as previous round] Monitor 
Cognitive Testing for the CE Recall Interview 3-77 

 

Results 3 

G1 Cell or landline phones or accessories. Do NOT include phone service fees.

G1 Cell or landline phones or their accessories. Do NOT include phone service fees.

The question wording changed over the course of testing.

Rationale for Revising the Wording

 Round 1 and 2 testing yielded some evidence of respondent confusion in interpreting 

the word “accessories” too broadly, including other devices that can access the internet, 

similar to a smart phone. We added the word “their” before accessories to tighten 

comprehension.

 In terms of reporting, several items were forgotten, including cell phones, cell phone 

cases, and chargers. These were memory issues and did not have any bearing on the 

question wording. High accuracy in categorization across all testing rounds indicated 

that participants seemed to have a clear understanding of the question meaning.

G4 Digital book readers or tablets

G4 Tablets or digital book readers. Do NOT include download or subscription fees.

Both the wording and order changed over the course of testing.

Rationale for Revising the Wording

 The initial testing results showed overly broad interpretation of the question to include 

the costs of activities conducted with a tablet or reader. Based on the comprehension 

results and reporting, several participants considered costs for books or movies, head 

phones, or internet service fees as applicable. In reporting, one person excluded an iPad, 

thinking it belonged under computers.

 We added and tested the exclusion instruction to focus comprehension on the devices 

themselves and to reduce the chance of double-counting associated expenses.

 We changed the wording to put tablets before digital book readers for emphasis and to 

reduce the likelihood of misreporting tablets as computers.

Rationale for Revising the Order

 We moved the question about digital book readers or tablets in order to group together 

two questions about devices that use mobile internet services. G4 was also moved to 
Cognitive Testing for the CE Recall Interview 3-78 

 

Results 3 

precede G2 about internet service away from home since these devices are often used 

with such services.

Remaining Issues

Monitor for evidence of a need to add more explicit exclusion text, e.g., Do NOT include fees for 

downloading books or music.

G2 Internet services away from home, such as internet cafes, and WiFi access fees in hotels, 

airports, or planes

G2 Internet services away from home, such as internet cafes, and WiFi access fees in 

hotels, airports, or planes

The question wording remained the same but the order changed over the course of testing.

Rationale for Revising the Wording

 In Round 1 testing, after hearing the words hotel and airplanes, one participant 

interpreted the question to include expenses related to travel and lodging and so offered 

a substantial misreport for hotel and airfare. There was also a misreport in Round 7 of a 

hot spot device.

 Based on a low number of overall reports for this item, we kept the wording the same in 

order to continue monitoring performance over multiple rounds.

Rationale for Revising the Order

 The order of this question was revised to improve flow, so that it followed both 

questions G1 and G4 about mobile devices that use internet services away from home.

Remaining Issues

Monitor in future testing whether to administer this question before the travel questions (if needed 

to deter respondents from reporting Wi-Fi fees with a hotel expense).
Cognitive Testing for the CE Recall Interview 3-79 

 

Results 3 

G5 Laptops or desktop computers, or equipment that you use with the computer. Do NOT 

include expenses for business use.

G5 Laptops or desktop computers, or equipment that you use with the computer. Do 

NOT include expenses for software or expenses for business use.

After testing concluded, the question wording and order remained the same.

Rationale for Revising the Wording

 The word “equipment” was intentionally used to lead participants to rule out software. 

However, results from the categorization activity suggested this was not effective. 

Multiple participants over the testing rounds judged software to be applicable, so the 

decision was to add software to the exclusion text. 

 In multiple rounds, a computer table (which qualifies as furniture) was also judged to be 

an applicable item.

Remaining Issues

We recommend administering this question after the furniture questions, to deter reporting of 

computer-related furniture for this question. 

G3 Portable memory storage for computers and other electronic devices

G3 Portable memory, such as flash drives, memory cards, or blank CDs or DVDs

Both the wording and order changed over the course of testing.

Rationale for Revising the Wording

 When reporting, several participants forgot blank CDs or DVDs.

 Comprehension results indicated that participants did not have a clear or uniform 

understanding of either “portable memory storage” or “computers and other electronic 

devices.” Participants miscategorized portable items that included memory storage or 

held an SD card, such as digital cameras or photo frames.

 We added examples to the wording to clarify the meaning of portable memory storage 

and reduce under-reporting of CD and DVDs.

 At the beginning of testing, hard drives were not considered applicable for this question 

and instead were assigned to question G5. This coding accounted for over-reports and 
Cognitive Testing for the CE Recall Interview 3-80 

 

Results 3 

difficulty in the comprehension activity. This was especially the case for external hard 

drives, which many participants considered to be portable. BLS decided that hard drives 

will now be counted as applicable for this question instead of G5; this will reduce the 

potential for over-reporting.

Rationale for Revising the Order

 The order of this question was not intentionally moved; it changed due to decisions 

about surrounding questions.

G13 Office devices for non-business use, such as calculators, fax machines, or paper shredders

G13 Fax machines, paper shredders, calculators or other office devices for non-business 

use

The question wording and order changed over the course of testing.

Rationale for Revising the Wording

 Comprehension results demonstrated a lack of understanding of “office devices,” as 

participants miscategorized not applicable items such as paper clips, rubber bands, pens, 

paper, calendars, and pencils. Participants seemed to be thinking more about “office 

supplies” than “office devices.”

 Reporting results indicated that the inclusion of “fax machines” as an example led to the 

over-report of charges associated with commercial faxing or fax services.

Rationale for Revising the Order

 We reordered this question in order to improve flow and combine it with other items 

coded in Expenditure Category 15 (Information and information processing other than 

internet services). The new order allows administering it before moving on to questions 

about audio and visual items. It was also moved to precede question G6 about setup, 

installation or repair of office devices covered by this question.

Remaining Issues

In further testing of the question it would be helpful to monitor whether “fax machines” as an 

example leads to any over-reporting of fax charges (there was an instance of this in testing). Also 

monitor the need for exclusion text: “Do NOT include office supplies or furniture.” 
Cognitive Testing for the CE Recall Interview 3-81 

 

Results 3 

G6 Any installations, repairs, or service contracts for personal use computers or equipment

G6 Setup, service, repairs, or warranties for computers, phones, tablets, or digital book 

readers

Both the wording and order changed over the course of testing.

Rationale for Revising the Wording

 Reporting and comprehension results were limited. Reporting in Rounds 1 and 2 yielded 

one accurate report. In Round 7 there were several over-reports of WiFi set up and no 

accurate reports. We added examples to the question text to increase accurate reports.

Rationale for Revising the Order

 In addition to other setup service costs, this question is meant to apply to office devices 

for non-business use, covered in G13. We revised the order, so that G6 will follow G13 

to deter under-reporting of associated costs.

Remaining Issues

BLS confirmed that this question needs to capture both installation and repairs and service contracts

for items covered in G13. In future testing, monitor reporting of applicable costs for office devices 

for non-business use to determine effectiveness of the question reordering.

G7 GPS devices, aside from those on cell phones or built into cars.

G7 GPS devices or their accessories. Do NOT include GPSs built into cell phones or 

cars.

The question wording changed over the course of testing.

Rationale for Revising the Wording

 We changed the question wording of “aside from” to “do NOT” to make the wording 

consistent with other questions and to reduce potential language comprehension issues 

with “aside from.”

 We added the wording “or their accessories” since GPS devices that are not built into 

cars or phones may have separate chargers.
Cognitive Testing for the CE Recall Interview 3-82 

 

Results 3 

G10 Video cameras or small personal music players

G10 Camcorders or small personal music players

Both the wording and order changed over the course of testing.

Rationale for Revising the Wording

 In naming examples, participants accurately named Mp3/Mp4 player, iPod charger, GO 

Pro, and video camera as applicable. Participants also incorrectly named digital cameras 

and headphones.

 We changed “video cameras” to “camcorders” to reduce the potential for misreporting 

digital cameras and other devices that record video.

 There was some evidence of confusion – several participants mentioned phones and 

digital cameras as examples of expenses for this question. 

Rationale for Revising the Order

 The order of this question changed only due to order decisions about surrounding 

questions.

Remaining Issues

In further testing, monitor whether phones and digital cameras are actually over-reported. 

G11 Stereos, radios, speakers, or sound components, including those installed in vehicles. Include 

charges for installation or setup.

G11 Stereos, radios, speakers, or sound components, including those installed in vehicles. 

Include charges for installation or setup.

The question wording remained the same but the order changed over the course of testing.

Rationale for Revising the Wording

 There was evidence across testing rounds that participants were not clear where to 

report satellite radio subscriptions. In comprehension activities, these expenses were 

judged as applicable.
Cognitive Testing for the CE Recall Interview 3-83 

 

Results 3 

 In all testing rounds, computer speakers were unanimously incorrectly judged as 

applicable. Since question G5 (computers and their equipment) precedes this question 

and is in the same Expenditure Category 81 (Video and Audio), we are not as concerned 

about the judgment results. Because of the length of the question in its current form, we 

are hesitant to lengthen it with additional exclusion text.

Rationale for Revising the Order

 To help determine if reordering reduced the potential for multiple reports of speakers, 

we revised the order so that this question would precede questions G9 and G8, about 

the various types of electronics with which speakers can be used, and follow G5 about 

computers and their equipment.

Remaining Issues

BLS indicated that satellite radio subscription fees will likely be captured in this category. In further 

testing of the question, monitor the need for exclusion text for satellite radio subscriptions, if these 

costs are decided to be captured in a different category.

G9 Any televisions for a home or vehicle, including equipment that you can use with them. 

Include charges for installation or setup.

G9 Any televisions for a home or vehicle, including equipment that you can use with 

them. Include charges for installation or setup. Do NOT include subscription fees 

for cable or satellite.

The question wording remained the same but the order changed over the course of testing.

Rationale for Revising the Wording

 The comprehension results showed evidence that the phrase “including equipment that 

you can use with them” is too vague and caused a few participants to think too broadly 

about what should be included for this question. Items judged applicable for this 

question included smartphones, speakers, TV stand, and mounting kit. The decision was 

to suggest a change in order (see below) and to monitor the comprehension evidence in 

subsequent testing.

 Results also showed that participants interpreted the phrase “installation or set up” too 

broadly, leading to confusion over the applicability of cable TV installation or cable 

service fees. We added exclusion text to reduce the possibility of over-reporting cable 

installation and fees.
Cognitive Testing for the CE Recall Interview 3-84 

 

Results 3 

Rationale for Revising the Order

 We revised the order to position G9 after question G11, to reduce the likelihood of 

multiple reports of speakers.

Remaining Issues

In further testing of the question, monitor whether there is a need to reduce the question length. 

The current wording includes both inclusion and exclusion text, leading to several participants 

asking for the full question to be repeated during testing. Also monitor the need to revise order to 

place this question after furniture questions in Cluster C.

G8 Any video game systems or accessories. Do NOT include actual video games.

G8 Any video game systems or their accessories. Do NOT include actual video games.

Both the wording and order changed over the course of testing.

Rationale for Revising the Wording

 Results for the categorization activity yielded evidence of participant confusion over 

applicable items for this question. In Rounds 1 and 2 this question had the highest 

percentage of “Don’t know/Not sure” judgments and one of the lowest percentages of 

accurately categorized items. Gaming furniture, television, and video game cartridge 

were judged applicable. We added “their” before accessories to help tighten wording.

Rationale for Revising the Order

 To reduce over-reporting of expenses for items associated with televisions and 

computers that could be considered video game system accessories, we moved this 

question to follow both G5 about computers, and G9 about televisions.

Remaining Issues

In further testing of the question we suggest administering G8 after questions about televisions and 

furniture to reduce multiple reports of these items. We hesitated to add text to an already long 

question, so suggest continuing to monitor the need to add more exclusion text about furniture. 
Cognitive Testing for the CE Recall Interview 3-85 

 

Results 3 

G12 Service contracts, maintenance, or repairs for any audio or video electronic equipment.

G12 Service contracts, maintenance, or repairs for any audio or video electronic 

equipment.

The question wording remained the same but the order changed over the course of testing.

Rationale for Revising the Order

 The order of this question was not intentionally moved but changed because of 

decisions made about surrounding questions. We suggest that the reordering of 

previous questions in this cluster will allow for a clearer understanding of this question 

in relation to others, and limit reporting to applicable expenditures.

Remaining Issues

We observed that a few participants reported expenses that overlapped with G6 (for installation 

repairs, or service contracts for personal computers or their equipment). The vague phrase “any 

audio or video electronic equipment” may have led them to over-report those expenses for this 

question. In further testing of the question, we suggest closely monitoring to see if reporting is more 

accurate due to the change in administration order.

3.8 Cluster H: Events 

This section summarizes the main results and recommendations for the 3 questions in Cluster H. 

The cluster was tested over the course of Rounds 2 – 4. Unlike testing for the other clusters, these 

3 questions were tested as “floaters” rather than together as a set. Participants scheduled (for other 

clusters) in Rounds 2 – 4 and also eligible for one of these questions completed it during their 

session. Per question, the number of participants was:

 Question 1 (Catered affairs) n = 8;

 Question 2 (Cemetery lots or vaults) n = 4; and

 Question 3 (Funeral services) n = 4.

Table 3-8 presents the question wording tested and the wording recommended after Round 4 

concluded. For detailed results and recommendations developed after the first testing, refer to the 

Cluster H Interim Memo.
Results 3 

Cognitive Testing for the CE Recall Interview 3-86 

 

Table 3-8. Cluster H: Summary table of tested question wording and order

Q # Tested Wording (Rounds 2 – 4) 

Recommended wording 

(when testing concluded) 

Testing 

status 

H. Events

H1 Catered affairs, including purchases, rentals, or 

entertainment [Same as previous round] Monitor

H3 Purchase or upkeep of cemetery lots or vaults [Same as previous round] Monitor

H2 Funeral services, burials, or cremation [Same as previous round] Monitor
Cognitive Testing for the CE Recall Interview 3-87 

 

Results 3 

H1 Catered affairs, including purchases, rentals, or entertainment

H1 Catered affairs, including purchases, rentals, or entertainment

The question wording and order remained the same after testing.

Remaining Issues

We recommended retaining the wording for additional testing, to evaluate its performance when 

administered to participants who have already answered questions about clothing and flowers.

To prevent double-reporting of clothing and flowers, we recommended administering this question 

after questions about clothing (questions A3, A4, and A6) and flowers (question C19) in a real recall 

interview context. These participants did not have a clear understanding of which wedding-related 

items fit the question. One participant excluded weddings altogether until she started the judgment 

task, which prompted her to think more broadly than her initial interpretation (which she limited to 

galas, fundraisers, and children’s events).

H3 Funeral services, burials, or cremation

H3 Funeral services, burials, or cremation

The question wording remained the same but the order changed after testing.

Rationale for Revising the Order

 In the comprehension activities, participants tended to lump together anything related 

to funerals or burial as applicable for this question. Although costs associated with 

purchase and upkeep were to be captured in H2), participants still named them as 

examples for this question, so we recommended reordering it to ask before H2.

Remaining Issues

In further testing of the question, monitor the need to add exclusion text, e.g., for items that should 

be reported elsewhere, such as food or clothing.

We also recommend, as noted below for Question H2, considering combining H2 and H3 into one 

(longer) question if further testing warrants that.
Cognitive Testing for the CE Recall Interview 4-88 

 

Recommendations 4 

H2 Purchase or upkeep of cemetery lots or vaults

H2 Purchase or upkeep of cemetery lots or vaults

The question wording remained the same but the order changed after testing.

Rationale for Revising the Order

 The order change was based on the recommendation to ask H3 first.

Remaining Issues

We suggested collecting more data and monitoring performance with additional participants. 

Participants had difficulty separating costs associated with the funeral and burial from those of 

purchase or upkeep. Among this set of participants, however, none had had upkeep expenses within 

the past three months. If the confusion persists, one option is to create one question to cover all 

content that is currently separated into Questions H2 and H3. The disadvantage of that option is 

potentially lengthy wording (to ensure coverage of all items), which may be more confusing for 

respondents than the current two shorter questions.
This page is intentionally blank
Cognitive Testing for the CE Recall Interview 4-1 

 

Westat completed a total of 85 cognitive interviews over the course of 9 rounds of testing. Chapter 

3 provided a high-level summary of those cognitive testing results. 

Summary of the question revisions. As described in Chapter 2, we drafted the 116 original 

questions as definitions where possible. For each question, if comprehension and/or reporting 

issues (such as reporting items that were not applicable, or excluding items that were applicable) 

surfaced, we considered the following as options for revisions for a subsequent round of testing.

 Revise the wording of the definition to address specific observed reporting and/or 

interpretation issues that surfaced;

 Use examples instead of, or in addition to, the definition; or

 Add exclusion text (to explicitly address a misinterpretation observed with multiple 

participants or judged likely to be significant for the population, e.g., over- or underreporting of large expenses).

If those options did not seem appropriate (e.g., a question was so broad that it was not possible to 

use a succinct set of examples, adding exclusion text made the question text very lengthy), or did not 

lead to improved performance in subsequent testing, a last resort was to split the question into two 

or more questions, each with a more specific focus. 

Through the iterative interview rounds, we tested the following types of revisions:

 As described above, changing the wording of expenditure questions 

 Merging two questions to form a higher-level expenditure question;

 Splitting a questions into two or more questions; and 

 Changing the order of administration of questions.

End result of the testing. Over the course of testing, of the 116 original questions, the wording 

changed (at some point) for 65 of the questions. Three questions were split into 2 or 3 separate 

questions, and two original questions were merged into a single question. When testing concluded, 

the total number of questions was 119.

Conclusions and Recommendations 4 
Cognitive Testing for the CE Recall Interview 4-2 

 

Recommendations 4 

For some questions, we observed one or both of the following:

 Participants reported an expense “too soon;” they thought of an item and did not know 

that it would be more appropriate to report it for a later question; or

 Comprehension results showed that a majority of participants judged items as applicable 

for a question, when they were actually applicable for a question that occurred later in 

the administration order.

In these cases, we recommended changing the administration order for a question and monitored 

whether reporting and comprehension improved in a subsequent testing round. The main concern 

with trying to ensure reporting of expenses for the intended questions was because of the mapping 

to Expenditure Categories. If expenses are reported for the “wrong” questions, or are reported 

multiple times (because recall of the same expense is cued by the wording of several questions), the 

mapping of expenses to the Expenditure Categories will be incorrect and/or an over-estimate of 

total expenses for the recall interview.

Question testing status. Table 4-1 shows, at the conclusion of testing, the status regarding 

recommendations for further testing of the expenditure questions. Over half of the questions (72) 

are considered to need more testing in order to monitor reporting accuracy and comprehension with 

a larger sample size and/or because questions around them changed in the last round they were 

tested. A significant number (42) are considered to need testing because changes were suggested for 

the wording or order when testing concluded for their cluster. Five of the questions are marked as 

not needing additional testing due to yielding strong reporting accuracy and/or comprehension 

evidence. These questions tend to ask about very specific expenses, such as GPSs that are not built 

into phones or cars.

Table 4-1. Question testing status at the conclusion of testing

Question testing status Number of questions 

Testing considered complete 5

More testing is recommended, to monitor the performance of the question

(due to wording and/or order changes to other questions within the cluster)

72

More testing is recommended, due to revisions to the wording and/or order of 

the question

42

Total 119 
Cognitive Testing for the CE Recall Interview 4-3 

 

Recommendations 4 

Recommendations for moving forward with expenditure questions. Assuming that BLS will 

proceed with using fewer, more general questions for the recall interview, there are a few cautions 

for the interview context that will improve reporting accuracy.

 Interviewers will need to be familiar with the full set of questions. The question order 

has been, and will likely continue to be, revised on the basis of empirical evidence. 

However, interviewers will need to be aware of questions that are most likely to provide 

overlapping item reports, so as to assign the items to correct Expenditure Category 

numbers and minimize over-reporting.

 The interview respondent instructions should clearly indicate that an item should be 

reported only once. This will also help to deter over-reporting. All respondents and 

interviewer instructions should be reviewed in light of recommended changes that are 

adopted. 

We also recommend a field experiment to compare performance of the original and revised 

interview. Problematic questions identified in a field test could be subjected to further cognitive 

testing to pinpoint issues and improve their performance. Another alternative is to use a “bridge 

sample” – where respondents would report for both versions of the interview in order to assess the 

impact of expenditure question changes.

Crosswalk. An additional final product of the testing was the creation of a crosswalk between the 

119 questions and the 38 Item Categories they are intended to measure. The crosswalk is provided 

as a separate document in Excel format.

Lessons learned in testing. BLS requested that Westat comment on the methods used in the 

sessions (e.g., which tasks worked best to assess comprehension), and to comment on which types 

of items seemed to work better with examples vs. definitions. Appendix 4 provides these 

observations.

Recommended question wording and order. Table 4-2 shows the tested clusters, with questions 

within each cluster ordered on the basis of the last time the cluster was tested. Clusters B, E, and F, 

and H were tested once. Cluster D was tested in two rounds and Clusters A, C, and G were each 

tested in a total of 3 rounds.

For each question, Table 4-2 also indicates whether more testing is recommended. Mainly due to the 

small n for testing, we recommend further testing for the majority of the questions. Even if the 

wording and/or order of an individual question did not change over the course of testing, reporting 
Cognitive Testing for the CE Recall Interview 4-4 

 

Recommendations 4 

and comprehension may be affected by changes made to preceding questions. Some questions had 

very little reporting and/or comprehension evidence to evaluate, so we recommended continuing to 

test them to judge their performance.

We offer a caveat: due to the large number of questions in the recall interview, it was not possible 

to cover all questions with participants. Therefore, we do not have a definitive recommendation for 

overall order of the questions. More testing will be needed to determine an optimal order for the full 

set of questions for the recall interview. Further testing will require a longer session time and/or an 

alternative approach that will allow covering more questions with each participant.
Recommendations 4 

Cognitive Testing for the CE Recall Interview 4-5 

 

Table 4-2. Original wording and recommended wording when testing concluded, by cluster

Q # Original wording 

Recommended wording 

(when testing concluded) 

Testing 

status 

A. Apparel and Personal Items

A1 Jewelry, watches, or parts for them Jewelry or watches, for men, women, or children Test revisions

A2 Watch or jewelry repair, including sizing or cleaning Care for watches or jewelry, such as repairs, sizing, cleaning, or 

engraving Monitor 

A3 Clothing rental or preservation, or clothing storage 

outside the home

Clothing rental, clothing preservation, or professional services for 

storing clothes

Monitor

A5 Any uniforms or costumes for men, women, or 

children. Do not include costs that are reimbursed.

Any uniforms or costumes for men, women, or children. Do NOT 

include shoes or costs that are reimbursed. Monitor

A6 Any type of dresses, for women or girls Any type of dresses, for women or girls. Do NOT include skirts. Monitor

A4 Any suits, sportcoats, or tailored jackets for men, 

women, or children

Any suits, sportcoats, or tailored jackets for men, women, or 

children. Do NOT include outerwear or individual tops or pants. Monitor

B. Repairs, replacements, upkeep of home; home operations

B1

Indoor flooring installation, repair, or replacement 

including wood, vinyl, tile, and wall-to-wall carpeting. 

Do NOT include expenses that will be reimbursed by 

someone outside of your household.

Indoor flooring installation, repair, or replacement including wood, 

vinyl, tile, and wall-to-wall carpeting. Do NOT include expenses that 

will be reimbursed by someone outside of your household.

Monitor

B2 Materials for termite or pest control. Do NOT include 

professional service fees.

Materials for termite or pest control. Do NOT include professional 

service fees. Monitor

B3 Any termite or pest control professional service fees Any termite or pest control professional service fees Monitor

B4

Housekeeping, lawn, or tree care services. Do NOT 

include services covered by management or 

maintenance fees.

Housekeeping, lawn, or tree care services. Do NOT include services 

covered by management or maintenance fees. Monitor

B5 Outdoor equipment or tools, including motorized, 

electric, or manual

Outdoor equipment or tools, including motorized, electric, or 

manual Monitor

B6 Service contracts, maintenance, or repairs for any 

outdoor equipment or tools

Service contracts, maintenance, or repairs for any outdoor 

equipment or tools Monitor

B7 Power or non-power tools, their accessories, or 

hardware Power or non-power tools, their accessories, or hardware Monitor

B8 Items used for hanging or storage in the home or yard

[B9] Indoor closet and storage items Test revisions

[B10] Items for storage in the garage or yard Test revisions

[B11] Other hardware not already mentioned Test revisions
Recommendations 4 

Cognitive Testing for the CE Recall Interview 4-6 

 

Table 4-2. Original wording and recommended wording when testing concluded, by cluster (continued)

Q # Original wording 

Recommended wording 

(when testing concluded) 

Testing 

status 

C. Household Linens and Furnishings

Part 1

C1 Any window coverings, including curtains, drapes, 

blinds, or shades

Any window coverings, including curtains, drapes, blinds or 

shades.

Do NOT include cost of hardware purchased separately.

Test revisions

C4 Bathroom linens, shower curtains, or bath mats Any towels or other bathroom linens, shower curtains, or bath 

mats Test revisions

C5 Bed linens, blankets, or pillows, for either adults or 

children Bed linens, blankets, or bed pillows, for either adults or children Monitor

C3 Slipcovers, decorative pillows, or cushions Slipcovers, decorative pillows, or cushions Monitor

C6 Kitchen, dining room, or any other room’s linens or 

fabric items

Linens or fabric items for the kitchen, dining room, or any other 

rooms not already covered Monitor

C2 Non-permanent floor or stair coverings Non-permanent floor or stair coverings, such as mats, rugs, 

treads or runners Monitor

C28 Electric floor or carpet cleaning equipment Electrical equipment for floor or carpet cleaning Test revisions

C25 Window or wall unit air conditioners Air conditioners installed into windows or walls Monitor

C30 Portable cooling or heating equipment Portable fans, heaters, humidifiers, or dehumidifiers Complete

C29 Indoor lamps, ceiling or wall lighting, or ceiling fans Indoor lamps, ceiling or wall lighting, or ceiling fans. Do NOT 

include light bulbs. Monitor 

C26 Air purifiers, water filters, carbon monoxide detectors, or 

smoke alarms

Air purifiers, water filters, carbon monoxide detectors, or smoke 

alarms Complete

C27 Burglar alarm equipment. Do NOT include regular 

security alarm service fees.

Burglar alarm equipment. Do NOT include regular security alarm 

service fees. Monitor 

C31 Items used to serve, consume, or store food and drink

[C35] Containers for storing foods and drinks. Do NOT include 

items you throw away. Monitor

[C36] Tableware for serving, eating, or drinking. Do NOT include 

items you throw away. Monitor

C32 Nonelectric cookware Nonelectric cookware Monitor

C33 Any other nonelectric kitchen gadgets, tools, or cooking 

utensils Any other nonelectric kitchen gadgets, tools, or cooking utensils Monitor 
Recommendations 4 

Cognitive Testing for the CE Recall Interview 4-7 

 

Table 4-2. Original wording and recommended wording when testing concluded, by cluster (continued)

Q # Original wording 

Recommended wording 

(when testing concluded) 

Testing 

status 

C. Household Linens and Furnishings

Part 2

C23 Dishwasher that is portable, not built-in Dishwasher that is portable, not built-in Complete

C24 Purchase, installation, or maintenance of a built-in 

dishwasher, garbage disposal, or range hood

Purchase, installation, or maintenance of a built-in dishwasher, 

garbage disposal, or range hood Monitor

C20 Small electrical kitchen appliances Small electrical kitchen appliances, including microwaves Monitor

C21

Any larger appliances for the kitchen or laundry room, 

excluding dishwashers. Include installation and haulaway costs. 

Larger kitchen or laundry appliances. Include installation and 

haul-away costs. Monitor

C22 Service contracts, maintenance, or repairs for those 

larger household appliances

Service contracts, maintenance, or repairs for those larger 

kitchen or laundry appliances Monitor

C19 Potted plants, fresh cut flowers, or flowers that can be 

worn or carried

Houseplants, fresh cut flowers, or flowers that can be worn or 

carried Monitor

C18 Household decorative items such as clocks, art, mirrors, 

and seasonal decorations

Household decorative items such as artificial flowers or plants, 

clocks, art, mirrors, or seasonal decorations Monitor

C8* Infants’ furniture Infants’ furniture

Test (based on 

resolution of 

category 

mapping)

C15 Any bedroom furniture, mattresses, or boxsprings Any bedroom furniture, mattresses, or boxsprings Monitor 

C10 Living, family, or recreation room furniture Living, family, or recreation room furniture Monitor 

C11 Dining room or kitchen furniture Dining room or kitchen furniture Monitor

C12 Office furniture for home use, including filing and 

storage equipment Filing cabinets, safes, or office furniture for home use Monitor

C16 Other furniture not already mentioned, for any other 

areas in the house

Other furniture not already mentioned, for any other areas inside 

the house Monitor

C7 Rented or leased any furniture Rented or leased any furniture. Do NOT include TV rental. Monitor

C17 Repairing, refinishing, or reupholstering furniture, 

including the cost for fabric

Repairing, refinishing, or reupholstering furniture, including the 

cost for fabric Monitor 
Recommendations 4 

Cognitive Testing for the CE Recall Interview 4-8 

 

Table 4-2. Original wording and recommended wording when testing concluded, by cluster (continued)

Q # Original wording 

Recommended wording 

(when testing concluded) 

Testing 

status 

C. Household Linens and Furnishings

Part 2

C14 Barbecue grills, outdoor lighting, or outdoor decorative 

items [Question was split into C38 and C39] [C38] Outdoor grills Test revisions

C13 Patio, porch, or outdoor furniture and covers Patio, porch, or outdoor furniture and covers Monitor

C14 Barbecue grills, outdoor lighting, or outdoor decorative 

items [Question was split into C38 and C39] [C39] Outdoor lighting or other items including decorations Test revisions

C34 Repair of household equipment or decorative items Repair of any household items we’ve covered, except appliances 

or furniture Monitor

D. Trips and Transportation

D1 Automobile service clubs such as AAA, or vehicle 

services such as OnStar and LoJack

Automobile service clubs such as AAA, or vehicle services such 

as OnStar and LoJack Complete

D3 Local tolls or electronic toll passes Local tolls or electronic toll passes Monitor 

D5 Any travel items, luggage, or any type of bags for 

carrying belongings 

Any travel items, luggage, or any type of bags for carrying 

belongings Monitor

D4 Any membership costs or other expenses related to 

vacation clubs or timeshare dues

Any membership costs or other expenses related to vacation 

clubs or timeshare dues Test revisions

Did you take any out-of-town trips since the first of [the 

ref month 3 months prior]?

During the period from [start of ref month – end of ref 

period], did you pay for anything related to any out-oftown trips?

IF YES TO EITHER: For the following questions, please 

think about money spent since the first of [the ref 

month 3 months prior] for out-of-town trips. Please do 

not include any costs that:

• You will be reimbursed for OR

• Were covered by a package deal.

D10 Lodging. Do not include expenses previously reported 

for vacation clubs and timeshares.

Cost of hotel or any lodging, including taxes. Do NOT include 

optional costs, such as parking, food, and Internet service. Test revisions

D6 Airfare, including baggage and change fees Airfare, including baggage and change fees. We’ll ask about 

other transportation costs next. Monitor
Recommendations 4 

Cognitive Testing for the CE Recall Interview 4-9 

 

Table 4-2. Original wording and recommended wording when testing concluded, by cluster (continued)

Q # Original wording 

Recommended wording 

(when testing concluded) 

Testing 

status 

D. Trips and Transportation

D7 Bus or train fare to reach your destination [D97] Transportation such as shuttles, taxis, buses, subways, 

trains, or ferries, at any time during a trip Monitor 

D9 Local transportation during a trip, such as shuttles, 

taxis, buses, subways or trains, and ferries

D8 Fares or tips for cruise ships or ferries Tickets, tips, or service charges for cruise ships Monitor

D13 Rented vehicles for recreational use. Do not include 

fuel.

Cost to rent an RV, boat, trailer, or any kind of vehicle for 

outdoor recreation. Do NOT include fuel. Test revisions

D12 Rental vehicles for getting around. Do not include fuel. Cost to rent a car, truck, motorcycle, or plane. Do NOT include 

fuel. Test revisions

D11 Gasoline, diesel, or any other vehicle fuels Gasoline, diesel, or any other vehicle fuels Monitor

D14 Parking fees Parking fees Complete

D15 Any tolls Any tolls Monitor 

D16 Meals, snacks, or drinks at restaurants, bars, or fast 

food places Meals, snacks, or drinks at restaurants, bars, or fast food places Monitor

D17

[Follow up question to D16]: How much of that amount 

was for alcohol? 

Alcohol at restaurants, bars, or fast food places

[Follow-up question to D16]: How much of that amount was for 

alcohol? 

Alcohol [at restaurants, bars, or fast food places]

Monitor

D18 Food or beverages at grocery or convenience stores Food or beverages from grocery or other stores Monitor

D19 [Follow up to D18]: How much of that amount was for 

alcohol? Alcohol [at grocery or convenience stores]

[Follow up to D18]: How much of that amount was for alcohol? 

Alcohol [from grocery or other stores] Monitor 

D21 Admission fees, tickets, tours, or other entertainment Tours, admission fees, or tickets for entertainment Test revisions

D20 Play sports, exercise, or rent any sports equipment To play sports, exercise, or rent any sports equipment Test revisions

D2 Mass transportation, commuter, or carpool services. Do 

NOT include expenses covered by your employer.

Mass transportation, commuter, or carpool services. Do NOT 

include expenses covered by your employer. Test revisions

E. Recreation

E1 Pets and their supplies, medical care, or other pet

services. Do NOT include pet food.

Pets, pet medical care, other pet services, or pet supplies. Do 

NOT include pet food. Test revisions

E3 Any equipment used for camping, hunting, or fishing. 

Include guns.

Equipment for camping, hunting, or fishing, including guns. Do 

NOT include vehicles or boats. Test revisions

E4 Winter sports equipment or gear Equipment for snow or ice sports. Do NOT include clothing. Test revisions
Recommendations 4 

Cognitive Testing for the CE Recall Interview 4-10 

 

Table 4-2. Original wording and recommended wording when testing concluded, by cluster (continued)

Q # Original wording 

Recommended wording 

(when testing concluded) 

Testing 

status 

E. Recreation

E5 Water sports equipment or gear Water sports equipment or gear. Do NOT include swimsuits or 

boats. Test revisions

E2 Protective eyewear or visual goods for magnification Protective goggles, binoculars, microscopes, or magnifying 

glasses Test revisions

E6 Bicycles, parts, equipment, or gear Bicycles, including parts or equipment for them Test revisions

E8 Health and exercise equipment Home exercise or workout equipment or machines Test revisions

E7 Sports and recreation goods or equipment Sports and recreation goods or equipment. Do NOT include 

shoes. Test revisions

E9 Service contracts, maintenance, or repairs for any sport 

or recreational equipment

Service contracts, maintenance, or repairs for any sport or 

recreational equipment Monitor 

E12 Docking or landing fees for boats or planes Docking or landing fees for boats or planes Monitor

E13 Any outboard motors Any outboard boat motors Test revisions

E17 Toys, games, or arts and crafts supplies Toys, games, or arts and crafts materials for adults or children. 

Do NOT include video or computer games. Test revisions

E10 Ping-pong, pool tables, or other similar recreation room 

items and accessories

Ping-pong tables, pool tables, or other similar recreation room 

items or their accessories Test revisions

E11 Playground equipment, sand boxes, or inflatable pools Playground equipment, sand boxes, or portable pools Test revisions

E14

Photography-related equipment, products, or 

professional services. Do NOT include computer 

software such as Photoshop.

Photography equipment, products, or professional services. Do 

NOT include phones, picture frames, or computer software. Test revisions

E15 Stamp or coin collecting Stamp or coin collecting Monitor 

E16 Sewing, knitting, or quilting machines, materials, or 

supplies Sewing, knitting, or quilting machines, materials, or supplies Monitor

E19 Any recreational or instructional classes or lessons Any recreational classes or lessons Test revisions

E18 Musical instruments, supplies, or accessories, including 

repairs Musical instruments, supplies, or accessories, including repairs Test revisions

E20 In-town admission fees, tickets, or tours Tours, admission fees, or tickets for entertainment Test revisions 
Recommendations 4 

Cognitive Testing for the CE Recall Interview 4-11 

 

Table 4-2. Original wording and recommended wording when testing concluded, by cluster (continued)

Q # Original wording 

Recommended wording 

(when testing concluded) 

Testing 

status 

E. Recreation

E21 Any souvenirs or memorabilia, purchased either in town 

or on a trip

Any souvenirs or memorabilia, purchased either in town or on a 

trip Monitor

For questions E23, E24, E25, E26, this text preceded the 

question wording: 

The next [4] questions ask about membership fees. 

Please do NOT include contributions to or membership 

in religious, professional, business, or other tax 

deductible organizations.

E23 Membership or entrance fees at golf courses, country 

clubs, health and fitness centers, or pools

Membership or entrance fees at golf courses, country clubs, 

health and fitness centers, or pools Monitor

E22 Fees for participating in leisure, recreational, or 

organized sports Fees for participating in leisure, recreational, or organized sports Monitor 

E24 Membership fees for online or in-store shopping clubs Membership fees required for online or in-store shopping clubs Test revisions

E26 Any matchmaking, speed dating, or online dating 

services Any matchmaking, speed dating, or online dating services Monitor

E25

Membership or other expenses related to any social, 

civic, fraternal, or service clubs, such as Boy Scouts, 

Kiwanis, AARP, fraternities, and honor societies

For the next question, please do NOT include contributions to or 

membership in religious, professional, business, or other tax 

deductible organizations.

Membership for any social, civic, fraternal, or service clubs

Test revisions

F. Children and infants

F4 Any child care services inside or outside the home Any child care services inside or outside the home Test revisions 

F3 Overnight camps, including transportation to and from 

the camps Overnight camps Test revisions

F1 Instructional day camps, such as basketball camp or 

acting camp

Day camps that teach specific skills, such as basketball camp or 

acting camp Test revisions

F2 General, non-instructional day camps Any other day camps not already mentioned Test revisions
Recommendations 4 

Cognitive Testing for the CE Recall Interview 4-12 

 

Table 4-2. Original wording and recommended wording when testing concluded, by cluster (continued)

Q # Original wording 

Recommended wording 

(when testing concluded) 

Testing 

status 

F. Children and infants

C9* Infants’ equipment Infants’ equipment

Test (based on 

resolution of 

category 

mapping)

F5 Any meals at school for preschool through high school 

age children

Any meals at school for preschool through high school age 

children Monitor

G. Electronics/Technology

G1 Cell or landline phones or accessories. Do NOT include 

phone service fees.

Cell or landline phones or their accessories. Do NOT include 

phone service fees. Monitor 

G4 Digital book readers or tablets

Tablets or digital book readers. Do NOT include download or 

subscription fees. Monitor 

G2 Internet services away from home, such as internet 

cafes, and WiFi access fees in hotels, airports, or planes

Internet services away from home, such as internet cafes, and 

Wi-Fi access fees in hotels, airports, or planes Monitor

G3 Portable memory storage for computers and other 

electronic devices

Portable memory, such as flash drives, external hard drives, 

memory cards, or blank CDs or DVDs Test revisions

G5

Laptops or desktop computers, or equipment that you 

use with the computer. Do NOT include expenses for 

business use.

Computers or equipment that you use with the computer. Do 

NOT include expenses for software or expenses for business use. Test revisions

G13 Office devices for non-business use, such as calculators, 

fax machines, or paper shredders

Fax machines, paper shredders, calculators or other office 

devices for non-business use Test revisions

G6 Any installations, repairs, or service contracts for 

personal use computers or equipment

Setup, service, repairs, or warranties for computers, phones, 

tablets, or digital book readers Monitor 

G7 GPS devices, aside from those on cell phones or built 

into cars

GPS devices or their accessories. Do NOT include GPSs built into 

cell phones or cars. Complete

G10 Video cameras or small personal music players Camcorders or small personal music players Monitor

G11

Stereos, radios, speakers, or sound components, 

including those installed in vehicles. Include charges for 

installation or setup.

Stereos, radios, speakers, or sound components, including those 

installed in vehicles. Include charges for installation or setup. Monitor 
Recommendations 4 

Cognitive Testing for the CE Recall Interview 4-13 

 

Table 4-2. Original wording and recommended wording when testing concluded, by cluster (continued)

Q # Original wording 

Recommended wording 

(when testing concluded) 

Testing 

status 

G. Electronics/Technology

G9

Any televisions for a home or vehicle, including 

equipment that you can use with them. Include charges 

for installation or setup.

Any televisions for a home or vehicle, including equipment that 

you can use with them. Include charges for installation or setup. 

Do NOT include subscription fees for cable or satellite.

Monitor 

G8 Any video game systems or accessories. Do NOT include 

actual video games.

Any video game systems or their accessories. Do NOT include 

actual video games. Monitor

G12 Service contracts, maintenance, or repairs for any audio 

or video electronic equipment

Service contracts, maintenance, or repairs for any audio or video 

electronic equipment Monitor 

H. Events

H1 Catered affairs, including purchases, rentals, or 

entertainment Catered affairs, including purchases, rentals, or entertainment Monitor

H3 Purchase or upkeep of cemetery lots or vaults Purchase or upkeep of cemetery lots or vaults Monitor

H2 Funeral services, burials, or cremation Funeral services, burials, or cremation Monitor

*At the conclusion of testing, Question C8 remains in Cluster C, and C9 remains in Cluster F. If BLS will be able to change the categorization scheme to code all infant-related items together, 

and ask in one question, then the placement of the combined question would need to be tested. We suggest starting with a combined question in Cluster C, where C8 is currently 

positioned.
This page is intentionally blank
Cognitive Testing for the CE Recall Interview R-1 

 

References 

 Beatty, P. (2010). Considerations Regarding the Use of Global Survey Questions. Downloaded 

September 1, 2015, from http://www.bls.gov/cex/methwrkshp_pap_beatty.pdf

Edgar, J. (2011). Global Questions Cognitive testing results. U.S. Bureau of Labor Statistics.

Downloaded July 1, 2014, from: http://www.bls.gov/cex/cesrvmeth_cloth.pdf

Gemini Project. (2010). Global Questions in the Consumer Expenditure Surveys Program.”

Washington, D.C.: Bureau of Labor Statistics, Division of Consumer Expenditures Surveys. 

Internal Document.

Goldenberg, K., and Steinberg, B. (2010). Global Questions Report Final. Internal Document.

Kopp, B. (2013). Use of financial records in the CE Survey. U.S. Bureau of Labor Statistics.

Downloaded July 1, 2014, from: http://www.bls.gov/cex/ce_symp2013_brandon.pdf

Kopp, B., and Yu, E. (2013). Summary report of the 2013 global questions cognitive testing study.

Internal Report.

Passero, B., Ryan, J., Cage, R., Stanley, S., and Garner, T. (2013). OPLC Requirements for the 

Consumer Expenditure Survey. Downloaded September 1, 2015, from

http://www.bls.gov/cex/ce_gemini_oplc.pdf

Peytchev, A. (2010). Global versus Specific Questions for the Consumer Expenditure Survey.

Downloaded September 1, 2015, from 

http://www.bls.gov/cex/methwrkshp_pp_peytchev.pdf
This page is intentionally blank
Appendix 1

Recruitment Screener
This page is intentionally blank
Cognitive Testing for the CE Recall Interview A1-1 

 

Appendix 1 

Recruitment Screener 

 Screener (in format for telephone screening; web survey screener has same items)

Thank you for your interest in our study. We need people to participate in an interview about things 

they have spent money on over the past 3 months. We are going to ask you some questions about 

yourself since we need to make sure that we include people with varying backgrounds. This will only 

take a few minutes.

1. What is your age?

|___|___|

AGE

IF AGE 17 OR YOUNGER .....................................................(GO TO END

STATEMENT ABOUT INELIGIBILITY DUE TO AGE)

AGES 18+ (GO TO 2)

NO NEED TO ASK IF OBVIOUS:

2. Are you male or female?

Male ....................................................................................1

Female................................................................................2

3. Are you of Hispanic, Latino, or Spanish origin?

YES.....................................................................................1

NO......................................................................................2

4. What is your race? Please select one or more.

White ..................................................................................1

Black or African American .............................................2

Asian...................................................................................3

Native Hawaiian or other Pacific Islander....................4

American Indian or Alaska Native ................................5

5. What is the highest level of education you have completed?

1st – 8th grade, some high school, but not a

diploma ..............................................................................1

12 grade/high school diploma or GED .......................2

Some college, Associate’s degree ...................................3

Bachelor’s degree..............................................................4

Master’s degree or higher................................................5
Cognitive Testing for the CE Recall Interview A1-2 

 

Appendix 1 

Recruitment Screener 

6. What is your marital status?

Single ..................................................................................1

Married...............................................................................2

Divorced ............................................................................3

Separated............................................................................1

Widowed............................................................................1

A member of an unmarried couple................................1

7. Do you live alone?

YES.....................................................................................1 (GO TO Question 9)

NO......................................................................................2

8. In your household...

a. How many other adults are there, besides you? 

_______ (number)

b. How many infants (2 years old or under) are there?

_______ (number)

c. How many other children (2 to 18 years old) are there?

_______ (number)

9. In the last 3 months, have you [or anyone in your household] purchased...

[List of expenditure categories with “common wording” and lures goes here,

with Yes/No answer choices for each]

10. What is your income – above $50,000 or below $50,000?

Below $50,000...................................................................1

Above $50,000 ..................................................................2

Rather not say ...................................................................3

11. What is your name?__________________________________

12. What is your address? ________________________________

________________________________

13. What is your e-mail address? ___________________________

14. What is your phone number? __________________________
Cognitive Testing for the CE Recall Interview A1-3 

 

Appendix 1 

Recruitment Screener 

[IF PHONE SCREENING]

We will be conducting interviews in [TESTING LOCATION, 

NAME AND ADDRESS OF FACILITY] during the days of 

[FROM WHEN TO WHEN]. Would you be available to meet at 

that location for a one-hour time slot on one of those days?

YES.....................................................................................1

NO......................................................................................2

[Thank you for your interest. {IF EARLY ROUND] We will be 

conducting interviews at that location again within the next few 

months. Would you be available to meet at that location for a

one-hour time slot in the future?

YES.....................................................................................1

NO......................................................................................2

Where did you see our ad for volunteers?

______________________________________________________

______________________________________________________

______________________________________________________

IF ELIGIBLE, AND DESIRABLE FOR SCHEDULING: It appears you are eligible for our 

study. We’d like to schedule you at XXXX. Let me read you the days and times I have available, and 

you can tell me what would be best for you. This will take about 1 hour of your time and we will pay 

you $40 cash. [We will send directions to the venue to you at [VERIFY – FROM SCREENER 

SURVEY – THE FULL NAME AND EMAIL OR REGULAR ADDRESS]. (It will include a map 

that shows you exactly where the room is).

I will send the directions out to you shortly. If you have to cancel your interview, please call back so 

that we can schedule someone in your place, OK? You can reach me at: [Recruiter’s Phone 

Number].

If you have any friends, relatives or acquaintances – who do not live with you – who would be 

interested in participating, please give them our contact information: studies@Westat.com or

call 1-800-XXX_XXXX.

IF NOT ELIGIBLE: Thank you so much for answering our questions. We will get back to you 

once we figure out how many people we will actually need for this study.

If you have any friends, relatives or acquaintances who do not live in your household, and would be 

interested in participating, please give them our contact information: studies@Westat.com or

all 1-800-XXX-XXXX.

END STATEMENT (INELIGIBLE)

IF NOT AGE 18+: It appears that you are not eligible for our study. Thank you very much for 

your interest and willingness to help out.
This page is intentionally blank
Appendix 2

Consent Form
This page is intentionally blank
Cognitive Testing for the CE Recall Interview A2-1 

 

Appendix 2 

Consent Form 

 CONSENT FORM

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide 

for statistical purposes only and will hold the information in confidence to the full extent permitted 

by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 

2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be 

disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of 

this form describes the conditions under which information related to this study will be used by BLS 

employees and agents.

During this research you may be audio and/or videotaped, or you may be observed. If you do not 

wish to be taped, you still may participate in this research.

We estimate it will take you an average of 45 minutes to participate in this research (ranging from 

30 minutes to 60 minutes).

Your participation in this research project is voluntary, and you have the right to stop at any time. 

If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently 

valid OMB control number. OMB control number is 1220-0141, and expires February 28, 2015.

-----------------------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant’s signature Date

___________________________________

Participant’s printed name

___________________________________

Researcher’s signature

OMB Control Number: 1220-0141

Expiration Date: February 28, 2015
Cognitive Testing for the CE Recall Interview A2-2 

 

Appendix 2 

Consent Form 

 

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is 

sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your 

voluntary participation is important to the success of this study and will enable the BLS to better understand the 

behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. 

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical 

purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the 

Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other 

applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.
Appendix 3

Session Protocol
This page is intentionally blank
Cognitive Testing for the CE Recall Interview A3-1 

 

Appendix 3 

Session Protocol 

 CE Recall Interview: Cognitive Testing Protocol

INTERVIEW PREP STEP

 Have laptop with Access form ready; enter the session number

I. Introduction 

Thank you for taking the time to help us out today. My name is ...... [and this is my colleague ( ) 

who will be taking notes for us today. We also have ( ) from the Bureau of Labor Statistics who is 

observing the session.]

II. Purpose and Consent Procedure

First, I’ll give you a little background about what we’ll be doing.

I will be asking you a series of questions from the Consumer Expenditure Survey, an ongoing survey 

that BLS conducts. The survey provides information about the buying habits of American 

consumers. This information is used for a lot of things, but its primary purpose is to help calculate 

inflation.

I’ll be asking you to answer the questions and share your reactions to them. The purpose of this 

session is to find out what you think each question is asking about, and how easy or hard it is to 

answer. We are not evaluating you. We are looking to improve the questions, so there are no right or 

wrong answers. All the information you give us will be kept completely confidential, and will be used 

to improve the survey questions.

Please read this consent form, and I’ll answer any questions you may have. [ANSWER 

QUESTIONS, IF ANY.] Please sign the form.

[START RECORDER AND GET ORAL PERMISSION TO RECORD; WHICH RECRUITER 

EXPLAINED AT TIME OF SCHEDULING CALL.] With your permission, I would like to 

record the interview in case I miss anything while taking notes. Do you give your permission to 

participate in this interview and to be recorded?

Do you have any [other] questions before we begin?
R number: ____ Interviewer Initials: ____ Interview Date and time: ____ Ref month fill: ___

Cognitive Testing for the CE Recall Interview A3-2 

 

Appendix 3 

Session Protocol 

 

General Probes, etc.

If a participant asks “What does that mean?” “Does that include...?” etc.: 

[Neutral responses]:

 “What does it mean to you?”

 “Whatever it means to you.”

 “What do you think it includes?”

 “Whatever you think it includes.”

If a participant hesitates for long time or says nothing when a response is needed:

 What are you thinking about here?

If a participant doesn’t know what cost to report:

 Your best estimate is fine.

If a participant appears to be waffling/having trouble:

 What makes it hard to [decide/answer]?

If a participant asks whether a certain item qualifies for reporting for an expenditure question:

 [There are no right or wrong answers.] What do you think?

If a participant goes into excruciating detail when giving details for items, costs OR has trouble 

remembering specifics:

 [Can you describe for me] how you would figure this out?

 How would you figure out what to say – for what you [bought/paid for], and how 

much it cost?

Part V lead-in for the Recheck: For any questions with no “Name 5” activity (and no judgment or 

list to show):

“Did talking about various recreation expenses for the past three months remind you of 

[any/other] recreation-related expenses for [question wording]?

[INTERVIEWER: GO THROUGH THE FOLLOWING SECTIONS III – V BASED ON THE 

CLUSTERS OF EXPENDITURE QUESTIONS DESIGNATED FOR TESTING IN THE 

PARTICULAR TESTING ROUND]
R number: ____ Interviewer Initials: ____ Interview Date and time: ____ Ref month fill: ___

Cognitive Testing for the CE Recall Interview A3-3 

 

Appendix 3 

Session Protocol 

 

III. Expenditure Questions: If Applicable, Total Expenditures

(REPORTING ACCURACY)

 FOLLOW ASSIGNED CLUSTER ORDER.

 ASK EVERY expenditure question designated for CLUSTERS IN testing round, to 

determine applicable expenditures.

1. Since the first of [the reference month 3 months prior], have you or any members 

of your household [FILL ADVANCE TEXT] [FILL QUESTION WORDING]?

 [FOR ANY “YES ANSWER]; RIGHT AWAY ASK

a. How much did you spend? [We’re now asking for one total across the three 

months]
R number: ____ Interviewer Initials: ____ Interview Date and time: ____ Ref month fill: ___

Cognitive Testing for the CE Recall Interview A3-4 

 

Appendix 3 

Session Protocol 

 

IV. Details for Expenses (REPORTING ACCURACY AND ESTIMATION ACCURACY)

A. [START BACK WITH P’S FIRST YES AND GO THROUGH EACH YES, ONE 

AT A TIME]. ASK:

1. You said you [or others in your household] spent money on [FILL].

a. What was one [item you (others) bought?/thing you (others) spent 

money on on/expense you (others) had]?

b. How much did it cost?

c. In what month was it [bought/paid for]?

2. Any other items?

[IF YES, REPEAT a. – c. UNTIL PARTICIPANT HAS NO MORE ITEMS TO 

REPORT FOR QUESTION]

a. What was another [item you (others) bought?/thing you (others) spent 

money on on/expense you (others) had]?

b. How much did it cost?

c. In what month was it [bought/paid for]?

Etc. for item(s) 3 – X

B. Was there anything you thought of, but decided not to [include/say] for [FILL]?

[IF YES]:

1. Can you say more about why you decided not to mention [FILL]? 

a. What was one item [you/others spent money on]?

b. How much did it cost?

c. n what month was it [bought/paid for]?

2. Any other items? 

[IF YES, REPEAT a. – c. UNTIL PARTICIPANT HAS NO MORE ITEMS TO 

REPORT FOR QUESTION]

a. What was another [item you (others) bought?/thing you (others) spent 

money on on/expense you (others) had]?

b. How much did it cost?

c. In what month was it [bought/paid for]?

[IF NEEDED] Etc. for item(s) 3 – X

Part B may be difficult for respondent. Move on quickly if respondent cannot answer readily.
R number: ____ Interviewer Initials: ____ Interview Date and time: ____ Ref month fill: ___

Cognitive Testing for the CE Recall Interview A3-5 

 

Appendix 3 

Session Protocol 

 

V. Comprehension

A. Name 5 Examples (COMPREHENSION)

[FOR INDICATED EXPENDITURE QUESTIONS], ASK:

1. What are 5 examples of expenses for... [FILL]?

[FOR THE VERY FIRST QUESTION TESTED WITH THIS PARTICIPANT, 

USE PROBE]: These can be either things you have bought or have never bought 

before.

[IF NEEDED]:

 What examples of expenses can you think of for ...[FILL]?

Make notes about whether participant has difficulty thinking of 5 examples.

B. Task, List (or only name 5) Then Recheck of Expenditures

(RECHECK REPORTING ACCURACY AND ESTIMATION ACCURACY)

[FOR EACH QUESTION, USE ACCESS PROGRAM TO FIRST:

 COLLECT INCLUDE/EXCLUDE JUDGMENT,

 SHOW LIST OF EXAMPLES, OR

 JUST SEE A PLACEHOLDER SCREEN]

[ONLY For questions that have INCLUDE/EXCLUDE JUDGMENT, say]:

1a. Please review this list and mark whether these belong, do not belong, and which 

you are not sure of.

[PARTICIPANT MAY SAY THINGS WORTH PROBING ON]

OR 

[FOR QUESTIONS THAT HAVE A LIST, SAY]:

1b. This is a list of items that apply for what’s in the box.
R number: ____ Interviewer Initials: ____ Interview Date and time: ____ Ref month fill: ___

Cognitive Testing for the CE Recall Interview A3-6 

 

Appendix 3 

Session Protocol 

 

[PARTICIPANT MAY SAY THINGS WORTH PROBING ON]

OR 

1c. [THERE WILL JUST BE A PLACEHOLDER SCREEN. IF THE NAME 5 

TASK WAS COMPLETED, SIMPLY CONTINUE TO #2A BELOW. 

OTHERWISE, SKIP TO #2B BELOW.]

[FOR ALL QUESTIONS WITH NAME 5, JUDGMENT, OR SHOW LIST, SAY]:

2a. Did/Does [looking at this list/thinking of 5 examples] remind you of [ any/other]

{expenses/items you’ve [bought/rented]} since [the first of the reference month 

3 months prior] that you didn’t previously tell us about?

OR 

[FOR QUESTIONS WITHOUT NAME 5, JUDGMENT, AND SHOW LIST,

SAY]:

2b. Do you remember [any/other] {expenses/items you’ve [bought/rented]} since 

[the first of the reference month 3 months prior] that you didn’t previously tell us 

about?

[FOR ANY “YES”], ASK:

3. Can you say more about that? [TRY TO ELICIT CAUSE SO WE CAN 

DETERMINE IF IT WAS A RECALL, COMPREHENSION, OR OTHER

ISSUE]

a. What was the [item you (others) bought?/thing you (others) spent 

money on on/expense you (others) had]?

b. How much did it cost?

c. In what month was it [bought/paid for]?

4. Any other items/things?

[IF YES, REPEAT a. – c. UNTIL PARTICIPANT HAS NO MORE ITEMS TO 

REPORT AS PREVIOUSLY NOT REPORTED]

V. Closing

We appreciate your help today. Thank you for your time.

[PROVIDE THE $40 INCENTIVE.]
Appendix 4

Lessons Learned and Observations on the Testing
This page is intentionally blank
Cognitive Testing for the CE Recall Interview A4-1 

 

Appendix 4 

Lessons Learned and Observations on the Testing 

 

BLS requested feedback from Westat about (1) which tasks worked well for diagnosing problematic 

question wording and (2) any insights into wording of the questions. These are provided below.

Feedback About Usefulness of Tasks 

We found these tasks to provide useful data for diagnosing question issues:

1. Checking exclusions: After itemizing expenses for a question, participants were asked if there 

were any expenses they thought about reporting. While most participants said “No” for most 

of the questions, the exclusions they did mention were helpful in identifying:

 What they correctly excluded (indicating the wording was working); and 

 Why they decided to exclude things they should not have (indicating an issue with the 

question wording).

2. Cued expenditure re-check: After completing the comprehension activity(ies) for each 

question, participants were asked if they had thought of any expense(s) they had not already 

told us about. Participants did have some forgotten items to report at this stage, which helped 

to identify if there was a question wording issue (that we might need to address) vs. purely a 

recall issue (that might or might not be addressed with a wording change to enhance recall). 

3. Probing: In the later rounds where we were trying to assess whether a specific wording change 

improved comprehension (or not), we found that probing about comprehension of specific 

phrases or words within the question was especially helpful. This provided direct evidence 

about the revisions, and we could look for similarities across participants in their verbalizations 

as well as identify misunderstandings. We also probed for examples (“What are some expenses 

that come to mind for this question [FILL]?”). This seemed to work better across participants 

– more like a natural conversation – and less like a “test” than the Name 5 Examples activity 

that we used in the earlier rounds of testing. 

4. Judgment task: One advantage of the judgment task was that it was possible to compare 

across participants, and across rounds, to see if there were improvements. Looking at these 

judgments helped to identify the types of expenses that were problematic. If “central” types 

of items were misjudged (vs. relatively more obscure or obsolete items), we attended to those 

in making wording changes or adding exclusion text. 

The ability to complete this task (as the Name 5 examples task below) was subject to individual 

differences; some participants carefully considered each item in making their assignments, 

while others tended to work quickly and judge most or all of the items as applicable. But this 
Appendix 4 

Lessons Learned and Observations on the Testing 

Cognitive Testing for the CE Recall Interview A4-2 

 

task is inherently less cognitively burdensome for participants since it is a “recognition” task, 

vs. a “recall” task. 

A disadvantage of the judgment task was that participants had to judge for one question at a 

time. They could judge items that were listed for more than one question as applicable for 

those questions; they were not limited to assigning applicability for just one question. A card 

sort approach could limit respondents – by requiring them to assign expenses to just one 

expenditure question. However, that approach would require more session time. 

We found this task to provide relatively less useful data for diagnosing question performance issues:

1. Name 5 examples: Participants tended to name items they had purchased themselves, despite 

hearing the instruction that examples could be things they had not bought. There were also 

individual differences; some participants were just better at naming examples than others (due 

to the cognitive burden of producing examples (vs. a recognition-type task).

We also found that for some of the participants, it was hard to switch back and forth from 

thinking about any examples of purchases (basically an “ever” reference period) to focusing on 

the past 3 months (for the expenditure re-check). For the first testing of each question, the 

flow in the comprehension section of the protocol was: 

1. Name 5 examples (ever, anytime)

2. Complete the judgment task (if the question had one)

3. Re-check expenditures (report of any other item expenses within the past 3 months 

not already reported for the question)

Then the participant repeated these steps for the next question. This required 

participants to switch between reference periods (ever, then 3 months) and the 

interviewer needed to keep participants on track.

Insights for Question Wording

The process for creating initial wording for the questions was described in the report, along with 

revision options/strategies. We found that each question needed to be evaluated individually; what 

seemed to work best for one question did not necessarily apply to other questions. However, in 

general, we found that:

 Examples worked well where the expenditure category is limited in breadth and the 

wording remained succinct. For example:

“Portable cooling or heating equipment” was changed to “Portable fans, heaters, 

humidifiers, or dehumidifiers.” This wording was not much longer but also helped 
Appendix 4 

Lessons Learned and Observations on the Testing 

Cognitive Testing for the CE Recall Interview A4-3 

 

participants realize that equipment to condition the air was applicable for the 

question.

 Definitions tended to work where:

o There is a wide variety of possible purchases;

o Using examples would have made the question wording very long; and/or 

o Instead of a standard word, respondents may use different words for the items. 

For example, there are many possible words for types of lights and many varieties of 

items used to decorate, so the wording was left as more of a definition for: “Lighting 

or other decorative items.” 

 Negatives are hard for people in an interviewer-administered interview. To train 

respondents, we placed exclusion text as a separate phrase, at the end of the question. 

We also applied a rule of thumb for question order, based on BLS input about that approach in the 

current recall interview: Order the questions that ask about more specific expenses first, to try to 

deter double-reporting that might occur if these were considered to be applicable for a more general 

question as well. 

For example, questions about dishwashers precede the questions about all other appliances, 

since portable and built-in dishwashers are asked about separately and should not also be 

reported for the questions about kitchen appliances.