1 

Global Questions Cognitive Testing Results

Jennifer Edgar

Office of Survey Methods Research

U.S. Bureau of Labor Statistics 

December 16, 2011

Background 

The use of global questions has been long discussed in the context of a redesigned Consumer 

Expenditure Survey (CE) to allow for collection of complete household spending in a timeefficient way. There are multiple possible uses of global questions, including in a split 

questionnaire design, as part of an allocation model to incorporate CE Diary Survey (CED) data 

into the Consumer Expenditure Quarterly Interview Survey (CEQ), or to replace detailed 

questions in a shortened bounding interview. 

The quality of the data collected by global questions has been addressed in past research such as

Creech et al., 2011; Crossley and Winter, 2011; Goldenberg & Steinberg, 2010; Weber, 2002; 

and RTI, 2000. These studies overall found that data collected from global questions tend to 

produce higher expenditure estimates than the associated detailed questions, but without a ‘gold 

standard,’ are unable to identify which question type yields better quality data. Caswell & 

O’Hara, 2010, found that global questions were able to collect high quality medical expenditure 

data, but this finding has not been reported in other studies. 

Additionally, these studies have not addressed the cognitive processes behind global questions, 

how respondents answer these questions. Without understanding the response strategies 

respondents use when answering global questions, the reason for the difference between global 

and detailed questions is unknown. Exploring the response process for global questions is a key 

step in evaluating the quality of the data they collect, and will inform other comparisons between 

global and detailed questions. 

As noted in presentations at the 2010 CE Methods Workshop, laboratory research can be used to 

evaluate global questions: “Some relatively straightforward laboratory investigations could 

provide a great deal of information about memory and response strategies, which could be 

informative about the relative advantages of global and specific questions” (Beatty, 2010). 

Similarly, Peytchev (2010) noted that “the decision to use global questions, and for which 

categories, needs to be informed by an understanding not only of the differences in estimates and 

their properties, but also by the causes for the differences.” Both experts suggested conducting

cognitive research to evaluate global questions. 
2 

Research Questions 

This study addressed the following research questions: 

1. What specific items do respondents include when responding to global questions, and 

how does their definition of the global category align with the intended meaning of the 

question (and the associated detailed questions)? For example do they include coats in 

clothing? Socks? Headbands? 

2. How do respondents arrive at their answer to global questions (e.g., do they estimate, 

identify specific purchases and add)? 

Method

Cognitive Interviews

Cognitive interviews were conducted in the Bureau of Labor Statistics’ Office of Survey 

Methods Research (OSMR) cognitive lab with participants who had made clothing purchases in 

the past month. Each cognitive interview session was divided into three main parts: 

1. Global clothing questions, with follow up: 

a. Follow up of global questions: 

i. Explanation of response process 

ii. Open-ended questions asking for details about clothing purchases

iii. Examples of items in each clothing category (clothing, footwear, 

jewelry and accessories)

2. CEQ clothing questions 

a. Debriefing of CEQ questions: 

i. Comparison of response accuracy between global and CEQ questions 

ii. Request for revision to global clothing responses 

3. Shopping behavior questions 

The CEQ clothing questions were taken directly from the current instrument, with the exception 

of layettes and diapers which were excluded. The remaining 19 clothing items were asked in the 

same order, with the same question wording, as is done in the production survey. 

The semi-structured debriefing questions following the global and CEQ clothing questions were 

designed to obtain information about the participant’s response process and reactions to the 

questions. For the global clothing questions, the debriefing asked participants to list the exact 

clothing items they had purchased within the last month, as well as to describe their thought 

process when answering the global questions. Participants were also asked to provide examples 

of items in each of the three global expenditure categories. For the CEQ clothing questions, the 

debriefing asked participants to compare the accuracy of their responses for the global clothing 
3 

and the CEQ clothing questions and to explain why they thought one was more or less accurate 

than the other. Participants were also given a chance to revise their response to the global 

clothing question if they felt it was inaccurate. 

The shopping behavior questions were designed to collect information about how participants 

purchase clothing, what types of stores, how often, and whether they go shopping with the 

intention of purchasing clothing or not. This information was collected to gain insight into 

participant shopping habits, which were thought to potentially impact their response process for 

the expenditure questions. 

Web Surveys

A separate group of participants completed a web survey using a combination of Survey Monkey 

and TryMyUI, an online usability testing system. A video of their computer screen (e.g., 

showing mouse movements, keyboard entries) and an audio recording of their voice while they 

completed the survey were captured through the TryMyUI service. To take advantage of large

sample size, and recognizing the limitations of web surveys in terms of administering semistructured debriefing questions, the survey included a slightly different set of tasks from the 

cognitive interviews, with an emphasis on close-ended questions rather than in-depth 

debriefings: 

1. Global clothing questions, with follow-up: 

a. Explanation of response process (think aloud) 

b. Open-ended questions asking for detailed about clothing purchases

2. CEQ clothing questions, with follow-up: 

a. Examples of items in each clothing category (clothing, footwear, jewelry and 

accessories)

3. Clothing categorization questions 

The same three global clothing, detailed clothing, CEQ clothing , and providing examples of 

items in each clothing category were used in both the cognitive interviews and web survey (See 

Appendix A for question wording). 

After completing all three global questions, participants were asked “Now, please describe out 

loud how you arrived at your answer for question 2. Explain what you thought about or what you 

remembered as you answered the question.” After answering the global clothing questions, 

participants were given the instruction “For the rest of the survey, please describe your thoughts 

as you answer the questions.” Additional reminders to ‘think aloud’ were given throughout the 

survey. Participants’ verbal responses were transcribed, and the data they entered into the survey 

was stored by Survey Monkey. 
4 

For the clothing categorization question, a list of 51 clothing items, selected from the CEQ 

Information Booklet was provided and participants were asked to categorize responses as either 

clothing, footwear, jewelry and accessories, or other. 

Global Questions

In both modes, the global clothing questions tested were: 

1. How much have you or any member of your household spent on clothing since the first of 

[refmonth]? 

2. How much have you or any member of your household spent on spent on footwear since 

the first of [refmonth]? 

3. How much have you or any member of your household spent on spent on jewelry and 

accessories since the first of [refmonth]? 

Participants

Twenty participants were recruited from the OSMR participant database for the cognitive 

interview study. Sixty-two participants were selected from the TryMyUI panel, and screened to 

ensure they had made a clothing purchase in the past month. Of these, 54 provided usable data 

(some participants did not complete the survey, were missing audio or had other technical 

problems). For both samples, participants were selected to represent a range of demographic 

characteristics (Table 1).

Table 1. Participant Demographics 

Cognitive Interviews

n=20

Web Surveys

n=54

Gender

Male 50% 52%

Female 50% 48%

Age

<35 30% 50%

36 – 55 45% 35%

56 > 25% 15%

Education

HS or Less 20% 9%

Some College 30% 33%

College Degree 35% 44%

Graduate Degree 15% 4%

 

Analysis
5 

The majority of analysis for this project is qualitative. Participants’ verbatim explanations of the 

process they used to answer the global questions were transcribed for both the cognitive 

interviews and web surveys. The responses were evaluated and categorized into a response 

strategy type (Table 2)1

. These strategies were based on the literature (see Edgar 2009) and a 

preliminary review of the data. 

Table 2. Response Strategies for Global Questions 

Response Strategy Definition 

Item Retrieval Participants retrieve information about specific items and report the 

sum of those events

Event Retrieval Participants use information from specific events (shopping trips) 

and report the sum of those

Retrieval and Adjustment Participants retrieve information about specific items or events and 

report the sum of those after adjusting the number 

Budget Participants use their planned budget number as a response

Guess Participants indicate they guessed

General Impression Participants retrieve a general impression of the event or amount

Receipts Participants referred to receipts to answer the question*

Not Codeable Participants did not give enough information to code

*although not asked to refer to records, some web participants appear to have done so, while 

some cognitive interview participants described remembering individual receipts when 

answering the question 

Analysis was also done on the expenditure data collected from the global, detailed and CEQ 

questions. For this analysis, data from the cognitive interviews and web surveys were collapsed. 

Finally, timing data from the surveys were analyzed. 

 

Results

Response strategies

The most commonly used strategy was item retrieval followed by event retrieval (Table 3). Both 

these strategies involve participants recalling specific items or events and using that information 

to arrive at an answer. Retrieval and adjustment was used by 14 percent of participants, with the 

rest of the strategies being used by fewer than 10 percent. Twenty percent of participants used a 

budget, general impression or guessing strategy, all strategies which are not based on recalling 

specific expenditure information. 

 

1

 Each participant response was categorized into only one response strategy. If a participant 

indicated using more than one strategy, a primary strategy was identified. 
6 

Table 3. Response Strategies Used

Strategy

Percent of 

Participants 

(N=74)

Item Retrieval 34

Event Retrieval 23

Retrieval and Adjustment 14

Budget 9

General Impression 8

Receipts 7

Guess 3

Not Codeable 4

To illustrate the response strategies, Table 4 gives a representative quote for each category. 

Table 4. Quotes from each Response Strategy 

Strategy Quote

Item Retrieval

“Well, I had to buy some clothes for my son for back to school because he 

started school in August. Purchased him t-shirts, jeans, underwear, socks. Just 

some basic items and I also bought my husband some socks, underwear, kind 

of thing”

Event Retrieval “Alright, I went to Kmart and I went to Walmart because I recently lost 30 

pounds and I needed to buy new small size shorts and pants.”

Retrieval and 

Adjustment 

“I bought a suit. The suit cost 500 dollars and the four shirts were 90 dollars 

apiece. So that is 360, that’s 860. And then there is socks. Ok fine, I’ll 

change this, it was more like 960.”

Budget “I remember how much it cost because we had a budget.”

General 

Impression

“Now I know this one is going to be a little tougher. I know the total is around 

200 and I know I bought two pairs of pants”

Receipts

“I took a look at my receipts that I purchased. I actually went to two stores in 

one day, for the whole month of July that is all I spent. I added the two 

receipts up and that is how I came up with 126.” 

Guess

“We did quite a bit of back to school shopping and I was just trying to come 

up with a number, cause there was quite a bit, I have two children. So I just 

roughly said, probably about 200 for each child is my guess”

Comprehension

Examples

Both web and cognitive interview participants were asked to give examples of each of the three 

global categories. Many web participants named 15 examples, the number of lines on the web 

form for the questions. The average number given was 13, ranging from 2 to 15. Cognitive 

interview participants on the other hand, averaged 9 examples, ranging from 6 to 10 items 
7 

named. Looking at the examples provided, and the order in which they occurred, provides 

insight into the most commonly recalled items in each category. 

For clothing, pants, jackets, jeans and socks were the items most commonly named first or 

second. For footwear, sandals and tennis shoes were most commonly named first or second, and 

earrings, bracelets, necklaces and rings were most commonly named first or second in the 

accessory category. 

Overall, there were some items named by a large percentage of participants (Table 5). These 

items may be thought of as exemplars of the category, items that may be synonymous with the 

category for a large number of participants. 

Table 5. Commonly Named Examples in Different Categories

Clothing

% of 

Participants 

Naming Footwear

% of 

Participants 

Naming Accessories

% of 

Participants 

Naming

sweater 52.0% tennis shoes 46.7% Necklaces 75.6%

socks 46.7% boots 44.0% earring 74.4%

jacket 30.7% flip flops 41.3% bracelet 67.9%

underwear 28.0% dress shoes 40.0% ring 65.4%

pants 22.7% slippers 28.0% watch 37.2%

shorts 21.3% running shoes 25.3% Scarf 29.5%

coats 20.0% sneakers 24.0% belt 26.9%

dresses 18.7% heels 20.0% ankle bracelet 19.2%

hat 18.7% flats 14.7% hats 16.7%

jeans 17.3% loafers 13.3% pin 14.1%

suit 17.3% Pumps 12.0% purse 14.1%

vests 17.3% shoes 10.7% toe rings 11.5%

shoes 16.0% work boots 10.7%

tie 16.0%

shirts 12.0%

t-shirt 10.7%

Categorization 

To explore how well participants understood the intended meaning of the global questions, the 

examples given for each category were coded as ‘correct’ or ‘incorrect’ (Table 6). A majority 

(86.2 percent) of the 427 clothing examples were correct, with all the incorrect examples being 

either footwear (4.2 percent) or accessories (9.6 percent). 
8 

A large majority of the 441 footwear examples was also correct. Only 10 percent of the footwear 

examples were incorrect, and most of these were related to the unique CE specification that 

athletic shoes (e.g., soccer cleats) are not included in the category. A few other examples were 

hosiery such as socks or stockings. 

Almost all of the 511 jewelry or accessory examples were correctly categorized. The few items 

that were incorrectly named were bath or make-up items (e.g., soap, lotion, lipstick). 

Table 6. Categorization of Examples 

Category Percent

Correct Clothing 86.2

Incorrect Clothing 13.8

Footwear 4.2

Accessories 9.6

Correct Footwear 98.2

Incorrect Footwear 10.2

Hosiery 1.8

Correct Accessories 98.6

Incorrect Accessories 1.8

As part of the web survey, participants were asked to classify a list of items into one of three 

global categories, or ‘other.’ The extent participants could correctly classify items is an 

indication of their comprehension of the intended meaning of the global questions. Twenty five 

items were correctly identified by all, or almost all participants (Table 7). 
9 

Table 7. Items correctly classified by at least 90 percent of participants

Clothing Footwear

Jewelry or 

Accessories

Athletic shirt Boots Hair accessories

Dress shirt Sandals Belts

Dresses Sneakers Purse

Men’s suit Tennis shoes Sunglasses

Skirts Bedroom slippers Scarves

Sweater Casual shoes

Down vest

Undershirt

Jacket

Tennis skirts

Night gown

Night shirt

Pajamas

Fur coat

Some items were inconsistently classified, indicating that participants had differing 

understandings of the categories. Most of the incorrectly classified items were from the clothing 

category, with an emphasis on hosiery and specialty clothes (e.g., ski suit, thermal underwear) 

(Table 8). 

Table 8. Percent of Participants Classifying Clothing Items Incorrectly

Clothing Footwear

Jewelry and 

Accessories Other

Apron 30.2 0.0 20.8 49.1

Bra 83.0 0.0 1.9 15.1

Halloween costume 34.0 0.0 5.7 60.4

Knee-highs 52.8 13.2 24.5 9.4

Leotard 86.8 0.0 3.8 9.4

Raincoat 82.7 0.0 5.8 11.5

Robe 83.0 0.0 5.7 11.3

Ski suit 73.1 0.0 1.9 25.0

Snow suit 72.5 0.0 3.9 23.5

Socks 63.5 19.2 15.4 1.9

Swimwear 80.8 0.0 0.0 19.2

Thermal underwear 88.2 0.0 5.9 5.9

Tights 66.7 0.0 25.5 7.8

Windbreaker 86.3 0.0 3.9 9.8

Winter coat 86.3 0.0 3.9 9.8

 
10 

There was also some confusion about accessory items (Table 9), with participants often putting 

them into an ‘other’ category rather than classifying them as an accessory.

Table 9. Percent of Participants Classifying Jewelry and Accessory Items Incorrectly

Clothing Footwear

Jewelry and 

Accessories Other

Ear muffs 3.8 0.0 81.1 15.1

Gloves 15.4 0.0 76.9 7.7

Handkerchiefs 0.0 0.0 81.1 18.9

Mittens 18.9 0.0 71.7 9.4

Shawl 25.0 0.0 61.5 13.5

Ties 15.7 0.0 82.4 2.0

Umbrellas 0.0 0.0 64.0 36.0

Wallet 0.0 0.0 84.3 15.7

In the past, CE did not include sports shoes in the footwear category, but has since changed this 

specific requirement. Although participants were not aware of the previous specification, nor 

were they told anything about sport shoes, some found something different about soccer cleats. 

Only 88.5 percent classified them as footwear, with the rest of the participants putting them into

either accessories or other (Table 10). 

Table 10. Percent of Participants Classifying Clothing Items Incorrectly

Clothing Footwear

Jewelry and 

Accessories Other

Soccer cleats 0.0 88.5 3.8 7.7

Expenditure Reports 

Finally, the expenditure items reported can be evaluated to explore comprehension of the global 

questions. The extent to which participants included purchases incorrectly is important in 

evaluating the effectiveness of global questions. If participants mistakenly report a large number 

of items in a global question, the likelihood that there will be significant error resulting from 

comprehension is high. If participants on the other hand only report expenditures that belong in 

the category, comprehension is not likely to be a significant source of error. 

All participants reported clothing purchases, while only 57 percent reported footwear and 31 

percent reported accessory purchases. Participants correctly reported their footwear and 

accessory purchases, but had some trouble reporting clothing purchases (Table 11). Eighty-six

percent of reported clothing purchases belonged in that category, with 9.0 percent of the reported 

purchases belonging in the footwear category, and 3.3 percent belonging in the accessories

category. 
11 

It is important to note that the footwear and accessory questions followed the global clothing 

question. If the order of these questions was reversed, the inclusion of footwear or accessories in 

the clothing question may be minimized.

Table 11. Categorization of Purchases 

Category Percent

Correct Clothing 87.3

Incorrect Clothing 12.7

Footwear 9.4

Accessories 3.3

Correct Footwear 100

Incorrect Footwear 0

Correct Accessories 100

Incorrect Accessories 0

Response Errors

In addition to potential comprehension errors, there are other response errors that may be more 

common with global questions. Consideration of other household members and the reference 

period may be less likely when determining the answer, and participants may be more likely to 

make a calculation error as a result of the multiple steps required to provide a response to the 

global questions. 

 

Proxy Reporting 

This study also provides the opportunity to explore errors that participants make while answering 

clothing questions, and compare the rate at which those errors are made by question type. 

Although we cannot say definitively that participants were thinking of other household members 

and simply did not mention them, looking at the number of participants (with households larger 

than 1 person) who did mention other people in their responses may indicate the extent to which 

they were reporting for people other than themselves. 

When naming the purchases included in their global responses, more than half mentioned 

another household member, while only 31 percent of participants mentioned another household 

member when answering the CEQ questions (Table 12). 

The reason for this difference is unclear. For both types of questions, participants were asked to 

name specific purchases. There may have been more dialogue when participants reported the 

items for the global question, or they simply may not have included other household members as 

often in the CEQ questions. 
12 

Table 12. Percent of Participants who Mentioned Another Household Member 

Percent 

Global Questions 55

CEQ Questions 31

Reference Period

In addition to including other household members in their responses, participants were instructed 

to report purchases made during the last month. Again, by identifying references to the past 

month in the comments made while reporting the items included in the detailed and CEQ 

questions, we can get a sense of how well participants followed this instruction (Table 13).

Again, participants were much more likely to mention the reference period when discussing the 

items included in their global purchases than in the CEQ. It is interesting to note that only 20 

percent of participants explicitly mentioned the reference period when explaining how they 

arrived at their answer, suggesting that this was not a key part of their response process. Not 

considering the reference period when calculating a global response may be a critical error, 

indicating that participants were not thinking through the task as carefully as it was intended. 

Table 13. Percent of Participants who Mentioned the Reference Period

Percent 

Global Questions 20

CEQ Questions 9

Calculation Errors 

Requiring respondents to add or do other mathematical functions when answering a survey 

question is one of the known weaknesses of global questions, and was found to be the case in 

this study as well. To the extent that participants explained their response process in enough 

detail to do so, these explanations were evaluated to identify calculation errors (Table 14). 

Eleven percent of participants made some sort of calculation error in the global questions. The 

impact of the calculation errors ranged in type and magnitude, resulting in a difference of a few 

dollars to over a hundred, with errors resulting in increases and decreases in the expenditure 

totals. It is interesting to note that there was also a calculation error found in the CEQ questions 

as well. 
13 

Table 14. Percent of Participants Making at least one Calculation Error 

Percent 

Global Questions 11

CEQ Questions 1

Qualifiers

Although not necessarily indicative of an error, the use of qualifiers when answering survey 

questions is often used as a sign of respondent uncertainty or estimation. To look for evidence of 

estimation, the transcripts were examined for use of qualifiers, e.g., “about,” “approximately.” 

For each type of question the response was marked if a qualifier was used when the participant 

answered the question (not when they explained their response strategy). As shown in Table 15, 

a large percent of participants spontaneously used a qualifier when answering the questions, 

almost half for the global clothing question. 

As noted above, not all participants reported answers for global footwear or accessories (57 and 

31 respectively). The percents in the table below are based on all participants, not just those who 

reported amounts. These percentages, therefore, may under-represent the amount of estimation 

occurring in reported totals, or may reflect participant certainty when reporting lack of spending. 

Table 15. Percent of Participants who used a Qualifier When Responding 

Percent

Global Clothing 48.1

Global Footwear 28.6

Global Accessories 16.0

CEQ Questions 38.2

When looking at the percent of participants who used a qualifier when explaining how they 

arrived at their global clothing answer, we see that 41 percent of all participants did (Table 16). 

There was a marked difference by mode, with 76.2 percent of web participants using a qualifier 

in their explanation and 46.9 percent of cognitive interview participants using one. 

Table 16. Percent of Participants who used a Qualifier During Global Clothing Explanation 

Percent

Web 76.2

Cognitive Interviews 41.1

Total 49.4

Another potential sign of estimation is the use of rounded numbers. For this analysis, all 

responses to the global and CEQ questions were coded as either rounded (at the 10’s level), not 
14 

rounded, or a range. Rounded responses were far more likely in the global questions than in the 

CEQ questions, with more than 60 percent of all responses being rounded (Table 17). Ranges 

were also slightly more common in the global questions, with more than 5 percent of global 

clothing responses being a range, and only 2.6% of CEQ responses being a range. 

Table 17. Percent of Rounded, Not Rounded and Range Responses 

Global 

Clothing

Global 

Footwear

Global 

Accessories

CEQ 

Clothing

Rounded 69.9 70.9 60.7 41.0

Not Rounded 24.7 27.3 35.7 56.4

Range 5.5 1.8 3.6 2.6

Timing

The time it took to complete the three global questions and the set of CEQ questions could be 

determined from the videos of the web participants (Table 18). Although it took longer to 

complete the CEQ questions, the average increase in time (55 seconds) is perhaps not as long as 

would have been expected given the number of questions (3 global versus 19 CEQ). 

Table 18. Time to complete (in seconds) 

Globals 

duration

CEQ 

Duration

Min 0:12 0:16

Max 2:55 4:41

Average 0:53 1:48

Median 0:46 1:46

Expenditure Reporting 

Participants provided expenditure amounts three times: in response to the global questions, in 

response to the follow-up to the global questions (“What did you include in that response?”) and 

in response to the CEQ clothing questions. These three sets of data allow us to look at 

inconsistencies across the questions. 
15 

Reporting Rates 

It is important to note that although all participants provided at least one clothing purchase, there 

were lower reporting rates for footwear; 74.3 percent, and accessories; 39.2 percent (Table 19). 

Table 19. Percent of participants providing answer to global questions 

Clothing Footwear Accessories

Web 100.0 75.9 33.3

Cognitive 

Interviews 100.0 70.0 55.0

Total 100.0 74.3 39.2

Average Expenditure Amounts 

Looking first at the responses to the clothing questions (Table 20), there are several interesting 

findings. The first is that the average global and detailed responses were almost identical. One 

explanation for this is that participants were consciously aligning their responses2

, since the 

detailed question immediately followed the global question. Another explanation, however, is 

that participants had done some sort of enumeration of specific items and used that to respond to 

both questions (despite this not being their response strategy reported). A final explanation is 

that participants worked to recall as many clothing expenditures as necessary to arrive at their 

global answer. This study is not able to determine which explanations, or combination of 

explanations, is the case here.

Table 20. Clothing Expenditure Reports 

Global Detailed CEQ*

Average $197.77 $196.13 $165.66

* Summed CEQ clothing questions, excluding footwear, watches and accessories 

 

The total expenditures reported for the CEQ clothing questions were found to be significantly 

lower than the total clothing expenditures reported for the global clothing questions (t-test, 

p=.005). Of the 74 participants, 45 (61 percent) gave higher global clothing responses than the 

total of their CEQ clothing responses. This result occurred for both the cognitive interview and 

web participants. 

To further explore this result, we can look at the distribution of the responses (Table 21). The 

trend was highly consistent across the quartiles, with the global and detailed expenditure totals 

being higher than the CEQ expenditure totals, with one exception: detailed was lower than the 

 

2 Some participants did go back and change their answer to the global questions after answering 

the detailed question; however, their original response is used in all analysis. 
16 

CEQ estimate in the 4th Quartile. It is interesting to note that there was at least one participant 

who did not report any clothing for the CEQ questions despite reporting $10 worth of clothing 

purchases for the global and detailed questions. 

Table 21. Distribution of Clothing Expenditure Reports 

Global Detailed CEQ*

Minimum $10.00 $10.00 $0.00

1st Quartile $65.00 $65.00 $60.00

2nd Quartile $150.00 $135.00 $120.00

3rd Quartile $250.00 $276.50 $196.25

4th Quartile $960.00 $810.00 $872.00

Maximum $960.00 $810.00 $60.00

Standard Deviation $180.91 $173.07 $175.45

* Summed CEQ clothing questions, excluding footwear, watches and accessories 

The same three expenditure amounts (global, detailed and CEQ) were provided for footwear and 

accessories (Table 22). The differences between the global and CEQ expenditures were not 

statistically significant, with the global question yielding higher expenditures in footwear but 

lower expenditures in accessories. For both categories, total CEQ expenditures were higher than 

the global for a majority of reporting participants (71 percent for footwear, 76 percent for 

accessories). The magnitude of the difference, however, was not significant, averaging out to be 

$3.04 in footwear and $1.57 in accessories. 

The higher CEQ expenditure total for accessories is likely related to a lack of comprehension of 

the category and the effectiveness of the CEQ questions for aiding recall. It is likely that the 

three specific questions about accessories, watches and jewelry are more meaningful and 

concrete than the global ‘accessory’ question, which previous analyses showed contains more 

conceptual ambiguity. 

Table 22. Footwear and Accessory Expenditure Reports 

Global Detailed CEQ*

Average Footwear $66.85 $64.57 $63.16

St. Deviation Footwear $54.03 $56.26 $54.04

Average Accessories $53.00 $31.42 $56.41

St Deviation Accessories $119.59 $32.17 $136.17

*CEQ Footwear consists of one question. Accessories is the sum of accessories, watches and 

jewelry. 

Differences in Expenditure Items Reported
17 

The detailed and CEQ expenditure reports provide the opportunity to identify specific 

differences between the question types across all expenditure categories (clothing, footwear, and 

accessories). Items that were listed either in the detailed questions following the global question 

or the CEQ questions but not both were flagged3

. The flagged expenditures were counted and 

total expenditure amounts were calculated (Table 23). 

Thirty-three participants listed at least one item in the global question but not the CEQ, and 33 

participants (with some overlap, but not complete) listed at least one item in the CEQ questions 

but not the global. On average, more items were added during the CEQ than during the global 

(0.92 versus 0.68), but this difference was not statistically significant (p>.05). 

The total expenditure amounts included in the CEQ but not the global questions were also higher 

($43.81 versus $23.94), but again this difference was not statistically different. There were some 

large expenditure amounts excluded from both question types, up to $245 in the global but not 

the CEQ and $810 in the CEQ but not the global. 

Table 23. Sum of differences in total expenditures between Global and CEQ Questions 

Number of Items 

in Global but not 

CEQ

Number of Items 

in CEQ but not 

Global

Sum of items in 

Global but not 

CEQ

Sum of items in 

CEQ but not 

Global

Average 0.68 0.92 $23.94 $43.81

Min 0.00 0.00 $0.00 $0.00

Max 4.00 5.00 $245.00 $810.00

St Dev 0.89 1.26 $46.13 $109.72

Since almost half the participants had at least one item which was excluded from one of the 

question types, this indicates that both question types are able to solicit expenditure information 

from participants, perhaps in different ways. Looking at the specific items excluded from each 

question type can provide insight into the differences between the questions (Table 24).

 

3 Items were only flagged if it was clear they were not reported for the other question. If there 

was any uncertainty due to inconsistent or missing item descriptions, the item was not flagged. 
18 

Table 24. Items Included in One Question Type but not Another 

Items included in 

Global but not CEQ

# of Participants 

Including*

Items included in 

CEQ but not Global

# of Participants 

Including**

Shirts 7 Undergarments 7

Socks 5 Swim suits 6

Shorts 3 Dresses 5

Skirts 3 Jewelry 5

Ties 3 Accessories 4

Dresses 2 Earrings 4

Jackets 2 Coats 3

Outfits 2 Skirts 3

T-shirts 2 Sweaters 3

Underwear 2 Watches 3

Jackets 2

Nightgowns 2

Pants 2

Uniforms 2

*There were an additional 16 items named by only one participant 

**There were an additional 8 items named by only one participant 

Looking at the items named in the global questions but not the CEQ questions, possible 

explanations can be developed. For some items, it seems likely the correct CEQ question was 

not meaningful enough to either stimulate recall of the item, or for the participant to know to 

report the item there. Examples of this may be socks (hosiery) and ties (accessories). For other 

items, where it is unlikely that participants misunderstood the category (e.g., jackets, dresses), it 

is possible that the number of CEQ questions read caused the participants not to hear or be able 

to process them all, and so they did not spend the time required on the response process to recall 

and report the items, or that they were not attending to all the categories. 

Looking at the items named in the CEQ questions but not the global questions, it seems likely 

that the detailed questions aided recall for items the participants otherwise did not think of. 

Examples of this are undergarments and swimsuits, categories which may not immediately occur 

to participants when considering the global category of ‘clothing.’ This is consistent with the 

comprehension findings of this study, there are certain types of items, including swimwear, 

which were not mentioned by any participants when giving examples of clothing. There was one 

participant who reported for other household members when answering the CEQ questions, 

suggesting that the question introduction, “Since the first of [refmonth], have you or any 
19 

members of your household...” may have added to her understanding of the task in a way the 

global question did not4

. 

Finally, there are some items that appear on both lists that some participants named only in one 

question type, while others only named it in the other question type (e.g., jackets and underwear). 

This finding may indicate that question type has a differential effect on participants, and that the 

effects noted above are not consistent across all participants. 

Cognitive Interview Debriefings

The cognitive interview participants answered a series of debriefing questions aimed at better 

understanding their experiences with the questions, and their clothing purchases. This 

information may be useful when making revisions to the clothing survey questions. 

Global versus CEQ 

Participants were asked which set of questions, global or CEQ, gave the most accurate answers. 

Ninety-five percent of participants felt that the CEQ questions yielded more accurate answers. 

The one participant who said the global questions were more accurate explained that she had one 

big trip and the total amount was just “something in my head,” but she didn’t know the price of 

each of the clothing items she bought. 

In general, participants said the CEQ questions made them think more carefully about their 

responses “Because it forces you to brainstorm and get down to the nitty gritty. Then when you 

get down to the specific items it forces you to think about the specific number.” Many also 

explained that the CEQ questions made them think of different items or categories they hadn’t 

thought of when answering the global question “what is clothing? It's not as specific, it could be 

something to one person but to me it’s something else. The more I think about it, the more 

clothing I think of.” Another participant commented “the categories offered stimulated memory 

better than trying to think through categories myself. "You really think about the detailed 

questions, if you don't mention skirt, I don't remember skirts" 

Some participants commented that the vagueness of the global questions influenced their 

answers “you ask a general question, so I gave a general answer.”

 

4

 Note that the global question also included “have you or any member of your household,” but 

the phrase occurs at the beginning of the question and may be easily forgotten as the participant 

calculates a response. 
20 

Shopping Patterns 

Cognitive interview participants were asked to describe how they typically buy clothing. This 

portion of the debriefing evolved from several discussions with participants about their 

purchases, which revealed that there was a relationship between how people responded to the 

global questions and how they shop for clothing. 

There were three main types of shopping behaviors: participants who planned big shopping 

trips, participants who pick up items as they see them, and participants who buy based on sales 

or coupons. The first two behaviors were more commonly reported than the last, with some 

participants noting that sales play a role in their behavior (e.g., scheduling big trips around sales, 

or picking up individual items because it was one sale). 

In addition to reporting how they typically buy clothing, participants also described whether they 

typically buy clothing items one at a time or if they stock up on clothing items. Most 

participants said they buy items ‘here or there,’ with about a quarter reporting that they do both 

(stock up and pick up individual items), and 30 percent saying that they stock up. 

Participants were also asked if they typically buy clothing from a clothing store, or from a store 

which sells more than just clothing, like a department store. Three-quarters of participants buy 

clothing from the latter. Some of the participants who reported buying clothing at a clothing 

only store referred to a specialty store, such as “Big-and-Tall.”

When asked how often they buy clothing, participants provided a range of answers (Table 25). 

No relationship was found between the reported frequency of reporting, reported shopping 

pattern or ‘stock-up’ behavior. 

Table 25. Reported Shopping Frequency. 

Frequency Percent

Monthly 30.0

Seasonally 20.0

Every 2 months 15.0

Every 2 weeks 10.0

Weekly 5.0

Every 3-6 months 5.0

Every 6 weeks 5.0

It depends 5.0

Conclusions 

Summary

This study was designed to explore how well participants’ understanding of global clothing 

questions align with the CE definitions, and how they arrive at answers to global clothing 
21 

questions. In addition to these goals, the data were explored to identify errors in reporting, the 

time it takes to answer both global and CEQ questions, and differences in expenditure reporting 

between question types. 

One of the key findings was that global clothing questions resulted in statistically higher 

expenditures when compared to the CEQ questions. On the other hand, global questions for 

footwear and accessories did not result in statistically higher expenditures when compared to the 

CEQ questions. A partial explanation appears to be the more ambiguous nature of the footwear 

and accessories categories. 

Overall participant understanding of the global clothing, footwear and accessory questions align 

well with the CE definitions of the categories. The majority were able to name purchases and 

examples in each category, and classify items into the correct category. There are some items 

that were problematic for participants, and which had higher rates of misclassification. 

The majority of participants identified either clothing items or shopping events when 

determining their answer to the global clothing question. The extent to which this yielded 

complete and accurate responses is unknown, but the differences between the global, detailed 

and CEQ clothing expenditure totals indicates that there is inaccuracy in the responses. 

Limitations

There are several significant limitations of this study which will reduce the applicability of the 

findings. Without validation or other benchmark data, the study is not able to identify which 

type of question yields more accurate answers. As noted above, the discrepancy between the 

three sets of expenditure totals indicates that there is inaccuracy, but this study is not able to 

identify which is more accurate. 

Another important consideration is the order the questions were presented; by the time 

participants received the CEQ questions, they had been asked about their purchases twice. This 

likely has the effect of improving CEQ reporting, but the effect cannot be confirmed and is a 

limitation of the study.

The focus of this study is clothing. Therefore, the information about response strategies and 

comprehension is only available for this category, and the extent to which these findings can be 

generalized to other expenditure categories is unknown. 

This study used a one month recall period. If global questions are used with a different, longer 

reference period, it is likely that participant response strategies would differ. 

This study cannot, nor can any cognitive lab study, evaluate the effectiveness of using global 

questions to prevent telescoping in future interviews. A field test would be required to evaluate 

that issue. 
22 

Implications

The finding that participants use a range of response strategies to arrive at an answer to global 

clothing questions, and the high percentage of participants who used a qualifier when answering 

the global questions, confirm that answering these types of questions is not a straightforward 

task, and the use of these types of questions should be carefully considered. 

Overall, participants understood the three clothing questions as intended, suggesting that the 

tested wording may be appropriate for future use. Although there was more misclassification 

within the footwear and jewelry and accessories questions, participants tended to correctly define 

these questions as well. 

The errors made by participants, however, particularly the lack of mention of the reference 

period, suggest that the introduction to the global questions should be carefully developed to 

ensure participants are considering the correct reference period (or any specific time period at 

all) when answering global questions. 

An unanticipated finding of statistically significant differences in expenditure totals between 

global and CEQ questions, even with a small sample size, confirms other studies showing 

differences in reporting levels between global and detailed questions. Given that one potential 

use of global clothing questions in the CEQ survey is to provide a picture of complete spending, 

while collecting the required detailed clothing data from the CED, these results raise concern 

about the comparability of the two data sources (global and detail). 

Another potential use of global questions is to prevent telescoping. As noted above, additional 

testing will be required to determine the effectiveness of this approach; however, the results from 

this study do suggest that caution be used in this approach. Twenty percent of participants in this 

study did not recall specific expenditure information when answering the global clothing 

question, and so the global question would not be effective in preventing telescoping for these 

participants. 

Future Work

Additional analysis will be conducted on this data to further explore differences in response 

strategies. Exploring response strategy by demographic characteristics and reporting shopping 

behavior may shed additional light on the results, or allow for more conclusions to be drawn 

about the use of global questions in the CE. 

 
23 

References

Caswell & O’Hara, 2010. Medical Out-of-Pocket Expenses, Poverty and the Uninsured. SEHDS 

Working Paper 2010-17, US Census Bureau. 

Creech, B., Smith, M., Davis, J., Tan, L., To., Fricker, S., & Gonzalez, J. (2011). Measurement 

Issues Study Final Report. BLS Internal Document

Crossley, T.F. & Winter, J. K. (2011). Asking households about expenditures: What have we 

learned? Paper presented at NBER CRIW workshop. 

Beatty, P. (2010). Considerations Regarding the Use of Global Survey Questions. 

Edgar, J. (2009). What does 'usual' usually mean? Presented at the 2009 American Associate for. 

Public Opinion Research Conference, Hollywood FL 

Goldenberg, K., & Steinberg, B. (2010). Global Questions Report Final. BLS Internal Document

Peytchev, A. (2010). Global versus Specific Questions for the Consumer Expenditure Survey. 

RTI (2000). CE Interview Global Questions Research: BLS Conference Presentation Report. 

BLS Internal Report. 

Weber, W. (2002). CE Diary Food-at-Home “Global-Detailed” Comparisons. BLS Internal 

Report. 
24 

Appendix A: Clothing Questions 

Global Questions

1. How much have you or any member of your household spent on clothing since the first of 

[refmonth]? 

2. How much have you or any member of your household spent on footwear since the first 

of [refmonth]? 

3. How much have you or any member of your household spent on jewelry and accessories 

since the first of [refmonth]? 

Open-ended detailed clothing question

1. What specific clothing items have you, or someone in your household, purchased since 

the first of [refmonth] and how much did each cost? 

CEQ Questions

Since the first of [refmonth], have you or has any member of your household purchased any of 

the following items, either for members of your household or for someone outside your 

household? If yes: What did you buy and how much did each cost? 

1. Coats, jackets or furs 

2. Sport coats or tailored jackets 

3. Suits 

4. Vests 

5. Sweaters or sweater sets 

6. Pants, jeans, or shorts 

7. Dresses 

8. Skirts 

9. Shirts, blouses, or tops 

10. Undergarments 

11. Hosiery 

12. Nightwear or loungewear 

13. Accessories 

13. Swimsuits or warm-up or ski suits 

14. Uniforms

15. Costumes 

16. Footwear, including athletic footwear 

17. Diapers 

18. Layettes 

19. Watches 

20. Jewelry