SUBJECTIVE SELF-ASSESSMENTS OF ECONOMIC WELL-BEING:

PRELIMINARY FINDINGS FROM MIAMI

Thesia Garner, Linda Stinson, and Stephanie Shipp

Thesia Garner, Bureau of Labor Statistics, 2 Massachusetts Ave. NE, Wash, DC 20212

Joint Statistical Meetings

Chicago, Illinois

August 5, 1996

Sections on: Government Statistics, Social Statistics, and Survey Research Methodology

* The authors would like to thank Gordon Fisher, Sylvia Fisher, David Johnson, Brian Kojetin, Mary

McCarthy, and Clyde Tucker for comments and suggestions leading to improvements in this manuscript.

We also thank Norman Bradburn and Lee Rainwater who served as our discussants at the 1996 Joint

Statistical Meetings. Thanks are also due to the University of Michigan Survey Research Center Team

member, Derby Miller, to Bob Groves for field data collection and suggestions, and to the Survey Design

Seminar taught by Bob Groves, Stanley Presser, and Mick Couper as part of the Joint Program in Survey

Methodology at the University of Maryland. The responsibility for errors remains with the authors. All

views expressed in this paper are those of the authors and do not reflect the views or policies of the Bureau

of Labor Statistics or the views of other BLS staff members.
SUBJECTIVE SELF-ASSESSMENTS OF ECONOMIC WELL-BEING

Thesia Garner, Linda Stinson, and Stephanie Shipp

Thesis Garner, Bureau of Labor Statistics, 2 Massachusetts Ave. NE, Wash, DC 20212

1. Introduction

Surveys have long been used to measure both objective and subjective phenomena. 

Objective realities, such as one's date of birth or educational attainment, are matters of public 

record. Such information can, theoretically, be obtained from anyone having access to the 

appropriate records or evidence. Subjective reality, on the other hand, is totally private, it can be 

reported only by the person experiencing it.

The focus of our study is economic well-being. Using "income" as a proxy for this, we are 

not looking at the objective dollar amount, recognized and audited by the IRS for example. Rather, 

we are concerned with respondents' private views of their income. These views may be included in 

one's subjective assessment of income and may help to determine whether or not one experiences a 

sense of economic well-being. Close friends or family members may possibly use one's behavior as 

indirect evidence and attempt to answer these questions. Ultimately, however, it is only the 

respondent who can give the decisive response.

The question then arises, what role does the subjective assessment of economic well-being 

and income, in particular, play in the larger scheme of social analysis? As early as 1881, 

economists recognized that the utility gained from any given commodity was not always 

independent of the consumption of other goods. For example, the value of butter
may be dependent upon one's ownership of bread. Likewise, the value of a right shoe may be 

dependent upon one's ownership of the matching left shoe. Along the same vein, the valve of 

one's income may, is fact, be tempered by the amount of one's expenses or by the particular 

configuration of commodities that one already owns. For example, a very small income may be 

adequate if one already has a. home that is paid-off or a garden in which to grow food.

Clearly it would be difficult to plot the utility function depicting the value of each 

individual's income based upon their unique circumstances. An alternative approach would be to 

ask respondents themselves to look inward and make a subjective self-assessment of the 

contribution made by their income to the attainment and maintenance of their social roles. Since 

most American adults are confronted daily with the delicate task of balancing their income and 

expenses, it seems reasonable to expect them to have a rather well-developed budgetary sense. 

Consequently, the task of evaluating that balance between income and expenses in the pursuit of 

their personal goals should not be unfamiliar to them.

Likewise, the use of the survey instrument as a venue for information collection should not 

be altogether foreign to most Americans. Surveys are frequently used to collect subjective 

information, such as attitudes, beliefs, values, and preferences. The major survey problem when 

measuring subjective phenomena is the susceptibility to nonsampling error. Any change, temporary 

or permanent, in or around the responding subject (mood, experience, weather) may alter that 

subject's perceptions, interpretations, and
conclusions- if only for a moment. Likewise, the context of the interview itself, the order in which

questions are presented, and even the characteristics of the interviewer may affect the respondent's 

subjective assessment. In such cases, it would be difficult, if not impossible, to determine whether 

changes in subjective assessments represent true variation resulting from a change within the 

subject or the intrusion of measurement error into the survey process. As a result, the effective use 

of subjective questions must absolutely depend upon (1) clear, precise, and unambiguous 

language, (2) non-arbitrary response categories, and. (3) clearly defined concepts. Only in this way 

can surveys overcome the vulnerability of subjective questions to measurement error.

For the purposes of this study, we tested the clarity and precision of four subjective 

self-assessment questions: the minimum income question (MIQ), the minimum spend question 

(MSQ), the income evaluation question (IEQ), and the delighted/terrible (DST) question. The 

texts of these questions are as follows:

Minimum Income Question (MIQ)

Living where you do now and meeting the expenses you 

consider necessary, what would be the smallest income (before 

any deductions) you and your family would need to make ends 

meet?

Minimum Spend Question (MSQ)

In your opinion, how much would you have to spend each 

month in order to provide the basic necessities for your 

family?

Income Evaluation Question (IEQ)

Which after-tax monthly income would you, in your

circumstances consider to be very bad?

bad? insufficient? sufficient? good? very good?
Delighted/Terrible Questions (D/T)

Which of the following categories best describes how you feel 

about your family income (or your own income if you are not 

living with relatives)? Do you feel delighted, pleased, mostly 

satisfied, mixed, mostly dissatisfied, unhappy, or terrible?

One of the key methodological issues is to understand bettor how respondents interpret 

these questions. Use of qualitative research techniques, such as in-depth interviews and focus 

groups provides insights into how respondents interpret such terms as 'minimum income,' 

'necessities,' and 'living where you do now.'

Once we gain a better understanding of these questions, greater progress can be mane to 

combine subjective assessments and objective measures of economic well-being. With this latter 

information we can, for example, test various consumption theories such as the life-cycle 

hypothesis and tile relative income hypothesis, develop equivalence scales, measure individual 

welfare functions, conduct tests of utility maximization in consumer behavior, quantify habit 

formation and preference formation, and produce sufficiency thresholds (See Appendix A for 

more information).

The remainder of this paper is organized into five major sections: previous research and 

U.S. Government involvement, study design, methodology, preliminary results, and summary and 

conclusions. In the results section we present results from one of the focus groups and ten 

cognitive interviews. Our plan is to collect data from respondents in five geographic areas across 

the U.S. The results in this version of the paper are only from one of the areas (Miami); thus, the 

results presented in this paper must be considered
preliminary. The study was designed by staff within the Bureau of Labor Statistics (BLS), with 

data collected by the BLS and the University of Michigan Institute for Survey Research. 

Funding is provided by the BLS and Bureau of the Census. 

H. Previous Research end U.S. Government Involvement

During the twenty years in which subjective questions have been used to assess the 

economic well-being of individuals and households, relatively little research has been done on the 

reliability or validity of these particular questions. However there are four notable exceptions.

First, the Delighted/Terrible (D/T) scale was developed and tested by Frank Andrews and 

Stephen Withey1in a series of quality of life surveys at the University of Michigan Institute for 

Survey Research during the 1970s. Based upon their tests, Andrews and Withey suggest that the 

D/T scale can account for 87 percent of the variance in respondents' feelings about their income.

Second, Kapteyn2focused his research primarily on the use of the IEQ and the D/T scale 

in measuring an individual's welfare function of income (WFI), although response to related

subjective questions were also examined. With respect to the IEQ, Kapteyn found that 

respondents preferred to answer in annual amounts. He suggested that respondents who answer in 

monthly or weekly amounts "forget" certain of their income like once-a

________________________

1 F. M. Andrews and S. B. Withey. Social Indicators of Well-Being. New York: Plenum Press, 

1976, p. 343. 

2 Arie Kapteyn, "Statistical Analysis of Response to Charlottesville Hothouse, Treatment A 

Attitudinal Module, June 6-7, 1979." Unpublished manuscript.
year-fringe benefits. He suggested a finer (more points) scale for the D/T response 

categories since less ties should improve the WFI model fit.

Third, Antonides and colleagues3 conducted a study to assess the reliability and validity of 

ten methods for measuring individual welfare functions of income using income evaluation 

questions. Data were collected from an experimental survey of approximately 400 households in the 

Netherlands in the Spring of 1979. Of the ten methods tested, they recommended a combination of 

two methods, that is, they recommend that individuals be asked numerical evaluations and 

corresponding income levels. Even this method raised concerns however, because they thought that 

the numerical evaluations might tempt respondents to provide income levels proportional to the 

numerical values which would tend to produce linear individual welfare functions. They 

recommended further testing of this method.

The fourth study was conducted under the direction of Morissette and Poulin4 of Statistics 

Canada. Respondents were surveyed in supplements to the 1983, 1986, 1987, and 1988 Survey of 

Consumer Finances to evaluate alternative low income measurement methods. Interviewees were 

asked both the minimum income and spend questions and an income evaluation question. Split 

samples were used to test question wording. The major findings concerning the minimum income 

types of questions include: (1) small differences in response resulted with increases in family size; 

(2) when the same question

___________________________________________________

3G. Antonides, A. Kapteyn, and T. Wansbeek, Reliability and Validity Assessment of Tin Methods 

for the Measurement of Individual Welfare Functions of Income, unpublished manuscript, Center 

for Economic Research, Tilburg, the Netherlands, 1986.

4R. Morissette and S. Poulin, Income Satisfaction Supplement. Summary of Four Survey Years, 

Statistics Canada, 1991.
wording was used over time, no notable changes in the minimum income required resulted; and (3) 

question wording on required income affected both the minimum income and, to a lesser degree, the 

equivalence scale; the researchers found a 32 percent difference in the levels produced from the 

income questions versus from the questions (using the 1988 split panel data), with the spend question 

producing lower responses.

Although not specifically designed to examine the impact of changes in survey 

methodology, other studies have been conducted using data from the MIQ and IEQ, and related 

questions which provide us with insight concerning subjective assessments. For example, 

Sounders and Matheson,5 using Australian data, note that "perceptions of an adequate income for 

oneself are shaped by cultural identities and their concomitant reference groups in ways which go 

far beyond the effects of immediate material and personal circumstances." Garner and de Vos,6

using data from the U.S. and the Netherlands, found that question wording, design of the survey, 

and data collection instruments are likely to contribute to variations in response. Based on 

econometric analysis, they found that differences for the two countries was likely related to what 

respondents consider necessary, and to their general interpretation of the MIQ.

During this same time period, the U.S. government moved forward and included 

subjective assessments of health status and health-related work limitations in federally

___________________________________

5Peter Saunders and George Matheson, Perceptions of Poverty, Income Adequacy and Living 

Standards in Australia. Social Policy Research Centre Reports and Proceedings, No. 99, The 

University of New South Wales, April 1992. 

6Thesia I. Garner and Klaas de Vos, "Income Sufficiency vs. Poverty: Results from the United States 

and the Netherlands," Journal of Population Economics, vol. 8, 1995 pp. 117-134. Also see: Klaas de 

Vos and Thesia I. Garner, "An Evaluation of Subjective Poverty Definitions: Comparing Results from 

the U.S. and the Netherlands," Review of Income and Wealth, Series 37, Number 3, September, 1991.
sponsored surveys. The move to include subjective measures of the impact of income and 

financial resources in U.S. government sponsored surveys, however, was not as readily 

endorsed.7 The first foray of subjective assessments of income into U.S. government surveys 

came when both the MIQ and the D/T question were included in the Bureau of the Census' 1979 

Research Panel of the Income Survey Development (ISDP) Program.8 The MIQ was also added 

to the 1982 Consumer Expenditure (CE) Interview Survey9 based upon the recommendation of 

the Expert Committee for the Bureau of Labor Statistics' (BLS) Family Budget Revisions.10 This 

committee suggested that in terms of measuring family budgets, there is “a general consensus 

about how much it takes for an ordinary family to ‘get along’ - perhaps not an exact figure, but 

rather a range or ‘band’ of total expenditure levels that contains what most people would agree is 

the ‘get along’ amount” (p. 8). Consequently, the committee recommended a major effort to 

evaluate and perfect a survey methodology that would permit a paradigm shift away from the 

notion that official experts can (and should) define what the populace needs in order to get along 

or prosper. The new measurement task would be directed instead toward finding stable, 

reproducible estimates of what ordinary people experience with their family budgets (p. 9). The

_____________________________

7 The result being that research into these questions has lagged behind that of the health-related 

questions. Yet researchers here in the U.S. and in other countries (mostly in Europe, Canada, and 

Australia) continued to use the subjective economic well-being questions to provide information 

on the situation of individuals and households. See Appendix B for selected references from the 

literature 

8 For a review of the history of self-assessments in potential federal household surveys, see Demon 

R. Vaughan, "Self-Assessments of Income Needs and Financial Circumstances: Two Decades of 

Seeking a Place in Federal Households Surveys," paper presented at the Joint Statistical Meetings, 

Sections on Government Statistics, Social Statistics, and Survey Research Methodology, Chicago, 

Illinois, August 5, 1996. 

9 See deVos and Garner (1991) and Garner and deVos (1995) for analyses of these data. 

10 Harold W. Watts. "Special Panel Suggests Changes in BLS Family Budget Program," Monthly 

Labor Review, December 1980, pp. 3-10.
committee, therefore, envisioned supplementing the absolute standards embodied in detailed lists 

of commodities with more relative standards based on the reported subjective experience of 

respondents.

Implementation of the committee recommendations was initiated by proposing the only 

existing subjective assessments of income available: the MIQ IEQ, and D/T question. However, 

when the survey package of questions was sent to the Office of Management and Budget (OMB) 

for approval, the MIQ question alone was accepted for inclusion in the CE and only within the 

last interview. Eventually, even this one question was dropped when the plan to revise the Family 

Budget program was discontinued.

Again in 1991, a formal proposal from an inter-agency team was presented to OMB to 

include the existing subjective assessment questions in the Survey of Income and Program 

Participation (SIPP), a Bureau of the Census survey. After consultation with BLS, the Bureau of 

the Census, and the Office of Senator Daniel Moynihan, OMB recommended "laboratory and field 

testing" prior to the inclusion of such questions in the SIPP. In the OMB memorandum, cited was 

research from the BLS, 11 one of the few U.S. statistical agencies that had direct experience with a 

MIQ. In this memorandum, the OMB reiterated its policy to support such survey development and 

testing in the statistical agencies. The Committee on National Statistics, 12 in their report 

Measuring Poverty, A New Approach, also suggested that more work needs to be conducted on 

these measures before they can be seriously considered for an official poverty measure. However, 

they did

_________________________________

11 De Vos and Garner (1991) and Garner and de Vos (1995).

12 Constance F. Citro and Robert T. Michael (eds.). Measuring Poverty: A New Approach.

Washington, D. C.: National Academy Press, 1995.
state that, "If such survey responses were available over time on a consistent basis,... they

could be used to provide useful information with which to evaluate the official

methodology for updating the thresholds." 13

In the autumn of 1994, a preliminary proposal was prepared within the BLS to explore the 

possibility of cognitive testing of subjective questions. By the spring of 1995, specific plans were 

detailed for a first round of cognitive tests. In the Spring of 1996, BLS management determined 

that this first stage of cognitive testing might proceed, with the understanding, that upon 

completion, there would be a review of the results and decisions made concerning further 

implementation. The format of the test and the questions to be asked were developed and 

pre-tested in the BLS cognitive lab. This pre-test helped to refine the protocols and improve the 

flow of the one-on-one interviews. Staff from the BLS and the Michigan Survey Research Center, 

under contract with BLS, are collecting the data from five areas in the U.S. Also during the 

autumn of 1994, the Bureau of the Census reconvened the SIPP Interagency Working Group to 

discuss possible topics for inclusion in the 9th wave interview of the 1993 SIPP panel (scheduled 

for fielding in the period October 1995, through January 1996).14 OMB agreed that the SIPP 

would be used to collect data for the proposed wave using the D/T question, the MIQ, and the 

MSQ. These data were collected by the Bureau of the Census.

________________________

13Citro and Michael (1995), pp. 136-I37.

14 See Robert Kominski and Kathleen Short, "Developing Extended Measures of Well-Being: 

Minimum Income," paper presented at the Joint Statistical Meetings, ," Sections on 

Government Statistics, Social Statistics, and Survey Research Methodology, Chicago, Illinois, 

August 5,1996.
In an effort to coordinate activities and to meet the OMB request, the BLS and Census 

Bureau are jointly funding the cognitive testing, as noted earlier. All of the cognitive work is being 

conducted or coordinated by the BLS. Once the SIPP field data are available, BLS will work with 

the Census Bureau in analyzing those data. Then results from the two parts of the project will be 

combined to provide an overall assessment of the subjective questions under examination.

III. Study Design

Our study was designed to address four main research questions:

1. How do respondents interpret such terms as "minimum income," "sufficient income," 

"necessary expenses, " and "monthly household income?" Since these terms are not defined 

for respondents, the range of possible interpretations needed to be determined.

2. Are there potential order effects when asking respondents to make subjective assessments? For 

example, when respondents are asked to assess satisfaction with their income, are their ratings 

affected by questions immediately preceding those asked about their expenditures? The 

possibility exists that consideration of one's expenses may create a comparison that may 

temporarily raise or lower one's evaluation of income.

3. How do respondents use response categories such as "good/bad," "sufficient/insufficient," and 

"delighted/terrible?" Are these the appropriate terms for assessing people's attitudes and 

emotions about their income? What metric should be used for the scale range?

4. How easy or difficult is it for respondents to make subjective assessments of their income? 

How accurately do respondents review their economic situation when making subjective 

assessments?

To address these four main research questions, a two-prong strategy of qualitative

testing consisting of cognitive interviews and focus groups is being used. The successful
application of such qualitative methodologies requires that as many differing viewpoints as 

possible be gathered, so that a full range of ideas and opinions may be observed. In order to 

extend the breadth of views expressed, our study is designed to include three types of faintly 

composition, three levels of income, and five sites spread across the United States. The result is a 

three-by-three design matrix (based on household type and income group) with a total of nine 

cells; not all sites are represented by each cell in the matrix. At least five (5) interviews and one 

(1) focus group are to be conducted in each cell.

Based on our own evaluations and earlier work, 15 we expected that spending patterns 

impinge upon one's attitude toward income, so we decided it was necessary to include a range 

of household types reflecting diverse expenditure patterns. We screened participants and 

categorized their household types as: 

1. single adults with no children under 18 years-of-age in the home,

2. adults (either single or sharing expenses with another adult) with children under 18

years-of-age in the home,

3. adults sharing expenses, but with no children under 18 years-of-age in the home.

We defined the levels of income as either (1) low, (2) medium, or (3) high. These are 

determined separately for each geographic area in which testing is conducted. Using July to 

December 1994 Income Percentile Data from the Current Population Survey (CPS),16 staff from 

the BLS 17 adjusted these estimates to the March 1996 level using the

_____________________________

15 Garner and de Vos (1995).

16 Special thanks are extended to Kathleen Short of the Census Bureau for producing these 

ranges. Earlier ranges were produced by Bill Passero using CEX data but we decided later to use 

the CPS data for selecting our subsamples. 

17 Thanks to Wolf Weber for this conversion.
CPI-U for all items.18 These estimates represent taxable household income in the appropriate 

metropolitan statistical areas and do not include the cash value of food stamps. The income 

ranges for each area are divided into thirds for each of the three household types (see Appendix C 

for the exact income ranges used).

The five sites are selected to provided national coverage, as well as some urban/rural 

differentiation.. The sites are Miami, Los Angeles, Detroit, Baltimore, and West Virginia. In this 

paper, we only present the results from ten interviews and one focus group from one site: Miami, 

Florida. The Miami focus group and cognitive interviews were conducted by a research 

psychologist from the Bureau of Labor Statistics during the month of May 1996. Data for the 

other eight cells are being collected this summer and autumn.

IV. Methodology

Our choice of two qualitative methodologies, focus groups and cognitive interviews, 

reflected our lack of even the most basic information concerning the ways that respondents would 

react to these types of questions, scales, and concepts. We had no prior hypotheses driving our 

data collection. Instead, we are attempting to (a) simply observe the procedures and strategies 

respondents use when making subjective assessments, (b) hear the language they use when 

discussing these topics, and, ultimately, (c) see these

__________________________________________________

18 No CPI-U data were available for the Detroit/Ann Arbor area for March 1996. The alternative 

for updating the income ranges was thus to average the CPI-Us for February 1996 and April 1996 

to make the adjustment.
issues from the respondents' perspective by encouraging and assisting them in articulating their 

own unique perspectives on the world. 

A. Focus Groups

In our focus groups, respondents are asked to discuss the concepts of "minimum 

income" among themselves. Specifically, we ask them:

 What amount of income would they need to "make ends meet" and how did they

 decide upon this amount?

o What expenses would be covered by "enough money to make ends meet?" What 

would the amount of money purchase?

o What things is their lives determine the amount of income a family needs "to make 

ends meet?

In addition, participants are asked to evaluate different "levels" of income. For

instance, they are asked to describe the differences between "good" and "bad" amounts of 

income. A similar discussion is raised for the distinctions between "sufficient" and

"insufficient" incomes. Participants are asked to consider what elements or events might

cause the dividing points between "good and bad" or between "sufficient and insufficient" 

incomes to fluctuate. Finally, participants are presented with the "delighted/terrible" scale and 

probed for their reaction to as well as their understanding and use of its response categories.

B. Cognitive Interview

For this study, several tasks are drawn from the repertoire of cognitive laboratory 

techniques. The interviews begin with a concurrent think-aloud task, through which the 

participants are able to describe their thoughts while answering either the MIQ and an
alternative "minimum spend" question. Participants are first "taught" to create an answer out loud 

by having the procedures explained and demonstrated. They are then requested to talk through all 

the issues and problems they had as they arrived at their answer to the question. As the answers 

emerge, the interviewer follows up with probing questions to gain additional details and 

understanding. Participants are asked to rate their confidence in the answer they provided and to 

paraphrase some of the concepts in their own words. Such strategies are generally used to identify 

difficulties in understanding question wording or concepts, recall strategies, and the participants' 

reactions to the question.19

A second technique used is an "income sorting" task directed toward assessing the ways 

that participants evaluate income. Two versions of the task are used for each participant. In each 

case, cards are arranged in front of the participant that are labeled either "Very good," "Good," 

"Bad," "Very Bad" or "Sufficient," "Insufficient." Participants are then handed a stack of cards 

with dollar amounts written on them, ranging from $250 to $6,000 in $250 increments. 

Participants are instructed to think about all the members of their household and their expenses 

and then to evaluate each dollar amount as monthly take-home pay. The evaluations are made by 

placing each dollar amount into an appropriate category. After all the cards have been sorted and 

the difficult gray areas between categories recognized, the interviewers probe the decisions. 

Participants are asked la discuss how they decided to categorize the dollar amounts, what the 

income amounts

___________________________________________________

19 T. J. DeMaio and J.M. Rothgeb. "Cognitive Interviewing Techniques in the Lab and in the 

Field." In Answering Questions, N. Schwarz and S. Sudman (eds.). San Francisco: Jossey-Bass 

Publishers, 1996, pp. 177-195.
categorized together had in common, and what their lives would look like with the varying categories 

of income.

A third task is a series of short answer questions using the "delighted/terrible" scale. 

"Satisfaction" assessments run the risk of being affected by preceding questions that make specific 

information salient, thereby creating temporary standards of comparisons, affecting judgments, and 

causing later responses to be higher or lower by comparison. So in order to test for this possibility, a 

series of questions is asked for which participants are to use the "delighted/terrible" scale to identify 

how they felt about expenses such as the cost of feeding their families, eating out in restaurants, 

buying clothes, health care, transportation, school tuition, and housing. Half of the participants are 

asked to assess their family incomes prior to expenditure assessments; half of the participants are 

asked to evaluate their family incomes after the evaluation of expenses. Participants are also asked to 

discuss the "delighted/terrible" scale and to describe the meaning of the various categories.

In closing, the interview participants answer a series of short debriefing questions. These 

questions probe their reactions to the interview itself, what they liked best and least during the 

interview, what was easiest and most difficult, and their ideas for other questions that we could ask to 

more fully understand their subjective experience of the yin and yang of income and expenses.
V. Preliminary Findings

The results presented in this section are from one of the nine cells being tested. As noted 

earlier, data for this one cell are based on ten interviews and one focus group conducted in Miami. 

The findings to date are categorized primarily into three main issues: ambiguity in the language, 

complexity in the questions, and "new" language. Each are addressed below, including examples. 

A. Ambiguity in the Questions

1. Minimum Income Question (MIQ) and Minimum Spend Question (MSQ).

Half of the respondents are asked the MIQ and the other half are asked the MSQ. To date, there do 

not appear to be any distinguishable differences, but the results show that both questions are 

plagued by the same problems.

At the beginning of each cognitive interview, respondents are presented with a "flunk 

aloud" task. They are asked to create an answer (out loud) to the MIQ and MSQ. Focus group 

participants are asked a similar question.

As respondents begin to generate their answers, it becomes apparent at once that these 

are extremely difficult questions. In order to answer the question, it is necessary for respondents 

to generate a complete list of their monthly expenses and estimate a dollar amount for each 

expense. In many cases, respondents stop generating items and declare their list of expenses 

"complete," only to revise their list in response to specific probes from the interviewer or as 

additional items surface throughout the course of the interview.
Additionally, dollar estimates generated prior to the itemized list of expenses greatly 

underestimate the amount of money needed to cover those expenses.

Among the focus group participants, there seems to be two distinct interpretations of the 

MIQ. On the one hand, about half of the participants aye indicating that the question is asking for 

the bare minimum with which to survive-"with no gravy," as one participant said.. These 

participants interpret the phrase "to make ends meet" as meeting the most basic needs, such as 

"food on the table and shelter in which to live." This group, when pressed for a second dollar 

estimate of the "absolute" minimum amount their family would need to make ends meet each 

month provides a dollar amount that falls within the range of their original estimate of necessary 

income. In other words, "making ends meet" is seen as equivalent to the "absolute minimum 

income."

A second large group of participants seems to be interpreting the question differently. For 

these participants, the amount of income needed to "make ends meet" includes some "gravy." This 

is seen most clearly when they provide a lower dollar estimate for the "absolute" minimum 

amount needed. When pressed to describe how they arrived at an estimate of what it would take 

"to make ends meet," one participant responded, "I just calculate what I would need not to have to 

worry about expenses at all" This point of view seems to interpret "making ends meet" as starting 

with the current level of expenses, including some expenses that might be jettisoned if income 

dipped to the "absolute minimum" amount. As another participant with this perspective explained, 

"I add up all my expenses ...and estimate what we might need for things like entertainment or
emergencies." For participants with this second interpretation, "absolute minimum" income 

needed to make ends meet would whittle away the "gravy" and include "just the house bills, but 

not really any money for leisure" In other words, it is "just what you need" with no thought of 

"putting anything aside"

Most of the focus group participants seem to feel that it is relatively easy to answer this 

type of question because their needs are "pretty stable month-to-month." One may speculate, 

therefore, that if the question did not include the vague, ambiguous phrase "to make ends meet," 

but rather clearly defined the parameters of the judgment, then respondents might tie able to 

provide a valid and consistent estimate. The challenge will be to convey to respondents the request 

for "minimum survival," while still allowing them the latitude to define what minimum survival 

would look like (and cost) for them.

As part of the MIQ and the MSQ, respondents are asked to consider "necessary 

expenses.'' When respondents are asked to list these expenses and explain why they consider 

certain items to be necessities, there is general consensus about the reasons--they are required for 

existence; they are essential for survival; without them you can't live.

However, there is less consensus about what is actually required for existence. Almost 

every respondent recognized two basics without which one could not survive: food and shelter.

But beyond those two items, there is less agreement. Focus group participants are also led 

through a two-step exercise of itemizing monthly expenses. First they are asked to identify those 

expenses that could be included as necessary for "making ends meet." They are then asked to go 

beyond that step and single-out only those expenses
that they would deem "absolutely necessary." While there is always some diversity within 

groups, there is remarkable consistency across groups. A composite of the lists of monthly 

expenses follows:

Monthly Expenses _ Necessary Monthly Expenses

1. Rent 1. Rent

2. Food 2. Food

3. Clothing 3. Clothing

4. Health Insurance 4. Health Insurance

5. Transportation 5. Transportation

6. Utilities 6. Utilities

'/. Household sundries 7. Household sundries

8. Credit card payments 8. Credit card payments

9. School expenses (where applicable) 9. School expenses (no consensus)

10. Garbage 10. Garbage (no consensus)

11. Grooming expenses (no consensus) 11. Grooming expenses (no consensus)

12. Pets 12. Pets (no consensus)

13. Telephone

14. Laundry

15. Automobile/insurance/maintenance

16. Cable (no consensus)

As seen here, the two lists are remarkably similar. While a few items are dropped from 

the second list, many respondents suggested that adjustments would probably be made by 

changing the quantity or quality of items, rather than omitting a category completely. For 

instance, those who included hair care expenses as necessary suggested that they could, if 

required, reduce the number of trips per month rather than omit this expense altogether. 

Likewise, clothing could be purchased from thrift shops or more careful use could be made of 

one's current wardrobe. Rather than go hungry, participants suggested that they would be likely 

to purchase cheaper food, buy bulk items, carry their lunches, and limit their meals eaten in 

restaurants. Other adjustments included the use of generic household products and finding 

additional persons with whom to share rent. The
lists generated from the cognitive interviews are almost identical to the focus group list. 

However, it is, important to note that no single interview respondent generated the entire list. 

One may conclude, therefore, that respondents will need cues in order to make an accurate 

estimate of their necessary expenses. And if this is the case, this composite list may supply a 

good starting point for these cues.

In some cases, a distinction is also made between "needs" and "expenses." For some

participants, one's "needs" pertain strictly to things necessary for survival. "Expenses" are thought 

to include both the basic requirements for survival and the various other things that one might see 

and desire to enhance the quality of life. Respondents also distinguish between contractual 

obligations, such as credit card bills, and necessary expenses. However, once the credit card 

purchase is incurred, the respondents tend to view the expense (or at least payment of the bill) as a 

necessary one.

In order to understand the ways respondents interpret the questions and to test out 

possible new wording, a series of probes are used to ask respondents to distinguish (1) the 

SMALLEST income, (2) the MINIMUM income, and (3) the ABSOLUTE MINIMUM 

INCOME needed to make ends meet. The amount of diversity here is almost staggering. The 

interpretations include:

• "Smallest" income is equal to "minimum" income.

• "Smallest' 'income is greater than "minimum" income.

• "Smallest" income is less than "minimum" income.

• "Absolute minimum" income is equal to "minimum" income.

• "Absolute minimum" income is less than "minimum" income. 
One component of the MIQ and the MSQ is the phrase "living where you do now ...." 

Respondents are asked to tell us in their own words what that phrase is asking them to consider. 

Once again, there is no consensus. The range of interpretations included my life circumstance or 

stage, my place in life; my house, apartment; my neighborhood; or my city.

One final difficulty that emerged is the interpretation of the word "you." One respondent 

asked the interviewer whether this question refers to the amount he personally spends or to the 

amount his family spends. Clearly, "you" is one of the great ambiguities in English, meaning both 

second person singular and plural. In order to have all respondents answering the same question, it 

would be absolutely necessary to clarify which of the two is being asked.

2. The "Delighted/Terrible" Scale.

In general, focus group participants and interview respondents seemed to like the use of emotional 

words in the "delighted/terrible" scale. However, when pushed to define or differentiate the teens, 

there is little consistency in the answers. Examples of the distinctions that are made between terms 

are as follows: "Pleased" would be like "I'm making it." "Delighted" would be like "Wow! I got a 

bonus!"

• "Pleased" is like "good." "Mostly satisfied" is like "sufficient"

• "Mostly dissatisfied" is like "OK," but "unhappy" means this job just isn't 

working out to give you enough money.

• "Terrible" is completely hopeless. 

Another problem is that the full scale of descriptors is not used. For example, some

respondents viewed mostly satisfied, mixed and mostly dissatisfied as one group. In
addition, no one reported being delighted with their income or having terrible income. For the 

most part, respondents appear to have collapsed the scale into three categories; pleased, mixed, 

and unhappy.

3. Income Evaluation Questions

The question asks for judgments along two separate dimensions: "good/bad" versus 

"sufficient/insufficient." For some participants, these dimensions are asking for extremely different 

judgments. On the one hand, some participants interpreted the "sufficient/insufficient" assessment 

as a judgment about what is absolutely necessary for survival. On the other hand, the "very 

good/very bad" dimension is seen as a judgment about the "quality" of life. In some cases there is 

minimal overlap, between "very bad" and "insufficient," in the sense that income may be defined 

as "very bad" precisely because it is "insufficient." However, this comparison did not hold for the 

assessment that income is "sufficient" because of the conceptual dualism between bad income that 

is, nevertheless, "sufficient for survival" versus good income that is "sufficient for some quality of 

life. " 

B. Complexity of the Questions

1. Minimum Income Question (MIQ) and Minimum Spend Question (MSQ).

There is a clear indication that the MIQ and MSQ are complex questions to answer. They ask 

respondents to make at least four judgments, for example:

• Does the question want me to include only the minimum expenses necessary for 

survival or is it asking for me to begin with my current expense level?

• What expenses are necessary for me to survive?

• How much money would I need in a month to cover each particular expense?

• What is the total for all of these expenses added together?
The respondent must first differentiate between the current level of expenditures and the 

minimum needed for survival. If the "minimum expenses necessary for survival" is the preferred 

piece of information, the follow-up estimate of the dollar amount is somewhat a matter of 

factual tabulation once the respondent has decided upon which elements to include. More work 

needs to be done to better cue respondents and to help them answer these questions. 

3. Income Evaluation Question (IEQ).

During the course of the focus group, participants are presented with a written version of the 

IEQ and are asked to discuss their responses. Many people reported having difficulty with this 

question for the following reasons:

• The task is cognitively complex. It requires that the respondent think about expenses 

in many different ways. One strategy reported is to begin with a "very good" amount 

and then whittle away amounts to fit the appropriate categories.

• It is difficult for some participants to conceptualize what would actually be sufficient 

for survival. As one respondent reported, since she had never experienced such dire 

straits, it is extremely "hard to decide what I need just to survive ...or even less than 

that." 

When asked to supply descriptions of what the categories "very good to very bad"

and "sufficient/insufficient" might mean when applied to income, the following ideas are

generated. For example, one respondent described "Very Good" income "It is enough to be

as comfortable as possible" from "Good" income which "It is an income that is "good"

for right now, the present circumstances. " Likewise, the conceptual distinctions between

"Very Bad" income and "Bad" income: "fit means you can't pay for rent and would have to

live in a shelter. " versus "It means there may sill be hope down the road that things may
be better. " Finally, the distinctions between "sufficient" and "insufficient" income: "It means 

having a little left over after you pay your monthly expenses. " versus "It is when yore can't pay 

your bills and you start using your credit card to pay things off. "

When asked which comparison they preferred, the "fiery good" to "very bad" judgment or 

the "sufficient" versus "insufficient" comparison, here is no consensus. However, there is some 

indication that which ever task is presented second is judged to be easier, suggesting a learning 

curve. When probed for the reasons they preferred one or the other scale, the following types of 

responses are given: "very good/very bad is easier because it is easier to identify an amount 

where I wouldn't have to worry about paying bills; " or "sufficient/insufficient is easier because 

you didn't have to think about as many things and make as many judgments."

In order to further understand the fundamental inconsistency of the response, each 

individual in the focus group is asked to provide a written response to the MIQ, an "absolute 

minimum amount of income needed" and the IEQ. In several instances, respondents are totally 

unable to generate dollar amounts. A second item of interest is the reliance upon whole numbers. 

In most cases, the estimates are increments of $1,000; in a few cases, the dollar amounts varied in 

increments of $100 or tens of dollars. From this, one gathers the impression that these are only 

very gross estimates. Some respondents place the "absolute minimum" amount within the range 

of "income needed to make ends meet" and others cite an "absolute minimum" amount that is 

lower than the range of income thought to be necessary to make ends meet.
C. "New"Language

Without prompting, respondents repeatedly referred to stress, anxiety, and worry, when 

talking about "bad" or "insufficient" income. With good income or spending, they spoke of 

"freedom," "security," "not having to worry," "being more relaxed." These words came only from 

the participants--interviewers did not prompt them in any way. Perhaps this is a fruitful avenue for 

designing future subjective assessment questions. For example, we might consider these types of 

questions: "How often, if ever, do you worry about paying your bills on time?" "To what extent, if 

at all, do outstanding bills/expenses cause you concern?" "How much more money would you need 

each month to be free from worry about your bills?"

VII. Conclusions and Final Comments

As the cognitive interview and focus group testing continues, we will look for 

consistency in our results across the five geographic areas as well as across income groups and 

family types. What have we learned from this initial testing?

 It is difficult for respondents to itemize all their expenses, especially on the spur 

of the moment. Providing the respondent with cues and lists should improve 

consistency in answers.

 Respondents experience difficulty in computing the income they would 

"need before deductions," they know their take-home or after-deduction

pay. 

 When asking the IEQ , there is no consensus about the meaning of the various 

response categories provided by the interviewer. 

 Respondents are asked to perform several tasks: (1) define the terms, (2) apply 

these definitions to their own lives, and (3) generate monthly estimates to convey 

this application. It seems that the heart of the subjective assessment is actually 

found in part two, the application of the categories to their unique situations. It 

might be feasible to supply respondents with
consistent definitions so that all respondents would, in essence, be answering the same 

question or performing the same task.

It may he difficult to achieve consistent answers from the D/T questions as the scale is 

difficult for respondents. Whey cope with this scale by dividing the scale into three 

broad groups.

In the short term, we will complete and assess the testing described above. In the long 

term, we would like to incorporate what we have learned into designing better questions and 

then conduct additional cognitive tests. Finally, assuming we are satisfied with the results of the 

cognitive testing, field testing would be required to determine reliability
APPENDIX A: USES OF SUBJECTIVE MEASURES

Subjective measures can be used to further explore the relationship between expenditures 

and income and why in the lower income groups, mean expenditures often exceed income (as 

published in BLS tabulations). Probably the two most frequent uses of the questions are for 

producing subjective poverty or sufficiency thresholds, and to estimate an individual's welfare 

function of income. Examples of the a subjective poverty line and individual welfare function are 

presented in Charts A1 and A2.

Subjective measures, as examined in this study, can also be used to produce equivalence 

scales. Equivalence scales are used to compare the income and expenditures for households or 

families of different sizes and with different needs. Equivalence scales represent the relative costs 

of living of households of different sizes and compositions that are otherwise similar. Thus 

differences in need for adults and children are accounted for as are economies of scale in the 

household. For example, if a family of two adults and one child can live as well as a single adult 

while spending only 50 percent more, then relative to the reference household with one adult, the 

equivalence scale value for a two-adult family with the one child is 1.5. The scale value is often 

defined as family size to some power (Se); the power value is also referred to as the elasticity of 

need. In the following table are some examples of ways to derive equivalence scale values. The 

first was suggested by Buhmann et al.20 , the second by the OECD21, and the third by the National

Academy of Sciences Poverty Panel: 22

_____________________________________________

20 Brigitte Buhmann, Lee Rainwater, Guenther Schmaus, and Timothy Smeeding. "Equivalence Scales, WellBeing, 

Inequality, and Poverty: Sensitivity Estimates Across Ten Countries Using the Luxembourg Income Study (LIS) 

Database. The Review of Income and Wealth 34(2), 1988, pp. 115-142.
Alternative equivalence scales and their corresponding elasticities are presented in Table Al.

21Organization for Economic Cooperation and Development. The OECD List of Social Indicators. Paris: 

Organization for Economic Cooperation and Development, 1982.

22 Citro and Michael (1995).
APPENDIX B: SELECTED LITERATURE ON 

SUBJECTIVE MEASURES OF WELL-BEING

Citro, Constance and Robert T. Michael, Measuring Poverty, A New Approach,

National Academy Press, 1995, pp. 134-140.

Bradbury, Bruce, "Family Size, Equivalence Scales, and Survey Evaluations of Income 

Well-Being," Journal of Social Policy, vol. 18, no. 3, July 1989, pp. 383-408.

Colasanto, D., Kapteyn, A. and van der Gaag, J., "Two Subjective Definitions of 

Poverty: Results from the Wisconsin Basic Needs Study, Journal of Human 

Resources, vol. 19. 1984, pp. 127-137.

Danziger, S., van der Gaag, J., Tanssig, M.K., and Smolensky, E., "The Direct

Measurement of Welfare: How Much Does it Cost to Make Ends Meet:" Review

of Economics acrd Statistics, vol. 66,1984, pp. 500-505.

Deleeck, H., "The Adequacy of the Social Security System in Belgium, 

1976-1985," Journal of Social Policy, vol.18, no.1, 1989, pp. 91-117.

Douthitt, R.A., MacDonald, M., and Mullis, R., "The Relationship Between Measures 

of Subjective and Economic Well-Being: A New Look," Social Indicators Research, 

vol. 26, 1992, pp. 407-422

Garner, T.I. and de Vos, K., "Income Sufficiency vs. Poverty: Results from the United 

States and the Netherlands,"' Journal of Population Economics, vol. 8, 1995 pp. 

117-134.

Hagenaars, A.J.M. and deVos, K., "The Definition and Measurement of Poverty," 

The Journal of Human Resources, vol. 23,1988, pp. 211-221

Hagenaars, A.J.M., The Perception of Poverty, Amsterdam: North Holland Publishers,

1956.

Kapteyn, A., Kooreman, P., and Willemse, B., "Some Methodological Issues in the 

Implementation of Subjective Poverty Definitions," The Journal of Human 

Resources, vol. 23(2), 1988, pp. 222-242.

MacDonald, M., "Consumption Theories and Consumers' Assessments of Subjective

Well-Being," The Journal of Consumer Affairs, vol., 26 (2), 1992.

Saunders, P., and Matheson, G., Perceptions of Poverty, Income Adequacy and 

Living Standards an Australia, Sots icy Research Centre, Reports and Proceedings, 

No. 99,
April, The University of New South Wales, Kensington, New South Wales, 

Australia, 1992

Stanovnik, T., "Perception of Poverty and Income Satisfaction," Journal of 

Economic Psychology, vol. 13, 1992, pp. 57-69.

Townsend, Peter and Dave Gordon, "What Is Enough? The Defintion of a Poverty Line," 

in Peter Townsend (ed.), The International Analysis of Poverty, London: Harvester 

Wheatsheaf, 1993.

Van den Bosch, K., Callan, T., Estivill, J., Hausman, P., Jeandidier, B., Muffels, R., 

and Yfantopoulos, J., "A Comparison of Poverty in Seven European Countries and 

Regions Using Subjective and Relative Measures," Journal of Population Economics, 

Vol. 6, 1993, pp. 235-2.59.

van Prang, B. M.S., "Ordinal and Cardinal Utility: An Integration of the Two

Dimensions of the Welfare Concept," Journal of Econometrics, vol. 50, 1991, pp. 

69-89.

Van Prang, B. M.S., and Van der Sar, N.L., "Empirical Uses of Subjective Measures 

of Well-Being," The Journal of Human Resources, vol. 23(2), 1988, pp. 193-210.

Van Prang, B.M.S., and Flik, R.J., "Poverty Line Concepts: An Application to 

Czechoslovakia," unpublished manuscript, Amsterdam, The Netherlands, 

1992;

deVos,K. and Garner, T.L, "An Evaluation of Subjective Poverty Definitions: 

Comparing Results from the U.S. and the Netherlands," Review of Income and 

Wealth, Series 37, Number 3, September, 1991.

Waldegrave, Charles and Paul Frater, "New Zealand: A Search for a National Poverty

Line," in Else Oyen, S. M. Miller, and Syed Abdus Samad (eds.), Poverty: A Global 

Review--Handbook on International. Poverty Research, Olso: Scandinavian 

University Press, 1996, pp. 177-182.

Walker, R., "Consensual Approaches to the Definition of Poverty: Towards an 

Alternative Methodology," Journal of Social Policy, vol. 16, part 2, April 1987, pp. 

213-226