MEASURING SUBJECTIVE

ECONOMIC WELL-BEING: An Economist’s Viewpoint

Thesia I. Garner, Linda Stinson, and Stephanie S. Shipp

Bureau of Labor Statistics, 2 Massachusetts Ave. NE, Wash, DC 20212

American Association for Public Opinion Research Conference

May 16-18, 1996

Norfolk, VA

May 13, 1996

Revised March 3, 1998

Subjective Assessments of Income and Expenses

Organizer: Linda Stinson (BLS)

Discussant: Seymour Sudman (University of Illinois)

The authors would like to thank Gordon Fisher, Sylvia Fisher, David Johnson, Brian Kojetin, Mary

McCarthy, and Clyde Tucker for comments and suggestions leading to improvements in this

manuscript . Special thanks are extended to Norman Bradburn, Lee Rainwater, and others for their

discussion of an earlier version of this paper that was presented at the 1996 Joint Statistical

Meetings, and to Kent Monroe for his discussion of the paper at the 1996 Association for Consumer

Research Conference. Thanks are also due to the University of Michigan Survey Research Center

Team member, Darby Miller, to Bob Groves for field data collection and suggestions, and to the

Survey Design Seminar taught by Bob Groves, Stanley Presser, and Mick Couper as part of the Joint

Program in Survey Methodology at the University of Maryland. The responsibility for errors

remains with the authors. All views expressed in this paper are those of the authors and do not

reflect the views or policies of the Bureau of Labor Statistics or the views of other BLS staff

members.
MEASURING SUBJECTIVE ECONOMIC WELL-BEING: AN ECONOMIST’S VIEW

Thesia I. Garner, Linda Stinson, and Stephanie S. Shipp

Bureau of Labor Statistics (BLS)

“...it is imperative to understand how a persons’ ratings of their own well-being relate to objective

measures of their circumstances.” (Duncan, 1973).

KEYWORDS: subjective economic well-being, equivalence scales, poverty thresholds

I. BACKGROUND

During the past twenty years subjective questions have been used to assess the

economic well-being of individuals and households. Such questions include:

1. “How much income does it take for you and your family to make ends meet?”

2. “How would you evaluate your income, say, on a scale from very good to very

bad?”

Data from these and similar questions have been used to produce equivalence scales,

subjective poverty thresholds, plots of utility curves, and as a benchmark for assessing

objective measures of well-being. This latter purpose cuts across all economic uses of

subjective assessments. For example, equivalence scales derived from subjective

measures are compared to equivalence scales derived using objective data. Subjective

measures provide information that helps determine the extent that objective measures are

in agreement with public perceptions (Citro and Michael, 1995).

The U.S. government became involved in the collection of subjective income data

in 1979 with the sponsorship of the Census’ Research Panel of the Income and Survey

Development Program. In 1982, a minimum income question was asked in the U.S.

Consumer Expenditure Interview Survey. However, before approving further data

collection, Office of Management and Budget (OMB) staff recommended that laboratory

and field testing be done in order to help us better understand what respondents are
2

thinking when they are faced with such questions. Up to the present time, there have

been only a few studies conducted which address this issue. The BLS responded to

OMB’s recommendation and began, in the spring of 1996, an in-depth investigation of

subjective economic well-being questions using cognitive techniques.

The purposes of this paper are

• to provide a brief overview of how data from such questions are used or can be used

by social scientists, particularly economists,

• to review previous studies which have been conducted related to question

interpretation, and

• to outline the study methodology employed by the Bureau of Labor Statistics, begun

in the spring of 1996, to learn more about what respondents think when they are asked

such questions.

For the purposes of this study, we tested the clarity and precision of four

subjective self-assessment questions: the minimum income question (MIQ), the

minimum spend question (MSQ), the income evaluation question (IEQ), and the

delighted/terrible (D/T) question. The texts of these questions are as follows:

Minimum Income Question (MIQ)

Living where you do now and meeting the expenses you

consider necessary, what would be the smallest income

(before any deductions) you and your family would need to

make ends meet?

Minimum Spend Question (MSQ)

In your opinion, how much would you have to spend each

month in order to provide the basic necessities for your

family?

Income Evaluation Question (IEQ)

Which after-tax monthly income would you, in your

circumstances consider to be very bad?

bad? insufficient? sufficient? good? very good?
3

Delighted/Terrible Question (D/T)

Which of the following categories best describes how you

feel about your family income (or your own income if you

are not living with relatives)? Do you feel delighted,

pleased, mostly satisfied, mixed, mostly dissatisfied,

unhappy, or terrible?

Subjective self-assessments of economic well-being and income, in particular, are

not new constructs of interest in social analysis. As early as 1881, economists recognized

that the utility or economic well-being gained from any given commodity was not always

independent of the consumption of other goods. For example, the value of butter may be

dependent upon one's ownership of bread. Likewise, the value of a right shoe may be

dependent upon one's ownership of the matching left shoe. Along the same vein, the

value of one's income may, in fact, be tempered by the amount of one's expenses or by the

particular configuration of commodities that one already owns. For example, a very

small income may be adequate if one already has a home that is paid-off or a garden in

which to grow food.

Clearly it would be difficult to plot the utility function depicting the value of each

individual’s income based upon their unique circumstances. An alternative approach

would be to ask respondents themselves to look inward and make a subjective selfassessment of the contribution made by their income to the attainment and maintenance

of their social roles. Since most American adults are confronted daily with the delicate

task of balancing their income and expenses, it seems reasonable to expect them to have a

rather well-developed budgetary sense. Consequently, the task of evaluating that balance

between income and expenses in the pursuit of their personal goals should not be

unfamiliar to them.
4

The major survey problem when measuring subjective phenomena such as income

adequacy is the susceptibility to nonsampling error. Any change, temporary or

permanent, in or around the responding subject (mood, experience, weather) may alter

that subject’s perceptions, interpretations, and conclusions— if only for a moment.

Likewise, the context of the interview itself, the order in which questions are presented,

and even the characteristics of the interviewer may affect the respondent’s subjective

assessment. In such cases, it would be difficult, if not impossible, to determine whether

changes in subjective assessments represent true variation resulting from a change within

the subject or the intrusion of measurement error into the survey process. As a result, the

effective use of subjective questions must absolutely depend upon (1) clear, precise, and

unambiguous language, (2) non-arbitrary response categories, and (3) clearly defined

concepts. Only in this way can surveys overcome the vulnerability of subjective

questions to measurement error.

One of the key methodological issues is to understand better how respondents

interpret these questions. Use of qualitative research techniques, such as in-depth

interviews and focus groups provides insights into how respondents interpret such terms

as ‘minimum income,’ ‘necessities,’ and ‘living where you do now.’

II. ECONOMIC USES OF SUBJECTIVE MEASURES:

THEORY

The minimum income question (MIQ) is used to develop what is called the subjective

poverty line (SPL) and the income evaluation question (IEQ) is used to estimate the
5

Leyden poverty line (LPL). The following explanation is from Kapteyn, Kooreman, and

Willmse (1988).

Subjective Poverty Line (SPL). The answer to the minimum income question (Y*) is

dependent on the respondent’s actual after-tax income (Y) and other demographic

characteristics (x) such as age, family composition, level of education and so on: Y =

f(Y, x). The function f is monotonically increasing in Y.

Figure 1. Subjective Poverty Line (MIQ)

450

Y*min

Ymin

Y

Ymin=Y

Ymin=f(Y;x) Y*min=f(Y*;x)

I

II

III

A

Y Y1 2

Y*min

The Subjective Poverty Line is defined as the point where Y*min = f(Y*, x), i.e. where the

two values intersect. This defines the point where families ‘make ends meet.’

Respondents with income below Y*min have greater needs than their income can meet,

while families with income above Y*min have more income than they need to ‘make ends

meet.’ The position of the function f(Y, x) depends on x, the demographic characteristics

of the family.

Leyden Poverty Line (LPL). A respondent’s welfare function of income (WFI) is

estimated from his answers to the income evaluation question that assigns various income

values to different evaluations of income from ‘very bad’ to ‘very good’.’ The chart

below illustrates this:
6

Figure 2. Leyden Poverty Line (IEQ)

U(y)

Y (U.S. dollars)

0.50

“very bad”

“bad”

“insufficient”

“sufficient”

“good”

“very good”

1.00

answers to IEQ used to estimate welfare function of income (WFI)

 y1 y2 y3 y4 y5 y6

0

0.45

[ ]

[ ]

U y

y u

y u

( )

[ln ]

 exp ( )

<

− <

≡ +×

α

μ σ σ

μ σ α

The income values given by the respondent for each of the six labels (very good, good,

sufficient, insufficient, bad, very bad) are assigned to the midpoint of the six intervals on

a zero-one scale. The relation between the income level z and its numerical evaluation on

a zero-one scale, U(z), can be approximated by a lognormal distribution (Van Praag,

1968). The LPL is based on the notion that poverty is a state of low utility. Someone is

defined as poor if his welfare function of income (WFI) is less than a the ‘welfare level’

set by politicians. Like the SPL, the LPL is also dependent on the characteristics of the

respondents. A basic assumption for the calculation of subjective income thresholds is

that every respondent understands the terminology in the same way (De Vos and Garner,

1991).

APPLICATIONS

Appendices 1 and 2 list several studies that use data from the three types of subjective

questions used in this study.

Developing poverty thresholds. Subjective measures are based upon the notion that the

opinions of people about their own situation should ultimately be the decisive factor in
7

defining poverty For example, the minimum income question is used to ask the opinions

of people with respect to the income level minimally necessary to make ends meet. The

Income Evaluation Question asks people to evaluate their income at various levels.

Data from these questions are used to develop subjective poverty thresholds and

thus eliminate the need for experts. Citro and Michael (1995) feel that the subjective

label is ‘unfortunate’ since all types of thresholds involve subjective elements1.

Subjective measures are criticized because poverty measures vary with type of questions

and methods used to collect information and often the samples used to collect these data

are small. As a result, thresholds derived using answers to subjective questions are

subject to large response variance and are sensitive to question wording, as the data below

indicate.

Table 1. Examples of Subjective Poverty Thresholds for Four-Person Families

in the U.S. Around 1980 Set by Various Methods (in constant 92 dollars)

Question asked and source Threshold

“Poverty”1 Gallup 1980 (Vaughan 1993) $15,895

MIQ2-Wis. BNS 1981 (Colasanto, Kapteyn, & Van der Gaag 1984) $12,160

MIQ3-ISDP 1979 (Danziger, Van der Gaag, Taussig, & Smolensky 1984) $24,680

MIQ4-CEX 1982 (de Vos & Garner 1991) $32,530

Source: Table 2-5, page 142, (Citro & Michael, 1995) 1

 Question asked: “People who have income below a certain level can be considered poor. That level is called the ‘poverty line.’

What amount of weekly income would you use as apoverty line for a family of four (husband, wife, and two children) in this

community?” Threshold based on values reported.

2

 Minimum income question asked but did not specify whether respondents were to answer in before- or after-tax income terms.

Threshold based on intersction method (Goedhart et al. 1977).

3

 Minimum income question asked: “Living where you do now adn meeting the expenses you consider necessary, what would be the

very smallest icnome you (and your family) would need to make ends meet?” “Is that per week, per month, or what?” “Wout that be

before or after taxes?” Threshold based on intersction method (Goedhart et al. 1977).

4

 Minimun income question asked: “Living where you do now and meeting the expenses you consider necessary, what would be the

samllest income (before any deductions) you and your family would need to make ends meet?” “Is that per week, per month, or

what?” Threshold based on intersction method (Goedhart et al. 1977).

 

1

 Page 134.
8

There is also the possibility that the results could be biased, at least for public opinion

surveys, if people realize that their answers could affect the poverty line and thus answer

differently than they otherwise would.2

Developing equivalence scales. Subjective measures, as examined in this study, can also

be used to produce equivalence scales. Equivalence scales are used to compare the

income and expenditures for households or families of different sizes and with different

needs. Equivalence scales represent the relative costs of living of households of different

sizes and compositions that are otherwise similar. Thus differences in need for adults and

children are accounted for as are economies of scale in the household. For example, if a

family of two adults and one child can live as well as a single adult while spending only

50 percent more, then relative to the reference household with one adult, the equivalence

scale value for a two-adult family with the one child is 1.5. The scale value is often

defined as family size to some power (Se); the power value is also referred to as the

elasticity of need. In the following table are some examples of ways to derive

equivalence scale values. The first was suggested by Buhmann et al.3

 , the second by the

OECD,4

 and the third by the National Academy of Sciences Poverty Panel:5

= (A+K)0.5 (1)

Scale Value = [1.0 + .0.7(A-1) + 0.5K]1.0 (2)

 

2

 Citro and Michael, 1995, page 50

3

 Brigitte Buhmann, Lee Rainwater, Guenther Schmaus, and Timothy Smeeding. “Equivalence Scales, Well-Being,

Inequality, and Poverty: Sensitivity Estimates Across Ten Countries Using the Luxembourg Income Study (LIS)

Database. The Review of Income and Wealth 34(2), 1988, pp. 115-142. 4

 Organization for Economic Cooperation and Development. The OECD List of Social Indicators. Paris:

Organization for Economic Cooperation and Development, 1982.

5

 Constance F. Citro and Robert T. Michael (eds.). Measuring Poverty: A New Approach. Washington, DC:

National Academy Press, 1995.
9

 = (A + 0.7K)0.65 to 0.75 (3)

Danziger et al (1984) use data from the 1979 Income Survey Development

Program (ISDP) Research Panel to evaluate the cost function of ‘making ends meet.’

They use a ‘direct approach’ to derive equivalence scales by using answers to the

Minimum Income Question: What is the smallest amount of money a family of four

needs to get along in your community? They compare their scales to those using an

‘indirect approach’ which derives equivalence scales from market behavior via survey

data. They postulate that the respondent’s answer to the MIQ (Y*) is systematically

influenced by their income (Y) and family composition fc:

Y* = f(Y, fc)

The ‘true’ income level associated with the welfare level ‘making ends meet:

Y-bar = f(Y-bar, fc)

Using a log-linear regression model, they estimate the minimum level of income ‘to make

ends meet:

ln Y* = 4.4764 + 0.3327 ln Y + 0.2078 ln fs - 0.2889 age-0.2503 female

Family size (fs) is used as a proxy for family composition. The coefficients from this

equation are used to construct a consistent level of well-being for families of different

sizes and composition. The equivalence scale is created by dividing the predicted income

for any family size by the predicted income for the reference family6

 

6

 Poverty thresholds are also derived from this method. For example, the estimated income level to ‘make ends meet’ in

1979 is $1261 for family of four with a male head of household under age 65, $699 for a single women, and $659 for

an elderly couple.
10

The Danziger et al results are presented below by family size and are compared to

the equivalence scales implicit in the official poverty line and to other equivalence scales

derived using subjective measures. Rainwater (1990) analyzed Gallup Poll data on the

same question as Danziger et al (1984)—what is the ‘smallest amount of money a family

of four needs each week to get along in this community?’ He also used a log-linear model

and regressed the log of minimum income on the log of family size and respondent’s age.

In the table below, the equivalence scales show the per person increment in

income needed to maintain a constant level of well-being as additional family members

are added. The scales derived using subjective measures show a higher increment in

income needed when adding a second person than the equivalence scales implicit in the

official poverty measures. For additions of three or more people, the per person

increment is about the same and more often lower.

Table 2. Equivalence Scale Increments: Alternative (relative to a scale value of 1.00

for a single adult family)

Base Scale

Fam

ily

Size

2 34 56

Official U.S. Poverty Census Bureau 0.29 0.26 0.40 0.35 0.27

Eurostat Eurostat 0.50 0.30 0.30 0.30 0.30

“OECD” OECD 0.70 0.50 0.50 0.50 0.50

ISDP 1979 MIQ Danziger et al (1984) 0.47 0.18 0.16 0.13 0.11

CEX 1982 - MIQ de Vos & Garner (1991) 0.37 0.23 0.24 0.12 NA

Gallup ‘get along’ 1980s Rainwater (1990) 0.26 0.18 0.15 0.12 0.11

CEX 1987-91-AID Johnson & Garner (1992) 0.39 0.29 0.24 0.21 0.18

CEX 1986-extended Engel Phipps & Garner (1994)

Source: Table 3-2 (Citro & Michael, 1995) & DeVos & Garner (1991)

Danziger et al (1984) sum up by stating the “direct measurement approach is most useful

for making relative welfare comparisons across households in various circumstances...the

new method has the conventional economic interpretation.” The equivalence scales
11

“resemble those obtained using the much more cumbersome (and more expensive

approach based on market behavior. . .The marginal pecuniary cost of including just a

few subjective income-evaluation questions in a typically elaborate household survey

questionnaire is virtually zero.” 7

It has been suggested that equivalence scales should be based more on plausibility

than empirical data.8 If true, then scales should be derived from answers to subjective

questions. VanPraag (1992?) sums it up nicely by stating that “With respect to

plausibility, it is obvious that the objective measures are based on expert opinions only,

while the subjective measures reflect the vox populi. Therefore we consider the

subjective measures as having a higher plausibility content.”

Using subjective measures to assess objective measures of well-being.

Citro and Michael (1995)9

 provide an example of how a time series of subjective poverty

thresholds provide information about the extent that other thresholds are in agreement

with public perception. Answers to Gallup Poll questions10 from 1947 to 1993,11

suggests that, on average, people perceived about the same poverty level for a four-person

family as the official threshold when it was first developed in the 1960s. In earlier

periods, however, the data suggest that people perceived the poverty level to be below the

 

7

 Page 540.

8

 Citro & Michael, 1995, page 174.

9

 Pages 34 & 138.

10 The Gallup Poll asked: “What is the smallest amount of money a family of four (husband, wife, and two children)

needs to each week to get along in this community?” In 1989, the Gallup Poll included a question about the poverty

line. Vaughn (1993) used the relationship of the average amounts for the poverty and the get-along questions in 1989

to construct a time series of subjective poverty thresholds from 1947 to 1989. A poverty line question in the 1992

Gallup Poll and the 1993 General Social Survey gave similar results.

11 Consistent questions and methodology were used between 1947 and 1989. Similar information was collected in the

1992 Gallup Poll and the 1993 General Social Survey.
12

official threshold. In later periods, the data suggest that people perceived the poverty

threshold to be higher than the official threshold. In comparing the answers to the

subjective Gallup questions and half the value of median after-tax income, the Gallup

answers were slightly higher from 1947 until 1963, and quite similar from 1963 to 1989.

The trend is quite similar for the entire period. It has been suggested that one of the

reasons the Orshansky-based thresholds were adopted is because of their relationship to

other measures at the time: the original 1963 threshold for a two-adult/two-child family

is very close to one-half the median after-tax four-person family income and to a

subjective four-person family threshold derived from Gallup Poll data.12 Answers to

subjective questions, such as the Gallup get-along questions, and the poverty levels,

respond to cyclical changes in the economy, declining during recessions and increasing

during expansions.

III. Previous Research and U.S. Government Involvement

During the twenty years in which subjective questions have been used to assess

the economic well-being of individuals and households, relatively little research has been

done on the reliability or validity of these particular questions. However there are four

notable exceptions.

First, the Delighted/Terrible (D/T) scale was developed and tested by Frank

Andrews and Stephen Withey13 in a series of quality of life surveys at the University of

Michigan Institute for Survey Research during the 1970s. Based upon their tests,

 

12 Citro, Connie, comments on paper presented during the 1996 Society of Government Economists Conference session

on Inequality and Poverty, November 22, 1996.

13 F. M. Andrews and S. B. Withey. Social Indicators of Well-Being. New York: Plenum Press, 1976, p. 343.
13

Andrews and Withey suggest that the D/T scale can account for 87 percent of the variance

in respondents’ feelings about their income.

Second, Kapteyn14 focused his research primarily on the use of the IEQ and the

D/T scale in measuring an individual’s welfare function of income (WFI), although

response to related subjective questions were also examined. With respect to the IEQ,

Kapteyn found that respondents preferred to answer in annual amounts. He suggested

that respondents who answer in monthly or weekly amounts “forget” certain of their

income like once-a-year-fringe benefits. He suggested a finer (more points) scale for the

D/T response categories since less ties should improve the WFI model fit.

Third, Antonides and colleagues15 conducted a study to assess the reliability and

validity of ten methods for measuring individual welfare functions of income using

income evaluation questions. Data were collected from an experimental survey of

approximately 400 households in the Netherlands in the Spring of 1979. Of the ten

methods tested, they recommended a combination of two methods, that is, they

recommend that individuals be asked numerical evaluations and corresponding income

levels. Even this method raised concerns however, because they thought that the

numerical evaluations might tempt respondents to provide income levels proportional to

the numerical values which would tend to produce linear individual welfare functions.

They recommended further testing of this method.

 

14 Arie Kapteyn, “Statistical Analysis of Response to Charlottesville Hothouse, Treatment A Attitudinal Module, June

6-7, 1979.” Unpublished manuscript.

15 G. Antonides, A. Kapteyn, and T. Wansbeek, Reliability and Validity Assessment of Ten Methods for the

Measurement of Individual Welfare Functions of Income, unpublished manuscript, Center for Economic Research,

Tilburg, the Netherlands, 1986.
14

The fourth study was conducted under the direction of Morissette and Poulin16 of

Statistics Canada. Respondents were surveyed in supplements to the 1983, 1986, 1987,

and 1988 Survey of Consumer Finances to evaluate alternative low income measurement

methods. Interviewees were asked both the minimum income and spend questions and an

income evaluation question. Split samples were used to test question wording. The

major findings concerning the minimum income types of questions include: (1) small

differences in response resulted with increases in family size; (2) when the same question

wording was used over time, no notable changes in the minimum income required

resulted; and (3) question wording on required income affected both the minimum

income and, to a lesser degree, the equivalence scale; the researchers found a 32 percent

difference in the levels produced from the income questions versus from the questions

(using the 1988 split panel data), with the spend question producing lower responses.

Although not specifically designed to examine the impact of changes in survey

methodology, other studies have been conducted using data from the MIQ and IEQ, and

related questions which provide us with insight concerning subjective assessments. For

example, Saunders and Matheson,17 using Australian data, note that “perceptions of an

adequate income for oneself are shaped by cultural identities and their concomitant

reference groups in ways which go far beyond the effects of immediate material and

personal circumstances.” Garner and de Vos,18 using data from the U.S. and the

 

16 R. Morissette and S. Poulin, Income Satisfaction Supplement: Summary of Four Survey Years, Statistics Canada,

1991.

17 Peter Saunders and George Matheson, Perceptions of Poverty, Income Adequacy and Living Standards in

Australia. Social Policy Research Centre Reports and Proceedings, No. 99, The University of New South Wales, April

1992.

18 Thesia I. Garner and Klaas de Vos, "Income Sufficiency vs. Poverty: Results from the United States and the

Netherlands," Journal of Population Economics, vol. 8, 1995 pp. 117-134. Also see: Klaas de Vos and Thesia I.
15

Netherlands, found that question wording, design of the survey, and data collection

instruments are likely to contribute to variations in response. Based on econometric

analysis, they found that differences for the two countries was likely related to what

respondents consider necessary, and to their general interpretation of the MIQ.

During this same time period, the U.S. government moved forward and included

subjective assessments of health status and health-related work limitations in federally

sponsored surveys. The move to include subjective measures of the impact of income and

financial resources in U.S. government sponsored surveys, however, was not as readily

endorsed.19 The first foray of subjective assessments of income into U.S. government

surveys came when both the MIQ and the D/T question were included in the Bureau of

the Census’ 1979 Research Panel of the Income Survey Development (ISDP) Program.20

The MIQ was also added to the 1982 Consumer Expenditure (CE) Interview Survey21

based upon the recommendation of the Expert Committee for the Bureau of Labor

Statistics’ (BLS) Family Budget Revisions.22 This committee suggested that in terms of

measuring family budgets, there is “a general consensus about how much it takes for an

ordinary family to ‘get along’—perhaps not an exact figure, but rather a range or ‘band’

of total expenditure levels that contains what most people would agree is the ‘get along’

 Garner, “"An Evaluation of Subjective Poverty Definitions: Comparing Results from the U.S. and the Netherlands,"

Review of Income and Wealth, Series 37, Number 3, September, 1991.

19 The result being that research into these questions has lagged behind that of the health-related questions. Yet

researchers here in the U.S. and in other countries (mostly in Europe, Canada, and Australia) continued to use the

subjective economic well-being questions to provide information on the situation of individuals and households. See

Appendix B for selected references from the literature.

20 For a review of the history of self-assessments in potential federal household surveys, see Denton R. Vaughan, “SelfAssessments of Income Needs and Financial Circumstances: Two Decades of Seeking a Place in Federal Households

Surveys,” paper presented at the Joint Statistical Meetings, Sections on Government Statistics, Social Statistics, and

Survey Research Methodology, Chicago, Illinois, August 5, 1996.

21 See deVos and Garner (1991) and Garner and deVos (1995) for analyses of these data.

22 Harold W. Watts. “Special Panel Suggests Changes in BLS Family Budget Program,” Monthly Labor Review,

December 1980, pp. 3-10.
16

amount” (p. 8). Consequently, the committee recommended a major effort to evaluate

and perfect a survey methodology that would permit a paradigm shift away from the

notion that official experts can (and should) define what the populace needs in order to

get along or prosper. The new measurement task would be directed instead toward

finding stable, reproducible estimates of what ordinary people experience with their

family budgets (p. 9). The committee, therefore, envisioned supplementing the absolute

standards embodied in detailed lists of commodities with more relative standards based

on the reported subjective experience of respondents.

Implementation of the committee recommendations was initiated by proposing the

only existing subjective assessments of income available: the MIQ, IEQ, and D/T

question. However, when the survey package of questions was sent to the Office of

Management and Budget (OMB) for approval, the MIQ question alone was accepted for

inclusion in the CE and only within the last interview. Eventually, even this one question

was dropped when the plan to revise the Family Budget program was discontinued.

Again in 1991, a formal proposal from an inter-agency team was presented to

OMB to include the existing subjective assessment questions in the Survey of Income and

Program Participation (SIPP), a Bureau of the Census survey. After consultation with

BLS, the Bureau of the Census, and the Office of Senator Daniel Moynihan, OMB

recommended “laboratory and field testing” prior to the inclusion of such questions in the

SIPP. In the OMB memorandum, cited was research from the BLS,23 one of the few

U.S. statistical agencies that had direct experience with a MIQ. In this memorandum, the

OMB reiterated its policy to support such survey development and testing in the statistical
17

agencies. The Committee on National Statistics,24 in their report Measuring Poverty, A

New Approach, also suggested that more work needs to be conducted on these measures

before they can be seriously considered for an official poverty measure. However, they

did state that, “If such survey responses were available over time on a consistent basis,...

they could be used to provide useful information with which to evaluate the official

methodology for updating the thresholds.”25

In the autumn of 1994, a preliminary proposal was prepared within the BLS to

explore the possibility of cognitive testing of subjective questions. By the spring of 1995,

specific plans were detailed for a first round of cognitive tests. In the Spring of 1996,

BLS management determined that this first stage of cognitive testing might proceed, with

the understanding, that upon completion, there would be a review of the results and

decisions made concerning further implementation. The format of the test and the

questions to be asked were developed and pre-tested in the BLS cognitive lab. This pretest helped to refine the protocols and improve the flow of the one-on-one interviews.

Staff from the BLS and the Michigan Survey Research Center, under contract with BLS,

collected the data from five areas in the U.S.

Also during the autumn of 1994, the Bureau of the Census reconvened the SIPP

Interagency Working Group to discuss possible topics for inclusion in the 9th wave

interview of the 1993 SIPP panel (scheduled for fielding in the period October 1995

 23 De Vos and Garner (1991) and Garner and de Vos (1995).

24 Constance F. Citro and Robert T. Michael (eds.). Measuring Poverty: A New Approach. Washington, D. C.:

National Academy Press, 1995.

25 Citro and Michael (1995), pp. 136-137.
18

through January 1996).26 OMB agreed that the SIPP would be used to collect data for the

proposed wave using the D/T question, the MIQ, and the MSQ. These data were

collected by the Bureau of the Census.

In an effort to coordinate activities and to meet the OMB request, the BLS and

Census Bureau jointly funded the cognitive testing, as noted earlier. All of the cognitive

work was conducted or coordinated by the BLS. Once the SIPP field data are available

(probably in late spring 1997), BLS will work with the Census Bureau in analyzing those

data. Then results from the two parts of the project will be combined to provide an

overall assessment of the subjective questions under examination.

IV. Study Design

Our study was designed to address four main research questions:

1. How do respondents interpret such terms as “minimum income,” “sufficient income,”

“necessary expenses, “ and “monthly household income?” Since these terms are not

defined for respondents, the range of possible interpretations needed to be determined.

2. Are there potential order effects when asking respondents to make subjective

assessments? For example, when respondents are asked to assess satisfaction with

their income, are their ratings affected by questions immediately preceding those

asked about their expenditures? The possibility exists that consideration of one’s

expenses may create a comparison that may temporarily raise or lower one’s

evaluation of income.

3. How do respondents use response categories such as “good/bad,”

“sufficient/insufficient,” and “delighted/terrible?” Are these the appropriate terms for

assessing people’s attitudes and emotions about their income? What metric should be

used for the scale range?

 

26 See Robert Kominski and Kathleen Short, “Developing Extended Measures of Well-Being: Minimum Income,”

paper presented at the Joint Statistical Meetings, ,” Sections on Government Statistics, Social Statistics, and Survey

Research Methodology, Chicago, Illinois, August 5, 1996.
19

4. How easy or difficult is it for respondents to make subjective assessments of their

income? How accurately do respondents review their economic situation when

making subjective assessments?

To address these four main research questions, a two-prong strategy of qualitative

testing consisting of cognitive interviews and focus groups was used. The successful

application of such qualitative methodologies requires that as many differing viewpoints

as possible be gathered, so that a full range of ideas and opinions may be observed. In

order to extend the breadth of views expressed, our study is designed to include three

types of family composition, three levels of income, and five sites spread across the

United States. The result is a three-by-three design matrix (based on household type and

income group) with a total of nine cells; not all sites are represented by each cell in the

matrix. At least five (5) interviews and one (1) focus group was conducted in each cell.

Based on our own evaluations and earlier work,27 we expected that spending

patterns impinge upon one’s attitude toward income, so we decided it was necessary to

include a range of household types reflecting diverse expenditure patterns. We screened

participants and categorized their household types as:

1. single adults with no children under 18 years-of-age in the home,

2. adults (either single or sharing expenses with another adult) with children under 18

years-of-age in the home,

3. adults sharing expenses, but with no children under 18 years-of-age in the home.

We defined the levels of income as either (1) low, (2) medium, or (3) high.

These are determined separately for each geographic area in which testing is conducted.

Using July to December 1994 Income Percentile Data from the Current Population

 

27 Garner and de Vos (1995).
20

Survey (CPS),28 staff from the BLS29 adjusted these estimates to the March 1996 level

using the CPI-U for all items.30 These estimates represent taxable household income in

the appropriate metropolitan statistical areas and do not include the cash value of food

stamps. The income ranges for each area are divided into thirds for each of the three

household types (see Appendix C for the exact income ranges used).

The five sites were selected to provided national coverage, as well as some

urban/rural differentiation. The sites were Miami, Los Angeles, Detroit, Baltimore, and

West Virginia. In this paper, we only present the results from ten interviews and one

focus group from one site: Miami, Florida. The Miami focus group and cognitive

interviews were conducted by a research psychologist from the Bureau of Labor Statistics

during the month of May 1996. Data for the other eight cells were collected during the

summer and autumn, but were not available for presentation at this time.

V. Methodology

Our choice of two qualitative methodologies, focus groups and cognitive

interviews, reflected our lack of even the most basic information concerning the ways that

respondents would react to these types of questions, scales, and concepts. We had no

prior hypotheses driving our data collection. Instead, we were attempting to (a) simply

observe the procedures and strategies respondents use when making subjective

assessments, (b) hear the language they use when discussing these topics, and, ultimately,

 

28 Special thanks are extended to Kathleen Short of the Census Bureau for producing these ranges. Earlier ranges were

produced by Bill Passero using CEX data but we decided later to use the CPS data for selecting our subsamples.

29 Thanks to Wolf Weber for this conversion.

30 No CPI-U data were available for the Detroit/Ann Arbor area for March 1996. The alternative for updating the

income ranges was thus to average the CPI-Us for February 1996 and April 1996 to make the adjustment.
21

( c) see these issues from the respondents’ perspective by encouraging and assisting them

in articulating their own unique perspectives on the world.

A. Focus Groups

In our focus groups, respondents were asked to discuss the concepts of “minimum

income” among themselves. Specifically, we asked them:

• What amount of income would they need to “make ends meet” and how did

they decide upon this amount?

• What expenses would be covered by “enough money to make ends meet?”

What would this amount of money purchase?

• What things in their lives determine the amount of income a family needs “to

make ends meet?”

In addition, participants were asked to evaluate different “levels” of income. For

instance, they were asked to describe the differences between “good” and “bad” amounts

of income. A similar discussion was raised for the distinctions between “sufficient” and

“insufficient” incomes. Participants were asked to consider what elements or events

might cause the dividing points between “good and bad” or between “sufficient and

insufficient” incomes to fluctuate. Finally, participants were presented with the

“delighted/terrible” scale and probed for their reaction to, as well as their understanding

and use of its response categories.

B. Cognitive Interviews

For this study, several tasks were drawn from the repertoire of cognitive

laboratory techniques. The interviews begin with a concurrent think-aloud task, through

which the participants were able to describe their thoughts while answering either the

MIQ and an alternative “minimum spend” question. Participants were first “taught” to
22

create an answer out loud by having the procedures explained and demonstrated. They

were then requested to talk through all the issues and problems they had as they arrived at

their answer to the question. As the answers emerge, the interviewer followed up with

probing questions to gain additional details and understanding. Participants were asked

to rate their confidence in the answer they provided and to paraphrase some of the

concepts in their own words. Such strategies are generally used to identify difficulties in

understanding question wording or concepts, recall strategies, and the participants’

reactions to the question.31

A second technique used was an “income sorting” task directed toward assessing

the ways that participants evaluate income. Two versions of the task were used for each

participant. In each case, cards were arranged in front of the participant that were labeled

either “Very good,” “Good,” “Bad,” “Very Bad” or “Sufficient,” “Insufficient.”

Participants were then handed a stack of cards with dollar amounts written on them,

ranging from $250 to $6,000 in $250 increments. Participants were instructed to think

about all the members of their household and their expenses and then to evaluate each

dollar amount as monthly take-home pay. The evaluations were made by placing each

dollar amount into an appropriate category. After all the cards had been sorted and the

difficult gray areas between categories recognized, the interviewers probed the decisions.

Participants were asked to discuss how they decided to categorize the dollar amounts,

what the income amounts categorized together had in common, and what their lives

would look like with the varying categories of income.

 

31 T. J. DeMaio and J. M. Rothgeb. “Cognitive Interviewing Techniques in the Lab and in the Field.” In Answering

Questions, N. Schwarz and S. Sudman (eds.). San Francisco: Jossey-Bass Publishers, 1996, pp. 177-195.
23

A third task was a series of short answer questions using the “delighted/terrible”

scale. “Satisfaction” assessments ran the risk of being affected by preceding questions

that make specific information salient, thereby creating temporary standards of

comparisons, affecting judgments, and causing later responses to be higher or lower by

comparison. So in order to test for this possibility, a series of questions was asked for

which participants were to use the “delighted/terrible” scale to identify how they felt

about expenses such as the cost of feeding their families, eating out in restaurants, buying

clothes, health care, transportation, school tuition, and housing. Half of the participants

were asked to assess their family incomes prior to expenditure assessments; half of the

participants were asked to evaluate their family incomes after the evaluation of expenses.

Participants were also asked to discuss the “delighted/terrible” scale and to describe the

meaning of the various categories.

In closing, the interview participants answered a series of short debriefing

questions. These questions probed their reactions to the interview itself, what they liked

best and least during the interview, what was easiest and most difficult, and their ideas for

other questions that we could ask to more fully understand their subjective experience of

the yin and yang of income and expenses.
24

APPENDIX I: LIST OF SELECTED LITERATURE BY

 SUBJECTIVE MEASURES OF WELL-BEING

Type of question Authors

Minimum income questions Colasanato, Kapteyn, van der Gaag (1984)

Danziger et al (1984)

DeVos & Garner (1991)

Goedhart (1977)

Kapteyn (1979)

Kapteyn, Koorman, Willmse (1988)

Van Praag (1992)

Vaughn (1993)

Income Evaluation Question Duncan (1975)

Kapteyn & Van Praag (1980)

Kapteyn (1979)

Kapteyn, Koorman, Willmse (1988)

Van Praag (1992)

Delighted-Terrible Dubnoff & Vaughn (1972)

Kapteyn (1979)

Kapteyn (1979)

Van Praag (1992)
25

APPENDIX II: SELECTED LITERATURE ON

 SUBJECTIVE MEASURES OF WELL-BEING

Citro, Constance and Robert T. Michael, Measuring Poverty, A New Approach, National

Academy Press, 1995, pp. 134-140.

Colasanto, D., Kapteyn, A. and van der Gaag, J., “Two Subjective Definitions of Poverty:

Results from the Wisconsin Basic Needs Study, Journal of Human Resources, vol. 19.

1984, pp. 127-137.

Danziger, S., van der Gaag, J., Taussig, M.K., and Smolensky, E., “The Direct

Measurement of Welfare Levels: How Much Does it Cost to Make Ends Meet:” Review

of Economics and Statistics, vol. 66, 1984, pp. 500-505.

Deleeck, H., “The Adequacy of the Social Security System in Belgium, 1976-1985,”

Journal of Social Policy, vol. 18, no. 1, 1989, pp. 91-117.

Douthitt, R.A., MacDonald, M., and Mullis, R., “The Relationship Between Measures of

Subjective and Economic Well-Being: A New Look,” Social Indicators Research, vol.

26, 1992, pp. 407-422

Dubnoff, S., Vaughn, D., & Lancaster, C., “Income Satisfaction Measures in

Equivalence Scale Applications,” ASA Proceedings, Business and Economics Section,

Washington DC, August 1981.

Duncan, O.T., “Does Money Buy Satisfaction?” Social Indicators Research 2, pages

267-274, December 1975.

Garner , T.I. and de Vos, K., "Income Sufficiency vs. Poverty: Results from the United

States and the Netherlands," Journal of Population Economics, vol. 8, 1995 pp. 117-

134.

Goedhart, T., Halberstadt, V., Kapteyn, A., and van Praag, B.M.S., “The Poverty Line:

Concept and Measurement,” Human Resources, vol. 12, no. 4, 1977, page 515.

Hagenaars, A.J.M. and de Vos, K., “The Definition and Measurement of Poverty,” The

Journal of Human Resources, vol. 23, 1988, pp. 211-221

Hagenaars, A.J.M., The Perception of Poverty, Amsterdam: North Holland Publishers,

1986.

Kapteyn, A., Kooreman, P., and Willemse, B., “Some Methodological Issues in the

Implementation of Subjective Poverty Definitions,” The Journal of Human Resources,

vol. 23(2), 1988, pp. 222-242.
26

Kapteyn, A., “Statistical Analysis of Response to Charlottesville Hothouse, Treatment of

a Attitudinal Module, June 6-7, 1979,” unpublished manuscript.

MacDonald, M., “Consumption Theories and Consumers’ Assessments of Subjective

Well-Being,” The Journal of Consumer Affairs, vol., 26 (2), 1992.

Saunders, P., and Matheson, G., Perceptions of Poverty, Income Adequacy and Living

Standards in Australia, Social Policy Research Centre, Reports and Proceedings, No. 99,

April, The University of New South Wales, Kensington, New South Wales, Australia,

1992

Stanovnik, T., “Perception of Poverty and Income Satisfaction,” Journal of Economic

Psychology, vol. 13, 1992, pp. 57-69.

Townsend, Peter and Dave Gordon,, “What Is Enough? The Defintion of a Poverty

Line,” in Peter Townsend (ed.), The International Analysis of Poverty, London:

Harvester Wheatsheaf, 1993.

Van den Bosch, K., Callan, T., Estivill, J., Hausman, P., Jeandidier, B., Muffels, R., and

Yfantopoulos, J., “A Comparison of Poverty in Seven European Countries and Regions

Using Subjective and Relative Measures,” Journal of Population Economics, Vol. 6,

1993, pp. 235-259.

van Praag, B. M.S., “Ordinal and Cardinal Utility: An Integration of the Two

Dimensions of the Welfare Concept,” Journal of Econometrics, vol. 50, 1991, pp. 69-

89.

Van Praag, B. M.S., and Van der Sar, N.L., “Empirical Uses of Subjective Measures of

Well-Being,” The Journal of Human Resources, vol. 23(2), 1988, pp. 193-210.

van Praag, B.M.S., and Flik, R.J., “Poverty Line Concepts: An Application to

Czechoslovakia,” unpublished manuscript, Amsterdam, The Netherlands, 1992;

deVos, K. and Garner, T.I., “"An Evaluation of Subjective Poverty Definitions:

Comparing Results from the U.S. and the Netherlands," Review of Income and Wealth,

Series 37, Number 3, September, 1991.

Waldegrave, Charles and Paul Frater, “New Zealand: A Search for a National Poverty

Line,” in Else Oyen, S. M. Miller, and Syed Abdus Samad (eds.), Poverty: A Global

Review--Handbook on International Poverty Research, Olso: Scandinavian University

Press, 1996, pp. 177-182.
27

Walker, R., “Consensual Approaches to the Definition of Poverty: Towards an

Alternative Methodology,” Journal of Social Policy, vol. 16, part 2, April 1987, pp.

213-226