EARNINGS CONCEPTS AND DATA AVAILABILITY FOR THE

CURRENT EMPLOYMENT STATISTICS SURVEY: FINDINGS FROM COGNITIVE INTERVIEWS

Karen L. Goldenberg and Jay Stewart, Bureau of Labor Statistics

Karen L. Goldenberg, BLS, 2 Massachusetts Avenue N.E., Room 4985, Washington, DC 20212

The views expressed here are those of the authors and

do not represent policy or the position of the Bureau of

Labor Statistics.

Key Words: Establishment survey, Payroll,

Earnings Concepts, Employer information systems,

Cognitive interviews

1. Introduction

The Current Employment Statistics (CES) Survey

is conducted monthly by the U.S. Bureau of Labor

Statistics (BLS), and is the source of national statistics

on month-to-month change in employment, hours and

earnings. At present, hours and earnings estimates are

computed for a subset of employees, and earnings are

based on the fairly limited concept of "payroll." BLS

recently completed a pilot study to explore the collection of hours and earnings data for all employees (AE),

and the collection of AE earnings based on a more

comprehensive earnings concept. This paper reports

the results of exploratory interviews conducted prior to

the pilot study. In these interviews we examined issues

underlying the current and proposed concepts.

2. Background and Definitions

CES is a monthly survey of approximately 400,000

business establishments. It is an ongoing survey, where

respondents report 6 or 7 data elements each month (the

number depends on the industry). Participants have a

wide array of reporting options and can submit their

data by mail, telephone interview, touchtone data entry

(TDE), fax, or electronic data interchange.

The primary CES data element is the number of

employees who worked or received pay for the pay

period that includes the 12th of the month. In addition,

the survey obtains the number of hours paid and the

total payroll paid to production workers (PW) in

manufacturing and construction industries, and to

nonsupervisory workers (NSW) in all other industries),

again for the pay period that includes the 12th of the

month. In this context, payroll (or total payroll) is

defined as the total dollar amount paid to PW/NSW

employees, before deductions, including overtime, paid

leave, commissions if paid at least once a month, and

bonuses if they are paid each pay period. This definition

of payroll excludes most other payments.

The more comprehensive earnings concept comes

from the ES-202 program, and is called earnings or

total earnings. ES-202 refers to the administrative files

generated by State Unemployment Insurance (UI)

programs. These records consist of quarterly statements

submitted by all U.S. employers to their State UI

offices. The statements show employee earnings along

with appropriate payroll taxes. Earnings refers to the

total amount paid by employers to their employees,

including commissions, bonuses, sick pay, the cash

value of payments in kind, and in some cases fringe

benefits or sick pay distributed by a third party.1 If

CES were to obtain earnings instead of payroll, we

would collect earnings for the entire previous month

instead of the current month, in order to ensure that the

figure included all payments.

The difference between the payroll and earnings

concepts is nonwage payments (NWP). NWP include

bonuses, awards, commissions paid less often than once

a month, profit sharing payments, severance pay, the

value of payments in kind (such as the use of a

company car), and lump sum payments. Until such time

as a decision might be made to change the earnings

concept, it is important that CES respondents recognize

the types of payments constitute NWP and so are not

part of "payroll."

3. Method

The findings presented here come from a series of

13 personal-visit interviews with current CES respondents. Interviews took place in the Washington, D.C.

metropolitan area between March and July, 1997.

Respondents represented a broad range of industries

and employment size classes. Members of the project

team conducted the interviews, which were audio-taped

and subsequently transcribed for analysis. Interviews

averaged about an hour.

We conducted semi-structured interviews, working

from a protocol that addressed topics relevant to the

planned pilot study. The protocol incorporated elements

of a respondent debriefing and a response analysis

survey. More specifically, we began the discussion

with a series of hypothetical questions asking respondents if they could report key data elements. Then we

considered those data elements in more detail. We

asked respondents to tell us, in their own words, how

they understood the terms BLS uses to describe CES

concepts (e.g., payroll, lump sum payments, nonwage

payments). Next, we asked whether employers made

specific types of payments to their employees (e.g.,

commissions, profit sharing), and we related that

 

1

 There are small variations in the specific components

of total earnings across states.
information to the definitions. We discussed the extent

to which the information in employers' existing data

systems supported current CES reporting. Finally, we

talked about availability of data on hours paid.

Because the number of cases is small, any

conclusions to be drawn from them are suggestive

rather than definitive. At the same time, the data

provide insights into respondent perceptions of CES's

fundamental economic concepts and show how changes

to the data collection process would be affected.

4. The Respondent Perspective

While it is important to define and clarify concepts,

the real question is what respondents do with those

definitions. First, how do they understand the words,

both those currently in use in CES and those proposed?

Second, how do respondents use the concepts? Are

they now reporting correctly? Finally, do the data

systems on which reporting is based support either the

current or the proposed CES concepts?

Earlier research demonstrated the collectibility of

earnings and hours data for AE (BLS, 1981; Rosen and

Farrell, 1986), although this research was based on payroll and not earnings. BLS has not previously studied

respondents' understanding of earnings concepts. As

with most establishment surveys, there is a basic

assumption that respondents have a good understanding

of economic concepts and can take data directly from

business records (e.g., Griffiths and Linacre, 1995). But

a growing body of literature challenges these assumptions by documenting respondent misunderstanding of

economic and other concepts (e.g., Christenson and

Tortora, 1995; Goldenberg et al., 1993; Jenkins, 1992).

One factor in respondent misunderstanding, at least

in self-administered surveys, is that the mechanism for

conveying definitions to respondents is a detailed set of

written instructions. However, respondents typically do

not read instructions, especially if they think they know

what is being requested.2 As a result, where terms have

a technical meaning, or have been specifically defined

for a survey, respondents may misinterpret what they

are being asked to do, resulting in measurement error.

5. Results: Earnings Concepts

What words should we use to convey the particular

concept that we want to collect? BLS has traditionally

phrased questions with its standard terminology and

then defined those terms on the data collection form,

but experience shows that that this practice is not

always effective.

Concepts in the abstract. Given the possible expansion of CES from collecting payroll for PW/NSW to

 

2

 In the research reported below, we asked respondents how

often they looked at the directions on the CES form. Most

respondents said "once or twice a year," or "never."

collecting both payroll for PW/NSW and earnings for

AE, we wanted to understand how these terms would be

perceived. The project team speculated that total

payroll, total earnings, and total wages might all be

words which employers could use to describe the total

amounts they paid to employees. We also considered

the phrase total compensation, which has a very different meaning to economists. To assess understanding of

these terms in the abstract, we asked respondents to

explain the terms in their own words. Later in the

interview we specifically placed the concepts in the

context of the CES questionnaire. From the answers

given, we considered whether:

• Respondent understanding and use of these terms

mirrors BLS usage

• Respondents could differentiate between them

• One term conveys a sense of the concept better than

another.

"They're all the same". Five of the 13 respondents

felt that total payroll, total earnings, and total wages all

meant the same thing and were basically interchangeable. Three others viewed total earnings and total wages

as the same but thought that these terms were different

from total payroll. One respondent equated payroll and

wages. Most respondents made a distinction between

these terms and compensation, although their descriptions of compensation varied widely.

Total payroll. Respondents' definitions of total

payroll were consistently closer to the ES-202 earnings

concept than to the BLS payroll concept. A few

referred to total dollars, gross pay, or expense to the

employer, with or without reference to "before taxes."

Two respondents itemized the specific types of earnings

they would include.

Total earnings. As noted, five respondents said

that total earnings was the same thing as total payroll.

A few respondents defined total earnings as the amount

earned by employees, which is consistent with the BLS

definition, while others referred to employer cost, or an

annual rate of pay that could be for an individual or for

the whole company. These ideas are not consistent

with the BLS definitions and point to a problem with

the terminology.

Total wages. Several respondents saw total wages

as the same thing as total payroll, but about half

understood the term to mean workers' individual rates

of pay. The term had no meaning for two respondents.

Total compensation. Six respondents viewed total

compensation as economists generally use the term, to

include wages and salaries plus the value of all benefits.

Others saw no difference between compensation and

earnings, regardless of the definition of earnings.

Clearest term. We asked which of the four basic

terms, if any, we should use to obtain inclusive reports

of earnings (i.e., the ES-202 concept). Most respon-
dents felt that either "payroll" or "earnings" would give

us what we wanted. Four respondents recommended

total payroll, thereby reinforcing the notion that "total

payroll" is what they call earnings under the ES-202

concept ." Five felt that "total earnings" was clearest.3

In short, at least in the abstract, the understanding

of earnings concepts among this small group of respondents differs from BLS definitions. Payroll and

earnings are more or less interchangeable, meaning that

to respondents payroll encompasses the bonus,

commission, and other payments that BLS specifically

excludes. Neither total wages nor total compensation

appear to be viable alternative terms.

"Total Payroll" in the CES Context. The cognitive

task now presented to employers on the CES form is

complex. In the general model of cognitive response to

survey questions, respondents must: (1) comprehend

the question, (2) search their memories to retrieve an

answer—or, for record-based data, the availability and

location of the information, (3) make judgments about

how best to report that answer, and then (4) communicate that answer to the source of the question

(Tourangeau, 1984), which in this case is the CES form,

interviewer, or Touchtone Data Entry system. In the

specific context of CES payroll reporting, first respondents must understand the terms payroll, overtime, and

lump sum payments, and the concepts underlying them.

Second, they must identify the employees who are

production or nonsupervisory workers. Third, they

must understand the concept of the pay period including

the 12th and relate it to their own pay schedules for the

appropriate group of workers. Finally, they must

connect all these elements. Only then can they locate

the appropriate records and report for the correct pay

period.

During the first three interviews, we observed that

respondent descriptions of total payroll did not match

the BLS definition. In the remaining interviews, we examined this discrepancy by reading the definition from

the CES form and asking respondents to paraphrase it:

"Report the total production [or nonsupervisory]

employee payroll, including overtime and excluding lump sum payments for the pay period that

includes the 12th of the month." In your own

words, what is the Bureau asking for?

Two respondents seemed surprised by the phrase

"excluding lump sum payments," indicating that they

had not noticed the exclusion. Others seemed to grasp

 

3 One term that did appear frequently was the word "gross."

This term surfaced when we asked about total payroll rather

than total earnings, and seemed to convey the inclusiveness

sought by the ES-202 concept. Consequently, we used it in

the pilot study when we asked for Total Gross Earnings for

the previous month.

the idea that lump sums were not part of the definition,

although it's not clear whether they were aware of it

prior to hearing the question. Several respondents

interpreted the question in terms of the ES-202 concept,

as they did earlier when we asked for the definition in

the abstract. Among these respondents, those that made

NWP or paid lump sums were as likely as those who

did not to mention to lump sump payments in their

interpretation of the payroll question. Finally, two

respondents focused on the workers covered, rather

than on remuneration to employees.

When they paraphrased the payroll definition, most

respondents focused on one or two elements rather than

on the entire concept. This suggests that the question

contains more material than can be readily grasped at

one time, and that the question wording may be

contributing to measurement error.

Nonwage Payments. The back of the CES form

contains a list of NWP that respondents should exclude

from the reported PW/NSW payroll amount. Some of

the excluded payments are general and some are

industry specific. The list of exclusions contains

bonuses (except those paid each pay period), retroactive

pay, annual pay for unused leave, payments in kind, pay

advances, tips, commissions not paid monthly, travel

expenses, and "lump sum payments."

Previous research on NWP. BLS has conducted

research in the past on supplements to wage and salary

earnings. The FY1989 CES Earnings Supplement Pilot

Test assessed the collectibility of nonwage payment

data and the quality of those data. Based on the

Response Analysis Survey (RAS) following the pilot

test, most respondents felt that they understood the

concept of nonwage payments. However, a significant

number of NWP went unreported in the pilot study and

were only uncovered during the RAS. Reasons for

nonreporting included lacking records with which to

ascertain dollar amounts, forgetting about the payments,

and most relevant for this purpose, misunderstanding

the payments to be reported (BLS, 1990).

Cognitive assessment of NWP terminology. The

current research addresses many of the reporting errors

and omissions identified in the earlier study. At different points in the interview, we asked respondents for

their interpretation of the terms "lump sum payment,"

"nonwage payment," "bonus," and "cash award." We

also asked whether employers paid bonuses, commissions, advances, retroactive pay, or severance pay, in an

effort to see if an understanding of these key terms was

consistent with behavior in reporting about them.

Lump sum payments and nonwage payments. The

phrase "lump sum payments" appears on the CES form

as part of the PW/NSW payroll column heading, so we

asked about it immediately after obtaining the respondent's interpretation of the form's payroll item. As used

here, "lump sum payments" is intended to cover pay-
ments made to employees in lieu of a salary increase, or

to supplement a smaller salary increase. Regardless of

the intended meaning, our respondents viewed lump

sum payments as various types of nonwage payments.

Nine interviewees specifically mentioned bonuses, and

three used the word commissions.

Are NWP different from lump sum payments? We

asked respondents to describe NWP in their own words.

This question was separated from the one on lump sum

payments by several minutes of discussion. Three of

the respondents used essentially the same words to

describe both concepts, while others listed specific

types of payments. On the other hand, four respondents

spoke of employer contributions to benefits, which are

not NWP for CES or ES-202 purposes. Perhaps the

term "lump sum payments" more clearly conveys to

respondents the sense of the exclusions, even if it

originally had a different meaning.

Bonuses and cash awards. The CES form instructs

respondents to exclude payments for bonuses and cash

awards, except for bonuses that are paid each pay

period. Respondents do have a good sense of bonuses.

In answer to our query, ten respondents described

bonuses as forms of additional compensation or extra

payments, in most cases linked to either the employee's

or the firm's performance. Two people actually used

the words "lump sum amount." Respondents were less

clear about cash awards.

6. Results: Data Systems

A key question underlying this research is whether

existing data systems can supply the earnings and hours

data for AE. Therefore, we explored the structure of

the information systems to see how well the systems

meet the purposes and definitions of CES. At the start

of the interview, we asked if respondents could supply:

• AE payroll (CES definition) for the pay period

including the 12th

• AE earnings (ES-202 definition) for the previous

month

• Total NWP for the previous month

• AE hours paid for the pay period including the 12th

• AE hours paid for the previous month.

For the most part, the respondents answered each

of these questions affirmatively. However, the response

to these "up front" questions is only part of the story.

Looking at both the initial answers and the more

detailed information that came out in the interview, we

see contradictions and refinements that negate the

earlier answer. Respondents who said they can give us

AE payroll for the pay period including the 12th are

often providing BLS-defined earnings rather than

"payroll." In addition, several who said they could

compile NWP seem to be answering in terms of

bonuses, rather than all types of NWP. When we

probed into the specifics of the data systems, several

respondents volunteered that what we wanted was

feasible, but would require either an intensive manual

operation or additional programming.

All employee payroll, pay period including the

12th. If CES is to collect data for AE, respondents

must have the ability to report AE payroll for the pay

period that contains the 12th of the month. Eleven

respondents first said that they could report this figure.

During the interview, however, we asked if the payroll

records for a single pay period contained one total that

met the BLS payroll definition and that covered all

employees. Faced with this more detailed question,

only five people gave an unqualified "yes" answer.

Three said that their payroll records included bonuses

or other lump sum payments, although in two cases it

would only be an issue once a year. The remaining

respondents either didn't know or said no, their records

did not meet this definition.

But even the employers who said their payroll

records coincide with the BLS definition told us that

bonuses are now included when they report PW/NSW

payroll, or would be if bonuses were paid during the

pay period including the 12th. Two others who said

their records meet the BLS definitions also told us that

the PW/NSW payroll data they now report would

include bonuses, except that they didn't pay bonuses in

the last year, or the bonuses appeared in a different pay

period. Bonuses would almost certainly be included if

we asked for AE payroll. This finding supports the

notion described above of payroll as an inclusive

concept.

We asked respondents whether they would be able

to exclude bonuses from their reported AE payroll

numbers. The answer was yes, but that it would require

changes to existing procedures, and would be a manual

task for some employers (especially smaller ones).

Larger organizations with programming staffs seemed

more willing to adapt their reporting.

Commissions are another element of payroll

reporting that can be problematic for some industries.

It seems that commissions appear in the pay system at

the time of payment, so they would fall into in the

payroll figure reported to CES, rightly or wrongly, if

that pay period happened to be the one with the 12th of

the month.

Four of the respondents we interviewed paid

commissions to their employees. While all four

reported correctly for PW/NSW, this would not have

been the case if they had been providing data for all

employees. Quarterly commissions to sales workers

would appear in the AE payroll once each quarter.

The respondents' firms occasionally made other

payments to their employees that we would label as

nonwage payments. Four different respondents mentioned payments to departing employees for accrued
leave. Payments in kind (e.g., car allowance) also

would appear in the payroll.

All employee earnings (ES-202 definition) for the

previous month. As noted, most of the employers we

interviewed said they could give us earnings for the

previous month. In fact, it appears that most of them

already do report earnings, but for one pay period rather

than the whole month. If we asked for monthly

earnings, they would generally sum weekly, biweekly,

or semimonthly figures. In some cases the addition

would also be across separate payroll groups (e.g. a

weekly and a biweekly payroll).

Total nonwage payments. Most respondents do not

compile a monthly summary of NWP. While these

respondents initially told us they could provide total

NWP for the preceding month, half said no when asked

specifically about a dollar total amount for all NWP.

Five mentioned a separate record of bonus payments

once or twice a year, and one said a total was available

because it would always be zero. Most respondents said

the totals could be tracked manually.

It's clear from these and other comments that the

questions were answered only in terms of bonuses. As

noted above, however, most employers made other

types of payments. If a separate collection of NWP is

important, we will have to ask for it in a different way.

Even then we may not get it. One respondent noted that

there are no identifiers associated with severance pay,

payments for accrued leave, or similar NWP, so they

could not be separated from the rest of the payroll.

We asked respondents whether the easiest time

period for compiling total NWP would be a month, a

quarter, the calendar year, the fiscal year, or something

else. Preferences were about equally spread across

month, year, and "doesn't matter." Again, though, this

understanding of NWP reporting equates NWP with

bonuses. For most respondents it does not include other

forms of nonwage payments.

Hours. Two of the economic indicators produced

by CES are average weekly hours and average hourly

earnings. These indicators require the collection of

total hours paid. "Hours paid" is relatively straightforward for workers paid by the hour, and all employers

with hourly-paid workers had adequate records to

provide the data, including overtime hours and paid

leave. Hours paid is more problematic for salaried

employees, but can usually be based on some standard.

The concept has less meaning, and therefore is more

difficult, for employees whose earnings are based on

commissions, piece rates, or miles driven.

Early in the interview, we asked respondents if

they would be able to give us hours paid information

for all employees. All but two respondents said "yes,"

both for the pay period including the 12th and for the

previous month. As a rule, hours for salaried

employees would be based on the number of hours in

the work week or the pay period. Two respondents said

they would report actual hours.

Hours reports for employees whose earnings are

solely commission-based are necessarily estimates. We

encountered one such situation where the employees

are personal-service workers whose income is a

percentage of the fees paid for services. The respondent

estimated hours paid from each employee's first and last

appointments on the daily schedule, but noted that there

were often gaps between those times. Commission sales

present a similar quandary, especially for big-ticket

items, such as real estate or securities, but we did not

interview anyone in these industries.

7. Discussion

Results from the small number of exploratory

interviews can only be viewed as suggestive, but there

are some patterns that emerge. First, respondents do not

distinguish between "payroll" and "earnings" as defined

by BLS. Rather, it appears that "payroll" is what goes

to the employee, including a variety of nonwage

payments that are specifically excluded by the CES

payroll definition.

Second, respondents have a fairly limited view of

nonwage payments and lump sum payments. Most

associate both of these terms with bonuses, even when

they offer severance pay, profit sharing, use of a

company car, or similar forms of nonwage compensation. In addition, while most respondents indicated

that they could provide total NWP monthly, it became

clear later that they were talking about bonuses and not

the other items. Further investigation revealed that preparing monthly NWP totals would be done manually

and that the NWP total would include some, but not all,

NWP. A 1989 BLS study found NWP to be about 3

percent of the total reported annual payroll (BLS

1990), which probably means 3 percent of "earnings."

Third, employer data systems can be significant

limiting factors. "Total payroll" is what comes out of

the information system, "total payroll" conforms more

closely to "earnings" than to "payroll", and "total

payroll" appears to be reported now for PW/NSW. In

some cases, respondents noted that payments could not

be eliminated from payroll because they were not coded

separately (e.g., accumulated vacation paid to an

employee who left the firm). Since 8 of the 13 respondents used payroll service bureaus, most of the people

we spoke to have little control over the way the data are

handled. On the other hand, large organizations with

internal programming staffs demonstrated the capability

and willingness to report data that match BLS concepts,

and large organizations by definition cover a larger

percentage of employment than small ones—thereby

reducing the effect of this limitation.

Fourth, the interaction of employer data systems

and misreported NWP is probably not too significant
most of the time. The concern arises from unusual

behavior that would increase payroll in a nonrandom

way—e.g. annual bonuses that tend to be concentrated

at certain times of year (especially December and

January). Employers seem to be more aware of bonuses

than other NWP and could probably more easily

subtract them. Quarterly commissions could present

another potential bias, unless respondents can be trained

to omit them. Most other NWP, however, would be

distributed across the year or paid on a recurring basis,

and so would not have a noticeable biasing effect on

reported payroll. After all, CES measures month-tomonth change in these variables.

Fifth, it appears that we can collect data on hours

paid for all employees.

The original question underlying this research was

whether respondents could give us AE earnings, AE

hours, and NWP. We conclude from these interviews

that the answer is "Yes, but...." Yes, the data can be

supplied—but the respondents are generally not attentive to the details of the concepts. Yes, there may be

measurement error—but the effect of it will generally

be small. If CES continues to collect "payroll" rather

than earnings, it would be useful to educate respondents

and encourage them to report data that more closely

match our definitions. To the extent that they cannot,

we could develop procedures to identify reported data

that includes NWP, so we can adjust for it.

ACKNOWLEDGEMENT

The authors acknowledge the contributions of David

Jaeger, a former member of the research team who was

heavily involved in the project and who conducted

several of the interviews.

REFERENCES

Bureau of Labor Statistics (BLS). 1983. Employer

Records Analysis Survey (ERAS) of 1981. Internal

report.

Bureau of Labor Statistics (BLS). 1990. Report on the

FY1989 CES Earnings Supplement Pilot Test.

Division of Monthly Industry and Employment

Statistics, September.

Christenson, Anders and Robert D. Tortora. 1995.

"Issues in Surveying Businesses: An International

Survey." In Brenda G. Cox et al., Business Survey

Methods. New York: Wiley, pp. 237-256.

Goldenberg, Karen L., Shail Butani, and Polly A.

Phipps. 1993. "Response analysis Surveys for

Assessing Response Errors in Establishment

Surveys." ICES: Proceedings of the International

Conference on Establishment Surveys. American

Statistical Association.

Griffiths, Greg and Susan Linacre. 1995. "Quality

Assurance for Business Surveys." In Brenda G. Cox

et al., Business Survey Methods. New York: Wiley,

pp. 673-690.

Jenkins, Cleo R. 1992. "Questionnaire Research in the

Schools and Staffing Survey: A cognitive

approach." Paper presented at the Joint Statistical

Meetings, American Statistical Association, August.

Rosen, Richard J. and John B. Farrell. 1986.

"Collection of New Data Types in the Current

Employment Statistics Program. Proceedings of the

Business and Economic Statistics Section. American

Statistical Association, pp. 508-513.

Tourangeau, Roger. 1984. "Cognitive Sciences and

Survey Methods." In T. Jabine et al., eds., Cognitive

Aspects of Survey Methodology: Building a Bridge

Between Disciplines. Washington, DC: National

Academy of Sciences.