Standing on the Outside, Looking In: Tapping Data

Users to Compare and Review Surveys

Leda Kydoniefs, Bureau of Labor Statistics

Linda L. Stinson, Bureau of Labor Statistics

U.S. Bureau of Labor Statistics

2 Massachusetts Ave., NE

Washington, DC 20212

telephone: 202-606-7523 FAX: 202-606-7426

Kydoniefs_L@BLS.gov

Paper prepared for the American Association for Public Opinion Research

Conference in St. Petersburg, Florida. Any opinions expressed in this paper

are those of the authors and do not constitute policy of the Bureau of Labor

Statistics.

May 15, 1999
2

Keywords: qualitative research, needs-assessment, survey-users, survey methodology, program

evaluation.

1.0 Introduction

It is not unusual for survey organizations to conduct periodic expert review of ongoing survey

programs; neither is it unusual for those involved in such a review to include behavioral scientists. In

fact, most behavioral scientists would agree that such an expert review would naturally include an

examination of the sample design, definitions of data elements, question wording, data collection

procedures, as well as publication criteria. The question arises, however, is there another way to

conduct such a review? What if the focus of the evaluation was redirected? That is to say, what if

attention was placed not solely on the substantive and methodological content of the surveys, but was

also turned toward the individuals, organizations, and establishments who use the resulting data? What

could be gained from such an endeavor? Would the information gained from data users substantially

add to the knowledge we already have from the “typical” internal review conducted by content

specialists and survey research experts?

A project recently emerged at the Bureau of Labor Statistics that provided us with an

opportunity to consider such an approach. Two surveys were undergoing programmatic review and

evaluation by the typical players: program office staff, office of field operations, office of publication,

and survey methodologists. The surveys bore certain similarities in that they were both establishmentbased, wage-related surveys. Our part of the project was to ask the more fundamental evaluation

question: “What do data users really want?” Protocols were developed for a series of discussion

groups and debriefing interviews of survey program office staff, regional staff and survey users.

Survey similarities permitted the use of the same protocols for users and staff of both surveys.

Questions were posed to identify the data users’ needs and understand their motivations for using one

survey or the other. In this way, it was hoped that fresh insights into the fundamental differences

between the two surveys would emerge by casting light into the applications of the data.

This paper presents the methodological and implementation decisions faced when designing

such an evaluation study. In particular, we examine the various methodological approaches one could

follow when pursuing answers to the multiple questions implied by the over-arching question, “What

do data users really want?” We also consider which methodological approach provides us with the

most extensive information about survey users given the limited time and resources available for this

project. Finally, we will also include our preliminary assessment of the effectiveness of the overall
3

approach -- that of contacting users of survey data-- and consider what new knowledge is added to that

already known by the two BLS survey programs.

2.0 Design and Methodology

The Office of Employment and Unemployment Statistics (OEUS) and the Office of Compensation and

Working Conditions (OCWC) along with the Office of Field Operations (OFO) and the Office of

Survey Methods Research (OSMR), at the request of the Commissioner of Labor Statistics, formed a

team in October 1999, to determine the most effective and efficient relationship between two

programs collecting wage and compensation data. The two programs were the Occupational

Employment Survey (OES) a part of OEUS, and the National Compensation Survey (NCS), a part of

OCWC. One of the specific tasks agreed upon by the team was to form a “Users’ Needs Assessment”

sub-team to assess the needs and desires of data users for occupational wage and compensation data.

Once reasonable user needs are identified that can be satisfied given existing resources, upper

management would decide what (if any) programmatic changes will need to be implemented in the two

programs. The ideal outcome would be for the two programs to collect data and produce final outputs

that are as seamless as possible from the perspective of data providers and data users.

2.1 Planning the Evaluation

The first task of the sub team was to design the scope of the evaluation -- identify what to evaluate,

how to go about it, whom to interview etc. The evaluation was to be built around access to two groups

of subjects. One group would include BLS staff who worked on the two surveys either in the national,

regional offices, or State offices and had access to users of BLS data. The other group would consist

of various direct users of the BLS survey data. A sampling mechanism of some sort needed to be

developed, as of course, did the nature and scope of the questions to be posed to these groups of

subjects, and the method by which the data would be collected. The work of the project was thus

further divided into four broad decision points:

Defining our Terms: Significant time was spent, as with all evaluation projects of this sort, in teasing

out exactly what was meant by the question “what do data users really want?” Were we to conduct

some sort of customer satisfaction survey to find out if users like what we provide? Or is the better

question that should be asked -- what would users want in such a survey independent of what we

provide? The first question would ask users to evaluate what we have, the second would ask users
4

their needs independent of the current surveys. We decided to develop a overall research protocol that

would hopefully yield a combination of the two types of answers.

Defining The Sample: Now that we had an idea of the goal of our endeavor, we needed to identify our

sample. The difficulty we faced was not in collecting information from our own regional staff and

State partners. Both these groups were easily defined and well known to us. We felt that meeting with

the 2 national program office staff, 4 regional and 4 state offices would suffice to give us an overview

of the kind of requests received from users. The difficulty lay in defining/identifying the sample of

users, that is what lists to use in developing the sample of users – lists of current users, past users (what

constitutes past in surveys that are in flux) and what about the ‘non users’, should they not be

contacted? We addressed this task by running a brainstorming session with the entire sub team. One

of the behavioral researchers acted as facilitator of the group and kept the members on task. The

brainstorming session was quite successful in identifying who, in theory, are the users of the two

surveys, what mechanism they typically use to ask their questions (phone, Internet, email, in person

etc.), to whom do we market the data we produce and, given the future direction of the two programs,

would it be more useful to contact current, past users, or potential users – individuals who don’t

necessarily currently use these BLS products. Once we were comfortable with having identified the

user lists, we had to decide how many users would be enough to contact given our time frame and

resources. Naturally, the method of data collection would, in large part, define the upper limit of our

users sample size (Kuzel, 1992).

How to Collect The Data: The decision of how to collect the data can best be made after you know the

general kinds of questions you want to ask, whether you will be asking people’s opinion or hoping that

a group discussion will reveal a richer data output (Merton, Fiske,and Kendall, 1990). We considered

all three possible methods : mail, telephone, and in-person discussions. Mail was immediately dropped

from consideration due to the additional time that would be required for mailing, follow-up, and final

receipt. Reaching BLS staff working at the national office was easy -- we met with them in two inperson groups, one group per survey program office. Regional staff and State staff were located in

different parts of the country so, for efficiency, we decided to conduct small, one – three person group

discussions via the telephone. The staff in each office were all located in a single room and the

researcher at the national office posed the questions via telephone which were audiotaped. We found

such a discussion fairly easy to conduct, though it requires additional vigilance and plenty of reminders

for staff to speak up, that nods of head cannot be heard over the telephone etc. Project staff considered

having direct, in-person access with data users to be of great importance, although methodologically,
5

there appeared to be no such need as the questions developed for the users were of a content/format

easily asked over the phone in one-to-one sessions. Nonetheless, all users were invited to attend

roundtable discussion groups at the BLS national office.

Choosing the Data Display and Designing the User Group Discussion Protocol As mentioned earlier,

the two programs (OES and NCS) under programmatic review are both establishment-based, wagerelated surveys. Users who call requesting data do not necessarily ask for a particular survey’s data –

they may simply call and request for example, the current entry-level wages for architects in a given

geographic location. As the goal of this project was to evaluate what users want, choosing the best

mechanism to combine and prepare the BLS data for the users’ review and evaluation during these

roundtable discussions became critical. Should all related wage data be collapsed into one seamless

format, or would it be best to present the two sets of program survey data separately? This task of

determining how best to “sort and organize the data for presentation is a common problem in

qualitative research (Wolcott, 1990) and one we struggled with repeatedly. The display of the data

would drive the specific questions to be asked of the users and the kind of dialog we could have during

the meetings. We had earlier made the decision we were interested in both what users want

independently from what BLS provides, yet we were also interested in obtaining an evaluation of the

usefulness of the data currently provided. Ultimately, the data display package developed and sent to

users for review prior to the roundtable discussions presented data separately by program. Not a

particularly surprising decision for a federal agency used to publishing data by survey program.

2.2 What We Did

Debriefing of BLS and State staff: We met with national BLS staff who answer user requests of both

surveys NCS and OES in two separate debriefing sessions, and spoke with regional staff for NCS and

State staff for OES in telephone debriefing sessions. State offices that collect OES data via a

cooperative agreement with BLS were approached, and staff in each office with experience in

answering telephone user calls/requests was solicited. We were especially interested to see if there

were differences in the nature of requests posed to the national BLS office versus requests that might

be posed to regional or our State partners. Each debriefing phone call was approximately two hours in

duration as were the in-person roundtable discussion groups. All were asked the same series of

questions presented in identical order. The topical questions posed to all staff and end users are

presented on the next page. Each question tended to have multiple follow-up probes which are not

listed in this paper due to space concerns.
6

Debriefing of NCS and OES Users: A total of 21 users attended one of four group roundtable

discussions. Some users were local working within the greater metropolitan DC area, others were

flown in from the northeast corridor. The users represented one of six specialties: compensation

specialists, economic and business researchers, unions, trade/professional associations, and educational

training specialists. They were solicited from the recent (within the past 12 months) logs of NCS and

OES users – i.e., individuals who called to request some particular OES or NCS data or product. Lists

of possible respondents were also obtained from the Labor Research Advisory Committee, the

Business Research Advisory Committee, American Compensation Conference attendance, and

presidents of Education Training profession associations.

The identified respondents were contacted via telephone and their participation solicited. Upon

consenting to participate in the study, respondents were sent the NCS/OES program data display

materials for review prior to the group discussion. As stated earlier, these materials were designed to

be the vehicle for discussion and participants were highly encouraged to spend some time reviewing

them. The materials represented certain carefully chosen tables of data from each of the NCS and OES

surveys along with questions for consideration. These questions are presented below along with the

questions posed to the BLS national, regional and State staff. Each user group discussion was

approximately 2 hours in duration and was parallel in format.

Though the questions for the two groups (left vs. right side of table) appear to be quite

different, they share certain essential characteristics: They both ask 1) where do users go to get the

wage data they need; 2) What specifically do they use of BLS wage-related data; and 3) Are there any

missing data gaps- wage data they would like to have and cannot get. The users are asked these

questions directly; the BLS and State staff are asked an indirect question: what do they think data

users want/would answer if asked these questions.
7

Topics for Discussion of National, Regional

BLS Staff and State Staff

Topics for Discussion of Users of BLS Data

Describe a typical call; what is a typical

scenario?

What types of (general) data do you use, where

do they come from, and how important are

they to you?

Why are users calling? Are there any data you would like to have and

cannot get?

Do you ever send them elsewhere, and if so,

where and for what?

Which BLS data do you use in your work, how

useful is it, and how much do you use it?

What do you personally give to the user? How much occupational detail do you need in

the wage data you use?

How well do you think customers needs are

being met?

How much industry detail do you need in the

wage data you use?

Do you have any ideas how the process of

meeting users requests might be improved?

How much geographic detail do you need in

the wage data you use?

Are there existing data elements that people

never seem to ask for?

What is your level of interest in benefits data?

If you received a request for a wage of a

particular occupation which survey (NCS,

OES) would you use and why?

What changes can BLS make to its wage

products to enhance their usefulness?

In conducting this evaluation project, we became interested in ascertaining which

methodological approach (programmatic review by survey program and BLS staff, or direct contact

with users) would provide us with the most extensive information about survey users given the limited

time and resources available for this project. Further, given the way the project evolved, we were also

able to observe whether there were differences in the quality and extent of data collected by discussion

method: via in-person groups or telephone debriefings. Naturally, in-person user discussion groups

have far greater resource costs associated with them – from the need to schedule specific session dates,

send letters of authorization, fed ex materials for the discussion groups, schedule conference rooms,

and all sorts of related travel and hotel costs. On the other hand, in-person discussion groups carry a

certain face validity that can go a long way towards the successful closure and ultimate acceptance of

an evaluation project. Furthermore, if one can afford to conduct both types of interviews – of BLS

staff as well as users, one could bolster the validity of the project by “data triangulation”...in our case,

the use of multiple sources of data collection (discussion groups and individual interviews) to see

whether they corroborate one another (Silverman, 1993)
8

III. Findings

By and large, all the national, regional, and state office staff presented a similar picture of OES

and NCS surveys and related user needs. That is quite reassuring when, in assessing user needs, you

are able to sample only a few regions and States and harbor concerns that you may not be seeing or

hearing the “whole picture”. There appeared to be no differences across regions, nor were there

observed differences in the quantity of ideas, examples, and data collected by an in-person method

(national office staff) versus the telephone debriefings of the regional and State staff. All groups lasted

approximately 2 hours, and staff took turns adding to each others ideas. It appeared that not being in

one room together had no adverse impact to the discussion.

The users as a group, presented a picture of their use of the BLS wage data and their wagerelated needs that was surprisingly consistent with that reported by BLS staff. Careful analysis of

points made by users revealed no significant differences between that known by BLS staff who have

direct and ongoing access to the users, and the users themselves. Naturally, users made their various

points with all the force of their conviction and thus their needs were very easy to “hear”. But the

points themselves were the same as the BLS staff who spoke in terms of “I think users want x, y...” “I

hear users asking about this point or that point...” “I believe that users would prefer if BLS did this

and such...”

If the central aim of this evaluation project was to obtain confirmation (data triangulation) of

the accuracy of BLS knowledge of the needs of the users, then the project was a rousing success. BLS

staff do in fact know a lot about the BLS users, and are able to report accurately on that knowledge. If,

however, the aim of the project was to evaluate the most efficient mechanism to conduct such a

programmatic review, then we learned something new. Perhaps others may say we simply confirmed

the obvious: When you have highly skilled and motivated in-house staff, trust their knowledge -- they

know what’s going on “out there”. The most significant gain we could observe from the expending

additional resources to contact users directly and bring them in for in-person roundtable discussions is

one of certainty. There can be no doubt about what users want if you ask them directly. Sometimes,

such a need for certainty is extremely important and if that is so, then the additional time needed to

obtain the such data can be worthwhile.

Finally, for those who absolutely have to know what it is the BLS staff and data users said of the

BLS wage data, we offer the following summary points:
9

• Users want more data, although their needs vary. Some cited a desire for more detailed

occupations, greater geographic specificity, and more timely data. Naturally, they use whatever is

available to them – i.e., there seems to be no place else they can go to fill data gaps though some

organizations conduct some small highly specialized surveys for their constituent groups.

• Users knowledgeable of the NCS and OES surveys found it confusing sometimes to know which

survey best provides the data elements they need and how the two surveys differ

• Users find the BLS data generally less accessible than they could be and requested a reorganization

the BLS Internet to simplify their data searches

• Users would like the BLS to be more proactive in informing them of changes (current and

expected) to their surveys

• Users would like more detailed explanations of the methodology, sampling, variances and

generally the technical aspects of the BLS wage surveys

IV. References

Kuzel, A. J. (1992) “Sampling in Qualitative Inquiry” pp. 31-44. In Doing Qualitative Research

edited by Benjamin Crabtree and William Miller. Newbury Park: Sage

Merton, R.,Fiske, M., and Kendall, P. (1990) The Focused Interview. A Manual of Problems and

Procedures, pp. 135-169. New York: Free Press

Silverman, D. (1993) Interpreting Qualitative Data. Methods for Analysing Talk, Text and Interaction,

pp 156-158 London: Sage Publications.

Wolcott, H. (1990) Writing up Qualitative Research. Newbury Park: Sage Publications.