Standing on the Outside, Looking In: Tapping Data Users to Compare and Review Surveys

Introduction It is not unusual for survey organizations to conduct periodic expert reviews of ongoing survey programs; neither is it unusual to reach beyond the program staff and include behavioral scientists in such reviews. An expert review of this type would generally include an examination of the sample design, definitions of data elements, question wording, data collection procedures, and the criteria for publication. The question arises, however, are there other ways to conduct such a review? What if the focus of the evaluation were redirected? That is to say, what if attention was placed not solely on the substantive and methodological content of the surveys, but was also focused upon the various individuals and organizations who use the resulting data? What could be learned from such an endeavor? Would the information gathered from data users substantially add to the knowledge already obtained from the "typical" internal review conducted by content specialists, survey research experts and behavioral scientists? A project recently emerged at the Bureau of Labor Statistics (BLS) that provided us with an opportunity to consider such an approach. In October of 1999, the BLS Offices of Employment and Unemployment Statistics (OEUS), Compensation and Working Conditions (OCWC), Field Operations (OFO), and Survey Methods Research (OSMR) formed a team to determine the most effective and efficient way to understand the needs of data users for occupational wage data. 1 The two BLS programs producing these data are the Occupational Employment Survey (OES), a part of OEUS, and the National Compensation Survey (NCS), a part of OCWC. These surveys have certain similarities in that they are both establishment-based, occupational wage surveys. The goal of the project was to address the fundamental question, "What do data users really want in terms of occupational wage data? " To answer this basic question, protocols were developed for a series of (1) debriefing interviews with survey program office, regional office, and state staff to identify the data requests they routinely receive and the types of assistance they are called upon to provide to data users and (2) roundtable discussions with survey data users themselves to identify specific ways they use the survey results and their motivations for choosing one survey or the other. In this way, it was hoped that fresh insights into the fundamental differences between the two surveys would emerge by casting light onto the applications of the data. This paper presents the methodological and implementation decisions faced when designing such an evaluation study. In particular, we examine the various methodological approaches one could follow when pursuing answers to the myriad of questions contained within the over-arching question, "What do data users want? " We also consider which of the two methodological approaches (i.e., talking with data users themselves versus talking with program staff who serve data users) provided us with the most extensive information about survey users needs given the limited time and resources typically available for such a project. Finally, we also include our preliminary assessment of the overall effectiveness of contacting users of survey data directly and consider what type of new knowledge this adds.