From Start to Pilot: A Multi-method Approach to the 

Comprehensive Redesign of an Economic Survey 

Questionnaire 

Alfred D. Tuttle, Rebecca L. Morrison, and Diane K. Willimack 

U.S. Census Bureau1

alfred.d.tuttle@census.gov 

Introduction 

In 2004, survey methodologists at the U.S. Census Bureau began a project with the U.S. Bureau of 

Economic Analysis (BEA) to aid in the evaluation and redesign of two of their self-administered survey 

questionnaires. These surveys, one annual and the other quarterly, collect foreign direct investment (FDI) 

data from U.S. companies that are foreign-owned. Although BEA goes to great lengths to assure that its 

surveys are not unduly burdensome, the forms may impose significant burden for some respondents, and 

data verification may be labor-intensive for BEA analysts. Some respondents expressed difficulties with 

comprehending the FDI concept and other particulars of the surveys, which may differ subtly but 

significantly from the ways they think about their companies. Our redesign strategies were aimed at 

facilitating respondents’ comprehension of the FDI reporting requirements, as well as reducing the overall 

burden of completing the forms. 

To maximize the effectiveness of our redesign, we undertook a comprehensive process using multiple 

research methods to gain a thorough understanding of the conceptual underpinnings of the surveys and the 

problems respondents encountered when completing them. The project was concluded when the 

redesigned questionnaire for the quarterly BE-605 was fielded in March 2007. This paper describes the 

various methods used throughout the project, and summarizes key findings from various qualitative and 

quantitative analyses. The focus will be on the BE-605 but will include relevant findings from research on 

the annual form (BE-15), because the two forms share the same conceptual bases and because the redesign 

strategies were developed to address similar reporting issues for each. 

Background – Form BE-605 

The BE-605 is a quarterly survey that collects data on cross-border transactions and equity positions 

between U.S. affiliates of foreign companies and their foreign parent companies and certain other affiliated 

foreign entities. The data are used in compiling the U.S. international transactions accounts, national 

income and product accounts, and the international investment position of the United States. The collection 

instrument is a self-administered form that can be mailed back to BEA or completed using BEA’s 

automated filing system. The target population consists of all U.S. business enterprises in which a foreign 

person or business entity owns or controls 10 percent or more of a U.S. business. At present, the BE-605 

universe is comprised of about 14,500 companies and other business entities. BEA mails the BE-605 

survey questionnaire to approximately 4,000 respondents each quarter. 

Aside from the basic ownership requirement (ownership of 10 percent or more of the voting securities), 

data collection on the BE-605 survey is based on a mixture of U.S. generally accepted accounting 

principles (U.S. GAAP) and economic accounting concepts. Although detailed instructions on reporting 

requirements are included with the questionnaire, respondents can often become confused. One oftenencountered point of confusion is the rules for consolidating entities on a single BE-605 report. U.S. 

generally accepted accounting principles (GAAP) normally require a worldwide consolidation, which 

would include all of a U.S. business entity’s majority-owned domestic and foreign operations. However, 

 

1

 This report is released to inform interested parties of research and to encourage discussion. Any views 

expressed on statistical or methodological issues are those of the authors and not necessarily those of the 

U.S. Census Bureau. 
the BE-605 requires that the foreign parts of the company must be excluded. This and other kinds of 

exceptions to U.S. GAAP sometimes lead to confusion and inaccurate reporting for some respondents. The 

difficulty in conveying these particular FDI rules to respondents in clear yet accurate terms was one of the 

main reasons why BEA decided to engage survey methodologists at the Census Bureau to undertake this 

research. It was hoped that by developing a more “user-friendly” questionnaire, with critical instructions 

presented more clearly to respondents, it could reduce the burden associated with completing the form and 

at the same time collect more accurate and consistent data. 

Methodology – stages of research, and findings from each stage

Our research took place in three major phases, each with its own specific goals. In each phase we 

employed the cognitive interview technique that is typical of survey questionnaire evaluation. We also 

used other techniques in order to “triangulate” on reporting issues and to obtain a broader perspective on 

the response processes involved in completing the BE-605. As the project progressed, the methods 

employed in each phase provided complementary findings that informed decisions and applications for 

succeeding phases, and the methods used in the later stages allowed us to evaluate and modify our design 

decisions. In this section we describe the methods employed at each stage of research. Note that nearly all 

methods described in this paper are qualitative in nature, and that samples drawn for the purpose of 

employing the various methods are not intended to be representative of the BE-605 survey sample. 

Statistical analyses are not appropriate for the samples and findings described herein. 

Phase 1 – Background investigations 

Our goals for the first phase of the project were to achieve a thorough understanding of the survey forms 

both in terms of their content and the types of data they are intended to collect, as well as of the sources of 

error arising from the form and the response process. To these ends we employed three different 

techniques with three specific research targets, which provided a broad foundation from which to proceed 

with subsequent stages. 

The first technique involved interviews with BEA survey analysts—the personnel who review survey data 

and prepare it for publication. We conducted three focus group interviews and one individual interview 

with analysts, segregating them according to which of the two surveys they worked on and excluding 

supervisors to avoid any bias that might have resulted from their presence (one supervisor was interviewed 

separately). Our goals for these interviews were two-fold: First, we gained a deeper understanding of the 

kinds of information the surveys attempt to collect and the reasons for the collections. Second, we learned 

about the kinds of reporting errors respondents tended to make, which errors were most significant in terms 

of data quality and analysts’ workloads, and what the analysts thought were the reasons for errors. 

During the course of each focus group interview, we asked participants to rank-order the survey items in 

terms of the difficulties associated with them, and this led to in-depth discussions of the nature of the items 

and what the analysts thought were the reasons respondents tended to have problems with them. The 

analysts’ rankings of problematic items allowed us in later steps in the research process to focus our 

attention on a manageable subset of important problems, rather than the entire form. Analysts gave us their 

impressions of the problems inherent in the design of the questionnaire, most importantly the use of 

language and terms unfamiliar to respondents and the complicated instructions accompanying the form. 

The analysts also alerted us to the fact that they frequently communicate with respondents about their 

reports, to ascertain the quality of submitted data and, more significantly, to explain to them what exactly is 

being requested by specific items, that is, how to report correctly. 

This last finding led us directly to our second method, observation of analysts’ telephone calls to 

respondents. We observed, with respondents’ permission, conference calls placed by analysts to 

respondents to clarify suspect or missing data. This approach allowed us to learn several things about the 

individual problems: First we heard each analyst’s explanation of the problems from her/his perspective 

(prior to the calls), and then, more significantly, we got to hear an explanation of each problem from the 

perspective of the respondents themselves. This technique exposed us to the types of language and terms 

used in the conversations by which respondents were able to understand the requirements of the survey, and 

actual points of confusion related to the survey. Additionally, we listened to analysts’ explanations of the 
problems to respondents and of ways to avoid them in future survey reports. We found that the analysts 

often compensated for the inability of the form to clarify adequately the reporting instructions by making 

these explanations themselves. We would later expand on these findings as we attempted to bridge the 

divide between FDI concepts and GAAP records structures to facilitate respondents’ correct understanding 

of survey items; these explanations informed the creation of concise instructions crucial to correct 

reporting, which we tested in the second phase of the project. We were limited in the information we could 

obtain during these conference calls, since we were observers and not participants. However, we were able 

to continue to explore these emergent themes in greater depth with our next technique. 

Third, we went out into the field and debriefed respondents who had recently completed the survey. 

Respondent debriefings are semi-structured interviews in which respondents are asked non-directive 

questions about the data items they reported, how they retrieved those data, and how they interpreted the 

items on the questionnaire. Unlike the earlier methods, these interviews allowed us to meet face-to-face 

with respondents in their office setting and explore their reporting processes for the BE-605 survey from 

their perspectives and in considerable depth. For the debriefings we created an interview protocol based on 

the insights gained from our interviews with analysts and our observations of analysts’ calls to respondents, 

as well as our experience with questionnaire design and knowledge of best practices in visual formatting 

principles. 

We conducted interviews with respondents at 28 companies. The debriefing sample was purposively 

selected by BEA staff, with the following considerations: geographic proximity, to maximize the number of 

interviews in a given location; economic sectors of interest, particularly those with more complicated 

reporting requirements, though a range of sectors were recruited for the sake of wide coverage; and 

company size, with a preference for larger companies having greater impact on aggregate data. BEA 

personnel had an advantage in setting up interviews because of their existing rapport with respondents. 

In the debriefings we were able to obtain a deeper understanding of the response process, especially 

respondents’ interpretations of survey items and the nature and availability of information in their records. 

We were also able to cover a broader range of items in the forms than had been possible in the earlier 

phases of research. The effectiveness of the respondent debriefing technique in general is limited to the 

extent that time passes between the survey response and the interview, making it more difficult for some 

respondents to recall interpretations of items and data retrieval processes. However, this is mitigated in 

establishment surveys by the retention by virtually all respondents of their survey reports and the 

supporting documents they generate in the course of responding to surveys, which allow the response 

process to be reconstructed, in many cases allowing considerable exploration of their responses. 

We found in respondent debriefings that the questionnaires were problematic for two main reasons. First, 

the way the forms were formatted made them difficult to complete. Both forms were printed on legal-sized 

paper, which made them difficult for many respondents to manipulate, photocopy, fax, and store. Both the 

quarterly and annual survey forms used small fonts, dense blocks of text, and dark lines separating items 

and sections which gave them a very crowded look, rendering them difficult to read and making important 

instructions easy to miss (see Figure 1). The forms also used complicated matrices in some sections, which 

were formatted such that respondents tended to overlook some important instructions. Perhaps most 

importantly, both forms had separate instructions, one located in the back of the form booklet and the other 

accompanying the form as a separate booklet. Both sets of instructions were quite lengthy and used the 

same small font and dense blocks of text as in the forms themselves, and tended to be ignored by 

respondents. The dense formatting, complex matrices, and separate instruction sections effectively 

obscured the important concepts that needed to be understood by respondents in order to report correctly. 

The second reason respondents historically have tended to find the forms difficult and to report incorrectly 

has to do with the consolidation rules used in reporting, which follow a combination of specific economic 

accounting rules and U.S. GAAP. The most significant difference between reporting rules and U.S. GAAP 

we found has to do with the way corporations tend to consolidate their subsidiaries and other holdings for 

financial accounting purposes compared with the economic accounting rules for reporting to BEA, as noted 

above. Also, FDI and economic accounting concepts require that relationships and transactions between 

the U.S. corporation and certain kinds of affiliated foreign companies be treated and reported in specific 
ways that can be, at times, confusing to some respondents. Following GAAP rather than FDI accounting 

rules has often resulted in the misreporting of significant amounts of revenues, expenses, liabilities, and 

assets. We found that though the BEA consolidation requirements are somewhat unusual from the 

perspective of corporate accountants, they are similar to those used by the Census Bureau and other 

statistical agencies and are not incomprehensible or impossible to fulfill, once they are adequately 

communicated. Insofar as critical FDI reporting requirements being located in separate instructions that 

tended to be overlooked by respondents, they were not effective in helping respondents to report correctly. 

Phase 2 – Redesign, Test, Revise, Test... Form Redesign and Cognitive Testing 

Our findings from the initial stage of background investigations informed the development of several 

specific and general recommendations for the redesign of the two forms. Our major recommendations 

included guidelines for more “respondent-friendly” formatting. Fundamentally, this involved using larger 

fonts and reducing the number of items per page to alleviate the crowded look of the forms and increase 

their readability. It also meant changing the format from legal-sized to letter-sized pages. We also 

recommended abandoning the separate instruction sections and instead placing instructions adjacent to and 

within relevant questions for easy reference (see Figure 2). Although these recommendations meant that 

the number of pages would increase, we thought this would be an acceptable trade-off for improving 

respondent understanding. 

Additionally, we recommended that the most critical instructions regarding FDI consolidation and foreign 

affiliated entities be converted into questions, which respondents are not likely to ignore. For example, 

from the consolidation instructions we isolated key criteria and made from them a series of yes/no 

questions, to create a step-by-step process by which respondents might arrive at the correct reporting unit 

for the form (see Figure 3). 

Our other major design innovation was the addition of small diagrams to enhance the comprehensibility of 

questions about affiliated entities (see Figure 3). We based these diagrams on the organizational charts that 

we found during our debriefing interviews were commonly used by respondents in preparing the BE-605 

and other survey reports. Many of them referred to such “org” charts in our discussions of the survey 

reporting unit during the respondent debriefings, so we thought they might be a useful device for conveying 

those requirements. 

We created drafts of redesigned sections of each form (not entire redesigned forms) to evaluate them. We 

selected the more problematic sections from each of the original forms, drawing upon the findings from 

both the respondent debriefings and analyst focus groups, and created redesigned mock-ups using a wordprocessing application. 

We tested the mock-ups in cognitive interviews with actual respondents from the two surveys. Cognitive 

interviews (so named because they are based on Tourangeau’s (1984) cognitive model of survey response) 

involve observing and probing respondents’ reactions to survey items, with the aim of uncovering problems 

of comprehension and, in the context of establishment surveys, learning whether respondents have access 

to requested data and whether the data are encoded in company records. Cognitive pre-testing of 

establishment questionnaires generally takes place at business locations, in respondents’ offices or in 

conference rooms, rather than in a laboratory setting typically used for pre-testing surveys of households 

and individuals. This is to accommodate the limitations of business respondents whose time is, after all, 

obligated to their employers and not to government survey researchers. These natural settings are much less 

controllable than cognitive testing laboratories, and researchers must often be somewhat flexible in 

covering all the topics in interview protocols. It is also difficult to get a respondent to retrieve and report 

actual data during a cognitive interview, which can require more time than that allotted for the interview 

and significant effort on the respondent’s part. Thus, cognitive interviews do not always fully replicate 

authentic survey responses and the findings from them are therefore somewhat artificial. However, the 

technique is effective at uncovering problems with comprehension of survey items, retrieval of data, and 

judgment in reporting answers. For a discussion of the literature on cognitive interviews and other pretesting techniques for establishment surveys see Willimack et al. (2004). 
The cognitive testing stage was an iterative process in which we used the findings from one round to revise 

the test drafts, and then tested the modifications in the next round. We conducted a total of five rounds of 

cognitive interviews with respondents at approximately 60 companies nationwide. As in the first phase, 

BEA personnel selected companies for interviews based on the same considerations described earlier, and 

they also recruited some companies with more complicated corporate structures to better test redesigned 

questions about reporting units and ownership structure. 

Overall, we found that the redesigned sections worked well. In the new format, the original content of the 

two surveys was found to be generally clear to respondents. Virtually all the respondents we interviewed 

preferred the more open design, although a few said they preferred the more compact original form 

(respondent preferences were investigated further in the next stage of the project, and are described later in 

the paper). Most respondents also preferred having instructions adjacent to relevant items. Several 

observed that this particular design innovation would lead to the form being longer (the original BE-605 

was four legal-sized pages long, not counting the separate instructions, and the redesigned form is sixteen 

letter-sized pages), but most thought this would be an acceptable trade-off given the usefulness and 

convenience of not having to look elsewhere for more information on particular items. 

Findings were mixed for another design innovation, the “organizational chart” diagrams used to modify 

instructions and questions (see Figure 3 for an example). Our intention was to use the diagrams to describe 

essential features of the various inter-company relationships. We initially hypothesized that respondents’ 

familiarity with such diagrams would make them useful in translating the specifics of the surveys into 

terms they would understand. However, even the simpler of the diagrams we created tended to be complex, 

from a cognitive perspective, and we revised them continuously across all testing iterations to compensate 

for unanticipated problems. For example, terminology found in diagram labels must match that found in a 

corresponding question exactly, or else respondents may be distracted or confused (e.g., “U.S. affiliate” in 

one and “this U.S. affiliate” in the other). Even when terminology was consistent, some respondents 

tended to generate novel (and unintended) interpretations based on their selective focus on details that 

seemed to fit their own corporate structure, which often resulted in incorrect responses. It should be noted, 

though, that the diagrams worked well for simpler questions describing less-complicated ownership 

structures. See Tuttle and Morrison (2006) for a detailed description of this portion of the project. 

Some items in the new sections created from key instructions proved difficult for respondents to 

comprehend – specifically, questions intended to convey FDI requirements for identifying and 

consolidating affiliated entities. Certain complicated and less common ownership conditions were difficult 

to reduce to key criteria, and became overly complicated when incorporated into questions. Although it 

was important to be able to clearly and simply describe these specific ownership conditions for the 

purposes of FDI reporting, in reality they apply only to a small minority of companies and were therefore 

confusing to most respondents. Also, while we found that most of the diagrams accompanying items in the 

new sections were fairly clear and were correctly interpreted by most respondents, the diagrams were 

unable to clarify the difficult questions, and only added to the confusion. These questions and their 

accompanying diagrams were extensively revised and refined. They were ultimately retained in the 

redesigned quarterly BE-605 form, but in one section of the annual form test draft, the series of questions 

and diagrams was abandoned for an alternative strategy using a single question and more elaborate 

instructions. 

Overall, the cognitive testing stage was considered a success. This stage informed the development of a 

complete, redesigned BE-605 form2

. The survey methodologists at the Census Bureau collaborated closely 

with BEA survey program personnel and with questionnaire design staff to resolve issues associated with 

portions of the form not included in the pre-testing, and we proceeded to the next phase, evaluating a 

redesigned BE-605 form in a pilot test. 

 

2 The redesigned BE-605 form that is now in production can be downloaded from BEA’s website: 

http://bea.gov/surveys/pdf/be605web.pdf. Note that on this version, background shading was removed in 

order to facilitate cleaner faxed copies, with the result that the white response fields do not stand out against 

the white page. With the exception of the removal of the background shading and some minor changes 

made following the pilot test, this is the version we evaluated in the final stage of the project.
Phase 3 – Pilot Test of BE-605 

The final phase of the research project was centered around a pilot test of the redesigned BE-605 

questionnaire. For the pilot test, BEA selected a sub-sample of companies to receive the new form as part 

of the regular data collection process, while the rest of the survey sample received the legacy form. The 

pilot test gave us advantages over the cognitive testing which, although extensive, was nevertheless limited 

by the somewhat artificial nature of the interview setting in that the intervention of an interviewer is not 

available to respondents completing a self-administered form. Moreover, the full final test version of the 

form was only evaluated with the members of the final testing round. The goals of pilot test, then, were to 

evaluate the redesigned form under “real world” conditions, that is, actually completed by respondents in 

the absence of an interviewer, and with a larger sample than could reasonably be obtained by conducting 

cognitive interviews alone. The role of the pilot test in our overall research plan was to assess the 

redesigned form in a larger, one-time “snapshot”; lacking the iterative aspect of cognitive testing, it would 

not be useful as a stand-alone method in developing a questionnaire. 

We performed two types of evaluation3

 of the piloted form: The first evaluation was in the form of 

respondent debriefings, and the other involved four survey questions appended to the form asking 

respondents about their preferences for features of the new and legacy forms. The pilot form was mailed to 

a sub-sample of 657 current respondents to the legacy BE-605 survey, in June 2006. After eliminating 

those reporting electronically, the pilot sample was selected using a random process featuring unequal 

probabilities of selection devised by BEA survey personnel. This effectively created a moderate preference 

for selection of firms from large countries and industries in terms of magnitude of reported figures, and that 

were newer (see the appendix for a description of the sample design for the pilot test).

A letter included with the mail-out advised respondents that completing the pilot version of the form was 

voluntary, but if they decided not to do so they would still be required by law to respond to the survey 

using the old form, which they would have to request from BEA or download from BEA’s website. 

Approximately 350 completed pilot forms were returned, a response rate of 53.8%. 

Shortly after completed forms began returning to BEA, we conducted 23 respondent debriefing interviews 

(13 in person and 10 via telephone; again, the sample was selected purposively by BEA). The information 

from respondents at this stage largely echoed the findings from earlier testing. Most respondents had 

favorable impressions of the new form and the new design features. Most preferred the reformatted 

instructions, found the diagrams helpful, and thought the letter-sized pages were easier to handle. Some 

respondents cited drawbacks to the new form; several noted that the redesigned form was longer, and a few 

said that the stapled booklet format made the new form harder to fax. While these qualitative evaluations 

are not statistically verifiable, on the whole they suggest that the redesign was favorably received and that it 

may reduce the burden of completing the form. 

Most respondents thought that the pilot form took longer to complete than the regular form, although we 

have reason to believe that this is because of their inexperience with the new format. Virtually all 

respondents reported that they completed the pilot form by looking at it side-by-side with their previous 

quarter BE-605 report and mapping the latter’s items to the new locations and numbering on the redesigned 

form. Once they make this adjustment and completing the new form becomes routine then it should not 

take longer to complete, since none of the substantive questions were changed and the new questions are 

few and relatively simple to answer. 

Some of the perceived increase in time of completion may have reflected time spent reading the 

instructions, which now were placed beside and within questions rather than being grouped together in a 

separate booklet, which could be easily overlooked. Those who said they read the instructions tended to do 

so because they wanted either to confirm that the content of the form had not changed or to ensure they 

knew how to complete specific items correctly. In so doing, a few respondents (even those with 

considerable prior experience in completing the BE-605) reported finding instructions they had not seen 

 

3 We attempted a third type of evaluation comparing error checks generated by BEA’s automated data 

editing system for each form type. However, our findings were inconclusive due to definitional and sample 

design complexities.
before. Therefore, we hypothesize that the improved formatting and accessibility of instructions will make 

some respondents more likely to take the time to read them, and that data quality will improve as a result. 

It should be noted that the bulk of the burden associated with reading instructions tends to occur when 

companies are asked to complete the initial reports for the first of several consecutive reference periods for 

which they are sampled. That is, once respondents comprehend the specific data requests of a new survey, 

they document and establish retrieval routines that reduce the necessity of remembering and/or interpreting 

the requests anew with each subsequent survey form they receive. Once respondents invest this initial 

effort, retrieving data for later reports should become less burdensome. Insofar as some respondents read 

the pilot form’s instructions and altered their reporting routines, they will not necessarily have to make a 

similar investment of time and effort in the future. 

 

 

 

The next evaluation we performed involved the addition of four supplemental questions at the end of the 

redesigned questionnaire. These questions asked respondents for their opinions on the new look and 

formatting of the form, the insertion of instructions and diagrams into the form, and the ease or difficulty of 

completing the pilot form compared to the legacy form. This method allowed is to expand our debriefing 

of respondents well beyond the 28 cases interviewed personally. 

Table 14

 presents simple tabulations of responses to these questions; the numbers ranged from 279 to 290. 

Generally, the feedback received from the debriefing questions was positive. Table 1 shows that 79.3% of 

respondents to the supplemental questions answered that they liked instructions placed near the questions in 

Table 1. Responses to evaluation questions included in pilot test draft. 

1. This form put instructions near the questions, while the usual BE-605 put 

them in a separate document. In which location do you prefer to find the 

instructions?

Frequency %

 Near the questions, like this form 230 79.3 

 In a separate document, like the usual BE-605 53 18.3 

 Not applicable — I haven’t completed the usual BE-605 7 2.4 

Total item responses 290 100.0 

2. This form has more open space than the usual BE-605, and it requires 

more pages. The usual BE-605 has less open space and requires fewer pages. 

Which arrangement do you prefer?

 More open space and more pages, like this form 136 48.7 

 Less open space and fewer pages, like the usual BE-605 131 47.0 

 Not applicable — I haven’t completed the usual BE-605 12 4.3 

Total item responses 279 100.0 

3. This form used small organizational charts in questions 6 through 14. 

Which of the following statements best reflects how well these charts aided 

your understanding of the corresponding questions?

 The charts were somewhat or very helpful. 185 65.4 

 The charts helped only a little. 33 11.7 

 The charts were confusing. 11 3.9 

 I did not need the charts to help me understand the question. 51 18.0 

Total item responses 283 100.0 

4. How much harder or easier was it to complete this form compared with 

the usual BE-605?

 A little or much harder 145 50.5 

 A little or much easier 122 42.5 

 Not applicable — I haven’t completed the usual BE-605 20 7.0 

Total item responses 287 100.0 

 

4

 These data are descriptive of respondents only. It is not appropriate to draw inferences to the target 

population. 
the new form, and 65.4% found the small “org charts” to be somewhat or very helpful. Respondents were 

evenly. Respondents were evenly split in their preferences for “more open space and more pages, like [the 

new] form” versus “less open space and fewer pages, like the usual BE-605,” at 48.7% and 47.0%, 

respectively. Although 50.5% of respondents found the new form harder to complete, some noted in the 

open-ended remarks section that the additional difficulty arose from the new format, which broke their 

reporting routines. Like the respondents debriefed personally, they also said that as they become more 

familiar with the new form and establish new reporting routines, it would become easier. Thus it appears 

that findings from the supplemental questions tend to corroborate results from the personal respondent 

debriefing interviews. 

Positive findings from our respondent debriefings lead us to hypothesize an overall reduction in reporting 

error for the redesigned questionnaire. Such an empirical evaluation was beyond the scope of our research 

and was not permitted by the design of our pilot test. 

Conclusions 

The field of establishment survey methodology poses particular challenges for its practitioners because of 

its position between two distinct areas: business financial accounting on one side, and economics on the 

other. Our role as methodologists at this intersection is often to bridge the gap between the two, and the 

authors’ involvement in this project was especially challenging because of the concepts underpinning the 

BE-605 survey. Recognizing this fact early on, we went to great lengths to attempt to establish a thorough 

understanding of the intentions of the survey designers and the problems faced by the analysts in the course 

of data collection as they saw them. As we gained this knowledge from survey insiders (a process which 

never really ended), we then took up the task of understanding the perspectives of respondents to learn 

where the disconnects were between the two systems of knowledge, in order to address these gaps 

systematically in our redesign and attempt to bridge them. 

The strength of our redesign lies in the multiple, complementary, and to some degree redundant methods of 

learning about the problems and of devising, evaluating, and revising solutions. Critical barriers to correct 

reporting were discovered and explored at length in the first phase of research, in the analyst focus groups, 

observations of telephone calls to respondents, and initial respondent debriefings. All of these methods 

contributed specific knowledge about the reporting issues, and each finding complemented and reinforced 

the others. Altogether, this knowledge, along with our experience with survey design and the expertise of 

BEA survey staff, gave us the means to devise potentially useful questionnaire design solutions. We went 

on to test our solutions quite extensively. Our iterative testing approach enabled us to make many fine 

adjustments and evaluate the design solutions, and we learned much, for example, about incorporating 

instructions and diagrams into questionnaires. Eventually, our testing led us to a completely redesigned 

survey form in which we had confidence, and the pilot test gave us the chance to perform a final evaluation 

of the form with a larger number of cases. It also permitted BEA to “handle” collected data from a subset 

of the sample before putting the form into production with thousands of cases, so that they could better 

prepare for implementation. 

While we recommend the use of multiple research methods to aid questionnaire design, to the extent that 

survey resources allow, future research is needed to quantify the value of this approach on improving data 

quality and reducing respondent burden. For example, one would hypothesize that improved data quality 

would result in fewer edit failures. However, findings from our respondent debriefings suggest that this 

benefit may not be realized until respondents re-establish their reporting routines. The effect of the 

redesigned questionnaire on error rates may be assessed by comparing successive reports over a period of 

time to reports of the period immediately prior to the implementation of the new questionnaire. 

Part of the success of this project lies in the confidence with which the survey managers put the redesigned 

form into production. Our role as researchers was to use our experience and knowledge of survey 

methodology to tap into survey analysts’ knowledge by asking questions and relying heavily on their 

knowledge in the redesign of the form. In the process, the survey managers and some analysts were able to 

participate in the investigative and testing process, see the survey from respondents’ perspectives, and 

understand their difficulties. The confidence they gained was a result of our multi-method approach, which 
provided both corroborating and complementary findings across methods. Their first-hand experience with 

both the problems inherent in the original questionnaire, seen in new light, and the opportunities offered by 

the redesign process, led to the survey staff’s full engagement as collaborators in the process rather than 

mere clients awaiting delivery of the final product. They were indispensable partners in the process, and 

have begun data collection with the new instrument with confidence in and in full ownership of the 

questionnaire. 

Acknowledgements 

The authors wish to acknowledge several people whose assistance and support of this research project and 

the writing of this paper are greatly appreciated. We would first like to acknowledge our clients and 

collaborators at BEA, Greg Fouch, Louise Ku-Graf, and Pete Fox, along with numerous other BEA 

personnel, whose contributions to the project were invaluable. We would also like to thank our Census 

Bureau peer reviewer, Broderick Oliver, for his time and thoughtful editorial suggestions. 

References 

Dillman, D. (2000) “Mail and Internet Surveys: The Tailored Design Method.” New York: John Wiley and 

 Sons. 

Edwards, W. S., and D. Cantor (1991) “Toward a Response Model in Establishment Surveys,” in P. P. 

Biemer, et al., eds., Measurement Error in Surveys, New York: John Wiley & Sons, pp. 211-233. 

Sudman, S., D. K. Willimack, E. Nichols and T. L. Mesenbourg (2000) “Exploratory Research at the U.S. 

Census Bureau on the Survey Response Process in Large Companies,” Proceedings of the Second 

International Conference on Establishment Surveys, American Statistical Association, pp. 327-337. 

Tourangeau, R. (1984) "Cognitive Science and Survey Methods" in T.B. Jabine, M.L. Straf, J.M. Tanur, 

 and R. Tourangeau (eds.), Cognitive Aspects of Survey Methodology: Building a Bridge Between 

 Disciplines. Washington, DC: National Academy Press, pp. 73-100. 

Tuttle, A. D. and R. L. Morrison (2006) “Using Graphical Elements to Convey Complex Concepts in an 

Economic Survey.” Proceedings of the Sixty-first Annual American Association of Public Opinion 

Research Conference. Public Opinion Quarterly, (2006)70: E1-E30. 

Willimack, D. K., L. Lyberg, J. Martin, L. Japec, and P. Whitridge (2004) “Evolution and Adaptation of 

Questionnaire Development, Evaluation, and Testing Methods for Establishment Surveys.” In S. 

Presser, J. M. Rothgeb, M. P. Couper, J. T. Lesser, E. Martin, J. Martin, and E. Singer (eds.), Methods 

for Testing and Evaluating Survey Questionnaires. New York: John Wiley & Sons, pp. 385-407. 
Figures and Tables 

Figure 1. BE-605 legacy form, page 1. (actual size 8 1⁄2 X 14”) 
Figure 2. Formatting of redesigned BE-605. Embedded instructions (left page and top right) adjacent to 

item (lower right). 
Figure 3. Formatting of redesigned BE-605. Consolidation instructions presented as survey items, with 

embedded instructions and organizational chart diagrams. 
Appendix: Sample Design and Selection for the BE-605 Pilot (prepared by Dan Yorgason, U.S. Bureau of 

Economic Analysis)

Each firm i was assigned a number, Si, between 0 and 1. Of these Si, the top 700 were ultimately included 

in the sample. The assignment of the Si was made according to a random process, but with unequal 

probabilities of selection. Each Si was computed as 

)/( ii zw

ii = uS

where 

1) ui is a uniform random variable. 

2) zi is the square of a weighted sum of the firm’s shares of overall income (INC), debt position (DP), 

equity position (EP), and “newness” (NEW): 

2

3. 75. ⎥

⎦

⎤ ⎢

⎣

⎡ = + + + ∑ ∑∑∑ i

i

i

i

i

i

i

i

i NEW

NEW

EP

EP

INC

INC

DP

DP kz , 

where the first three terms use absolute values for each firm, and k is a constant. 

3) wi is determined according to the following random process: 

( )

( ) ⎥

⎦

⎤ ⎢

⎣

⎡ =

4

3

4

3

max vinici i

vinici

i

www

www

w , 

with wci computed as 

c

ci

ci N

o

w = , where oci is the position of firm i in a randomly ordered list of all 

firms from country c, and Nc is the number of firms in country c. wni and wvi are computed in the same 

manner as wci, but use NAICS industry and vintage rather than country. 

As ui is between zero and one, the effect of raising this random variable to the (1/zi)th power is to 

preference relatively large, new firms. The effect of the randomization described in 3) is to generate a 

likelihood of representation by each country, industry, and vintage. It also creates a moderate preference 

for selection of firms from large countries, industries, and vintages.