1 

The Business R&D and Innovation Survey: 

Lessons Learned from the Redesign of the Survey of Industrial 

Research and Development 

Richard S. Hough 

U.S. Census Bureau 

Brandon Shackelford 

Twin Ravens Consulting 

John Jankowski, Ray Wolfe 

National Science Foundation 

Federal Committee on Statistical Methodology Research Conference 

January 10-12, 2012

Background 

Before 1953, federal statistics could tell you how many widget-making plants there were in the U.S. and what their 

sales of widgets were. What they couldn‘t tell you was the amount of research and development companies were 

performing to ―build a better widget‖—or to develop any other new or improved good or service. However, since 

1953 the Survey of Industrial Research and Development (SIRD) has collected information, primarily financial in 

nature, on the systematic work companies were undertaking with the goal of discovering new knowledge or using 

existing knowledge to develop new or improved goods and services. The SIRD was sponsored by the National 

Science Foundation‘s (NSF) National Center for Science and Engineering Statistics and was conducted by the U.S. 

Census Bureau in accord with an interagency agreement through 2007. 

Over the past 50+ years, NSF and the U.S. Census Bureau added and deleted content from the SIRD based on input 

from companies responding to the survey as well as users of the data, but the fundamental concepts and structure of 

the survey were largely untouched. More recently, prompted by recommendations from the 2005 Committee on 

National Statistics (CNSTAT) Report, Measuring Research and Development Expenditures in the U.S. Economy, 

NSF and the U.S. Census Bureau concluded a full-scale redesign of the SIRD. The goal of the redesign was to 

produce high quality, relevant data on R&D in the business sector that took into account the changing reality of 

R&D and innovation writ large. This paper will discuss the redesign project and some of the most significant 

changes that were implemented in the 2008 pilot of the redesigned survey, titled the Business R&D and Innovation 

Survey (BRDIS). 

Five Phase Redesign 

A full-scale survey redesign involves a range of required tasks. In order to simplify planning and management, the 

SIRD interagency redesign team organized these tasks into five phases: 

Phase 1: Define data/information needs and priorities. To accomplish the goals of this phase the 

interagency team conducted two data user needs workshops (federal and non-federal data users); held three 

panel meetings of R&D managers and executives from a variety of U.S. firms; conducted a series of 

innovation workshops; reviewed existing literature that referenced the survey data; and evaluated the data 

needs of NSF, the Bureau of Economic Analysis (BEA), and the U.S. Census Bureau. 

Phase 2: Identify data sources and availability. The team conducted record keeping studies, respondent 
 

2 

debriefings, cognitive interviews, response studies of the current SIRD and used supplemental surveys to

gather information from the current set of respondents. The team also consulted with a panel of business

experts to better understand what information firms can provide and are willing to provide. 

 

Phase 3: Evaluate current operations and methodology. During this phase, the team examined the 

industry classification, imputation and editing methodologies implemented in the SIRD and identified areas 

where new methods and technologies would improve the results. 

Phase 4: Implementation and testing of new content and methods. This phase synthesized the 

information gathered during the first three phases to implement the new Business R&D and Innovation

Survey. A critical component of this phase is the cognitive testing of proposed survey content with 

potential respondents. 

Phase 5: Evaluation of the pilot results. This is the post collection phase where the reported data is 

analyzed for accuracy and consistency with desired concepts. The critical component of this phase is the 

post survey interviews conducted with respondents. 

The first three phases of the redesign plan concluded in November of 2006. At that point, the team began the 

processes necessary to design the BRDIS and the systems required to support the collection and dissemination of 

BRDIS data. The pilot survey was fielded in January 2009 using a sample of about 40,000 companies. The 

remainder of this paper will discuss the major changes that were implemented for the pilot survey and the results of 

the testing done after the pilot was fielded. We will also discuss some changes that were made to the 2009 BRDIS as 

a result of the pilot survey. 

Changes to Survey Content 

Data users in the public and private sectors had strong opinions regarding what content should be collected in a new 

survey of business R&D and the list of potential new data items was voluminous. The redesign team evaluated the 

need for different types of data as well as the reasonableness of collecting consistent data from companies without 

introducing burden on them out of proportion to the value of the data. This evaluation resulted in numerous 

proposed changes to the content of the BRDIS relative to the SIRD. 

Adopting a definition of R&D based on accounting standards. The primary economic concept the 

survey wants to measure is based on a definition of R&D presented in the Frascati Manual (2002). The 

Frascati definition focuses on R&D conducted or performed by the companies. The companies account for 

R&D based on the accounting definition presented in the Statement of Financial Accounting Standards No.

2 (FASB 2) Accounting for Research and Development Costs – 1974. Rather than ask companies to report 

based on a definition of R&D that differed from that used for financial reporting and tax purposes, the

BRDIS asked companies to start with the definition that they understand and it derives the performance 

based concept from that information. 

Collecting data for worldwide company operations as well as domestic operations. The SIRD focused 

on domestic U.S. R&D, the BRDIS collects data that reflects the global nature of R&D and innovation 

including country-level detail for offshore R&D activity. The BRDIS also collects data on the outsourcing

of R&D both domestic and foreign. 

Collecting business segment detail. A criticism of the SIRD was that each company‘s reported data was 

assigned to one industry. The BRDIS maintains the company-level focus, but requests companies to report

six figures (worldwide R&D expense, domestic R&D paid for by the company, domestic R&D 

performance, worldwide and domestic sales or revenues, and R&D paid for by 3rd parties) by business 

segment. The BRDIS utilizes the model employed by the Annual Capital Expenditures Survey (ACES) to 

collect this sub-company data.. 

Capital expenditures for R&D operations. To support the efforts of the Bureau of Economic Analysis 
 

3 

(BEA) to explore the impact of including R&D as part of the national accounts the BRDIS is requesting 

companies to report the portion of their capital expenditures that are in support of R&D operations. 

Collecting more detail about the R&D workforce. The BRDIS collects data on full time equivalent 

(FTE) R&D employment and R&D headcounts. Companies are asked to report R&D headcount data by 

gender, occupational category, and level of educational attainment. Companies are also asked to report the 

number of U.S. R&D employees working under a visa (H-1B, L-1, etc.). 

Using R&D as an indicator of change within the company. The BRDIS asks companies to estimate the 

share of their R&D that is directed towards new areas of business for the company as well as the share of 

R&D that involves science and technology (S&T) that is new to the company. These measures reflect the 

efforts of the company to grow in new directions and to learn from and adapt to the ever-expanding base of 

science and technology. 

Gauging the R&D strategy of companies and the potential impact of R&D on the market. The BRDIS 

asks companies to estimate the share of their R&D that is in areas of science and technology new to their 

marketplace. This question identifies the perceived importance of the strategy of technology leadership 

within different industries and types of companies. This question also reflects the relative effort, in the 

words of one company respondent, devoted to ―revolutionary versus evolutionary‖ R&D. 

Identifying R&D directed to application areas of particular national interest. Industry-level detail is 

useful to economists, but many data users require information about areas of interest that spans multiple

industries and sectors. The BRDIS asks companies to report the share of their R&D that is directed towards 

several of these application areas, such as health, defense, energy, environmental protection, and 

agriculture. 

Collecting basic project-level data. It has been noted that the fundamental unit of analysis for R&D is not 

the firm, line of business, or establishment, but the R&D project. The BRDIS does not request companies 

to report at this level of detail, but it does gather information on counts of R&D projects within companies, 

including the number of new projects started and the number of R&D projects that moved into 

commercialization. 

Measuring intellectual property and technology transfer activities and attitudes. The BRDIS gauges 

the relative importance of different types of IP protection to companies; asks what percent of their 

discoveries they attempt to patent; asks whether the company participated in specific technology transfer

activities; asks about the company‘s practices with respect to licensing technology both in and out; and asks 

for counts of the number of collaborative R&D agreements the company is a party to and who the partners 

were (universities, suppliers, government agencies, etc.). 

Collecting innovation data. The BRDIS will be used as a platform to test the collection of innovation 

metrics from U.S. businesses. The 2008 – 2010 BRDIS cycles have collected innovation data using the 

questions from the European Community Innovation Survey (CIS). The interagency team is currently 

reviewing the data results and planning for an innovation section to be added to future BRDIS cycles. 

A Sectional Survey Booklet 

The bullets above are a good representation of the range of data the BRDIS collects. Because the various data span a 

variety of subject matter areas, from financial to human resources to technical to legal, it is unlikely one person or 

organization within each company will be best able to answer all the questions. The BRDIS was designed as a 

sectional survey booklet in order to facilitate obtaining information from the people within each company who have 

the best understanding of the concepts and definitions being presented and access to the information necessary to 

provide the most accurate response. Each section has the look and feel of an individual form. From the Census 

Bureau‘s business help site, the respondent has the ability to download each section separately and distribute them to 

the appropriate person/department within their company. Each section has instructions that explain the key concepts 

to be reported and suggests the appropriate department within the company to request the information. The web 
 

4 

version of the survey was developed using this same approach. The sections have been defined by grouping 

questions based on subject matter areas within the company and currently include: 1) Company Information – 

requests information on company ownership, business segments, revenues and innovation; 2) Financial Schedule A - 

focused on company R&D expenses; 3) Financial Schedule B – focused on costs for R&D that was funded, paid for 

or reimbursed by others; 4) Management and Strategy of R&D – collects information about the characteristics of the 

R&D; 5) Human Resources – focusing on employees who work on R&D activities; 6) Intellectual Property and 

Technology Transfer – focusing on patenting and other activities regarding intellectual property. This sectional 

approach is a departure from the SIRD that was formatted as one survey for one respondent. 

The sectional survey booklet design also allowed for the incorporation of instructional material to be included where 

they are appropriate. The SIRD mailed a separate instructional booklet with the survey instrument. The instruction 

book provided specific question by question guidance. The BRDIS imbedded the most important instructions into 

the questionnaire where the question is actually located and provides an on-line tool for question by question 

specific instructions. The design also implemented visual design features that include color coded sections and 

shading of the instructional areas. 

The incredible complexity of R&D and the ever-changing nature of science and technology makes it is a challenge 

to anticipate what questions will be most relevant to decision makers in the future. The sectional design approach of 

the survey will allow for future survey enhancements such as a section focused on innovation or sections focused on 

specific industries (pharmaceutical R&D for example). It will allow for sections that may not be necessary to collect 

every cycle to be rotated in and out of production easily. 

Pre-Survey Testing 

The interagency team implemented a testing strategy that included five rounds of testing with a panel of SIRD 

respondents. During the first four rounds the team tested the cognitive understanding of the key concepts presented 

in the individual sections of the survey, the fifth round was where the entire survey booklet was presented to the 

panel of respondents. 

Round one of the pre-survey testing was focused on the initial draft of the financial section. Respondents were 

provided a copy of the financial section with instructions. As respondents worked through the form, they were asked 

non-leading questions about their interpretations of the concepts and how they would go about retrieving the data. 

Their responses were non-directively probed at length until a thorough and accurate understanding of the response 

process was obtained. The key results from round 1 were: 

Respondents have a different definition for R&D than that presented on the survey. Respondents track 

R&D expenses from an organizational/accounting perspective. Respondents seem to understand what 

corporate entity pays for the R&D. Some respondents were including activities such as technical support in 

their R&D expense and this is not consistent with the BRDIS (Frascati) definition of R&D. 

Respondents take a global perspective when reporting R&D expense. Companies are global entities and 

they track expenses at the global level first. It is difficult for some respondents (of multinational 

companies) to break out the R&D expense that was performed in the U.S. 

Foreign owned companies will have a more difficult time reporting R&D performance. Foreign companies 

may allocate R&D expense to the U.S. entity but the U.S. entity would not know where the R&D was 

actually performed. 

Round two of the pre-survey testing focused on a revised draft of the financial section and on the initial draft of the 

R&D personnel section. The same protocol was implemented as in round one. Human resources personnel were 

available for most of the discussions and where they were not available, written comments on the questions were 

provided to the financial respondent. The key results from round two were: 

Worldwide R&D expense was found to be understood by most respondents. The specific include and 

exclude instructions helped to clarify the item for respondents. The discussions about the include and 

exclude instructions also highlighted some of the limitations within company records. Some respondents

would not be able to distinguish between internal and external funding sources. Foreign owned company 

respondents would have their total worldwide R&D expense allocated to them from the parent. 
 

5 

Respondents may not be able to distinguish purchased R&D services from other purchased services in their

accounting records. 

Companies performing R&D in foreign locations would be able to report the R&D by country. 

Respondents explained that they keep accounting records by country for tax purposes and the breakout of 

R&D by country would not be difficult for them to report. 

Interpretation and source of R&D employment data varied. Respondents have access to lists of job titles/ 

descriptions/series and they align that information with the categories requested on BRDIS. For example, 

respondents would report all employees listed as engineers or design engineers as R&D employees. 

Obtaining worldwide employment data will be difficult for some respondents. Some respondents did not 

have access to the HR records of the foreign entities and these questions would require the respondent to 

request the data from those entities. 

Some companies did not track the types of information requested on BRDIS. Some companies did not have 

the educational data in their HR systems. 

 

Round three of the pre-survey testing focused on the R&D Management and Strategy section. The same protocol 

that was used in the previous rounds was implemented in round three. Appropriate R&D personnel were not 

available for all of the interviews. The key results from round three were: 

Finding the appropriate R&D personnel within the companies will be burdensome for survey coordinator. 

Most respondents to the BRDIS are located in the accounting departments within the companies. The 

questions in this section will require an understanding of the companies R&D strategies and decision 

making processes. For larger more diverse companies, this may be even harder to accomplish. 

In order to characterize R&D costs as requested in this section, many respondents stated that they would 

need to rely on project based accounting. For companies that have many projects, this would be too 

burdensome. 

Round four of the pre-survey testing focused on the Intellectual Property (IP) section. The protocol was consistent 

with that implemented in previous rounds. Appropriate IP personnel were available for most of the interviews. The 

key results from round three were: 

Some respondents found the requested data items to be extremely sensitive. Some of the information 

requested is not even released to the financial personnel. 

Reporting data on R&D agreements and contracts for licensing IP would be very burdensome. These items 

tend not to be centrally tracked and respondents would need to send the questions to multiple operating

units. Further, to report accurately, this section would require the review of all of the contracts to determine 

if they are still active. 

Round five of the pre-survey testing was the first time that respondents were able to see the entire survey booklet 

together. This round of testing included respondents that participated on the panel and were familiar with the survey 

concepts and new respondents who had never seen the concepts prior to the interviews. The protocol was consistent 

with that implemented in previous rounds. The key results from round five were: 

The BRDIS imposes considerable burden on companies. Completing BRDIS will require a survey 

coordinator to gather the information from the appropriate departments within the organization and compile 

a consolidated company report. This process will be more labor intensive for the larger, more diverse 

companies. 

Respondents had difficulty understanding the difference between R&D expense and R&D performance. 

The layout and format of the BRDIS booklet was well received by respondents. They found the appearance 

pleasing, especially the use of color. They liked the organization of the form into sections and the 

formatting of the questions and instructional material. 
 

6 

The 2008 BRDIS 

The interagency team relied on the information gathered during the pre-survey testing to guide content and forms 

design decisions for the 2008 BRDIS. The team also approved implementing recommendations to change 

methodologies and survey processes based on results of phase 3 of the project. A new classification methodology 

was developed to utilize the business code data collected on the 2008 BRDIS. New imputation and editing programs 

were written and tested. The 2008 BRDIS migrated to the Integrated Computer Assisted Data Entry (ICADE) 

system and utilized a web based survey instrument for the first time. The team decided to discontinue the SIRD and 

administer the redesigned survey to the entire sample. The 2008 BRDIS pilot had the following goals: 

Publish U.S. level estimates for total R&D performance, R&D by source of funds, sales, and total R&D 

FTEs for 2008. 

Conduct comprehensive post survey testing of the new survey content, survey design and the contact 

strategies. 

Establish and/or improve working relationships with large R&D performing companies. 

The 2008 BRDIS represented a significant change from the 2007 SIRD for respondents. The 2007 SIRD long form 

(RD-1) was 18 pages in length compared to the 2008 BRDIS long form booklet (BRDI-1) that was 56 pages in 

length. The 2007 SIRD short form (RD-1A) was 7 pages in length compared to the BRDIS short form booklet 

(BRDI-1A) that was 28 pages in length. The number of data items processed increased from 342 in 2007 to 717 in 

2008. The electronic collection migrated to an internet based reporting instrument that was significantly different 

from the 2007 SIRD. The new survey also required increased editing and additional programs needed to be written 

to support the new data collection. The 2007 SIRD required 8 derived variable calculations compared to 166 for the 

2008 BRDIS. The 2007 SIRD needed 338 edit specifications compared to 924 in the 2008 BRDIS. The 2007 SIRD 

needed 412 imputation specifications compared to 1479 in the 2008 BRDIS. The 2008 BRDIS represented a 

significant increase in burden to respondents and required additional programming and support resources to upgrade 

the processing systems. 

Account Manager Program 

Given the number of changes and the challenges that the new structure would present, significant resources were 

devoted to communicating with respondents about the new survey. A draft of the new survey instrument was

complete and made available for respondents to review in Oct. 2008. Top performing R&D companies were sent a 

pre-survey letter that directed them to a web site where the new form was available and we also requested that they 

verify the name of the survey coordinator that we had in our database from the SIRD. A communication plan was 

implemented that involved communications via mail, email and telephone to mitigate the potential impact on the 

response rate that the increase in burden may have. The purpose of these communications was to encourage timely 

and accurate responses, find ways to facilitate the reporting process and to address respondent questions and 

concerns. To accomplish all of this, the communication strategy for the 2008 BRDIS involved three distinct 

processes: 1) pre-survey contacts, 2) telephone assistance provided by survey analysts through our account manager 

program, and 3) post survey follow-ups that included debriefing meetings held at the companies. 

In terms of dollar expenditures, a relatively small number of companies represent a large share of total R&D. 

Because the distribution of R&D expenditures across companies is so skew, survey responses from the large R&D 

companies have a disproportionate impact on published estimates. In order to increase the quality of data collected 

from these large companies the Census Bureau established an account manager program through which the top 

R&D companies (based on prior year reported or imputed data) were assigned a survey analyst to act as a single 

point of contact for all communications. The goal of the program was to establish strong working relationships with 

the most significant respondents to the survey to insure high quality R&D estimates. Account managers proactively 

contacted their companies to offer assistance rather than waiting for the company to initiate contact. Account 

managers offered their assistance in explaining survey concepts; consolidating data from units of the company; and 

recruiting assistance from parts of the company with which the survey respondent was unfamiliar. 
7 

The 2008 BRDIS account manager program was a compilation of communications including a minimum of four 

phone calls. The first phone call, made approximately two weeks prior to survey mail –out, was used as an 

introduction to help build rapport. The analyst introduced him or herself, explained the BRDIS, and discussed the 

Business Help Site (BHS). The BHS is an online tool created to help respondents better understand, complete and 

submit the BRDIS. It includes Excel worksheets designed to help companies with decentralized record keeping 

systems collect information related to specific sections of the Business R&D and Innovation Survey. If the 

respondent chooses, they also have the option of submitting an electronic version of the form programmed with 

appropriate skip patterns and simultaneous editing to assist the respondent in filling out only those questions 

applicable to that specific company. 

The second call, approximately two weeks after mail-out, was done to make sure the respondent received the survey, 

answer any questions they may have, and again remind them of the tools on the BHS. Call three was concerning the 

approaching due date for the survey. The analyst will confirm that the BRDIS has yet to be submitted and offer their 

assistance in completing the form in a timely manner. Once the company has submitted a completed form, the 

analyst will follow up with a “Thank- you” call. The cooperation of the company is very important to us and we 

want them to know our appreciation. This call is also used to ask any follow up questions necessary in regards to the 

responses and data submitted by the respondent. 

The four specific calls are attempted for each company; however, some analyst to respondent contact occurs many 

more times, as well as contact with some companies is never successfully made at all in the process. The staff does 

not acknowledge leaving voicemails as having made  ̳contact‘ with the respondent unless or until the respondent 

makes a return phone call. 

Through the account manager program, survey analysts were able to learn a great deal about the reporting processes 

of the largest companies in the survey. The inclusion of companies in the account manager program was shown to 

be a significant predictor of response (Hough, Keller, Curcio, and Wilkinson 2011). 

Initial Edit Results 

The 2008 BRDIS micro data review process included the review of an edit failure listing. The initial output from the 

edit programs revealed a systematic problem within NAICS 5417 (R&D services) and 3254 (Pharmaceutical 

Manufacturing) industry groups. The initial output revealed that more than 60% of the cases that had reported and 

were classified in NAICS 5417 (R&D services) had reported no R&D costs in the yes/no checkbox question. The 

team also saw similar reporting issues when reviewing reported cases in NAICS 3254 (Pharmaceutical 

Manufacturing). The BRDIS analytical staff conducted a review of publically available information regarding the 

business activities of these respondents and concluded that the reporting problem was causing significant missing 

data in the world wide funded R&D question. 

The interagency team developed a proactive plan to address the reporting issues for the 2008 BRDIS. The plan 

needed to deal with the cases that had already reported to the survey and also included a communication strategy for 

cases in these industry groups that were still non-response. First, the team identified the largest cases that had 

reported and assigned them to a survey analyst at the Census Bureau‘s headquarters. These cases were contacted and 

the analysts asked specific questions about their business and the survey definition of R&D. These contacts revealed 

a problem with how the respondents were interpreting the R&D instructions contained on the questionnaire and in 

some instances, not reading the instructional material at all. The respondents did not understand how they were to 

report R&D costs associated with collaborative ventures and partnerships. Respondents did not classify activities 

performed under a contractual agreement as R&D as defined in BRDIS. For most of these companies, all of the 

contractual agreements would be for R&D as defined by the survey. The analysts were able to help the respondents 

understand how to properly report the R&D costs associated with their contractual agreements and were able to get a 

majority of the respondents to report properly. A call unit was established at the U.S. Census Bureau‘s National 

Processing Center to contact the smaller cases that had this reporting issue with the goal to identify and have the 

respondent correct the missing data. Similar issues were found in the smaller companies as well. For the cases that 

were non-response when the edit program was run, the team sent an instructional letter to the respondents at these 

companies. The instructions were written using the specific scenarios that were uncovered during the follow-up with 

the companies who did not report properly. The follow-up strategy was implemented throughout the 2008 BRDIS 

processing cycle and these companies were monitored very closely throughout the cycle. 
8 

The interagency team implemented changes to the 2009 BRDIS questionnaire to attempt to address this reporting 

issue for future survey cycles. See the Improvement to the 2009 BRDIS section of this paper for more details about 

the specific changes made. 

2008 BRDIS Post Survey Debriefings 

The interagency team implemented a strategy that focused on the most significant R&D performing companies in 

the survey. The structure of the interviews was tailored to the specific reporting issues of the companies being 

interviewed. A thorough review of the companies report was conducted and an agenda was developed and sent to 

the respondent prior to the meeting. The interagency team was interested in learning the process that these 

respondents undertook to fulfill the 2008 BRDIS reporting requirement. The primary goal of the interviews was to 

understand the composition of reported data, respondents‘ interpretations of the items, and respondent burden. 

Ideally, respondents were contacted and scheduled for debriefing within four weeks of receiving their responses, so 

that they would be more likely to remember specifics about their experiences with the survey. 

Companies were selected purposively, primarily with regard to the magnitude of reported annual R&D expenditures 

but also to maximize industry coverage and to meet with some respondents who participated on the panel for the 

pre-survey testing. We also met with some of the NAICS 5417 companies who did not report the world wide funded 

R&D question properly. For most of the interviews, we were successful in having all of the employees at the 

company who worked on completing the survey present at the interviews and we were able to walk through the 

reporting process in great detail. The key results from the post survey debriefings were: 

Most respondents worked in the accounting or finance divisions within the companies. They had to 

coordinate with multiple people from multiple divisions within the company to gather the information 

requested on the survey. Few of the respondents had extensive knowledge of their companies R&D 

activities. Some respondents were only responsible for the consolidation of the information from specific 

business units within the organizational structure. Some respondents had a difficult time finding 

appropriate personnel who were willing to assist with the report. 

Most respondents reported using data pulled directly from company accounting records or they derived the 

total from multiple accounting sources. They did not attempt to adjust their reported figures based on the 

include/exclude instructions contained on the form. 

Several respondents used the electronic versions of the form available on the business help site to gather the 

data internally. They reported that these versions were very useful in the reporting process. They did 

comment that the ability to simply upload the spreadsheets into the on line instrument would save them 

from having to type all of the information and would save them even more time. 

It was observed that the large amount of text within questions and embedded instructions contributed to 

them not being read completely. Some of the instructions were unnecessary or irrelevant and this lead to 

some key instructions being overlooked. 

Some respondents were interpreting domestic sales as sales to domestic customers, not sales that originated 

from domestic operations. Accounting records for these companies are based on where the sale is booked 

not where the shipment originated from. 

Worldwide R&D expense is the concept that most respondents are familiar with. Most report the data 

directly from their records or financial statements. These respondents were very familiar with U.S. GAAP 

(Generally Accepted Accounting Principles) reporting and SEC (Securities and Exchange Commission) 

requirements. Some respondents used the IRS R&D tax credit form to report to BRDIS. 

Respondents defined “R&D agreements” differently across companies. Some respondents thought of only 

government contracts applying to this item. Several others reported the number of agreements in which they 

paid third parties to perform R&D. 

Respondents did not track R&D costs contained within contractual agreements. Some respondents should 

have reported positive data to the worldwide funded R&D question but did not because they did not 

interpret the activities done under these contracts to be R&D. Respondents talked about ―costs of goods 

sold‖ when referring to the specific costs that the survey is attempting to collect. Some respondents 

commented that they had hundreds of contracts so to report this with any level of accuracy would take too 

much time.
 

9 

The 2008 BRDIS post survey debriefings provided the interagency team with extremely valuable information on 

how the survey concepts were being interpreted and, most importantly, how data was being reported by some of the 

most significant R&D respondents in the sample. The meetings were also very valuable in establishing a strong 

working relationship with these companies. The information was analyzed and used to make improvements for the 

2009 BRDIS cycle. 

2008 BRDIS Results - Response Rates/Patterns 

For the calculation of response rates for the 2008 BRDIS, response cases would have met the following criteria: If 

they answered yes to question 1-4(R&D yes/no question), then they would need to have worldwide R&D expense > 

0 or worldwide funded R&D costs > 0. If they answered no to question 1-4, then they would need to have a response 

to the innovation question 6-1. The 2008 BRDIS achieved a 76.6 % overall response rate. The long form (BRDI-1) 

achieved a 74.5 % response rate and the short form (BRDI-1A) achieved a 77.0 % response rate. Cases that were 

included in the account manager program achieved a 93% response rate. 

Figure 1 shows the 2008 BRDIS response pattern compared to the 2007 SIRD. The vertical red line represents the 

due date reminder letter mailed to all non-response cases with no extension data in the processing system at that 

time. The two vertical green lines represent form follow-up mailings and the purple line represents the beginning of 

non-response follow-up calls conducted by the National Processing Center in Jeffersonville, IN. 

The graph above shows that the response pattern for the 2008 BRDIS was relatively consistent with the 2007 SIRD. 

It should be noted that the 2008 BRDIS was mailed in late January of 2009 whereas the 2007 SIRD was mailed in 

Mid-March of 2008. The difference in the mail date would explain the lag in the 2008 BRDIS response rate for the 

long form cases. Long form recipients are some of the larger multi-national companies in the sample. These 

companies will usually report annual surveys after they file their SEC required reports and the annual reports that 

they prepare for their shareholders. 

2008 BRDIS Results – Coverage rates

The sample frame was the Census Bureau‘s business register at the enterprise level. Company level data for Total 

R&D was extracted from the 2003-2007 cycles of the Survey of Industrial Research and Development (SIRD) and a 

longitudinal series created for each company present. The series for any specific company could have had as little 

as one year of data or as much as five years of data. In addition, the data for any given year could have been reported 

or imputed. 
 

10 

For companies with historical data (from the 2003-2007 SIRD), the initial value for the MOS was defined as the 

most recent value for Total R&D that was greater than zero. If there were no historical data from the SIRD for a 

company, we looked at R&D from three alternate sources: Compustat, Bureau of Economic Analysis (BEA), and the 

Company Organization Survey (COS). If there were no historical values greater than zero and at least one zero value 

did exist, then the company was assigned a zero value for the initial MOS. Again, this zero MOS could have been 

based on any response to the 2003-2007 cycles of the SIRD. 

 

The BRDIS sample design contains three distinct components based on available R&D information as previously 

described and business register information available. The first group sampled was the known R&D performers; the 

measure of size (MOS) for this group was R&D from a previous cycle or alternative source. Companies that had 

R&D greater than $3 mil. were sampled with certainty and the remaining cases were sampled using a stratified 

probability proportionate to size (PPS) design. The second group sampled were the cases where we had a reported 

zero value for R&D from a previous cycle; these cases were sampled using a stratified random sample design. The 

third group were cases where we did not have any data on the R&D at the company; these cases were sampled using 

a stratified PPS design where the MOS was payroll from the business register. 

The coverage rate for each group was defined as the total MOS of cases that satisfied the response criteria as defined 

in the previous section divided by the total measure of size. The 2008 BRDIS coverage rates were: 

 Group 1 Known R&D performers: Coverage rate = .90 

 Group 2 Known zero cases: Coverage rate = .77 

 Group 3 Unknown R&D: Coverage rate = .80 

Improvements made to the 2009 BRDIS 

Based on the results of the 2008 BRDIS pilot, several changes were implemented in 2009 to address the 

reporting/conceptual issues that were observed. R&D is a rare activity among companies and the majority of 

responses to BRDIS (and its predecessor the SIRD) are from companies that indicate they have no R&D. Out of a 

desire to minimize the burden of the survey on these companies a yes/no checkbox question was included at the 

beginning of the 2008 BRDIS asking whether the company had R&D. The review of 2008 BRDIS data revealed that 

many respondents incorrectly checked ―No‖ on this question. These respondents included both smaller, private 

companies unfamiliar with accounting definitions of R&D (one respondent related their thought process as: ―We 

don‘t have any scientists in white lab coats, so we must not have any R&D‖) as well as larger companies in the 

business of providing services to customers that BRDIS would consider to be R&D, but that from an accounting 

standpoint were not classified as R&D expenses. 

The R&D screener question was dropped from the 2009 survey and all companies were asked to answer the 

questions on R&D costs. This form design change forced respondents to process more information about how 

BRDIS defined R&D before they reported whether or not their company had R&D. The goal was to expose 

respondents who might have checked ―No‖ to the screener question with more instructions and survey context that 

might challenge preconceptions about R&D that differ from the BRDIS definitions. The risk inherent in this 

approach is that it invites a higher incidence of false-positive reports, but identifying false-positive R&D cases is 

much easier than identifying false-negative R&D cases in BRDIS. 

Although BRDIS‘ definition of R&D is closely aligned with accounting definitions of R&D, there are important 

areas where the concepts measured by BRDIS differ from how companies track their R&D. The 2008 BRDIS 

addressed these differences through include and exclude instructions in its two main R&D questions. The review of 

2008 BRDIS data revealed that many respondents were not following the include/exclude instructions when

reporting their R&D figures. During debriefing interviews some respondents indicated that they did not think they 

needed to read the detailed instructions because they ―already knew what R&D expense was‖. Others skimmed the 

instructions and focused on the points that confirmed what they were already inclined to report. Yet others assumed 

that since many of the instructions did not apply to their company, none of them did (some instructions, such as one 

involving clinical trials, are industry-specific). These failures to follow include/exclude instructions resulted in 

significant corrections to individual company reports, but there was no easy way to gauge the scale of the problem 

across the entire survey. 
 

11 

For the 2009 BRDIS, the include/exclude instructions were turned into a series of questions, each requiring a 

response from the company. This approach was based on the premise that the economic concepts requested by 

BRDIS do not always conform to the R&D measures tracked by companies. Rather than directly ask for concepts 

that may diverge from respondent preconceptions about R&D, the approach in 2009 guided respondents to derive 

amounts that conformed to the BRDIS definition of R&D. 

An important goal of BRDIS is to collect information about R&D from the organizations that actually perform the 

R&D as opposed to simply paying for it. In 2008 BRDIS introduced this concept early in the form by asking 

companies to report ―R&D performed by your company‖ versus ―R&D performed by others‖. Debriefings indicated 

that some companies did not interpret and report these concepts consistent with BRDIS definitions. For the 2009 

BRDIS the concept of ―R&D performed by others‖ was derived from the sum of two R&D costs known to be 

tracked by companies: payments to business partners for collaborative R&D and purchased R&D services. 

The results detailed in this paper show that it is risky to assume that detailed survey instructions will result in 

respondents reporting more accurate data. In fact, the perception among survey designers that a concept requires 

detailed explanation on a survey should be a warning that the concept should be treated carefully to mitigate 

measurement error. The changes made to the 2009 BRDIS were premised on the guiding principle that company 

respondents will report more accurately and consistently to questions that tie directly to concepts and records they 

are familiar with than to questions about unfamiliar concepts or concepts that differ in subtle, but important ways 

from how the respondent thinks. As simple as this may sound, it is not always easy to identify the types of 

measurement error highlighted in this paper. The 2008 BRDIS went through multiple rounds of cognitive testing, 

but its strengths and weaknesses were not fully revealed until the survey data was reviewed and companies were 

asked to explain their responses. Were it not for the fact that BRDIS was a pilot survey in 2008 and 2009 and 

therefore had additional resources devoted to data review and company debriefings, there may not have been enough 

evidence to justify the form design changes described in this paper. 

Acknowledgements 

The redesign of the Survey of Industrial Research and Development began in 2005. The project was governed by a 

steering committee that consisted of senior managers from the National Science Foundation and the US Census 

Bureau. The make-up of that committee changed multiple times over the duration of the project and each manager 

brought a unique perspective and years of experience in conducting and managing data collections. We would like 

to take this opportunity to thank Dr. Lynda Carlson, Jeri Mulrow, Jock Black, Tom Mesenbourg, Harvey Monk, Bill 

Bostic, Tom Zabelsky, Paul Hsen, Ron Lee, Peter Gibson, Dave Kinyon, and Kim Moore for their contributions to 

the steering committee. 

This paper utilized multiple reports that were prepared for the steering committee during the redesign process. We 

would like to thank Diane Willimack, Kristen Stettler, Dave Tuttle, Grace O‘Neil, La Toya Thomas, Mike Flaherty, 

Stacey Cole, Thomas Falconer, Francisco Moris, and Fran Featherston for their hard work and thoughtful 

suggestions throughout the redesign process. 

References 

Hough, Keller, Curcio and Wilkinson (2011), Characteristic of Large Company Respondents to an Economic

Survey. Paper presented at the ASA Joint Statistical Meetings, Miami, FL 

Shackelford and Heilig (2011), Effect of Form Design on Reporting of Complex Concepts in a Survey of 

Businesses: Evidence from the Business R&D and Innovation Survey, Paper presented at the ASA Joint Statistical 

Meetings, Miami, FL 

Jankowski, Wolfe, Carlson et al (2010), The Redesign of the National Science Foundation‘s Business R&D and 

Innovation Survey: What Worked and What Didn‘t in Trying to Measure Global R&D, Intellectual Property, 

Technology Transfer, and Innovation, Presented at the AEA Meetings, Atlanta, GA