Multi-mode survey response process for 

a complex economic survey 

Lessons Learned from the Evolution of the Business 

R&D and Innovation Survey’s User Interface

“We have an app for that”

Richard S Hough

U.S. Census Bureau

Brandon Shackelford

Twin Ravens Consulting

Federal Committee on Statistical Methodology Research Conference

Washington D.C.

January 10, 2012
Overview

• Business R&D and Innovation Survey 

(BRDIS) Background

• BRDIS – A challenging survey to complete

• What we learned from respondents

• What we did to facilitate the response process

• BRDIS Reporting Trends

• Future Considerations

2
Background on BRDIS

• Primary federal source of information on the R&D activity of 

businesses in the U.S.

• Conducted under a joint partnership agreement between the 

U.S. Census Bureau and the National Science Foundation 

(NSF)

• Fielded as a full scale pilot in 2008

Replaced the Survey of Industrial R&D (SIRD)

Mailed to ~ 40,000 U.S. based companies

Survey collected under Title 13 authority

Response is required by law 

Response is confidential by law

3
BRDIS Population & Reporting Unit

• Target population: All for-profit companies with U.S. 

operations that are classified in non-farm industries 

and have 5 or more U.S. employees

• Reporting unit is the addressed company (not 

establishment)

Companies report data for all foreign and domestic 

subsidiaries where the company has majority ownership

4
BRDIS Challenges 

• Survey content expanded significantly from the SIRD

 Data Items increased from 342 in 2007 to 717 in 2008

 The 2007 RD-1 form was 17 pages – The 2008 BRDI-1 form was 56 

pages. The 2007 RD-1A form was 7 pages – The 2008 BRDI-1A was 

28 pages

 Content was expanded into more non-accounting areas (R&D 

Management and Strategy, Patent information, IP protection questions, 

Technology Transfer questions, expanded HR detail questions and 

Innovation questions) 

 Not all content applies to every company

5
BRDIS Challenges 

• R&D is a highly concentrated activity in the U.S.

• Very diverse organizational structures – Varying degrees 

of data access

• Survey burden for these companies dramatically 

increased from 2007 to 2008

• Most respondents must coordinate the response 

process and/or consolidate data from different 

divisions or business units

6
Lessons Learned 

• Data requested located in different places 

within the companies

• Respondents were scanning the form to 

create PDF version that could be emailed to 

colleagues

• Respondents were programming 

spreadsheets versions that could be emailed 

to colleagues

* Format did not include the relevant instructions that 

are on the questionnaire

• Some respondents needed to consolidate the 

information to compile a company report

7
“We have an app for that”

• Designed the questionnaire to be a color coded 

sectional booklet

• Created PDF versions of each questionnaire and 

made them available on our web site

• Programmed Excel Spreadsheets of each individual 

section and made them available on our web site

• Programmed a consolidator spreadsheet and made it 

available on our web site

8
Internet Reporting 

Lessons Learned

• Most respondents prefer to use the web 

instrument

• Respondents were using paper/excel/PDF 

formats to gather data within the 

organization

• Respondents were then entering the data 

from the various formats into the web 

instrument

9
Paper vs. Excel

10
Excel Error Checking

11
Consolidator Tool

Dept. 1 Dept. 2 Dept. 3

Consolidated 

Report

12
“We have an app for that”

• Programmed an upload capability for the 2009 

BRDIS

 System populates the answers if proper excel format is used

13
Use of Excel Tools for reporting the 

2010 BRDIS

• The 2010 BRDIS sample was ~ 43,000 companies

 We mailed ~ 2,600 BRDI-1 forms

• The R&D Expense section was downloaded a total of 

2,374 times with 1,402 of those from the BRDI-1 form

 Consolidator was downloaded 543 times

• Respondents complemented the spreadsheets during post 

survey interviews 

14
BRDIS Internet reporting trends

0

10

20

30

40

50

60

70

80

90

100

2007* 2008 2009 2010**

% of all responses 

(unweighted)

% of BRDI-1 form 

responses (unweighted)

% U.S. R&D reported 

(weighted)

15

* The 2007 SIRD electronic instrument was not web based

** The 2010 data are preliminary at this point
Web Survey = Fewer Observed 

Reporting Errors

• Fewer balancing errors from online reports

• Fewer “path-related” errors from online reports (skip 

patterns are automated)

• Online reports are more “complete” than paper 

reports (significantly less imputed data from online 

reports)

16
Testing the Impact of Web Reporting

• Generalized Ordered Regression Model (Adua and 

Sharp 2010) (Hough, Keller, Wilkinson, Curcio 2011)

• Dependent variable coded at three levels

 Level 1 = no imputation

 Level 2 = < 15% of data items imputed

 Level 3 = > 15% of data items imputed 

• Web reporters were more than 3X more likely to have 

zero imputations than paper 

17
Future Considerations

• 2011 Long form (BRDI-1) respondents will be 

directed to report on line. No form in initial mailing.

• Explore ways to integrate more context-based 

instructions, edits, and guidance into the web survey 

and electronic tools

As noted earlier, not all content applies to all respondents 

(industry-specific items, foreign-owned companies, govt. 

contractors, etc.)

18
Contact Information

Richard Hough, Chief

Research, Development and Innovation Surveys Branch

U.S. Census Bureau

(301) 763-4823

richard.s.hough@census.gov

19
Testing the Impact of Web Reporting 

Regression Results

N = 1,092,819 (23,782) 

Estimate Std Error Prob > ChiSq Log Odds

Intercept[1] -6.46 0.0207 <.0001

Intercept[2] 3.49 0.0070 <.0001

Web Reporter 1.28 0.0161 <.0001 0.2788

Account Manager Company 1.05 0.2218 <.0001 0.3489

Manufacturing -0.19 0.0193 <.0001 1.215

Funded R&D -2.34 0.0285 <.0001 10.405

Outsourced R&D -0.89 0.0329 <.0001 2.4269

Company reports to the ACES 0.26 0.072 0.0003 0.7708

Public Owned Companies -0.26 0.1060 0.0141 1.2970

Foreign Owned Companies -0.65 0.0983 <.0001 1.9070

20
Use of Excel Tools for reporting the 

2010 BRDIS

21

Total Downloads All Form Types BRDI - 1

Section 1 4,010 2,342

Section 2 2,374 1,402

Section 3 1,573 862

Section 4 1,536 843

Section 5 1,550 849

Section 6 1,575 850

Section 7 1,509 803

Consolidator 543

Total 14,670 7,951