# Presentation: 399347

## From Start to Pilot: Comprehensive Redesign of a Questionnaire Measuring Foreign Direct Investment 

- Alfred D. Tuttle
**Rebecca L. Morrison**
- United States Census Bureau
- David H. Galler
- United States Bureau of
- Economic Analysis

## Overview – Redesign of BEA Form BE-605 

- Reimbursable project with BEA
- Multiple, intensive stages of research, design, testing, revision
- Qualitative and quantitative evaluations
- New BE-605 form mailed out in March 2007

## Outline

- Background – BEA Form BE-605
- Summary of stages of research
- Findings
- Next steps

## Background

- BE-605 (quarterly) and BE-15 (annual)
- Foreign Direct Investment (FDI) – _>_10% of voting rights in an incorporated entity (or equivalent interest in unincorporated entity)
- Sample – foreign-owned businesses   in US

## Research stages and methods

- Background investigations
- Expert review of form
- Focus groups with survey analysts
- Observations of callbacks
- Respondent debriefings

## Research stages and methods

- Cognitive testing of mockups

## Research stages and methods

- Pilot test
- Respondent debriefings
- Evaluation questions

## Findings from background research

- Data collection and design issues:
- Dense formatting
- Economic accounting concepts differ from US GAAP
- Separate instructions

## Recommendations

- Formatting recommendations
- Letter size pages (not legal)
- More “open” visual design
- Incorporate instructions into form:
- within questions
- adjacent to questions on facing page
- key instructions become questions
- Diagrams to augment questions

## Original form

## Recommendation: More “open” design

## Recommendation: Embedded instructions

## Recommendation: Adjacent instructions

## Recommendations: Questions from instructions, and diagrams

## Cognitive testing

- Problematic sections mocked-up and tested (not entire forms)
- 5 rounds of cognitive interviews
- Iterative process:
- mock-ups revised between rounds
- design features modified & re-tested

## Findings from cognitive testing

- New design features were effective:
- Majority of respondents liked the more open format
- Embedded instructions would likely improve data quality
- Some *“Instructions****Questions”* and diagrams required extensive modification, though most did not

## Developing 605 pilot

- Redesigned entire BE-605 form
- Close collaboration with BEA survey staff

## Pilot test

- Mailed 2nd quarter 2006
- Sent to sub-sample (n = 653; survey sample ~4,000)
- Completing pilot form optional
- Response rate: 53% (350 returned forms) during initial 8-week processing period

## Pilot form evaluation

- Respondent debriefings
- Evaluation questions in form

## Findings – R debriefings

- Favorable impressions of new form:
- More pleasing to the eye
- White reporting spaces easier to see against colored background
- Larger font, bold text, more space
- Letter size pages easier to handle
- Diagrams were helpful
- Embedded instructions helpful

## Findings – R debriefings

- Drawbacks:
- More pages
- Booklet format makes it harder to fax
- Took longer to complete than original form (disruption of reporting routine)

## Limitations

- Stymied by experienced Rs, reporting routines...
- ...though even experienced Rs were helped by new formatting and instructions.

## Findings – Evaluation questions

- Four questions appended to survey
- Each item received between 279 and 290 responses

## Findings – Evaluation questions

- Instruction placement?
- 	79.3% - Near questions
- 	18.3% - In separate booklet
- 	2.4%   -	Not applicable – I haven’t completed the 		usual BE-605

## Findings – Evaluation questions

- Open space and number of pages?
- 	48.7% - less open space, fewer pages
- 	47% - more open space, more pages
- 	4.3%  -  Not applicable – I haven’t completed the 		       usual BE-605

## Findings – Evaluation questions

- Use of diagrams?
- 	65.4% - very helpful or somewhat 			helpful
- 	11.7% - only a little helpful
- 	18% - did not need them
- 	3.9% - confusing

## Findings – Evaluation questions

- Easy/difficult compared to usual form?
- 	50.5% - harder
- 	42.5% - easier
- 	7%	-  Not applicable – I haven’t completed the 		   usual BE-605

## Findings – Evaluation questions

- Instructions preferred near questions
- Organization charts helpful
- Easier or more difficult?
- Preference for more/fewer pages?

## Summary

- Redesigned BE-605 went into production in March 2007
- Recommended analysis of edit checks during form processing

## Summary

- Multi-method approach to comprehensive survey redevelopment

## Thanks!

- Alfred D. Tuttle
- alfred.d.tuttle@census.gov
- Rebecca L. Morrison
- rebecca.l.morrison@census.gov
- David Galler
- david.galler@bea.gov