BUILDING USABILITY INTO ELECTRONIC 

DATA-COLLECTION FORMS FOR 

ECONOMIC CENSUSES AND SURVEYS[1] 

Elizabeth D. Murphy, Elizabeth M. Nichols, Amy E. Anderson, 

M. Diane Harley, and Kimberly D. Pressley 

U.S. Census Bureau, Washington, DC 20233 

Abstract 

Because the economic census[2] is the primary source of information about the Nation’s economy, collecting detailed economic data from business

 establishments is one of the Census Bureau’s key activities. In planning for the 2002 Economic Census, the Census Bureau intends to offer all 

respondents an electronic alternative to paper forms. Providing an electronic alternative requires translation of over 650 paper formats into

 Computerized Self-Administered Questionnaires (CSAQs). Because of the differences between paper and the electronic medium, and because of

 the capabilities made available by the electronic technology, the translation process is not simply a one-to-one mapping. Consideration must be

 given to the cognitive and usability issues associated with electronic questionnaires to ensure that respondents who choose to report by CSAQ find

 it to be an effective and efficient means of providing their data. As an aid to developers, the Census Bureau developed a style guide documenting

 numerous design decisions. This paper describes the process of developing the user-interface style guide with input from cognitive and usability

 testing. Informed by user-interface design principles, standards, best practices, and the results of low-fidelity-prototype testing, a style guide of this

 kind is a tool for building usability into electronic forms for data collection. 

Keywords: usability, style guide, Computerized Self-Administered Questionnaires, cognitive testing, usability testing 

Introduction 

The economic census provides a detailed portrait of the Nation's economy once every five years. Data collected in the 

Economic Census are essential for government, business, and the public[3]. For example, business establishments use

 Census data to make business decisions and to evaluate their performance. Trade associations and news media study

 economic indicators and projections based on Census data. Legislators at local, state, and national levels use Census

 data in the preparation and evaluation of new laws. Consultants and researchers use Census data to analyze changes in

 industrial structure and location. To meet these many and diverse needs, the data collected must be as complete and

 accurate as possible. 

In previous economic censuses, the Census Bureau has mailed out paper questionnaires to collect economic data from

 business establishments. In 1997, an electronic form was offered to a selection of large companies in the retail sector.

 According to plans for 2002, the Census Bureau intends to offer all business respondents the option of reporting their

 data on paper or by electronic means. The electronic forms are called Computerized Self-Administered Questionnaires

 (CSAQs). Electronic technology offers the opportunity to streamline the process of collecting data. However, any

 change in data-collecting techniques must result from a detailed design-and-development process that includes user

 testing. 

There are more than 650 versions of the economic census form, each customized to particular industries. To convert the 

paper forms into a set of consistent electronic questionnaires, developers need detailed specifications on such topics as

 font style and layout. These specifications must cover the hundreds of design decisions that would otherwise need to be

 made “on the fly” by individual developers. A style guide documents design rules to a level of detail that ensures

 consistency across developers. Since the universe of possible design rules is enormous, the design team must consider

 and rule out many options before reaching consensus on the content of a style guide. 

Development of the 2002 Economic Census Style Guide 

The purpose of a style guide is to foster consistency across forms and across the various programmers who may be

 working on different pieces of a software development project, in this case, a CSAQ. To ensure quality and consistency 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 in the design of more than 650 electronic forms for the economic census, a cross-divisional team was assembled to

 develop a style guide for Windows-based, economic CSAQs. Working in a top-down fashion, the team determined that

 the style guide should include design rules for layout and screen design, navigation, graphics, error handling, help, 

feedback from users, and other internal feedback mechanisms. 

Once the initial set of topics to be covered had been identified and grouped, members of the team signed up to develop

 various sections of the style guide. The team met weekly to discuss and review the draft content for the various

 sections. Content evolved gradually as team members, individually and collectively, consulted existing usability

 principles, user-interface design standards and best practices in creating design rules and sample screens. The project's

 developer advised the team on alternative approaches that were technically feasible. In some instances, cognitive

 testing and usability testing were conducted to help the team decide between alternative design rules. 

Usability Principles 

By definition, principles are stated at a high level of abstraction and offer general, top-down guidance. The style guide

 team consulted the following kinds of usability principles, which were collected from two major sources of expertise in

 usability: Nielsen (1993) and Shneiderman (1998): 

• Ease of navigation through the user interface: Users should be able to move easily from their point of origin 

to their destination. They should not have to work their way down menu hierarchies and back up again to 

get to where they want to be. Users who need to complete forms for multiple units should be able to use 

shortcuts, such as alternative keyboard commands, to navigate quickly within the user interface. 

• Ease of training/learning: The user interface should reflect an understanding of the user's thought process 

and the logical flow of the user's work. Required training should not be lengthy. 

• Consistency of "look and feel" within and across applications: Consistency is the key to building usability 

into an automated system. Everything from terminology to system feedback should be phrased and located 

consistently from screen to screen. System behavior should be consistent from one instance to the next. 

Consistency in look and feel allows the user to build up expectations with the reasonable hope that they will 

be met. Consistency within and across electronic questionnaires allows the user to build expertise quickly. 

Although principles define general criteria for a design, developers need much greater specificity about the attributes of

 design elements. Thus, the style-guide team consulted more detailed user-interface guidelines and "best practices," e.g.,

 conventions used by other CSAQ-development teams. This continued the top-down approach to identification of topics

 and style-guide content. 

User-Interface Design Guidelines 

In the absence of any standards for the design of electronic questionnaires, the team consulted available user-interface

 design guidelines (e.g., Compuware, 1998, Microsoft, 1995). The team also consulted an internal Census Bureau report

 on research into user-interface guidelines. The guidelines and the research report helped the team identify topics to

 cover, provided a general overview of approaches to the design and layout of visual elements, and gave ranges of

 specific guidance on some design elements. 

Best Practices in User-Interface Design 

Although the literature on human-computer interaction was the primary source of best practices, the team often referred

 to design techniques used successfully in other Census Bureau economic CSAQs, such as the Annual Survey of 

Manufactures (ASM), the Company Organizational Survey (COS)[4], and the Quarterly Financial Report (QFR). For

example, the COS uses what we have termed a "stacked design" in formatting the questions on status, payroll, and 

employment (Figure 1): 
 

 

 

 

 

 

 

 

 

 

 

 

 Status of establishment...

 Employees during pay period...

 Payroll for first quarter...

 Total annual payroll...

 Figure 1. Example of a stacked format as an

 alternative to horizontal strings of items; recommended

 to fit all items within the viewable screen space. 

Using a stacked format deviates from the horizontal layout used on the paper form. However, this format eliminates the

 need for the respondent to scroll to the right. Because respondents are likely to miss the subtle cues for horizontal

 scrolling, they are likely to omit items that are not completely visible within the computer screen’s display area. Thus,

 the stacked format is preferable for the electronic form. 

The error-handling mechanism implemented in the QFR is another design element adopted for the 2002 Economic

 Census Style Guide. In the QFR, an icon consisting of a red circle with a white X was used successfully to identify edit

 failures (i.e., answers that are out of predefined ranges or inconsistent with prior answers). As in QFR, this icon will

 appear to the right of an edit-failing data field in the electronic forms for the 2002 Economic Census. Clicking on the

 icon will open a dialog window containing the edit message, which will provide at least an explanation of the problem.

 The icon will disappear once the respondent deals with the edit failure. 

In developing wording for edit messages, the team referred to wording that had been used for ASM, COS, and QFR

 edits as well as the 1997 Economic Census. In those CSAQs, some messages were preceded by the word "Warning" or

 "Error," depending on the severity of the discrepancy. Based on recommendations in the literature, an early team

 decision for the 2002 Style Guide was not to use the word "error" in any message because of its negative implications

 (e.g., Shneiderman, 1998). The general feeling was that a respondent's data can be correct even if certain entries

 technically fail an edit check. 

Giving the respondent a Remarks section to explain anything that fails an edit check is expected to help the Census

 Bureau understand the respondent's rationale for not changing values in response to edit messages. In contrast to the

 paper form, where respondents often write explanations right beside an entry instead of going to the Remarks section, 

respondents in the electronic mode can enter explanations only in the Remarks section. 

Cognitive and Usability Testing in the Laboratory 

To resolve some user-interface design issues, the style-guide team needed further guidance beyond the available

 literature and best practices. In those cases, alternative user-interface designs were prototyped and submitted to

 cognitive and usability testing. Results helped to further define the contents of the style guide. For example, members

 of the team tested proposed wording of edit messages for the electronic forms. 

Method: We conducted ten concurrent, cognitive, think-aloud laboratory interviews to test respondent comprehension,

 task difficulty, and sensitivity of proposed edits for the 2002 Economic Census electronic forms. Using paper 

prototypes, we also performed preliminary usability testing on the presentation of the edits. Test participants were

 members of the Census Bureau's economic staff. 

We gave participants a partially completed paper questionnaire mimicking an on-line mortgage application. We asked

 participants to pretend that they were reviewing their own application. Participants read through the questionnaire 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 aloud. 

In order to “fail the edits,” some pre-filled answers did not make sense. For example, one question on the mortgage 

form asked respondents to indicate the percentage of their down payments and closing costs that would come from

 several types of funds (e.g., savings, investments). The pre-filled data totaled 110%, thus tripping the edit message, 

"Sum of down payment percentages is 110%. It should be 100%. If values are correct, explain discrepancy in the

 remarks section." We tested several types of edit messages. 

Scattered through the questionnaire were three questions for which participants were to receive preventive edits. That

 is, the edit message for each of these questions was to be presented immediately instead of later, at the end of the

 session. Immediately after the participant read each of the three questions designated for preventive edits, the test

 administrator explained how those questions were to be answered, and the edits were presented on paper. Once

 participants completed reading the entire questionnaire, we gave them two sheets of paper with twelve edit messages

 relating to the responses they had just read. The participant's task was to find the item referenced by an edit message,

 and then to correct the answer to that item or go to the Remarks section and explain why the answer was correct. 

In evaluating the edit messages, we needed to determine the importance of and placement of three attributes in the

 message: item number, item topic, and actual response. When the item topic was present, we varied its capitalization.

 For example, for several of the variable names, we used the following capitalization styles: TOTAL HOME COST,

 Homeowner's Fee, and found a new home. That is, variable names were given in all caps, initial caps, or no caps. We

 found that the following confusing edit message was the most confusing to participants: "Since you reported found a

 new home in item 15, please complete signed a sales contract in item 16," where "found a new home" and "signed a

 sales contract" were actually variable names inserted into the sentence. Deciphering this sentence was especially

 burdensome to respondents. 

Results and Recommendations: We developed a set of general recommendations on the content and style of edit

 messages. These recommendations may apply in other contexts beyond the 2002 Economic Style Guide. The following

 examples are selected from the full set of recommendations (Nichols, Murphy, & Anderson, 2001): 

• Provide a location, a description of the problem, and an action to take. Several error messages were written 

with key pieces of information missing, such as the location of the problem or what action was needed to 

resolve that particular error. Respondents reacted by spending extra time determining the location of the error 

and interpreting their course of action. Participants commented that working with the error messages would 

have been easier if these key pieces of information had been included. When the location of the problematic 

question was not readily available, respondents used other pieces of information within the edit to help give 

them "clues" as to where the problem was. The most readily used clues were subject-matter content that was 

inserted into the message, or the actual amounts reported in answer to the troublesome questions. 

• Beware of mismatches between language used in the messages and the questions. The subject-matter 

content that was provided in the error messages was intended to match back to the content in the actual 

questions. For testing purposes, the subject-matter content that was provided in the error messages was slightly 

altered to see how participants would react. When respondents encountered this mismatch, they reported 

"thinking twice" about whether they were working with the correct question. This mismatch was another 

obstacle for the respondent. 

• Avoid jargon. Avoiding the use of jargon is a standard usability recommendation when designing a computer 

application. This rule also extends to the wording that is used in edit messages. When presented with jargon, 

respondents noted that they were unfamiliar with its meaning. 

• Be polite. Instead of phrasing the messages as commands, the word "please" was used to introduce instructions 

for respondents. It was intended to convey a pleasant tone and to reduce any negative perception of the Census 

Bureau as an imperative authority. 
 

 

 

 

 

 

 

• Be sensitive to the respondent. Certain words that are synonymous with error messages can be very judgmental 

and imply that the user is wrong. Words typically associated with error messages that should be avoided are 

"error", "discrepancy", and "mistake." 

• Beware of offering only one of many possible solutions. Several of the messages used in testing included 

reminders such as "please check for typing mistakes." These messages were meant to help the respondent 

figure out what could have gone wrong. Participants spent more time than necessary focusing on these

reminders. The reminders seemed to be another hurdle that the respondents encountered in working with error 

messages. Reminders often led respondents down the garden path and away from thinking about the real

problem. 

• Capitalize variable names. Variable names referring to subject-matter content were used in many of the error 

messages. To the respondent, the variable name was a very important part of the error message, especially 

when it was the only piece of information given in the message. Participants noted that they preferred seeing 

the variable name presented in all uppercase because that style made this key piece of information stand out 

more. 

• Keep messages brief. Participants noted that lengthy messages were more difficult to deal with than shorter 

messages that seemed to "get to the point" quicker. It is important to ensure that messages contain all of the 

necessary pieces of information in a succinct way. 

• Use good grammar. Participants stumbled when awkward grammar was used in a few error messages. The 

difficulties resulting from insertion of variable names that did not "read" in context suggest that variable names 

encoded in the metadata need to make sense in any context in which they are inserted for the respondent. 

• Present edit messages as soon as possible. Several respondents felt added burden when presented with their 

mistakes at the very end rather than at the point where the mistake had occurred. They would rather have dealt 

with any errors that occurred at the time and place where they occurred. 

• Format data-entry fields to prevent errors. Participants noted that an edit could have been very easily 

avoided if the format that the application required for an answer was presented as part of the actual question. 

This convention is widely used in several computer applications and on web pages on the Internet. 

Based on test participants' performance and verbal feedback, we also developed recommendations for wording the

 various kinds of edit messages. The following table provides examples of the wording that was tested and the generic

 re-wording recommended after testing (Nichols, Murphy, & Anderson, 2001): 

Table 1. Examples of Tested Wording and Recommended Wording 

Tested Wording Recommended Wording

Homeowner's Fee is $1,200. Typical values are 

between $25 and $300. Please check for typing 

mistakes. If entries are correct, explain 

discrepancy in the remarks section. 

In Item X, typical values are between A and B. 

You reported C. If this value is correct, please 

explain in the remarks section. 

Since you reported found a new home in item 15, 

please complete signed a sales contract in item 

16. 

Please complete Item X, since you completed 

Item Y. 

Sum of down payment percentages is 110%. It 

should be 100%. If values are correct, explain 

discrepancy in the remarks section. 

In Item X, the values you reported add up to A%. 

The total should be 100%. 
 

 

 

 

 

 

 

 

 

 

 

The recommended, generic wording for the various types of edit messages is documented in the style guide. 

Review of Form Prototypes 

As prototypes were developed for the paper versions of the economic forms, a programmer built corresponding

 prototypes of the electronic form. These electronic prototypes were revised according to style-guide team consensus.

 Decisions made during team discussions were documented for inclusion in the style guide. In some cases, situations

 arising from the prototypes called for topics to be added to the style guide (e.g., retention of column headers,

 indentation, presentation of running totals). This bottom-up approach complemented the top-down approach taken

 earlier in the evolution of the style guide. 

As necessary, the team called in subject-matter experts from the various economic areas (e.g., Retail, Manufacturing) to

 assist in making decisions about necessary departures from the design of the paper forms. For example, to avoid

 horizontal scrolling, the grid design of the paper form for the manufacturing survey needed to be streamlined to fit the

 limited width of the computer screen. A representative from the manufacturing subject-matter area attended style-guide

 team meetings and participated in developing several options that departed from the corresponding paper form. One

 option was to use the stacked design that has been used successfully by the COS (Figure 1). Other manufacturing 

subject-matter experts agreed to this design change, and their representative brought this decision back to the styleguide team. Rather than risk loss of data, the electronic form was re-designed to present all the items within the 

horizontal display area of the screen. 

Cognitive and Usability Testing in the Field 

In the second round of user testing, a partially functioning electronic prototype of the retail form was administered to

 participants at several companies in the Washington, DC Metropolitan Area. Accompanied by one or two other team

 members, a survey researcher visited business locations and administered the prototype questionnaire to participants.

 The prototype CSAQ was administered to participants using a laptop from the Census Bureau, i.e., it was not loaded 

onto the participants' own personal computers (PCs) at their worksites. During each test session, the laptop's screen

 was videotaped to record the participant's actual navigational path through the prototype CSAQ. 

Researchers wanted to determine whether the electronic format would support the respondent in comprehending what

 data to provide and where to enter data. Of interest was the respondent's use of navigational features. As shown in 

Figure 2, three navigational features were embedded in the prototype: a set of menu options in the left navigation bar,

 “next” and “previous” navigational buttons, and hyperlinks for skip instructions and data verification. 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 2. Prototype of electronic form for the 2002 Economic Census showing a hyperlinked skip instruction (...go to

 29), hyperlinked menu options in the left navigation bar, and navigational buttons at the lower right. 

Participants in the testing used all three of these navigational features appropriately. As expected, the hyperlinked skip 

instructions were the least likely to be used, and the next and previous buttons were most likely to be used. 

Another issue of interest in testing was the method provided for respondents to select answers. Experience in QFR

 usability testing had indicated that combining radio-button functionality with tabbing can have serious, undesired

 consequences. For example, tabbing from one question to the next can cause the respondent's selected radio button to

 jump from "Yes" to "No" without the respondent noticing this change. Thus, for "select one" choices, the style-guide

 team is considering checkboxes instead of radio buttons. 

The difficulty with checkboxes is that multiple responses can be selected. In test situations, participants have quickly

 discovered that they can select more than one option. As of this writing, all test participants have noticed the

 instruction to select only one response and have assumed initially that the computer will allow them to check only one.

 To change a response, however, participants have found that they need to perform the additional step of unchecking a

 box to proceed. One participant assumed that this "feature" would be corrected in the final form. The team has 

documented the desired behavior of checkboxes and other user-interface elements in the style guide. 

Summary 

Development of the 2000 Economic Census Style Guide drew upon several different but related methods. Consulting

 usability principles and user-interface design guidelines got the team started in a top-down direction, but it was

 necessary to test alternatives and use bottom-up approaches in formulating design rules at the required level of 

specificity. Past results (what worked and what did not work for participants) from usability testing of other CSAQ's 

was valuable in informing team decisions. Cognitive and usability testing were useful in providing human performance

 data and observations on which to base decisions. Once the style guide is complete, for designers will use it in

 conjunction with forms-design software to design the CSAQs for the 2002 Economic Census. Revision of the Style

 Guide for future use can occur as necessary now that the foundation has been laid. 

References 

Compuware, Inc. (1998). GUI Guidelines (Help template). Madison, WI: Author. 

Microsoft, Inc. (1995). The Windows Interface Guidelines for Software Design. Redmond, WA: Author. 

Nichols, E., Murphy, B., and Anderson, A. E. (2001). Results from Cognitive and Usability Testing of Edit Messages for the 2002 Economic

 Census (Human-Computer Interaction Memorandum Series #39). Washington, DC: U. S. Census Bureau. 

Nielsen, J. (1993). Usability Engineering. New York: AP Professional/Academic Press. 

Shneiderman, B. (1998). Designing the User Interface (3rd ed.). Reading, MA: Addison-Wesley. 

[1] This paper reports the results of research and analysis undertaken by Census Bureau staff. It has undergone a Census Bureau review more

limited in scope than that given to official Census Bureau publications. This report is released to inform interested parties of ongoing research and

to encourage discussion of work in progress. The authors are grateful to Kent Marquis, Erica Olmsted, and Diane Willimack for their constructive

reviews.

[2] The economic census is mandated by law under Title 13 of the United States Code (sections 131, 191, and 224).

[3] http://www.census.gov/epcd/www/econ97.html

[4] Officially known as the Report of Organization Survey.