Usability Issues Associated with Converting Establishment Surveys to 

Web-Based Data Collection 

Jean E. Fox 

William Mockovak 

Sylvia K. Fisher 

Christine Rho 

Office of Survey Methods Research, Bureau of Labor Statistics 

1. Introduction 

In an effort to combat non-response, survey managers continually seek new ways to encourage 

respondents to participate in their surveys. One approach is to offer respondents the option of selecting 

from multiple reporting modes so that they can select the mode they prefer. The Internet is one of the 

newest modes available and offers a variety of benefits. For example, respondents can access the 

Internet easily from their desktop PCs, so they can complete the survey at their convenience. Properly 

designed surveys can introduce instructions, edits, and help screens that simplify the respondents’ task 

by guiding them through the completion process. From a survey manager’s point of view, the Internet 

eliminates or reduces data entry costs, because respondents enter data themselves. Further, Web surveys 

can check data as the respondent works, so the need for follow-up phone calls or post-data collection 

processes is minimized. With these obvious benefits, the Internet offers the potential for enhancing 

response rates, improving data quality, and improving timeliness of reporting. In addition, the potential 

for cost savings also exists, although in some cases offering an additional data collection mode might 

actually increase costs. 

On the other hand, there are some possible drawbacks to Web data collection. One major problem is 

that the use of multiple data-collection modes complicates data integration and survey operations, such 

as follow-up efforts. Further, developing Web surveys can actually raise up-front costs. The cost of 

building, maintaining, and integrating different systems is expensive. Moreover, preliminary research 

with Web surveys indicates that rather than enhancing response rates, offering multiple modes can 

actually depress overall response rates (Griffin et al. 2001). For a detailed discussion of these and other 

problems, see Fricker and Schonlau (2002). 

In establishment surveys, the Internet is likely to be one of several reporting options that may include 

mail, phone, and fax. While respondents may select another reporting mode if a Web survey is too 

difficult to complete, they may also decide not to report at all. Further, since respondents often 

participate in more than one government survey, a respondent may generalize from a negative 

experience on one Web survey to others, even though different agencies may be involved. Therefore, to 

encourage participation, survey managers need to design Web surveys that will provide as positive an 

experience as possible for the widest range of respondents. A key element of that design is ensuring the 

usability of Web surveys. 

This paper focuses on the usability of dedicated Web-based government surveys, where usability is 

defined as the effectiveness, efficiency, and satisfaction experienced by respondents as they provide the 

requested survey data. At BLS, we are dedicated to developing usable Web surveys. This paper 

describes our experiences and lessons learned in designing Web surveys for establishments. 
2. Usability Issues in Web Survey Design 

 

As with any new technology, early attempts to develop Web surveys have relied largely on existing 

conventions for Web design, coupled with research on designing surveys for other modes, and the 

personal preferences of designers. This heuristic approach is understandable, because research 

regarding the design of large government Web-based surveys is still limited. However, after some 

experience at BLS, we have identified several important issues related to the usability of Web surveys. 

Following are some of the design considerations and constraints that we believe federal survey managers 

should be aware of when considering the use of Web-based surveys. 

2.1 Importance of Standardization across Surveys 

Many government agencies conduct numerous establishment surveys, which means that in some cases, 

the same establishment (and respondent) responds to more than one survey. From a respondent’s 

perspective, it is logical to expect that the look and feel of all Web surveys from the same agency will be 

similar. To accommodate respondents and allow for adequate security, the Bureau of Labor Statistics 

offers a common portal or gateway into its data collection Website, called the “Internet Data Collection 

Facility” (IDCF). 

In addition to a common gateway, the IDCF requires that all surveys meet internal standards for user 

interfaces.1

 One of the challenges of applying these standards was that the early adopters (i.e., surveys 

introducing Web collection first) were designing their Web survey as the standards were being 

developed. Therefore, these survey managers had the extra responsibility of providing input to 

determine appropriate standards. On the other hand, later adopters were faced with some established 

standards that were not quite appropriate for their purposes. Once standards are in place, they are often 

difficult and costly to change. At BLS, we are just beginning the process of reviewing our standards. 

We expect that support for changes will come from research, from respondents, and from requests made 

by survey managers using Web-based data collection. 

2.2 Consistency across Survey Data-Collection Modes 

Research has found that different modes of data collection for identical content can produce different 

results (e.g., Dillman, 2000; Dillman et al., in press). As noted by Couper2, design of Web surveys is 

important because they are self-administered, interactive, visual, potentially multimedia, and are 

distributed over a wide variety of hardware and software systems. This last characteristic is especially 

important because the most carefully laid out design can appear quite different depending on the 

respondent’s hardware and software configurations. 

Therefore, if a survey uses multiple data collection modes, survey managers need to ensure that 

comparable data are being collected using the different modes. Since federal establishment surveys deal 

largely with reports of factual information, some survey managers may discount research findings on 

multi-modal differences, because these studies have dealt primarily with attitude questions or question 

formats not typically used in establishment surveys. However, caution is warranted. Assuming that 

different data collection modes do not affect the reporting or accuracy of establishment data may be a 

questionable hypothesis until the necessary research is done. 

1 GUI and HTML Standards. Internal Bureau of Labor Statistics document. 2

 Workbook for JPSM seminar in Web Survey Design, February 18-19, 2003. 
2.2.1 Creating a Unique Design for the Web vs. Reproducing the Paper Form 

Some survey managers make the immediate assumption that the best Web design when converting a 

paper form to the Web is one that simply adopts an electronic copy of the paper form already in use. 

The argument for this approach is that respondents who are already familiar with the paper form will 

transfer their knowledge of the paper form to the Web version of the form and, therefore, have little 

difficulty completing the Web version. Also, it may be tempting to believe that using an electronic copy 

of the paper form will result in similar data collection results across all collection modes. However, as 

mentioned above, the representation of the form may be affected by the respondent’s hardware and 

software configurations. At a minimum, a computer screen and a piece of paper are very different types 

of displays and may require different types of behaviors from the respondent. 

The “direct copy” approach would seem to work best when the form is fairly simple, it can be displayed 

with little or no scrolling, and screen display concerns have been addressed. Surveys that are longer and 

more complex often need a different interface for the Web version to avoid usability concerns. These 

surveys can also take advantage of automated skip patterns and edits to streamline the respondent’s 

effort. 

Another concern is that the direct copy approach may discourage Web reporting. If respondents are 

completing exactly the same form, they might wonder why they should expend the additional effort 

necessary to enter data on a computer, which requires the additional step of signing or logging on. 

Since the Web and paper are two different modes, they each have their own advantages, which should 

be exploited. For example, paper allows more of the survey to appear on a single page, and affords 

more flexibility in layout and formatting. The Web allows you to walk respondents through the process 

using automated skip patterns, exposing them only to the relevant parts of the survey, and also providing 

validation checks, where appropriate. Our experience at BLS has been that program managers prefer to 

start with the “direct copy” approach, but then once they see the actual product, readily make the 

transition to designs that take better advantage of the computer. 

2.3 Security and Confidentiality on the Web 

Our gateway requires identical log-on procedures for all surveys, but two security options are offered: 

(1) Personal ID Number (PIN) and password or (2) digital certificate. A digital certificate offers a 

higher degree of security, but is somewhat complicated for respondents to obtain. Initially, digital 

certificates were confusing to users, but after usability testing and a change of vendors, the process was 

simplified substantially. 

Although easier to use, the PIN & password approach also presents possible difficulties. The log-on 

information must be sent to respondents, which, in itself, presents some security concerns. Existing 

security requirements also demand the creation of a fairly complicated permanent password (it must 

meet multiple criteria) that many users are not used to, and which many find confusing. Finally, 

respondents must be able to recall permanent passwords for future access to the system. To help 

minimize confusion with temporary passwords, we have found that it helps to provide passwords that do 

not contain 0 (zero) or o (oh), or 1 (one), l (el), or I (eye), as they may be difficult to differentiate. 

Although necessary to protect respondents’ confidentiality, Web security procedures introduce an 

additional hurdle compared to other response modes. In addition to increasing respondent burden, the 

net impact of security procedures associated with Web reporting is that these gateway functions will 

increase operational demands on the surveys and require a larger support or help staff. For example, 
Web reporting for the Current Employment Statistics survey generates ‘trouble tickets’ from about 15 

percent of the sample each month, versus about 4 percent for the long-established touchtone data entry 

help desk.3

2.4 Validation Checks 

Obviously, paper forms lack any type of validation checks or edits. Therefore, one might assume that 

any editing done in a Web form would automatically result in improved data quality, as well as save 

money by reducing the number of follow-up phone calls. On the other hand, a delicate balance exists 

between the survey designer’s need for the highest possible data quality and the burden imposed on a 

respondent when trying to respond to edits. If the scale tips too far, the overuse or improper use of edits 

could lead to frustration, increased burden, and either possible premature exits from the survey or 

refusals to report in the future. What is important to keep in mind is that edits are critical to the overall 

design and should not be viewed an afterthought to be dealt with as a last step in the design process. 

Although the use of some edits may seem perfectly justified, another issue concerns their enforceability. 

Surveys use both hard and soft edits to distinguish between required and recommended changes. If a 

hard edit is triggered, respondents must address the problem to continue. On the other hand, if a soft

edit is triggered, respondents are notified that there may be a problem, but they are not required to make 

any changes. A related question regarding edits in Web surveys is when they should be used. 

Possibilities include (1) immediately after an entry is made, (2) after a table (grid) of entries is 

completed, (3) after a complete screen of entries, or (4) at the very end of a survey, when the respondent 

submits the data. Each option imposes different demands on the respondent. 

Edits can be implemented in several different ways. For example, the edit message could appear in a 

separate window (pop-up box), as text next to the entry field, or on a separate page. A common problem 

when edit messages are displayed on the same screen is that respondents may fail to see them, even 

when different color text is used. When this happens, respondents think they either failed to click a 

button properly or that the same screen has redisplayed in error, so they simply click Continue again. In 

general, it is usually better to let respondents know about problems or potential problems as soon as 

possible. However, some edits can only be run when respondents indicate that they are finished, such as 

checks for consistent data across multiple entries. 

Because there is a lack of research that addresses the general issues of how and when to use survey edits, 

there is no ideal solution at this time. However, some general guidelines may be helpful. For example, 

to be useful, edits must be noticed, read, understood, and then acted upon. Moreover, they cannot be 

overly burdensome. With these common sense goals in mind, the following general design guidelines 

are proposed: 

• Take steps to ensure that edit messages are noticed (e.g., through good screen design). 

• Use plain English (avoid jargon), and keep the explanatory message as brief as possible. 

• Give control to users. Allow them to either change the answer or leave it as is, and to move on 

when ready. 

• Consider offering a comment box, so the respondent can explain the entry. 

• Err on the side of introducing too few edits into the initial Web survey. Study the resulting data 

and then gradually introduce edits into future releases to see if data quality issues are addressed. 

 

3

 Personal communication with Richard Rosen, Program Manager for the Current Employment Statistics program. 
Admittedly, these are rough guidelines, but until definitive research is done, they provide a useful 

starting point. 

2.5 Navigating Among Survey Questions 

Any Web application requires some basic level of navigation. An important point to keep in mind is 

that respondents do not approach the task of completing a Web survey with the same expectations 

elicited by other on-line tasks, such as ordering merchandise or searching for information about a topic, 

tasks which people do because they want to (Schober, Conrad, Ehlen, and Fricker, 2003). Numerous 

observations in usability tests indicate that when respondents encounter a survey question or survey 

form, they expect a structured task, where a question is posed and they provide an answer. 

Respondents also approach tasks with behaviors and expectations acquired from previous experience on 

the Web. For example, they are likely to do things like use the Back button on the browser, click X to 

close the application, click on underlined words for additional information, or try to tab among multiple 

answer fields. Moreover, many respondents will know how to use radio buttons and check boxes for 

choosing answers. Therefore, a Web survey designer must expect that respondents are likely to engage 

in expected, conventional behaviors when completing a Web survey. If the interface responds 

differently, then respondent confusion is likely, which could lead to error. 

In terms of navigating a survey questionnaire or form, at a minimum the respondent should be able to 

move from one question or answer field to another, either automatically (cursor is controlled by the 

survey) or with the use of a mouse or tab key; leave (exit) or be able to close the survey before it is 

completed and return to the same point; back up to a previously answered question, and either review or 

change the answer; access question-specific help and return to the same point in the survey; and access a 

survey home page from within the survey as a navigational anchor. 

2.6 Section 508 Requirements 

A critical issue in the development of government Web surveys is the need to meet Section 508 

requirements. Section 508 is a federal law that requires electronic and information technology 

(including Web sites) that are developed, procured, maintained or used by the federal government to be 

accessible to people with disabilities. This law also applies to self-administered, Web-based surveys, 

where a major concern is the accessibility by visually challenged respondents. Some of the most 

important guidelines are the following:4

 

• images must have equivalent ALT text (text assigned to a graphic, which screen readers can 

interpret) 

• color and stylesheets must not be mandatory to view the site, 

• data tables must be properly coded with headers, 

• frames should be named properly, 

• alternatives must be provided for scripting languages, 

• if plug-ins are used, a link to the download page must be included, 

• forms should be coded properly and logically, and 

• repetitive navigation should be coded so that it can be skipped. 

 

4

 See http://www.webaim.org/standards/508/checklist for a Section 508 checklist. 
2.7 Other Screen Design Principles and Issues 

There are a variety of other screen design issues that affect the usability of Web surveys. Although there 

are many sources of guidelines for designing web applications, very few focus on surveys. Dillman and 

Bowker (2001) is one of the few sources that specifically address web surveys. They propose 14 

principles for guiding Web survey design and addressing four major sources of survey error. 

Unfortunately, many issues specific to web surveys have not yet been resolved. For example, designers 

of federal surveys may face the following questions: 

• Which is better, a page-by-page approach or a single scrolling page? 

• Which labels should be used for navigation buttons (e.g., “Submit” vs “Save” vs “Continue”)? 

• What is the optimal way to present on-screen instructions or validation checks? 

• How should “help” functions be presented? 

At present, the best we can do for many of these questions is to follow general web design practices. 

Sources for helpful information include the Research-Based Web Design and Usability Guidelines 

(available online at www.usability.gov), Shaping Web Usability by Albert N. Badre (2002), and 

Designing Web Usability by Jakob Nielsen. Other useful sources include Nielsen and Tahir (2002), 

Spool, Scanlon, Snyder, DeAngelo, and Schroeder (1998). There are many other good books on 

designing and building web pages as well. We have included some general web design guidelines 

relevant to web surveys in Appendix A. 

3. Incorporating Usability into Your Development Culture 

Many large survey development projects follow the “waterfall lifecycle,” where one stage of 

development follows the previous, with little or no feedback to repair problems in earlier stages (Royce, 

1970). For example the development team would (1) prepare the specifications, (2) design the survey 

and associated database, (3) build the system and program the survey, (4) test it, and (5) deploy it. 

Fraser (2002) adds that in newer incarnations of the waterfall model, the development team would have 

a contractor conduct a usability test just before deployment to ensure that user requirements were met. 

If you follow this approach, you will probably find that the resulting survey may still contain numerous 

usability problems, potentially leading to inaccurate data or non-response. 

Boehm (1988) noted that the waterfall lifecycle model is inflexible and risky. He found that the cost of 

making changes increases exponentially from one stage to the next, and, the likelihood of having major 

problems is very high because there are few opportunities to fix problems as they occur. Fraser (2002) 

added that the waterfall approach to usability testing leads to a poor user interface and reliance on 

outsiders. Since production schedules will often not permit major changes late in development, a 

common strategy using the waterfall approach is to make the easy, more cosmetic changes in the current 

release, but to save the major changes for the next release. Unfortunately, this list often grows rapidly as 

users find more and more problems. 

In response, Boehm (1988) created the “Spiral” lifecycle model. In this model, development starts 

small, at the “center of the circle.” Each cycle around the center consists of analysis, development, and 

testing. Each cycle, or iteration, includes more and more functionality, building on previous iterations. 
This method not only helps developers find problems earlier, but also makes it possible to incorporate 

feedback from users before development is nearly final. 

One process often utilized as a way to focus on the users is known as user-centered design (Norman and 

Draper, 1986). Within BLS, survey programs are strongly encouraged to implement a user-centered 

design approach (Fox, 2001; Mockovak and Fox, 2002). Simply defined, this process ensures that the 

needs of the users are incorporated into the design of the software from the beginning and throughout 

the design process. User-center design activities can be easily incorporated into the development, 

testing, and feedback cycles of the spiral model. 

Gould (1988) lays out four principles for designing usable systems: 

 (1) Early and continual focus on users 

 (2) Integrated design (coordinate work on all components) 

 (3) Early and continual user testing 

 (4) Iterative Design 

It is important that the user-centered design activities be led by those who are skilled and experienced in 

the field of usability. They have to know (1) which methods are appropriate at each stage of 

development, specifically within a particular development schedule, (2) how to implement the methods, 

(3) how to interpret the results, and (4) how to present the results to the development team and 

management. Although the methods themselves may seem straightforward, these four challenges 

highlight the specialized skills that are required. For example, Nielsen and Molich (1990) created the 

“Heuristic Evaluation” as a method that could be applied by developers knowledgeable about the 

product but not necessarily about usability. Nielsen (1992) later found that the method was most 

successful when conducted by usability specialists. 

Incorporating a user-centered design approach does not have to be expensive. It is always possible to 

incorporate some level of effort within the expected development schedule. When the effort is made 

early on, it can prevent expensive problems later in development. 

Usability testing is one of the most common tools of user-centered design. Ultimately, it is highly 

desirable to test actual users, but users with roughly comparable characteristics will suffice in early 

stages of the testing. Moreover, in most situations it is not necessary to test a large number of users. For 

example, Virzi (1992) argues that five users will often be satisfactory, with diminishing returns from 

testing additional users. Members of the development team should observe the usability test, so that 

they can see firsthand the types of difficulties that users are encountering. 

The frequency of usability testing depends on the application being built. At BLS, we generally test 

Web surveys every 2-3 months or so, giving developers the time to incorporate design changes. As 

Gould (1988) suggests, we have also found that it is critical to test all parts of the system. Therefore, we 

also evaluate advance mailings, instructional brochures, or e-mails, since they will directly impact use of 

the system and the users’ experiences. 

4. Summary 

To make effective use of the Web for conducting surveys, survey managers must address a variety of 

issues, many of which are currently unresolved. This paper summarizes key issues that survey managers 

should be aware of when they design new surveys for Web collection or convert existing surveys. At 

BLS, we have found that a process called user-centered design is extremely helpful to ensure that Web 
surveys are easy for respondents to use. This paper provides some basic suggestions for incorporating a 

user-centered design approach into the development of government surveys. 

References 

Badre, A.N. (2002). Shaping Web Usability: Interaction Design in Context. New York: Addison-Wesley. 

Boehm, B. (1988). A spiral model of software development and enhancement. IEEE Computer, 21(5). 61-72. 

Dillman, D.A. (2000). Mail and Internet Surveys: The Tailored Design Method (second ed.). John Wiley & Sons. 

Dillman, D.A., and Bowker, D.K. (2001). The Web Questionnaire Challenge to Survey Methodologists. In Dimensions of 

Internet Science. Ulf-Dietrich Reips & Michael Bosnjak (Editors). Lengerich, Germany: Pabst Science Publishers. 

Dillman, D.A., Phelps, G., Tortora, R., Swift, K., Kohrell, J., and Berck, J. (in press). Response Rate and Measurement 

Differences in Mixed Mode Surveys: Using Mail, Telephone, Interactive Voice Response, and the Internet. Draft 

document available at 

http://survey.sesrc.wsu.edu/dillman/papers/Mixed%20Mode%20ppr%20_with%20Gallup_%20POQ.pdf. 

Fox, J.E. (2001). Usability methods for designing a computer-assisted data collection instrument for the CPI. Proceedings of 

the FCSM Conference, Washington, D.C. 

Fraser, J. (2002). The culture of usability: How to spend less and get more from your usability-testing program. New 

Architect. Available at http://www.newarchitectmag.com/documents/s=7457/na0802b/index.html. 

Fricker, R.D., and Schonlau, M. (2002). Advantages and disadvantages of Internet research surveys: Evidence from the 

literature. RAND document, available at http://www.schonlau.net/publication/02fieldmethods.pdf. 

Gould, J.D. (1988). How to design usable systems. In Helander, M. (Ed.), Handbook of Human-Computer Interaction. 

North-Holland: Elsevier Science Publishers, 757-789. 

Griffin, D.H., Fischer, D.P., and Morgan, M.T. (2001). Testing an internet option for the American Community Survey.” 

Paper presented at the Annual Conference of the American Association for Public Opinion Research, Montreal, Canada, 

also available at http://www.census.gov/acs/www/Downloads/ACS/Paper29.pdf. 

Mockovak, W. and Fox, J. (2002). Approaches for Incorporating User-Centered Design into CAI Development. 

Proceedings of the International Conference on Questionnaire Development, Evaluation, and Testing Methods 

(QDET), Charleston, SC. 

National Cancer Institute. (2002). Research-Based Web Design and Usability Guidelines. Available online at 

http://www.usability.gov/guidelines/index.html. 

Nielsen, J. (2000). Designing Web Usability: The Practice of Simplicity. Indianapolis: New Riders Publishing. 

Nielsen, J. (1992). Finding usability problems through heuristic evaluation. In CHI ‘92, 373-380. 

Nielsen, J. and Molich, R. (1990). Heuristic evaluation of user interfaces. In CHI ‘90, 249-256. 

Nielsen, J. and Tahir, M. (2002). Homepage Usability: 50 Websites Deconstructed. Indianapolis: New Riders Publishing. 

Norman, D.A. and Draper, S.W. (1986). User Centered System Design: New Perspectives on Human-Computer Interaction

Hillsdale, NJ: Lawrence Erlbaum Associates. 

Royce, W.W. (1970). Managing the development of large software systems: Concepts and techniques, Proceedings of IEEE 

WESTCON, Los Angeles, 1-9. 

Schober, M., Conrad, F., Ehlen, P. & Fricker, S. (2003). How Web surveys differ from other kinds of user interfaces. Paper 

presented at The 58th Annual Conference of the American Association of Public Opinion Research, Nashville, TN. 

Spool, J. Scanlon, T., Snyder, C., DeAngelo, T., Schroeder, W. (1998). Web Site Usability. Morgan Kaufmann 

Virzi, R.A. (1992). Refining the test phase of usability evaluation: How many subjects is enough? Human Factors, 42(4), 

457-468. 
Appendix A 

These are some general web design guidelines that may be useful to those designing web surveys for 

federal agencies. 

Basic Web guidelines

• Be consistent within the web site. 

• Use standard interface controls as they are supposed to be used (e.g., radio buttons for “Check 

one” and check boxes for “Check all that apply”). 

• Use a simple URL that people can remember. 

• Limit requirements on User names and passwords to keep them simple; avoid 1 (one), i (eye), 

and l (el), as well as 0 (zero) and O (big oh) and o (little oh). However, your agency may have 

specific security requirements you must follow. 

• Consider how the page will print out. Some respondents may print it, fill it out, then transcribe 

the data. 

• Provide information on privacy, confidentiality, and ADA issues. 

• Provide links to sites that show how your “products” (i.e., data) might be useful to the 

respondent. 

• Provide easy access to help (e.g., an email address or phone number). 

Navigation 

• Make it obvious what respondents should do next. 

• Label links clearly so respondents understand them. 

• Make it easy to correct mistakes. 

Layout 

• Put important information at the top, left-hand side of the page. 

• Limit the use of graphics (minimizes download time and helps meet Section 508 requirements). 

• Do not use animation, unless it helps the respondents. 

• Eliminate horizontal scrolling. 

• Minimize vertical scrolling where possible. 

Data Entry 

• Use appropriate data entry tools (e.g., radio buttons vs. check boxes). 

• Make text boxes large enough to accommodate the longest possible response (use scrolling on 

very long fields). 

• Drop-down lists can be helpful in limiting the responses to valid values. However, some 

responses may be “hidden” if respondents don’t scroll the list. 

• Label each data entry field clearly. 
• Don’t make respondents enter data twice (e.g., birthdate and age) unless you are using it to verify 

a value. 

• If respondents need responses from one question to answer another, display the questions 

together. 

Text 

• Use short, simple sentences and paragraphs. 

• Avoid jargon and acronyms, unless they are very familiar to the respondents. 

• Use well-designed headings to guide the respondents. 

• Use a sans serif font, as the serifs don’t display well. 

• Don’t make the lines of text too long, as it’s hard to find your place when going from one line to 

the next. 

Color 

• Use high contrast colors (e.g., black text on a white background). 

• Use color coding appropriately and consistently, without overusing color. 

• Do not rely on color coding as the only way to convey information.