Identifying the Intended Navigational Path of an Establishment Survey 

Cleo Redline1

National Science Foundation 

4201 Wilson Boulevard, Arlington, VA 22230, credline@nsf.gov 

A desk stapler has sixteen parts, a household iron fifteen, the simple bathtub-shower combination twenty-three. You can’t

believe these simple objects have so many parts? 

Donald A. Norman, “The Design of Everyday Things” 

Introduction

We do not often see it described this way, but a self-administered survey instrument is really a physical object with many 

parts that need to work in unison for the express purpose of collecting information. These parts can be viewed along a 

continuum from the micro to the macro. At the micro level, there is color and brightness, shape and location of information. 

At a more macro level, there are the questions, instructions, definitions, and response categories. At an even more macro 

perspective, however, there is the entire mailing package—the outgoing envelope, cover letter, questionnaire, mailback 

envelope, possibly brochures or anything else that might be sent to respondents to aid and persuade them to answer and 

return the survey. 

Making a survey instrument work, at both the micro and macro level, such that the parts are transparent to the respondents 

and the respondents can accurately and efficiently operate the survey instrument (defined as performing the tasks in the order 

respondents are instructed to perform them) is a monumental undertaking. This is especially true in the case of an 

establishment survey such as the Survey of Graduate Students and Postdoctorates in Science and Engineering (GSS) because 

of the large number of complicated parts that respondents are required to handle. 

 

 

The objectives of this paper are: (1) to synthesize the literature that informed efforts to evaluate and improve the GSS, (2) to

describe what were discovered to be the major parts of the GSS, (3) to show how the survey has been redesigned thus far and 

why, (4) and to extract the major principles that were used to redesign the survey. 

Background 

The literature is replete with design principles (e.g., Powell, 2002; Dillman, 2000; Couper, 1994; Norman, 1990; Wright and 

Barnard, 1975), but as Tourangeau (2000) points out the principles are vague enough that applying them is still as much an 

art as it is a science. Tourangeau also says that for this same reason it may be difficult to test them empirically or to 

determine the gains from questionnaires that embody them. On the other hand, we have made strides that make it worth at 

least attempting to synthesize what we have learned to date and worth attempting to extend it to establishment surveys. 

Take, for instance, the following principle: 

1. Use the visual elements of brightness, color, shape, and location in a consistent manner to define the desired 

navigational path for respondents to follow when answering the questionnaire (Jenkins and Dillman, 1997). 

It has since been shown that brightness, color, shape and location are visual elements of graphic language, and that graphic 

language is only one of the languages that comprises a questionnaire. Additional languages include the: (1) symbolic 

language, (2) numeric, and (3) verbal (Redline and Dillman, 2002). It is also now clear that ‘in an effective manner,’ not ‘in a 

consistent manner’ is the overarching goal really; consistency is only one means to this end. Finally, it has also become clear 

that it is necessary, but not sufficient, to think in terms of the questionnaire only. As Jenkins (1997) points out, improving the 

navigational qualities of a questionnaire requires paying attention to the entire mailing package. However, even that may not 

be sufficient, because as was discovered while redesigning the GSS, respondents may toggle between two modes: paper and 

Web. Thus, the following more encompassing and overarching revised principle emerges: 

 

1

 The views expressed in this paper are those of the author and do not necessarily reflect the views of the National Science 

Foundation. The author would like to thank her colleagues, Jeri Mulrow and Emilda Rivers, for their assistance and 

ingenuity throughout this project, and Scott Crawford (from Survey Sciences Group, LLC) for his advice and programming. 
2. Use the verbal, numeric, symbolic, and graphic languages (i.e., visual elements of brightness, color, shape, 

and location) in a consistent an effective manner to define the intended navigational path for respondents to 

follow when answering the survey (mailing package and/or Web questionnaire). 

Identifying the Intended Navigational Path 

A key phrase in the above design principle, a phrase that has remained stable from the first iteration of the principle to the 

second is “define the intended navigational path.” By definition, the navigational path is under the control of the interviewer 

in an interviewer-administered survey, whereas this path is under the control of the respondent in a self-administered survey. 

Four important tasks that respondents must handle in a self-administered survey that would be under the control of the 

interviewer in an interviewer-administered survey are: 1) starting correctly; (2) moving around; (3) reading and following 

instructions, definitions, or directions and; (4) answering in terms of the correct reporting unit. The addition of these tasks in 

a self-administered instrument translates into greater complexity for respondents, which in turn translates into greater 

respondent burden and potential for error. These tasks are not necessarily mutually exclusive, but are separated here for the 

sake of discussion. 

Starting Correctly. The interaction that occurs within the first few seconds of a survey has long been thought critical to 

cooperation rates (Groves et al., 2005). According to sociolinguistics, one of the three key components of an interview is its

opening interaction because this interaction sets the stage for the interview as a social encounter (Pan, 2003). The other two 

components are obtaining/giving information and maintaining the interaction. In cognitive interviews with three variants of a 

decennial census questionnaire (with approximately 20 respondents per variant), there was a large difference in how well 

respondents began the questionnaires. In one case, 4 percent of the respondents started by looking at the back page of the 

questionnaire, whereas 73 percent started by looking at the back page of another (Dillman et al., 1996). Respondents were 

asked in a debriefing at the end of the interview to recommend which questionnaire they suggested be used in Census 2000. 

One of the largest reasons they provided for their choice was the ease with which they were able to start the questionnaire. 

Forty-five percent favored the variant that allowed them to get started correctly compared to 22 percent for the one that did 

not (Redline, 1997). In the best-case scenario, a delay in starting correctly, however small or recoverable the delay may be, is 

a waste of respondents’ valuable time. In the worse case scenario, it will result in unit nonresponse. Numerous studies in 

Web surveys have demonstrated respondents’ inabilities to start correctly as a result of being provided ambiguous IDs or 

passwords, some of which have clearly been shown to lead to unit non-response (Fox et al, 2003; Couper et al, 2001). 

Moving Around. Schwartz is generally credited with demonstrating that a survey adheres to the tenets of conversation 

(e.g., 1997), as put forth by Grice (1975). Because of its visual and physical nature, however, a self-administered survey is 

more like having that conversation while driving. Powell (2002) discusses many issues relevant to Website use: for 

example, site organization structures, navigational theory, navigational bars and the advantages and disadvantages of their 

possible placement, and the design of buttons. This information is discussed, however, in the context of Web sites, whose 

users have a different goal than respondents to a survey. Web site users tend to be searching for information, whereas with 

surveys, respondents are answering queries. As Fox et al (2003) have noted, there is less directly applicable information 

regarding the design of Web surveys. A finding for which evidence does seem to be accumulating, however, is that moving 

from screen to screen (or by extension, from page to page) leads to a disruption in visual continuity (Mockovich, 2005), 

which has been shown to lead to response errors in paper questionnaires (Redline et al., 2005; Featherston and Moy, 1990). 

Thus, moving from screen to screen or page to page (or even from the bottom of one column to the top of the next one) 

appears to be more susceptible to error than many would have us believe. 

Reading and Following Instructions/Definitions/Directions. In a self-administered survey, respondents often need to read 

instructions, definitions, and directions critical to their understanding the questions and items, but they do not for what appear 

to be a number of, not necessarily mutually exclusive, reasons. One is because the information is often located physically out 

of sight (Redline et al., 2005; Tourangeau et al., 2003). Information may be far from where the respondent’s attention is 

presently. It may be in a separate booklet, for instance, or behind a link. Or, as has been demonstrated with skip instructions, 

it may be simply outside the respondent’s immediate attention (or foveal view in eye-movement terminology) (Redline and 

Lankford, 2001). Or, as has been shown with edit messages in Web surveys or branching instructions in paper questionnaires, 

it may be located after a disruption to visual continuity (Mockovich, 2005; Redline et al., 2005). A second possibility, 

however, is that the information is physically within sight, but is located among such a dense amount of information as to be 

rendered undetectable (Redline, 2003). A third possibility is that other aspects of the information’s graphic presentation (e.g., 

its color or size) give respondents the impression that the information is optional (Redline, 1997). A good example of this 

comes from one of the Census 2000 test questionnaires in which the “Getting Started” information on the inside cover of the 

questionnaire was designed to take the place of the cover letter in the mailing package. This design failed, however, because 
the getting started information looked like unimportant background information. It was composed of blue lettering, while 

everything else was in black, and it was put outside the gold background area in what looked like unimportant floating white 

space. 

 

Reasons endogenous to respondents, however, may compound the exogenous reasons provided above. That is, respondents 

may not perceive a need for the information (Redline et al, 2005; Frohlich, 1986). Or they may not have the cognitive ability

to successfully process all of the information they are provided, or for various other reasons, they may lack the motivation to

do so (Krosnick, 1991). 

In terms of Principle 2 above, the numeric, symbolic and graphic languages need to unambiguously indicate what to read and 

the importance of reading it, and the verbal language needs to attract respondent’s attention, maintain their attention, and 

convey accurate meaning—not an easy undertaking. Furthermore, it is important that the methods used to identify respondent 

errors in self-administered surveys allow us to distinguish between respondents’ not reading information, and the reasons for 

their not reading it, as opposed to reading it and not comprehending it. We need to correct errors that arise as a result of both, 

but the solutions may differ. 

Identifying the Reporting Unit. In a household or establishment survey, one of the first tasks respondents must undertake is

to understand about whom or what they are being asked to report. Research has shown that answering in terms of the wrong 

reporting unit may be the result of respondents not starting the questionnaire correctly or not reading the necessary 

instructions. However, research has also shown that respondents may not read instructions and still report correctly, or they 

may read and understand the instructions, but not agree with them; or in the case of an establishment survey, it is conceivable

that their records will not be in keeping with the request. Thus, it is clear that identifying and correcting the problems that 

arise here is not easy. 

An example of not identifying the reporting unit correctly comes from cognitive research with the Public School 1991-92 

Field Test Questionnaire for the Schools and Staffing Survey (SASS) (Jenkins et al., 1992). The cover page of this 

questionnaire contained a very important instruction in the paragraphs on the left-hand side of the page that told respondents 

to “Please complete this questionnaire with information about the SCHOOL named on the label.” Only about half of the 20 

respondents ever read this instruction and looked at SCHOOL named on the label. Not reading the instruction mattered when 

respondents had alternative definitions of the school for which they could report, and got the wrong impression from the rest 

of the questionnaire regarding the unit for which they should report. The tendency was to mistakenly report for the entire 

school system (K-12) when they were really only supposed to report for the portion of the school over which they had direct 

charge (e.g., the high school). 

The population count question (typically referred to as the pop count question) in the decennial census questionnaire has 

demonstrated similar tendencies. In cognitive interviews with the three variants of the decennial census questionnaire 

mentioned earlier, 15 percent of the respondents left the pop count question blank in the first variant, whereas only 7 and 6 

percent left this question blank in the second and third variant, respectively. A large-scale mailout/mailback test of the three 

variants verified the fact that respondents tended to overlook the pop count question in the first variant of the questionnaire

(Leslie, 1997). In hindsight, the pop count question appears like a mass of instructions in the left-hand corner relative to the 

name of Person 1 question, especially in the first variant of the questionnaire. These ‘instructions’ appear either unimportant 

or cognitively demanding in comparison to the much more salient and easy-to-answer Person 1 question. 

The instructions in the pop count question are referred to as the residency rule instructions. Importantly, vignette research 

into the residency rule instructions has shown that respondents’ definitions of who should be reported as living or staying in 

their household is influenced by the residency rule instructions in some cases and not others. Gerber et al. (1996) refer to the 

cases in which the instructions have no effect as intuitive. For example, respondents do not need to be reminded to include 

permanent household members that are temporarily away. In contrast, the rules are necessary in counter intuitive situations 

to counter respondents’ erroneous preconceived definitions of who should be included and who should not. For example, 

according to census rules, someone’s mother who has been placed in a nursing home on a trial basis is supposed to be 

reported at the nursing home. A vignette regarding this situation showed the largest percentage gain (27 percent) in correct 

answers between the ‘with instruction’ and ‘without instruction’ conditions, demonstrating that the instruction helped. 

Tourangeau et al. (2003) examined the complex everyday concepts of residence and disability and found similar results--that

participants were better at classifying vignettes that closely matched a definition (central instances) than ones that only partly 

matched it (peripheral instances). 
 

Methodology 

This knowledge informed efforts to redesign the GSS. The GSS collects data from all U.S. institutions offering graduate 

programs in science, engineering, or selected health-related fields. Data published from the survey include graduate student 

enrollment by race, ethnicity, by gender, by type of financial support, by discipline, and by full-time or by part-time status. 

Data are also published on the number of postdoctorates and non-faculty researchers by gender, citizenship and by discipline. 

The GSS is a visually administered survey (with both a paper and Web component). 

The redesign efforts consisted of conducting a round of cognitive/usability interviews with the original GSS, redesigning the 

survey, and conducting a second round of interviews with a redesigned portion of the survey. Twelve in-depth 

cognitive/usability interviews were conducted in the DC Metropolitan area and four in Florida in 2002 to learn more about 

how respondents navigated through and understood the original GSS. Interviews were conducted in the respondents’ offices 

using the concurrent ‘think aloud’ interviewing method. Respondents were handed the paper mailing package and asked to 

report their thoughts aloud as they went about the process of actually handling the mailing package and responding to the 

survey questions. The mailing package provided instructions for accessing the Web survey. Respondents were encouraged 

to answer the survey by whatever mode they would normally answer it, and to retrieve the appropriate records to answer the 

survey. Being as interested in what respondents read as well as how well they understood what they read, respondents were 

asked to read aloud in addition to thinking aloud. When respondents fell silent or could not be understood, neutral, thinkaloud probes were used as prescribed by Ericson and Simon (1980). Specific debriefing questions were held until the 

respondent had completed the think-aloud portion of the interview. In hindsight, the interviewing method was an attempt to 

use a method that would later be described by Hak et al. (2004) and discussed by Willis (2004, pg. 265). This method was 

designed to evaluate the cognitive aspects of answering the questions as well as the usability aspects of operating a visually 

administered questionnaire (either paper or Web). Interviews were both audio and video-recorded with the respondent’s 

permission. The typical time for completion of the interview was two hours. 

NSF contracted to have the survey redesigned in 2003-2004. Both the original and revised versions of the Web surveys were 

created in Cold Fusion (with possibly some Java script added) and were developed to run in Sequel Server. The paper 

version of the survey was created in PageMaker. 

NSF contracted to have a second round of cognitive/usability interviews conducted in 2005 with a redesigned portion of the 

GSS. Eight schools were recruited from Chicago and Connecticut, respectively. However, due to cancellations, 5 interviews 

were actually conducted in Chicago and 7 interviews in Connecticut. The same interviewing method was used in the second 

round of interviews as had been used in the first, except this time around, respondents were sent an email an hour in advance

of the interview, providing them with a link to the Web survey. As before, respondents were handed the paper mailing 

package at the beginning of the interview. Rather than asking respondents to complete the entire survey, they were asked to 

focus on a portion of the survey. And in addition to the cognitive/usability portion of the interview, a reconciliation was 

conducted at the end of the interview. That is, in advance of the interview, differences between the information provided in 

last year’s GSS and information gathered from the institution’s Website were discussed with respondents at end of the 

interview. The entire interview took about an hour. (Refer to Abt Associates, Inc., 2005, for a full reporting of this 

research.) 

Results and Discussion 

Identifying the Intended Navigational Path 

One of the most important findings to come out of the first round of cognitive/usability interviews with the original survey 

was that the GSS is really composed of three separate parts or instruments. Part 1 is meant to elicit a list of departments in

science, engineering, or selected health fields from survey coordinators, that is, to create the departmental frame. Part 2 is

forwarded to a respondent to collect the data at the departmental level. Part 3 is a database management tool meant to assist 

survey coordinators monitor (either their own or the departments’) data collection process. Furthermore, two of these parts 

were administered both by paper and by Web. Thus, it became evident that there were five parts or instruments, not just one. 

It also became evident that the overarching problem with the survey was that respondents were being asked to carry out many 

tasks (refer to Table 1 for a listing of the tasks) and that the overall design of the survey did not highlight and reduce 

competition for respondent’s attention to one manageable task at a time—in other words, it did not help respondents navigate 

through the information correctly, beginning with getting started correctly. 
Another important finding to come out of the cognitive/usability interviews with the original survey was that although the 

majority of respondents submitted their survey by Web (for instance, about 75 percent of the respondents submitted their 

survey by Web at the time of the interviews), many of the respondents in the cognitive/usability interviews actually

responded to the paper version of the survey, and simply transferred this information to the Web. In these cases, the true 

interface between the survey questions and that of the respondents was the paper questionnaire, with the Web survey serving 

as a data dissemination tool. 

Signaling the Intended Navigational Path 

Starting Correctly. In the original design of the paper questionnaire, Part 1 (listing science, engineering, and selected 

health-related departments) was downplayed and Part 2 (the questionnaire) was highlighted. Part 1 was printed portrait-style 

on standard white loose-leaf sheets of 8.5” x 11” paper stapled together in the upper left-hand corner. Its cover page was 

laden with instructions (full of text) that gave no visual clue that a task lay beneath (see Figure 1), that of listing the 

departmental frame (See Figure 2). Part 1 competed with the many other standard white mailing pieces that were included in 

the mailing package that were also text laden, similar to the way the residency rules in the pop count question of the 

decennial census questionnaire competed with the name question or the instruction on the cover of the Schools and Staffing 

Survey competed with the first question on that survey. As a result, a critical component of the GSS, the development of its 

frame, got less attention in the first round of interviews than was later assumed necessary to ensure correct reporting. In 

contrast, Part 2 was printed on colored paper stock in the form of a booklet and drew respondents’ attention (see Figure 3). 

The Web version of the survey, on the other hand, was problematic for two different reasons: (1) it had an entirely different 

look and feel from that of the paper and (2) information on its getting started pages (accessing the Web survey and home 

page) competed for the respondents’ attention. These problems will be described in greater detail next, and the solutions that

were developed. 

First, a new architecture with a new look and feel were conceptualized to better express or afford the parts in both the paper 

and Web and a prototype of Part 1 was developed. Visual design principles were used to make the two parts look as though 

they were supposed to function as a unit. Both were printed landscape in booklet formats with colored backgrounds, beige in 

the case of Part 1 (refer to Figure 4), and blue, in the case of Part 2 (refer to Figure 5), with the survey’s heading and 

navigational bar expressed in white print against a dark blue background that spanned the top and right hand side of both 

booklets. In this conceptualization, the two parts were joined visually by their many similarities (that is, parallel 

construction), but were also distinguishable by the color of their background and by their salutations. The beige survey was 

addressed to the survey coordinator, the blue to the survey respondent. Since respondents tended to fill out the paper version 

of the survey and simply transfer this information to the Web in the first round of cognitive/usability interviews, the same 

look and feel was applied to both the paper and Web versions of the survey. 

Costs played a significant role in determining which part received which color. The beige questionnaire required three colors 

to print: beige for the background, dark blue for the navigational bar, and black print for the text. Because three colors are 

more expensive to print and fewer coordinator questionnaires are printed than departmental questionnaires, the coordinator 

questionnaire was assigned the beige color. The blue questionnaire required only two colors to print: different tints of blue

for the background and navigational bar, and black print for the text. Because two colors are less expensive to print, this 

color combination was assigned to the departmental questionnaire. It is interesting to note that because introducing color on 

the Web does not cost anything, originally a highly multi-colored design was proposed for the Web (see Figure 6). One of 

the goals of the redesign was to maintain consistency between the paper and Web versions of the survey, so that respondents 

could effectively navigate back and forth between the two modes. For this reason, and because the unrestrained use of color 

competes for respondents’ attention, color choices were restrained in the redesigned version of the survey (refer to Figures 

4,5, 12, and 13 for examples). 

Ensuring that the paper and Web surveys looked similar so that respondents would be able to navigate between them required 

a great deal of coordination and discussion between professionals with very different backgrounds—those who ordinarily 

worked on paper documents (editors) and who rarely, if ever, spoke to those who worked on the Web (programmers). These 

two spoke two very different languages, and rarely thought about how a change they made in one of the modes would affect 

the other. It required a great deal of effort to ensure that differences between the two modes were introduced only after 

careful considerations on both sides. 
Suggesting that Web surveys look similar to their paper counterparts is contrary to conventional advice now. The opposite 

advice, suggesting that Web surveys be different from their paper counterparts may have come about for two reasons: one is 

when paper questionnaires are badly designed, it is unreasonable to repeat what was a bad design to begin with. The second 

is the assumption that respondents answer by one mode or the other. If the GSS is any indication, respondents to 

establishment surveys may be answering the paper and submitting by Web. Thus, establishment surveys may need to be 

careful not to think of the two modes in isolation, but to ensure that they operate together. 

To save space, only screens from the Web survey are shown and discussed in the remainder of this paper. It should be noted, 

however, that parallel pages from the paper survey tend to be identical, except for differences in functionality (differences 

due to buttons, links, etc.). 

In the original Web survey, the welcome screen presented respondents with two links, one for the survey coordinators (school 

login screen) and another for the departmental respondents (department login screen). Refer to Figure 7. Respondents were 

often confused between these two links and chose the wrong one. The instruction preceding these links, “To login, please 

click School Login Screen or Department Login Screen” did not provide respondents with any additional information over 

what they knew already, which violated the conversational maxim of quantity (Grice, 1975). Furthermore, the login 

information, which is obviously the point of this screen, competed with the rest of the information on this page because it was

placed in the same size and boldness as the rest of the information. Highlighted areas, such as the forget-your-password 

question beneath the login links, are likely to distract respondents’ attention from the primary goal of logging in. Beyond the 

password section, the “Introduction” and “Why These Data Are So Important’ sections are meant to persuade respondents’ to 

complete the survey. These sections are either going to compete with respondents logging in or be totally ignored, neither of 

which serves a fruitful purpose at this point in time. This information should be processed, in other words presented, before 

reaching the login stage, perhaps in the cover letter. 

Getting to the first question of the Web survey (the contact item) is twice as long in the original survey than the redesigned 

version. In the original, respondents must first respond to the login screen (Figure 8), a quick contact screen (Figure 9), and a 

main menu screen (Figure 10) before reaching the contact screen (Figure 11). On the main menu screen, they must select the 

number “1” from the middle of the page and from among many links and other pieces of information. Most of this 

information is premature at this point in time, violating the conversational maxim of relevance (Grice, 1975). The status 

legend, the upload and download your data links are not necessary until respondents get to the point where they are entering 

data, much later in the process. Furthermore, the navigational bars, the status legend, and other links are taking up valuable

real estate: the top and left of the screen. Eye-movement research has demonstrated that all else being equal, the top and left 

of a page are more attended to than the bottom and right (Brandt, 1945). Thus, placing the navigational bar and other 

information at the top and left makes perfect sense if the purpose is to facilitate their use. However, since the primary 

purpose of a self-administered survey is to read and answer questions, it makes more sense to place the questions in the top 

and left-hand side of the page, and to relegate the navigational bar to a supporting role on the right. 

In the redesigned survey, respondents are provided with a url, which automatically takes them to the correct login screen (see 

Figure 12). Furthermore, the screen they are taken to is highly simplified in comparison to the original. It highlights the task 

of logging in and rightfully reduces the original forget-your-password question to a supportive role (If you have forgotten 

your password...). It does this by boldly welcoming respondents and asking them to enter their User ID and password. The 

‘Welcome’ salutation was purposely made big and bold and placed in the upper left-hand corner of the beige background, 

with nothing to its left or below it to distract respondents’ attention from the primary task of logging in. The language here is 

consistent with the wording used in the cover letter, and with the sociolinguistic notion of this as an opening interaction. In 

other words, the computer jargon “to login” was purposely removed from the redesigned version and replaced with “to begin 

answering the online questionnaire.” 

The redesigned survey simplified and reduced getting to item 1 dramatically (refer to Figure 13). After logging in, the 

number 1 immediately presents itself in the upper left-hand corner of the page followed by the wording of the item. To 

further attract respondents’ attention, the item wording was made bolder and larger than the rest of the information on the 

page, and clearly bolder and larger than it was in the original version of the survey. 

The headings in the upper-left hand corner of the original survey were relocated because cognitive research has shown that 

respondents often read these headings in lieu of the items beneath them and draw the wrong interpretations as a result 

(Jenkins and Dillman, 1993). These findings are in direct opposition to conventional wisdom and research in the 

instructional text area, which has suggested that headings help readers understand text better (e.g., Hartley, 1981). However,
one might reasonably conclude after reading this research more closely that the information conveyed by the heading is 

necessary, not necessarily the heading itself. Thus, an effort was made in the redesigned version to include all necessary 

information in the body of the item/questions. Besides helping readers understand the text better, headings serve another 

purpose, according to the instructional text research: they help readers to scan, select, or retrieve materials more easily. 

Hartley (1981) suggests that headings may be more usable when placed in the margin, although he also notes that this has not 

been subjected to research. 

The redesigned survey makes use of these ideas. Rather than using the headings to supply pertinent information, they are 

used in the navigational bar in the right-hand margin to guide respondents through the survey. The navigational bar’s design 

is predicated on the notion that items listed vertically signal greater independence than those listed horizontally (Jenkins and 

Dillman, 1997), along with the hypothesis that lists of action items are a commonly understood convention. 

Finally, visual principles were used to indicate which item in the list is activated. This was accomplished by making the 

activated item look the same as the body of the screen (black text against a beige background). As a result, the selected item

looks visually connected to the body of the screen, while the rest of the navigational bar is reduced to background 

information (white text on a dark blue background). 

Moving Around. The cognitive interviews revealed that moving from screen to screen in the original survey was 

convoluted. In the original survey, respondents were presented with two buttons at the bottom of the contact information 

page: “Save and Refresh” and “Cancel,” neither of which moved respondents to the next screen or question, as expected 

(see Figure 11). To move to the next screen they had to use the navigational bar to return to the main menu and choose 

where to go from there (which was not always evident), or choose where to go next from the navigational bar itself, or hit 

“Save and Refresh,” followed by “Cancel,” which then took them back to the main menu screen. Later screens presented 

even more complicated buttons from which to choose (e.g., as many as 8) in addition to the navigational bars at the top and 

bottom of the screens. This is an example of a hierarchical or ‘tree structure’ organization, with the home page serving as the

‘root’ of the ‘tree structure’ (Powell, 2002). In this organizational structure, backtracking is necessary to move forward. 

Furthermore, respondents have full control over the order in which they process information, which is not good if the point is 

to process information in a predetermined order. In this organizational structure, navigating through the survey is 

unnecessarily competing with actually answering the survey because it requires cognitive effort. 

In the redesigned version of the survey, “Previous” and “Next” buttons were consistently placed at the bottom of the screen, 

not always in the same position on a screen, as is commonly recommended, but always the same distance beneath the last 

piece of information on the screen (see Figure 13). The point of placing them here is two-fold: (1) because respondents need 

to traverse this information to reach the buttons, thus they may be more likely to notice this information, or (2) if they do read 

the last piece of information, this is an efficient use of their eye-movements, and consequently, their time, for they simply 

need to continue down a little ways before reaching the ‘next button.’ 

The ‘Previous’ and ‘Next’ buttons have been designed to serve as the primary means of navigating through the redesigned 

survey, with the navigational bar available in case respondents want to skip further ahead or back or exit the survey, and 

come back. This is an example of a linear-with-options organizational structure. This organization best mimics the paper 

survey in that it provides a structure for answering the survey in a pre-determined linear order, but it also allows respondents 

to freely move around, if they so choose. In this organizational structure, answering the survey is emphasized over 

navigating through the survey. 

Reading Instructions/Definitions/Directions. The two modes of the original survey diverged greatly in the amount of 

instructions/definitions/directions provided to respondents in Item 2 “ Maintain Your List of Departments.” To accurately 

maintain their list of departments, survey coordinators need to know which departments should be included according to 

NSF’s rules, and which should be excluded. As can be seen in Figure 14, however, the original Web version did not provide 

the survey coordinators with any guidance about which departments to include or exclude from the list, at least not without 

searching elsewhere for this information, and not one respondent in the first round of cognitive/usability interviews looked 

elsewhere for this information. The first instruction on the screen here “This screen allows you to maintain your list of 

departments” does not provide respondents with any additional information over what they were able to gather from the 

heading already. Similarly, in the response matrix, there is a column labeled ‘department name, ‘ but no explanation as to 

what precisely that means. 
Supplying an exhaustive list of rules, however, is not the answer either, especially if this list does not look as though it is part 

of the navigational path. The paper version of the survey (refer back to Figure 1) supplied the survey coordinators with so 

many instructions and in such a way (on white loose-leaf paper when the ‘survey’ itself was a colored booklet) that for all 

practical purposes it may have not been supplied. There were approximately 537 words on this page, and still not all the 

information that was needed is on this page. Some of it was buried elsewhere. Thus, in the redesigned version of the survey,

an attempt was made to extract and reorganize the most critical of the eligibility rules and to place this information in the 

navigational path. 

The redesigned version has approximately 303 words, a decrease from the original (see Figures 15 and 16 for the Web 

version). The first instruction on this screen is numbered ‘2,’ and is placed in the same position on this page/screen as the

number ‘1’ had been in the preceding page/screen. The first instruction here attempts to convey to respondents that they are 

to answer three questions (A, B, and C) to update their list of departments in the table below, and questions that were only 

implied in the original were made explicit. In addition, columns that confused respondents in the cognitive/usability 

interviews, and for which no good purpose could be established, like the ‘locked’ column was removed. The locked column 

is in a spatially and temporally important position, the first column of the table, but it is not clear what its purpose is. And the 

second column was very confusing. No respondents in the cognitive/usability interviews referred to their departments by ID, 

but by their own names for their departments (e.g., Chemistry), which doesn’t show up until the third column of the matrix. 

In the redesigned version of the survey, department name was placed first, and what was an implied question in the original 

version of the survey was made explicit in the redesigned version: “In column A, what are the names of your school’s

science, engineering, and health-related graduate departments for 2004?” An attempt was made to link the question above 

the matrix to the correct column in the matrix by placing the prepositional phrase at the beginning of the question: “In 

column A’. Furthermore, question A contains a very important piece of conceptual information that is not explicitly stated in 

the original version of the Web survey—the fact that this survey is supposed to collect information from health-related 

departments in addition to science and engineering. 

Another piece of information that was missing from the original survey was the reference period. Schaeffer and Presser 

(2004) suggest that the reference period should be given at the beginning of a question (so that respondents do not construct 

their own before hearing the investigator’s), and that respondents hear it before they formulate what they believe to be the 

intent of the question and that it be given in abbreviated form, and in a parallel location in subsequent questions. In this case 

that would have meant doubling up on prepositional phrases at the beginning of the sentence “In column A, in 2004,” which 

seemed awkward. Furthermore, it would have been difficult placing the reference period in a parallel position in the rules 

that follow, so the reference period was repeatedly placed at the end of instructions instead. This is an example of the oftendifficult choices survey designers must make. Besides creating a more complete, explicit question, the most relevant 

instructions pertaining to this question were divided by function (delete, remain or add) and listed in bulleted format under 

the question here. 

A comparison of the table in the original version of the Web survey with the original paper version suggests that what is 

called “ID” in the Web version is referred to as “Department Code” in the paper version. This code is an effort on NSF’s part 

to summarize departments into higher-order categories known as disciplines. The redesigned version did away with the code 

altogether, which has virtually no meaning for respondents. In the redesigned paper version of the survey, respondents are 

asked to refer to a list of disciplines, and in the Web, they are asked to select the discipline that best describes their 

department from a drop-down box. 

One gets the impression from the heading in the original version of item 2 (maintain your list of departments) that this item is 

going to contain one task; however, closer inspection of the table reveals that respondents are actually being asked to conduct

two: (1) to update their list of departments, and (2) to update the names of the departments’ coordinators (refer to Figure 14). 

The redesigned version of the survey decomposed item 2 further. The first task (as shown in Figures 15 and 16) asks 

respondents to update their list of departments. The second task is not shown here, but is hinted at in the navigational bar. It 

invites departments to respond, and it is in this next task (or screen) that respondents are asked to update the names of the 

departments’ coordinators. 

Thus, an attempt was made to place a reasonable amount of information in the navigational path (e.g., making the listing 

form and eligibility rules look as though they are a part of the survey); reorganize the information in a way that is keeping 

with respondents’ own schemas about the information (e.g., place department name first); connect physically disconnected 

but conceptually related information (e.g., use “in column A”); replace unfamiliar jargon with potentially more 
understandable terms (e.g., replace ‘ID” or “Department Code” with “Discipline”); supply missing but important information 

(e.g., the reference period); and decompose the task into its constituent tasks (e.g., list departments then solicit information to 

aid in inviting departments to respond). 

The second round of cognitive/usability interviews with the redesigned survey suggested that respondents did start more 

easily and correctly, navigate through the form with greater ease, and had a better understanding of what was being asked of 

them. Respondents deleted, added, and disentangled departments far more in the second round of interviews than in the first, 

suggesting that the changes made above were a step in the right direction. For example, one respondent noticed 5 dental 

departments that had been defunct for the past 5 years and deleted them during the course of the interview. (It is interesting

to note that after participating in the cognitive/usability interview, she deleted 22 departments from the actual GSS.) 

Defining the Reporting Unit. It was difficult to tell from the first round of interviews what were respondents’ definitions 

for their reporting units. Instead of attending to Part 1 of the GSS in the first round of interviews, respondents had a tendency 

to accept whatever list they were provided with to start, and to focus on what they thought was the real purpose of the survey,

the questions in Part 2. A few respondents, particularly those new to the survey, still had a tendency to view the pre-assigned 

list as correct in the second round of interviews; however, in general, a better understanding of respondents’ definitions for 

their reporting unit was garnered from the second round of interviews for at least two reasons: (1) because an emphasis was 

placed on inducing respondents to read Part 1 in the redesigned version of the survey and (2) because the second round of 

interviews deliberately focused on this part of the survey. 

Item 2 of Part 1 asks respondents to report a highly complex construct: departments, programs or research centers with 

graduate students, postdoctorates, or non-faculty research staff in science, engineering, and selected health-related fields. 

By most people’s standards, the concept of postdoctorate alone is complicated, so the above construct is clearly highly 

complicated and multifaceted. As borne out by previous research regarding complex concepts (e.g., Gerber et al, 1996; 

Tourangeau et al., 2003), respondents’ schemas were sometimes in keeping with NSF’s definitions, and sometimes not. The 

second round of cognitive/usability interviews suggested that respondents had little difficulty reporting what Gerber et al. 

(1996) refers to as intuitive and Tourangeau et al. (2003) refer to as central instances. Examples of these in the GSS are 

physics and chemistry departments. However, respondents tended to have problems knowing whether to report what Gerber 

et al. (1996) refer to as ambiguous and Tourangeau et al. (2003) refer to as peripheral. A good example of this in the GSS is 

research centers. Finally, there were cases in the GSS that Gerber et al. (1996) refer to as counterintuitive, that is, 

respondents’ definitions did not appear to agree with NSF’s official definition of science, engineering, and health-related. An 

example of this was acupuncture. 

Respondents not understanding NSF’s technical definition for what to include and what to exclude, and not listing the 

institution’s organizational elements correctly are likely to lead to coverage error at the second stage level. In general, 

respondents seemed to understand the redesigned version of the survey better in the second round of cognitive/usability 

interviews than in the first, although there were still problems. Respondents still came to the task with expectations (for 

example, believing that the pre-assigned list provided to them was accurate) and questions (wondering what the rules were 

about research centers and health fields) that need to be addressed in future renditions of this item. 

Research shows that decomposing multifaceted or compound tasks into their constituent tasks and organizing information in 

ways that are in keeping with respondents’ schemas or that cue their memories are helpful. Thus, we need to learn more 

about respondents’ schemas in the future, and what cues will aid them access these schemas. It may prove beneficial to ask 

respondents to report their research centers first, then programs, and then departments (under the assumption that research 

centers are the least central to respondents’ thinking about this issue, and therefore, the most likely to be forgotten or 

misunderstood). Also, the eligibility rules are now broken down by function (delete from the list, leave on the list, add to the 

list). Is this the best ordering of these functions? Might it be beneficial to separate these into separate questions, each 

referring to a separate response table? However, what is best if respondents are filling out the paper questionnaire, and 

transferring this information to the Web? Further research is needed to help us answer these difficult questions. 

It should be noted that a pattern appears to be emerging from household and establishment surveys (Jenkins et al, 1992; 

Gerber et al., 1996), which when put together with the research of Tourangeau et al. (2003) suggests that respondents can 

have multiple definitions for their reporting units and that they need to clearly understand what the survey is requesting (that 

is the survey’s eligibility rules) in order to report correctly. The findings of the research seem to point in the same direction: 

household and establishment surveys, which have a global instruction buried at the beginning of the survey aimed at 

respondents’ understanding the unit for which they are to report, may not be elicit accurate reports. These surveys may 
benefit from following the GSS’ model of listing out their organizations’ elements (e.g., members of a household; 

subsidiaries, affiliates, or branches of a business; or departments of an institution) first, followed by requesting data 

from/about those elements. 

Redesign Principles. Many changes were made to the GSS to induce respondents to start it correctly and to better 

understand the tasks they were being asked to perform in the order they were being asked to perform them, with a focus on 

getting respondents to properly attend to the critical task of defining their reporting unit. The major principle used to guide 

these changes was provided at the beginning of this paper: use the verbal, numeric, symbolic, and graphic languages in an 

effective manner to define the intended navigational path for respondents to follow when answering a self-administered 

survey (Jenkins and Dillman, 1997). Table 2 attempts to delineate this principle into the sub-principles that were used to 

redesign the GSS. These sub-principles may provide even more concrete guidance for improving a survey, or they may serve 

as input for hypothesis testing under controlled, scientific conditions. Harking back to the major principle, these subprinciples attempt to define how to signal the navigational path more ‘effectively.’ 

Conclusions and Future Research 

Starting with the end in mind does not mean starting at the end. In the case of the GSS, the end is to elicit accurate counts of 

graduate students, postdoctorates, and non-faculty research staff in science, engineering, and health-related departments, 

programs, and research centers. The results of previous survey research suggest that to reach this goal, we need to start much

further back. We need to help respondents: start the GSS correctly; move through the survey in a systematic order; read the 

questions, instructions, definitions, and directions in non-critical items so that by the time they reach a critical item, they are 

primed to read information which will aid them to accurately answer it with as little effort as possible. In other words, we 

need to identify and define the navigational path for respondents. 

The first round of cognitive/usability interviews with the original survey confirmed the fact that the navigational path was not 

clear, and that respondents had a tendency to overlook Part 1 of the survey (and more specifically, Item 2 in Part 1). This 

item elicits the survey’s departmental frame—or more specifically, a list of science, engineering, and health-related 

departments, programs, or research centers that have graduate students, postdoctorates, or non-faculty research staff. A 

second round of cognitive/usability interviews with the redesigned version of the survey suggested that respondents did start 

the redesigned survey more easily and correctly, navigate through it with greater ease, and had a better understanding of what 

was being asked of them in this item. Respondents deleted, added, and disentangled departments more in the second round 

of interviews than the first. 

An important finding to come out of this research was that respondents often answered the paper version of the GSS and then 

simply transferred this information to the Web. Thus, the navigational path of the redesigned survey was in effect doubled—

respondents not only needed to navigate within a mode, but between two modes. Contrary to conventional advice, it became 

imperative to maintain consistency between the two modes so that respondents could navigate back and forth between the 

two modes without getting lost. 

Another important finding to come out of the redesign efforts was that in order to answer the GSS correctly, respondents 

must understand a very complex and multifaceted technical construct, that of departments, programs or research centers with 

graduate students, postdoctorates, or non-faculty research staff in science, engineering, and selected health-related fields. It 

is clear that more research needs to be conducted before this construct will be successfully conveyed to respondents. 

Finally, one of the most important findings to come out of this research is that instructions provided at the beginning of 

household and establishment surveys, aimed at helping respondents’ understand the unit for which they are to report, are 

often inadequate. In all likelihood, these surveys would benefit from creating an additional step (or part) at the beginning of 

the survey, in which respondents are asked to enumerate the organization’s elements according to some rules (e.g., members 

of a household; subsidiaries, affiliates, or branches of a business). Thus, we end where we began: a self-administered survey

instrument is really a physical object with many parts that need to work in concert for the express purpose of collecting 

information. It is clear that we need to continue to understand these parts ourselves, and to help respondents navigate 

through them and understand them as well. 
 Abt Associates, Inc. 2005. NSF-GSS Final Report: Cognitive/Usability Testing of Module One in the Spring of 2005. 

Report Prepared for National Science Foundation, Arlington, VA 22230. 

 

 Brandt, H.F. 1945. The Psychology of Seeing. New York: The Philosophical Library. 

 Couper, M. 1994. What can CAI learn from HCI? Paper presented at the COPAFS Seminar on New Directions in 

Statistical Methodology. June 1994, Bethesda, MD. 

 Couper, M., Traugott, M.W, and Lamias, M.J. 2001. Web Survey Design. Public Opinion Quarterly. 65:2. 

 Dillman, D. 2000. Mail and Internet Surveys: The Tailored Design Method. John Wiley & Sons: New York. 

 Dillman, D., Redline, C., Martin, B., and DeMaio, T. 1996. Cognitive and Motivational Properties of Three Proposed 

Decennial Census Forms. Report Prepared for the U.S. Bureau of the Census. Washington, D.C. 

 Ericson, K.A., and Simon, H.A. 1980. Verbal Reports as Data. Psychological Review. 87: 215-251. 

 Featherston, F.and Moy, L. 1990. Item Nonresponse in Mail Surveys. Paper Presented at the International Conference 

on Measurement Errors in Surveys, Tucson, Arizona. 

 Fox, J.E., Mockovak, W., Fisher, S.K, and Rho, C. 2003. Usability Issues Associated with Converting Establishment 

Surveys to Web-Based Data Collection. Proceedings of the Federal Committee of Statistical Methodology. Washington, 

D.C. 

 Frohlick, D. 1986. On the Organizations of Form-filling Behavior. Information Design Journal. 5:43-59. 

 Gerber, E., Wellens, T.R., and Keeley, C. 1996. Who Lives Here? The Use of Vignettes in Household Roster Research. 

Proceedings of the Section on Survey Research Methods, (pp 962-967) Alexandria, VA: American Statistical Association 

 Grice, H.P. 1975. Logic and Conversation. In P. Cole (ed.) Syntax and Semantics. Vol 3. (pp. 41-58) New York: 

Academic Press. 

 Groves, R.M., Fowler, F.J., Couper, M.P., Lepkowski, J.M., Singer, E., and Tourangeau, R., 2004. Survey Methodology. 

John Wiley & Sons: Hoboken, New Jersey. 

 Hak, T., van der Veer, K. and Jansen, H. 2004. The Three-Step Test-Interveiw (TSTI): An observational instrument for 

pre-testing self-completion questionnaires. (ERIM Report ERS-2004-029-ORG). Rotterdom: Erasmus Research Institute of 

Management. Retrieved January 05, 2006 from hdl.handle.net/1765/1265. 

 Hartley, J. 1981. Eighty ways of Improving Instructional Text. IEEE Transactions on Professional Communication, Vol. 

PC-24, No. 1, March 1981. 

 Hunter, J., Redline, C., Morrison, R., Willimack, D., Sangster, R. 2005. Broadening the Horizons of Q-bank: Expanding 

the Framework to Encompass Self-Administered and Establishment Surveys. Paper prepared for presentation at the 

American Statistical Association, Minneapolis. 

 Jenkins, C. and Dillman, D. 1997. Towards a Theory of Self-Administered Questionnaire Design. In L. Lyberg, P. 

Biemer, M. Collins, E. DeLeeuw, C. Dippo, N. Schwarz, & D. Trewin. (Eds). Survey Measurement and Process Quality

(pp. 165-196). New York: Wiley. 

 Jenkins, C. 1997. Improving the Navigational Qualities of the Decennial Census Short Form Requires Paying Attention 

to the Entire Mailing Package. Proceedings of the Section on Survey Research Methods, Alexandria , VA: American 

Statistical Association. 
 Jenkins, C., Ciochetto, S. and Davis, W. 1992. Results of Cognitive Research on the Public School 1991-92 Field Test

Questionnaire for the Schools and Staffing Survey (SASS-3A). Report Prepared for the National Center for Education 

Statistics. Washington, D.C. 

 Krosnick, J.A. 1991. Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys. 

Applied Cognitive Psychology, 5:213-236. 

 Leslie, T.F. 1997. Comparing Two Approaches to Questionnaire Design: Official Government Versus Public 

Information Design. Proceedings of the Section on Survey Research Methods, American Association of Political Opinion 

Research. (pp. 336-341) Alexandria, VA: American Statistical Association . 

 Mockovich, B. 2005. An Evaluation of Different Design Options for Presenting Edit Messages in Web Forms. Paper 

Presented at FedCASIC, Bureau of Labor Statistics: Washington, DC. 

 Norman, D. 1990. The Design of Everyday Things. Doubleday: New York. 

 Pan, Y. 2003. The Role of Sociolinguistics in the Development and Conduct of Federal Surveys. Proceedings of the 

Federal Committee of Statistical Methodology. Washington, D.C. 

 Powell, T.A. 2002. Web Design: The Complete Reference. Second Edition. McGraw Hill/Osborne: New York. 

 Redline, C., Dillman, D., Carley-Baxter, C., Creecy, R. 2005. Factors that Influence the Reading and Comprehension of 

Branching Instructions in Self-Administered Questionnaires, Allgemeines Statistisches Archiv. 89, 21-38. 

 Redline, C. 2003. Discussion of Tourangeau, Couper, and Conrad’s paper, The Impact of the Visible. The Second 

Seminar on the Funding Opportunity in Survey Methodology. Research Subcommittee of the Federal Committee on 

Statistical Methodology. Washington, DC. 

 Redline, C. and Dillman, D. 2002. The Influence of Alternative Visual Designs on Respondents' Performance with 

Branching Instructions in Self-Administered Questionnaires, in Groves, R., Dillman, D., Eltinge, E., and Little, R. (eds.) 

Survey Nonresponse. New York: John Wiley and Sons, Inc. 

 Redline, C. and Lankford, C. 2001. Eye-movement Analysis: a New Tool for Evaluating the Design of Visually 

Administered Instruments (paper and Web), Proceedings of the Section on Survey Research Methods, Alexandria: VA: 

American Statistical Association. 

 Schaeffer, N. C. and Presser, S. 2003. The Science of Asking Questions. Annual Review of Sociology, 29: 65-88. 

 Schuman, H. and Presser, S. 1981. Questions and Answers in Attitude Surveys: Experiments on Question Form, 

Wording, and Context. San Diego: Academic Press. 

 Schwartz, N. 1997. Questionnaire Design: The Rocky Road from Concepts to Answers, in Lyberg, L., Biemer, P., 

Collins, M. de Leeuw, E., Dippo, C., Schwarz, N., Trewin, D. (eds.) Survey Measurement and Process Quality New York: 

John Wiley & Sons, Inc. 

 Tourangeau, R., Conrad, F., Arens, Z., Fricker, S., Sunghee, L., and Smith, E. 2003. Everyday Concepts and 

Classification Errors: Judgments of Disability and Residence. Paper presented at the American Association of Public 

Opinion Research, Phoenix, Arizona. 

 Tourangeau , R., Couper, M., and Conrad, F. 2003. The Impact of the Visible. Paper presented at the Second Seminar 

on the Funding Opportunity in Survey Methodology. Research Subcommittee of the Federal Committee on Statistical 

Methodology. Washington, DC. June, 2003. 

 Tourangeau, R., Rips, L., and Rasinksi, K. 2000. The Psychology of Survey Response. Cambridge University Press: 

Cambridge. 
 Willis, G. 2005. Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, California: Sage 

Publications. 

 Wright, P., and Barnard, P. 1975. Just Fill in This Form—A Review for Designers. Applied Ergonomics, 6:213-220. 
Table 1. A Listing of the Self-Administered Tasks in the GSS by Type and Number of Respondents and Mode of 

Administration 

Self-Administered Tasks Respondent 

Type 

Number 

of Respondents 

Mode 

of Administration 

1. Email cover letter 

Read email 

Link to Web survey, or 

 go to mailing package, or both 

 

 

Survey Coordinator 1 Web 

2. Mailing package envelope 

 Read envelope 

Open envelope 

Take out contents 

Survey Coordinator 1 Paper 

3. Questionnaire cover letter 

Read letter 

Link to Web survey or 

 Go to Module 1 or both 

 

 

 

 

 

 

 

 

 

Survey Coordinator 1 Paper 

4. Part 1: Survey Coordinator 1 Paper or Web 

 4.1. Review the contact information 

 Read information 

 Comprehend 

 Retrieve information 

 Act: respond 

4.2. Enumerate the sampling frame 

(science, engineering and selected healthrelated departments) 

 Read information 

 Comprehend 

 Retrieve information 

 Respond 

Distribute the questionnaire(s)/ 

invite departments to respond 

5. Part 2: Survey Coordinator 

or 

Department Respondent 

1 up to as many 

respondents as 

departments 

Paper or Web 

Read cover letter 

Read instructions and definitions 

Respond to department questionnaire 

 Read items 

 Comprehend 

 Retrieve information 

 Judgment 

 Respond 

 Return or submit dept. questionnaire 

6. Part 3: Survey Coordinator 1 

View department progress Web 

Contact Departments about Corrections Paper or Web 

7. Return or Submit Survey Paper or Web 
Table 2. Design principles, divided by language type. 

Graphic 

• Location

o Up and left of page/screen indicates the beginning, thus place all information in the order it will be used 

temporally, beginning with upper left

• Place numbers to the left of text and text to the left of action items

• Place response categories to the left of check boxes

• Place check boxes to the left of skip instructions

• Place navigational bar to far right of screen (because in essence it is a skip instruction)

• Place definitions/instructions/error messages within close proximity of where they are to be acted upon

• Bottom and right of page/screen indicates end, thus place next button last on screen just beneath the last piece of 

information to be read in the lower right side

o Place conceptually related information horizontally (rows)

o Place conceptually unrelated information vertically (columns)

• Brightness and color

o Use high contrast and/or color coding

• Use light background

• Light beige background (20 percent saturation) for coordinators, light blue background 

(20 percent saturation) for departments

• Use dark text, preferably black

• Black text (100 percent saturation)

• Use red error messages to distinguish them as different from other text

• Use blue Web links because this is convention

o Use size and boldness to convey hierarchy (i.e., establish order of importance).

• Ensure that size and boldness do not conflict with location, but work in concert.

o Use white for action items (radio buttons, text boxes, drop down boxes, navigational links, etc.)

o Do not rely on color only to convey meaning

o Minimize changes in color: less is more

• Shape

o Use san-serif font

o Minimize changes in font under the auspices that less is more 

Symbolic 

• Use symbols in addition to verbal messages

• Use familiar symbols

• Beware of ambiguity

Numeric 

• Use numbers to logically order questions for respondents

• Beware of ambiguity (for example, using ‘1’ more than once)

Verbal 

• Use a conversational (social interaction) style, as opposed to computer jargon

• Present only necessary and relevant information (Maxims of Quantity and Relevance (Grice, 1975))

• Use short, active, affirmative sentences/questions (Maxim of Manner (Grice, 1975))

o One clause

o Active rather than passive

o Affirmative rather than negative

o Decompose questions: ask questions about one thing at a time

• Use familiar words (rather than unfamiliar words, numbers, or symbols) (Maxim of Manner (Grice, 1975))

• Beware of ambiguities (Maxim of Manner (Grice, 1975))
Figure 1. Part 1’s original cover page. 
Figure 2. Part 1’s original departmental listing page. 
Figure 3. Part 2’s original cover page. 
Figure 4. Part 1’s redesigned cover page. 

Figure 5. Part 2’s redesigned cover page. 
Figure 6. Unrestrained use of color proposed for Web version of the survey. 

Figure 7. Original welcome screen.
Figure 8. Original login screen.

Figure 9. Original quick contact screen. 
Figure 10. Original main menu screen. 

Figure 11. Original contact screen. 
Figure 12. Redesigned welcome/login screen. 

Figure 13. Redesigned contact item screen.
Figure 14. Original list of departments screen. 

Figure 15. Top of the Redesigned List of Departments Screen. 
Figure 16. Bottom of the Redesigned List of Departments Screen.