Linear Navigation Guideline

Elizabeth Nichols

Center for Behavioral Science Methods, U.S. Census Bureau

Federal Committee on Statistical Methodology

October 27, 2022

This presentation is released to inform interested parties of research and to encourage discussion. 

The views expressed are those of the authors and not those of the U.S. Census Bureau. 

The disclosure review number for this presentation: CBDRB-FY22-CBSM002-043. 1
Sample of Census Bureau questionnaires

American Community Survey - left

2
Sample of Census Bureau questionnaires

Business Survey - middle

3
Sample of Census Bureau questionnaires

2020 Census - right

4
Literature

• Placement of forward and backward navigation did not affect breakoff 

rates – Couper, Baker, & Mechling (2011)

• Placement of forward button to the right of the backward button 

increases user satisfaction and is preferred with no time-on-task 

differences noted – Romano Bergstrom, Erdman, & Lakhe (2016)

5
Web survey design issue

• For forward and backward navigation buttons we needed to know the 

optimal:

• Placement

• Color

• Size

• Label

6
User Experience Design Factors

• Mental models = how the user expects the design to work

• Typically based on what the user knows from past experiences

• Fitts’ law = is about how time on task is affected by size and distance

• Bigger and closer targets take less time to engage

• Challenge: We suspected that time on task would be affected by the 

content of the survey questions we asked. We wanted to take that 

“cognitive factor” out of the equation. 

7
Method

• Series of A/B experiments using nonprobability panels

• Reside in the U.S. and opted in to participate in studies.

• 400 participants in Experiment 1

• 39% Male/61% Female

• Age (mean=47 years old; range 18-88) 

• 5% Hispanic origin/95% not of Hispanic origin

• 77% White only/23% nonwhite only

• 16.5% High school or less/83.5% more than high school

• 30% Mobile/ 70% PC

• 515 participants in Experiment 2 confirmation experiment

• PC/Mac only

• Similar breakdowns but more High School or less

• 34% High school or less / 66% more than high school 

8
Placement experiment

4 navigation button placements & 3 questions

12 screens altogether

Centered with the radio button 

question

Either side with the 

dropdown question

Right-aligned buttons with the text 

entry question

Left-aligned buttons

9
How we tried to remove the “cognitive factor” 

from the questions and uncover the user’s mental 

model

10
UX Placement winner

Faster selection

Centered Either side

Other notes:

Over 97% of participants selected the 

button on the right to go forward.

Participants preferred the either side 

navigation buttons.

Participants preferred buttons closer to 

the question. This matches Fitts’ law. 

The centered buttons could be the 

same in larger and smaller devices so 

we went with that design with forward 

navigation on the right and backwards 

navigation on the left. 11
Color and size experiment

15 screens altogether

12
UX Color and size selection

There was not a clear UX winner; but there were less usable designs.

Do not:

Make the Previous button bigger than the Next button

Make the Previous button blue when the Next button is white

Use same sized buttons if the buttons are the same color

13
Confirmation experiment

Color, label and size

Measured time-on-task and accuracy of button selected

Showed 32 of 64 combinations - randomized

14
Guideline

• White and Colored button to match the branding of the survey 

• Centered

• Default labels of Next and Back with the ability to choose an 

alternative or to add arrows in addition to text

15
Challenges & Compromise

• Small differences or no differences in time and accuracy once labeling 

and colors were in place

• Survey area stakeholders had strong opinions on labels

• Compromise to include default labels but then allow for alternatives

16
Thank you!

Elizabeth Nichols

elizabeth.may.nichols@census.gov

17
Designing Edit Validations

Shelley Feuer

Center for Behavioral Science Methods, U.S. Census Bureau

Federal Committee on Statistical Methodology

October 27, 2022

This presentation is released to inform interested parties of research and to encourage discussion. 

The views expressed are those of the authors and not those of the U.S. Census Bureau. 

The disclosure review number for this presentation:CBDRB-FY22-CBSM002-012
Background

• Edits/validations are programmed into web surveys to catch possible errors or 

inconsistencies in the data

• The validations are triggered once the respondent attempts to leave the survey 

page and navigate to another page

• If an edit/validation is triggered, a message appears to the respondent alerting 

them to the issue

• Edits/validations are typically used to 

• remind respondents of missing answers

• compare the answers to different questions

• signal that an answer is out of range
Types of data entry errors

• Hard edits

• Prevent respondent from answering more survey questions or submitting 

data

• E.g., entering an address for the 2020 Census

• Soft edits

• Do not prevent respondent from answering more survey questions or 

submitting data without changing responses 

• E.g., inconsistent age and date of birth 
Design questions

• How should these alerts appear? 

• Length of alert message?

• Placement/location?

• Color? 

• Different for each type of alert (e.g., soft vs. hard)?

21
2014 Census Test

• Usability testing of census online instrument

• Think aloud protocol 

• Tested three edit messages for the race/Hispanic origin item

22
Findings

• Race edits were not noticed when activated

• Very few race edits were invoked during the actual sessions

• No pattern with one type of edit message helping more than 

another edit message

• During the debriefing these edits even caused some 

confusion 

23
Findings

24

Content and color

• Long messages can be ignored, particularly in green

• Conflicting content/color “Red tells you that you skipped something, but message says it’s OK to go on”

Color

• Green

• Ignore

• Positive

• Not an error

• Red

• Alerts user that there’s an issue

• Made an error
2016 Census Test

25

• Green message (soft) and red outline

• Participants do not pick on nuance of soft vs. hard
2020 Census: Soft edits

26
2020 Census: Hard edit

27
Challenges & Compromise

• Functionality of messages present programming challenges

• Difficult to test experimentally 

• Have been unable to design and test edit messages due to software 

limitations 

• Rely on United States Web Design System (USWDS) guidance 

https://designsystem.digital.gov/components/alert

• More research will occur on this topic in FY23 using the American 

Community Survey panel
USWDS

29
Guidelines

Hard edits (error)

• Messages should appear with red background color

• Use sparingly 

• Only for information survey needs to allow respondent to continue

Soft edits (warning)

• Messages should appear with yellow background color
Guidelines

Informational alert

• Informational messages should appear with blue background color • Informational messages are rare within surveys

Success alert

• Messages should appear with green background color • These are rare within surveys

31
Guidelines

• The alert message is at the top of the survey screen

• Outline the corresponding Dropdowns, 

Combo boxes, and write-in fields in the same color

32

Please provide your full address.
Challenges & Compromise

• The alert is the message that appears when a validation/edit is 

triggered

• Information messaging that needs to be communicated to the 

respondent

• Software limitations prevented systematic design testing (e.g., of 

color and placement of alert)

• Rely on USWDS guidance 

• More research on edit validations ahead in FY23

33
Thank you!

Shelley Feuer

shelley.b.feuer@census.gov

34
Web Survey Branding 

Jonathan Katz

Center for Behavioral Science Methods, U.S. Census Bureau

Federal Committee on Statistical Methodology 

October 27, 2022

35

This presentation is released to inform interested parties of research and to encourage discussion. 

The views expressed are those of the authors and not those of the U.S. Census Bureau. 

The disclosure review number for this presentation: CBDRB-FY22-CBSM002-015.
36

Banner 

Footer

Wrapper 
Examples of wrappers used in surveys 

37
Goals 

• Wanted to create a uniform look-and-feel for all survey wrappers.

• With the help of Public Information Office, designed survey wrappers 

to embed in testing that would inform guidelines

• Research Questions:

1. Where did users expect these select survey features to be

located?

2. What did users think of redesigned survey wrappers? 

38
Methods

• Conducted unmoderated testing in Qualtrics 

• Series of A/B experiments using nonprobability panels 

• 795 participants 

• 38% Male/62% Female

• Age (mean=46 years old; range 14-90) 

• 12% Hispanic origin/88% not of Hispanic origin

• 74% White only/26% nonwhite only

• 43% High school or less/57% more than high school

• 30% Mobile/ 70% PC

39
Where did users expect these survey features to 

be located? 

1. Logo of the survey sponsor

2. Name of the survey

3. Contact us

4. Frequently asked questions (FAQ)

5. Language toggle 

40
41

• “Typically, surveys 

have the logo of the 

survey sponsor. 

Where do you 

expect to see that 

logo on the page? 

(Click or touch the 

image.)”
Heat map results: Where do you expect to see that logo on the page?

42

*The dark red 

signifies the 

most 

frequently 

clicked (or 

touched, if 

mobile) area
43

Results: Where would you expect to find a Contact us link?
44

Results: Where would you look for the language toggle feature?
What did users think of redesigned survey 

wrappers? 

45
Census Planning Survey 

46
Redesign 1 – People Design

47
Redesign 2 – Logo Only Design

48
Question 1: Which redesign did participants prefer? 

49

OR

Question 2: Did they prefer the chosen redesign or the original design? 

OR Preference from Q1 

above
Census Planning Survey Preference (Redesigns) 

50

61.9%

23.8%

14.3%

0%

10%

20%

30%

40%

50%

60%

70%

People Design Logo Only Design No Preference

Percent of Respondents

Preference for People versus Logo Only

• Chi-square test between 

conditions

• Χ2 (2, N = 265) = 100.76, 

p < 0.01 

• Users preferred the 

people design over the 

logo only design
Census Planning Survey Preference (People versus 

Original design) 

51

67.2%

19.1%

13.7%

0%

10%

20%

30%

40%

50%

60%

70%

80%

People design Original design No preference

Percent of Respondents

• Chi-square test between Preference for people versus original design

conditions

• Χ2 (2, N = 183) = 95.344, 

p < 0.01 

• Users preferred the 

people design over the 

original design
What did we recommend for the final 

guideline? 

52
53
Challenges

• Many respondents skipped the questions asking them to click (or 

touch) where they expected select features.

• Sometimes it was not exactly clear where select survey features 

should be located on the wrapper

• Does preference for the people images lead to better data quality? 

54
Thank you!

Jonathan Katz

jonathan.m.katz@census.gov

55
Progress Indicators

Rachel Horwitz

Demographic Statistical Methods Division, U.S. Census Bureau

Federal Committee on Statistical Methodology

October 27, 2022

*Any views expressed are my own and not those of the U.S. Census Bureau. 56
What is a progress indicator?

• Visual display to inform respondents how they are progressing 

through a survey

• Typically displayed across the top, in the top corner or in the bottom 

corner

• Can provide a percent complete, number of questions (question 2 out 

of 10), or no textual reference

Source: Heerwegh and Loosveldt, 2006

57
Why use progress indicators?

• Social exchange theory

• Respondent agrees to participate and in exchange they receive information 

on the burden of the survey and how they are progressing (Villar, et al., 2013)

• Keep respondents motivated

• Respondents may be more likely to complete if they see they are making 

progress (Villar, et al., 2013)

• If respondents are feeling fatigue, they may see they are almost finished and 

complete the survey (Heerwegh and Loosveldt, 2006)

58
Web survey design issue

• Goal is to set an expectation of time

• Most surveys are not linear

• Not every respondent sees every question

• Questionnaire length can vary depending on pathing

• This results in unpredictable progress indicators

• Static – jump around depending on pathing

• Dynamic – start the progress slow until paths have been established

59
Method

• Conducted a literature review to determine effectiveness

• Google Scholar

• Conference Proceedings

• Books

• Sources

• Experiments

• Meta-analysis

• Usability testing

• Experiments date back to 2001

60
What the research found

• Progress indicators work when

• Respondents believe a task will take longer than it actually ends up taking 

(Yan, et al., 2010)

• If the survey is promised to be short and is actually short (Yan, et al., 2010)

• Start off fast but later slow down (Villar, et al., 2013)

• Progress indicators don’t work when 

• Respondents don’t see early progress (slow to fast) (Villar, et al., 2013; 

Crawford et al, 2021)

• They jump around as pathing changes (Kaczmirek, 2008)

• In other cases, there is no effect (Couper et al., 2001; Yan, et al., 

2010; Villar et al., 2013; Crawford et al., 2021)

61
Guideline

• Do not include progress indicators for survey completion status

• Progress indicators are unlikely to help and may reduce completes

62
Challenges & Compromise

• Some surveys want a way to communicate expectations

• Solution – section progress indicators

• Allows respondents know where they are and what’s to come

• Is not influenced by pathing

• Indicator was not problematic in 2020 Census usability testing (Nichols, et al., 

2017; Olmsted-Hawala, 2018; Olmsted-Hawala, et al., 2019; Olmsted-Hawala, 

et al., 2020)

Source: 2020 Census 63
Thank you!

Rachel Horwitz

rachel.t.horwitz@census.gov

64
Personalizing Questions for Online SelfResponse Modes: Allowing respondents to 

enter in timeframes that fits their situation

Erica Olmsted-Hawala

Center for Behavioral Science Methods, U.S. Census Bureau

Federal Committee on Statistical Methodology 

October 27, 2022

65

This presentation is released to inform interested parties of research and to encourage discussion. 

The views expressed are those of the authors and not those of the U.S. Census Bureau. 

The disclosure review number for this presentation: CBDRB-FY23-CBSM002-001
Income question 

user issues

• The American Community 

Survey (ACS) asks the 

income question over past 

12 months

• Timeframe is “annually”

• People may think about 

the amount of money the 

earn in a different time 

period

• Daily, weekly, monthly, 
We noticed in user 

testing that 

participants may 

have been 

reporting different 

time frames –

67
Example of cost 

of water and 

sewer in the ACS: 

timeframe past 

12 months
Expert review of ACS

• The ACS had an expert review 

done by an outside vendor 

(contractor) and they 

recommended harnessing the 

benefits of online survey design 

for these types of questions.

• They suggested that if we know 

the timeframe, we can ask the 

question that best fits the 

respondent and may get better 

data. 

• Here is an example of what the 

contractor recommended...

69
National Household Education Survey (NHES) 

example of time spent at daycare

70
NHES example of day care costs

71
What’s the 

best design 

for a survey 

question with 

a dynamic 

timeframe? 

• ACS was not able to test this 

topic in the production setting 

• So – our team decided to test 

alternate ways of asking 

timeframe questions

• Focused on personalizing the 

question based on the 

respondent's situation

72
Method

• Two Rounds - modifying the design in each round

• Split-panel survey with 2 conditions

• Between-subjects design

• Three main questions where we captured data:

• Questions on paying for household electricity (amount paid)

• Questions asking for water and sewer (amount paid)

• Question asking about respondent wages (amount earned)

• Follow up question that calculated the income amount for a year and asked if the 

amount was correct, if not, open text field to input the correct amount.
Data collection

• Online survey using Qualtrics platform

• Non-probability panel

• Members of U.S. public who signed up to participate in surveys through a private 

panel provider

• Sample 

• Round 1: 520 completed surveys 

• Round 2: 508 completed surveys

• Quotas set for region and participant age

• No IE browser & PC only

• Data collected: 

• Round 1: June 8-16, 2020

• Round 2: September 2-10, 2020

74
Dropdown design Response options visible at all times

75

Round 1 design
Outcome of first 

test

• It takes more time for users to 

answer the questions with 

response options visible at all 

times than it does to answer the 

question with dropdown design

• There was slightly more missing 

data in the dropdown design. 

• So - we retested with a tweak to 

the dropdown design: switch the 

dropdown fields: first ask for the 

timeframe and second ask for 

the amount

76
• Dropdown design • Response visible at all times

77

Round 2 design
Summary

• Accuracy

• Swapping the two dropdown boxes didn’t 

seem to matter. There was still a slight 

difference in item missingness 

• Frequency was more often left blank in the 

dropdown design than in the response 

visible on screen at all times

• Efficiency

• Takes more time for design questions 

visible on screen at all times

78
Guideline: Personalizing questions to fit 

respondent situation

• When asking question where the answer may come in different 

timeframes – consider allowing respondents to choose the best 

timeframe for them.

• We tested two different “choose how to respond” designs

• Mixed results

• Either design could work, depending on question

• But we recommend the design where responses are visible on screen at all 

times.

79
We recommend: Response visible 

at all times

80
Thank you!

Erica Olmsted-Hawala

Erica.L.Olmsted.Hawala@census.gov

81
Input Field Formatting:

Monetary values

Temika Holland

Economic Statistical Methods Division, U.S. Census Bureau

Federal Committee on Statistical Methodology

October 27, 2022

82

This presentation is released to inform interested parties of research and to encourage discussion. 

The views expressed are those of the authors and not those of the U.S. Census Bureau. 

The disclosure review number for this presentation CBDRB-FY22-ESMD009-005.
Entering Dollar Values

83
Entering Dollar Values

84
Prior Research

• Research suggests that 

providing a visual template 

reduces reporting errors 

• (Couper et. al, 2011)

85
Template example:

Household survey 

Example from the American Community Survey

86
Template example:

Business Survey

Example from the Annual Capital Expenditures Survey

87
Template inside the input field Template outside the input field

88
It is unclear which placement of the 

template is helpful to respondents–

in terms of efficiency, accuracy, 

perceived burden/difficulty

89
Methodology: Participants 

• 515 participants

• 44% Male/ 56% Female

• Age (mean=46 years old; range = 18-96)

• 9% Hispanic origin/91% not of Hispanic origin

• 76.5% White only/ 23.5% non-White only

• 34% High school or less / 66% more than high school 

• .5% Mobile/99.5% PC or Mac

90
Methodology: Study Design

• Between-subjects design

• Two versions of the formatting template placement

• Outside the field (n=256)

• Inside the field (n=259)

• Each version contained four tasks

• Whole dollar

• Thousands

91
Challenge

•Accuracy performance metric

• True value

Our solution

Tasks:

• ‘Simple’ addition problems

• Whole dollar

• Thousands

• Audio stimuli 

• Whole dollar

• Thousands

92
Outside the Field: Whole Dollar

• Task 1: ‘Simple’ addition • Task 2: Audio

93
Outside the Field: Thousands

• Task 3: ‘Simple’ addition • Task 4: Audio

94
Inside the Field: Whole Dollar

• Task 1: ‘Simple’ addition • Task 2: Audio

95
Inside the Field: Thousands

• Task 3: ‘Simple’ addition • Task 4: Audio

96
Results

97
Accuracy

37.48%

21.75%

42.52%

31.46%

39.42%

31.65%

40.78%

39.03%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

Whole dollar_audio *Thousands_audio Whole dollar_add *Thousands_add

Outside the field Inside the field *p<.05

98
Completion Time

• Mixed Model Analysis of log of time

• Longer completion times when format 

is inside the field

• Whole dollar (p<.01)

• Thousands (p=.07)

99
Ease/Difficulty

For the questions you just answered, how easy or difficult was it for you to enter the correct amount using the formatting of the box?

28.79%

14.20%

3.31% 3.50%

22.76%

17.51%

3.11%

6.81%

0%

5%

10%

15%

20%

25%

30%

35%

Extremely easy Somewhat easy Neither easy nor difficult Somewhat difficult/extremely difficult

Outside field Inside field

100
Preference

37.48%

30.87%

25.05%

0%

5%

10%

15%

20%

25%

30%

35%

40%

Outside the field Inside the field No preference

101
Summary

• Formatting inside the field may be helpful when requesting large 

values (thousands) as this resulted in greater accuracy

• Formatting inside the field takes longer to complete when entering 

small values (whole dollar)

• Formatting outside the field was perceived as less difficult 

102
Additional findings 

• Requiring participants to add values 

took longer to complete

• Communicating values via audio 

stimuli took less time for task 

completion 

• Entering smaller values took longer

to complete

103
The Guideline

•For monetary data 

rounded, place the format 

(“.00” “,000.00”) outside 

the field to the right, and 

the $ symbol outside the 

field to the left

•Do NOT allow entry of a 

decimal.

104
Thank you!

Temika Holland

temika.holland@census.gov

105
Questions

• Lora Rosenberger – U.S. Census Bureau Enterprise solution

• Elizabeth Nichols – Navigation buttons

• Shelley Feuer – Edit validations

• Branding – Jonathan Katz

• Progress Indicators – Rachel Horwitz

• Personalizing Questions – Erica Olmsted-Hawala

• Input field formatting for monetary values – Temika Holland

106
Request a copy of the web survey design 

guidelines