Jean E. Fox

Bureau of Labor Statistics

     Discussant

FCSM

October 27, 2022

FCSM Session:Web Survey Design Standards for the Enterprise Data Collection System at the U.S. Census Bureau

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Wow!

Huge effort

Exciting research

Great to have all this together

Often work that is put off

Too difficult

Too “minor”

HIPPO decisions

Ambitious schedule – 80 surveys in 8 years?

‹#› — U.S. Bureau of Labor Statistics • bls.gov

HIPPO = Highest Paid Person’s Opinion

2

Overview of DICE - Miller

Data Ingest and Collection for the Enterprise (DICE) program 

Vision:  “To provide consistent, cost-effective, efficient, standard, and scalable data collection operations and technical solutions that support all directorate needs”

Impressive collaboration and agreement

Focus of this session is on the Questionnaire Design and Metadata System (QDM) and Electronic Data Collection 

Good framework for the session

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Leslie Miller

3

DICE Web Standards Team

Mission

Develop empirically-based web survey design standards that maximize data quality and minimize respondent burden

Research: We programmed 9 separate experiments using Qualtrics to test 3 to 5 different design issues in each one. 

Outcome

Design standards that can be implemented for all Census Bureau web surveys

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Linear Navigation - Nichols

Often overlooked design components

Nice experimental design to get people to answer a question and select a button

Recommendation seems logical and consistent with modern web design

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Beth Nichols

5

Edit Validations - Feuer

Generally hard to study edits (they’re not always triggered)

Nice idea to have participants say what they would do if they saw the message.

Good recommendation to go with the USWDS guidelines

Looking forward to their next study.

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Shelley Feuer

6

Web Branding - Katz

Great use of the heat maps 

Next step might be to see if people can find the features

Seems likely, but with banner blindness people may miss them

Would like more information about the reasons for preferring the “people” version vs the “logo only” design.

It might be about placement of the logo.

Curious about the evolution from the preferred “people” wrapper to the final example

It’s quite a bit different

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Jonathan Katz

7

Progress Indicators - Horwitz

Studies on progress indicators

Researchers:  Don’t use progress indicators.

Survey owners:  Let’s use progress indicators.

What does it mean that progress indicators don’t work?

Are respondent annoyed?  Or do they drop off?

Recommendation is a good compromise

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Rachel Horwitz

8

Personalizing Questions – Olmsted-Hawala

Allowing people to select the timeframe to report

Great value in this approach

I’d hoped the final design would have done better

Can edits warn of some possible frequency mismatch?

Confirmation of annual rate seems helpful

Especially when the frequency is missing

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Erica Olmsted-Hawala

9

Inputting Monetary Values - Holland

Good approach to testing so you know the correct response

Challenging results to make a recommendation

No clear story

Recommendation to use the formatting outside the field seems reasonable

May be worth doing some moderated studies next to explore:

Why the overall accuracies were all fairly low (<50%)

Get more insight into what made the outside the field formatting easier.

‹#› — U.S. Bureau of Labor Statistics • bls.gov

Temika Holland

10

Looking to the Future

Explore differences for cell phone users

Publish and Present

Your guidelines

Journal article bringing all this together

With more information about methodologies and sample sizes

And about the quality of the participants recruited through Qualtrics

How implementation goes

Do you have to make exceptions to the guidelines?

Share more research and progress on the whole DICE program

‹#› — U.S. Bureau of Labor Statistics • bls.gov

11

Jean E. Fox

Fox.Jean@bls.gov

Contact Information

‹#› — U.S. Bureau of Labor Statistics • bls.gov

# image1.jpeg

# image2.png

# image3.png

# image5.png

# image4.png