Evaluating Previously Reported Data in the Census of Agriculture: Results from Usability Testing

For

2021 FCSM Research and Policy Conference

November 4, 2021| Washington, DC

By

Heather Ridolfo and Ken Pick

www.eia.gov

U.S. Energy Information Administration

Independent Statistics & Analysis

1

Disclaimer

All research presented today was conducted at the National Agricultural Statistics Service (NASS)

The findings and conclusions in this presentation are those of the authors and should not be construed to represent any official USDA or U.S. Government determination or policy

Heather Ridolfo, 2021 FCSM Research and Policy Conference

November 4, 2021

2

2

Background

Establishment respondents are surveyed frequently, sometimes multiple times per year

Responding can feel burdensome and redundant

Use of previously reported data (PRD) may ease that burden

PRD in the 2022 Census of Agriculture Content Test web questionnaire

Experiment 

Usability testing

Heather Ridolfo, 2021 FCSM Research and Policy Conference

November 4, 2021

3

3

Research Questions

Does displaying PRD assist respondents?

Does displaying PRD impact efficiency?

Does displaying PRD impact measurement error/data quality?

Heather Ridolfo, 2021 FCSM Research and Policy Conference

November 4, 2021

4

4

Methodology

Nine usability interviews in April–May 2020

Conducted remotely using WebEx and Zoom

Respondents used their own devices/browsers to view survey

PRD presented as prefilled answers

Usability Testing focused on PRD messaging and PRD use in 5 sections

Heather Ridolfo, 2021 FCSM Research and Policy Conference

November 4, 2021

5

5

PRD Messaging

6

Heather Ridolfo, 2021 FCSM Research and Policy Conference

November 4, 2021

6

First PRD Statement

Heather Ridolfo, 2021 FCSM Research and Policy Conference

November 4, 2021

7

7

Research Questions

Heather Ridolfo, 2021 FCSM Research and Policy Conference

November 4, 2021

8

8

PRD Statements – Recommendation and Outcome

Recommendation:

Keep a statement on a separate screen at the beginning of the survey alerting respondents of the use of PRD and setting respondents’ expectations on how the PRD will be displayed and how they can use it

Outcome: 

Revised first PRD statement instructing respondents to correct information that is no longer accurate

Second PRD statement only appears on first screen with PRD

Heather Ridolfo, 2021 FCSM Research and Policy Conference

November 4, 2021

9

9

Reactions to and Utility of PRD

10

Heather Ridolfo, 2021 FCSM Research and Policy Conference

November 4, 2021

Positive Aspects of PRD

Respondents generally like the idea of PRD

When no change, PRD appears to reduce reporting burden 

(retrieval and judgement steps in response model)

When no change, PRD appears to improve efficiency 

(comprehension and communication steps in response model)

		“Anything that saves me time is good.” 

Heather Ridolfo, 2021 FCSM Research and Policy Conference

November 4, 2021

11

11

Example of Where PRD Worked Well

Heather Ridolfo, 2021 FCSM Research and Policy Conference

November 4, 2021

12

12

Negative Aspects of PRD

When change occurred, PRD appeared to increase response burden and respondents questioned its source

Changed occurred for different reasons:

Information has changed

Different respondent reported in the past

Respondent entered information incorrectly in past (comprehension issue, typing issues, etc.)

Errors in reading PRD into web instrument

Use of PRD reduced the reading of questions and instructions (may reduce data quality)

If answered incorrectly in the past, errors will not be corrected

May not capture change

Current presentation of PRD did not align with respondents’ expectations

Web functionality worked against PRD

Heather Ridolfo, 2021 FCSM Research and Policy Conference

November 4, 2021

13

13

Where PRD Did Not Work Well: Example 1

Heather Ridolfo, 2021 FCSM Research and Policy Conference

November 4, 2021

14

14

Where PRD Did Not Work Well: Example 2

Heather Ridolfo, 2021 FCSM Research and Policy Conference

November 4, 2021

15

15

Where PRD Did Not Work Well: Example 2

“I’ve probably answered this with ‘one’ all along... my daughters would be insulted.” 

Heather Ridolfo, 2021 FCSM Research and Policy Conference

November 4, 2021

16

16

Screen 1

Screen 2

Heather Ridolfo, 2021 FCSM Research and Policy Conference

November 4, 2021

Where PRD Did Not Work Well: Example 2

17

17

Where PRD Did Not Work Well: Example 2

Heather Ridolfo, 2021 FCSM Research and Policy Conference

November 4, 2021

18

18

Summary

Respondents had positive reaction to PRD when there was no change

Some items can be easily answered without PRD

Respondents did not always read full questions and instructions when PRD present

When current answer is different from PRD, burden is increased and data quality may be affected

Heather Ridolfo, 2021 FCSM Research and Policy Conference

November 4, 2021

19

19

PRD Recommendations

Recommendations:

Conduct analysis to understand the variability of data from year to year, or census to census, before deciding on where to use PRD

When using PRD, use it for all questions in a section, if appropriate

Conduct additional review of dependent interviewing literature on how to present PRD in web form

Align web functionality with PRD functionality

Outcome:

PRD removed from the personal characteristics section

Heather Ridolfo, 2021 FCSM Research and Policy Conference

November 4, 2021

20

20

Thank you!

Heather Ridolfo

Heather.Ridolfo@eia.gov

Ken Pick

Kenneth.Pick@eia.gov

Heather Ridolfo, 2021 FCSM Research and Policy Conference

November 4, 2021

21

21

# image2.png

# image3.png

# image4.png

# image5.png

# image6.png

# image7.jpg

# image8.jpg

# image9.png

# image10.jpg

# image11.png

# image1.jpeg