# Presentation: 517108

## Testing EIA Surveys

- ASA Committee on Energy Statistics
- April, 2004
- Stan Freedman
- Robert Rutchik

## EIA Testing Methods

- Pre-survey design visits
- Cognitive interviews
- Without respondent debriefings
- With respondent debriefings
- Respondent debriefings on completed survey forms
- Usability testing

**Notes:**
EIA has used 4 testing methods for it’s surveys.

Pre-survey design visits – Meetings with potential survey respondents prior to survey development to determine if they have the data that EIA needs

Cognitive interviews – Meetings with respondents to test draft forms for comprehension and navigation
Without debriefing
With debriefing – expand cognitive interview to include discussions of record keeping, timeliness of response, questions about respondent burden and data accuracy.

Respondent debriefings – Draft surveys are completed prior to the site visits and respondents are debriefed on comprehension, navigation, business record keeping practices, burden and accuracy

Usability testing – Used on electronic surveys to test navigation functionality.

## Pre-survey Design Visits

- Conducted prior to survey design
- Review business record keeping practices and definitions
- Determine the accuracy and timeliness with which the respondent can report
- Modifications may be made to requirements based on data availability

**Notes:**
Conducted prior to survey design
Used during the development of the survey form to match requirements to what the respondents have

Review business record keeping practices and definitions
Units of measure
Frequency of collection and recording
Level of aggregation
Correspondence of companies definitions with EIA’s definition
Issues of data confidentiality

Determine the timeliness and accuracy of respondent reporting
How soon after the report period can the respondent provide data
How accurate are these data – estimates, final measures or something in-between

Modifications may be made to requirements based on data availability
Original concepts behind the survey will be adjusted to fit what’s available.

## Cognitive Interviews Without Debriefings

- Respondents do not see the questionnaire prior to the interview
- Structured protocol
- Test question wording and interpretation
- Test understanding of instructions and definitions
- Test navigation

**Notes:**
Classic cognitive interview

Gauge respondents first take on the questionnaire and instructions

Most formal type of interview we do at EIA
Very little interaction with the respondent
Probes are design primarily to get the respondent to continue thinking out loud and to provide their reactions to the survey

Respondents are not debriefed on “What the right answer is”.

## Same initial steps as in interviews without debriefings plus
Extensive discussion about record keeping 
Questions about data measurement and accuracy
Respondent is asked for alternative question wording and alternative measures for the survey concepts 

- Same initial steps as in interviews without debriefings plus
- Extensive discussion about record keeping
- Questions about data measurement and accuracy
- Respondent is asked for alternative question wording and alternative measures for the survey concepts

**Notes:**
Same initial steps as in interviews without debriefings plus
Same as discussed in previous slide

Respondent does not see the form prior to the interview

Debriefing often takes the form of the pre-survey design visit.
Extensive discussion about record keeping 
Questions about data measurement and accuracy
Respondent is asked for alternative question wording and alternative measures for the survey concepts 

Combine aspects of cognitive interviewing and pre-survey design visits in the same session.

Often used when there has not been time for pre-survey design visits.

## Respondent Debriefings on Completed Survey Forms

- Respondents receive survey form prior to the interview and are asked to complete it
- Questions asked are similar to pre-survey design visits:
- Do respondents have data and can they provide it
- Review definitions and instructions
- Review measurements points for the data
- Review business record keeping practices
- Determine accuracy and timeliness

**Notes:**
Major difference with previous methods is that the respondent is asked to complete a proposed survey form prior to our visits.

We have found that respondents are more likely to read instructions and take time to understand them than when presented with the form “cold”.  More “realistic” to the way a respondent completes one of our surveys.

Important in business surveys, because they rely heavily on linking cognition with company business record keeping practices.

## Mostly done on browser based survey
Test navigation
Test if respondent can find and use functional features:
Save files
Print
Submit

- Mostly done on browser based survey
- Test navigation
- Test if respondent can find and use functional features:
- Save files
- Print
- Submit

**Notes:**
Conducted on final surveys that have been internet enabled

Main purpose is to test navigation and functionality of what the respondent sees on the computer.

Tests functionality of the system “in the field”

## What are some of the strengths and weaknesses of what EIA has been doing?
What could we be doing to evaluate the effectiveness of the methods we have been using?
What additional techniques could EIA use for developing and testing establishment surveys?

- What are some of the strengths and weaknesses of what EIA has been doing?
- What could we be doing to evaluate the effectiveness of the methods we have been using?
- What additional techniques could EIA use for developing and testing establishment surveys?