# Presentation: 305861

## Introduction to Designing a High Quality Energy Survey

- Dr. Nancy J. Kirkendall
- Energy Information Administraton

## Outline -- today

- Overview of Survey Process steps and sources of error
- Concept of Total Survey Error
- Defining Frames and Respondents
- Controlling Measurement Error

## Outline -- tomorrow

- Processing Error
- Nonresponse and Imputation
- Quality Control Methods
- Evaluating Survey Quality

## Steps in Designing a Survey

- Identify needed information
- Consider flow of product in industry to identify points of concentration – and the survey population
- Goal is to describe attributes of the population by conducting a survey
- Access to the population requires a **frame** (eg a list of the population entities.)

## Frames and Sampling

- Watch for** Frame Errors** (omissions) -- usually contribute to understatement of totals
- Determine whether a sample is needed.  (Traditional sampling requires a sample design with a probability mechanism)
**Sampling Error** results from estimating at total from a sample (variance easily estimated, no bias)

## Measurement

- Identify respondents – work with them to gain cooperation, assist in development of form they understand requesting data they have.
- Watch for **Measurement error --** results if form is confusing, respondent does not have data.
- Result may be bias or increased variance.

## Collection and Processing

- Design form, send to respondents, collect by mail or telephone
- Enter data into Processing System.
- Watch for **Processing Error** -- occurs during data entry, cleaning and tabulation.
- Result may be increased variance, or increased bias.

## Nonresponse

- Watch for **Nonresponse Error** --  the error due to failure of some respondents to submit information.
- May result in bias or increased variance depending on information available for imputation.
- Imputation is an estimate you prepare for a missing respondent.

## Total Survey Error

- Accuracy – one aspect of quality
- Quantifying accuracy
- Sampling error
- Coverage and nonresponse error
- Specification and measurement error
- Data processing error
- Evaluation methods

## Focus on Accuracy

- Data must be of sufficient accuracy for decision-making
- Other dimensions (timeliness and cost) are constraints
- “Sufficiently accurate” difficult to define
- Depends on the many contributions to survey error – or *total survey error*, as well as uses of the data

## What is data accuracy?

- Closeness to “truth”
- Absence of “error”
- Need to understand survey error in order to achieve accuracy in survey design

## Components of Total Survey Error

- Sampling error – due to interviewing sample rather than entire population
- Nonsampling error
- Frame error
- Nonresponse error
- Processing error
- Measurement/ specification error

## Nonsampling errors -- detail

- Frame error
- Omissions
- Inclusions
- Duplicates
- Nonresponse error
- No form received
- Item not filled in

## Nonsampling Errors (Cont’d)

- Processing Error
- Data entry, Coding, Editing, Tabulating
- Measurement Error
- Respondent understanding
- Form complexity
- Data available in company
- Interviewer effect (if telephone)

## Impact of Nonsampling Errors on Estimates

- Sampling error usually the only variance cited
- Nonsampling error adds to sampling error, but is usually unknown and must be controlled.
- Systematic – errors that tend to be in the same direction – makes estimate too high or too low – called bias.
- Variable – errors that tend to be in different directions – increased variance, greater uncertainty.

## Total Survey Error

**Large variance, **
**small bias**

**Small variance,**
**large bias**

## Truth is unknown

**Large variance **

**Small variance**

## What is Mean Square Error?

- Average distance squared from “hits” to bullseye equals the sum of
- Average distance squared from hits to center of hits (bias squared)
- Distance squared between center of hits and bullseye (variance)
- Or  MSE = bias2 + variance

## Risk of Bias and Variance by Error Source

## What is mean squared error

- MSE = bias2 + variance
- MSE = (biasframe +biasnonresp + biasprocess
- 			+bias meas )2  + Varsamp + Varnonresp
- 			+ Varmeas

## Survey Design Objective

- Minimize MSE of estimates
- Subject to cost
- OR
- Minimize cost
- subject to minimum MSE

## Conclusions

- Survey design involves allocation of resources using incomplete and imperfect information
- Objective should be to minimize total survey error subject to cost constraints

## Frame Coverage

## Defining the Scope of Survey and Frame

- Define what you want to measure, determine  which part of industry is optimal, where data exists in company
- 	-- custody versus equity?
- 	-- company versus establishment?
- 	-- specific person who knows or has data
- 			knowledge versus willingness 			versus authority to cooperate
- 	-- when are data available?

## Energy Industries Highly Skewed

- 11% of all establishments account for 70% of all business
- Sample design must take frame size (counts) and respondent size (skewness) into account
- Size is an important analysis variable

## Energy Companies

- May need more than name and address—eg. industry type, geography, size
- Difficult to track “births” and “deaths” of establishments
- When is an entity “in operation”?
- How should mergers be handled?
- Frames get out-of-date

## Types of Frame Errors

-  Coverage errors
- Missing units
- Duplicatates
- Extranegous units
-  Classification errors
- Industry type (SIC, NAICS)
- Geography
- Size

## Types of Frame Errors (Cont’d)

- Contact errors
- Address incomplete or incorrect
- Contact name incorrect
- Phone number incorrect
- If, frame not current there are errors

## Frame coverage issues

- New operations, small establishments
- Changes
- Definition of “in operation”
- Births, deaths
- Unit of interest?  Location, administrative, legal?

## Coverage Bias

** ****Not on Frame**

**On Frame**

**Total Population**

## Frame undercoverage – the impact

- Estimated totals will be too small by the amount missing on the frame
- Sometimes can identify types of undercoverage to assess impact
- Comparisons with other data sources may reveal frame undercoverage

## Controling Measurement Error

## Determine Collection Mode

- Self-administered questionnaire most common, (except for end-user surveys)
- Mail
- Electronic (including fax, email)
- Internet
- Telephone interviewing
- For survey with few questions
- For nonresponse followup

## Collaborate with Respondent to reduce measurement error

- Identify appropriate respondent at establishment.
- May require preliminary contact with establishment
- Or, as part of initial contact to solicit cooperation
- Send materials to respondent
- Form and introductory materials

## Factors In Response Process

- Respondent knowledge, authority, desire to respond
- Respondent’s assignment of priority for this response (voluntary versus mandatory, competing tasks, sensitivity of information)
- Respondent’s comprehension of requested information (unit of response, specific data)
- Ease of data retrieval from records, number of others involved in getting data
- Company policy on response

## Measurement error

- Interviewer (usually by telephone)
- Can help with comprehension
- More difficult for respondent to find data from records if requested by phone phone.
- Respondent
- Data requested not in company records, or does not exactly conform to records
- Others involved in obtaining information
- Turnover in record-keeper/respondent

## Measurement Error cont’d

- Form
- Appearance counts
- Use definitions on forms (minimize need to go to instructions)
- Facilitate computations
- Involve subject matter expert in developing wording of questions
- Use cognitive methods with respondents to verify concepts, make sure question wording consistent, and assure requested information available in company records

## A fancy way to say talk to people and use the information they provide 
Work with experts to understand the issues
Work with industry to gain their approval
Work with the person who fills out the form to make sure they understand and have data

- Work with experts to understand the issues
- Work with industry to gain their approval
- Work with the person who fills out the form to make sure they understand and have data

- Cognitive Methods

## Some Techniques for Survey Evaluation

| Evaluation Method | Stage | Purpose |
| --- | --- | --- |
| Expert review of questionnaire
  unstructured
  structured
 | Design | Identify problems with questionnaire layout, format question wording, order and instructions |

## Some Techniques for Survey Evaluation

| Evaluation Method | Stage | Purpose |
| --- | --- | --- |
| Cognitive methods | Design/Pretest | Evaluate one or more stages of response process |