# Presentation: 083432

## External Review of Survey Programs: A Progress Report

- By
- Brenda G. Cox
- Battelle Memorial Institute
- and
- Nancy J. Kirkendall
- Energy Information Administration

## This research is being conducted under a subcontract Battelle has with Z, INC. from the Energy Information Administration (EIA).  The views expressed are the authors’ and do not necessarily reflect the official position of EIA.

- This research is being conducted under a subcontract Battelle has with Z, INC. from the Energy Information Administration (EIA).  The views expressed are the authors’ and do not necessarily reflect the official position of EIA.

## Overview of Study Activities

- Develop Survey and Program Evaluation Templates 
- Test of Concept of Survey Evaluation Template 
- Test of Concept of Program Evaluation Template 
- Produce Penultimate Templates
- Document Findings

## Program Evaluation Template

- Description of Program
- Program Objectives
- Target Populations
- Sampling Frames
- Program Design
- Data Collection and Processing
- Data Analysis, Products, and Documentation
- Summary of Findings

## Test of Concept Conditions

- Only publicly available documentation used for the evaluation
- Limited or no input from or interactions with program staff at EIA
- Written summaries of findings shared with program staff to obtain comments and corrections

## Template Testing: Survey 

- Six Petroleum Marketing Surveys Evaluated
- Survey Methods  Summarized and Critiqued For Each Item on Template
- Recommendations or Endorsements Provided
- Results Documented for EIA Staff Review

## Testing Results:  Survey

- The Survey Evaluation Process Worked
- Recommendations Need to Be Labeled and Presented for Each Item
- Design Attributes and Flaws Were Repeated Across Surveys
- Lack of Survey Methods Reports Impeded Evaluation

## Typical Survey Findings

- Survey Objectives Need to Be More Completely Specified
- Target Population Definitions Need to Be Made More Specific

## Typical Survey Findings

- Coverage and Frame Updating Procedures Should Be Specified
- Insufficient Information to Evaluate Editing Procedures
- Mailout Package Is Well Designed But Needs Tweaking

## Testing Results: Program

- No Overall Program Documentation
- Survey Evaluations Were More Important Input than Originally Thought
-  Variations Across Surveys Both Intentional and Due to Happenstance

## Program Findings

- Program-Level Conceptual Design Plan Needed
- Population Definitions and Reporting Conventions Should Be Standardized Across Surveys
- Methods Documentation Needed for Program and Surveys

## Remaining Activities

- Revision to Create Penultimate Templates Based Upon Results
- Documentation of Templates and their Use

## Questions for the Committee

- Are there additional items that should be included in the Program Template?
- Do some Program Template items need to be modified or expanded?
- Are there Program Template items that should be deleted?

## Contact Information

- Brenda G. Cox:  (703) 875-2983 or _[coxb@battelle.org](mailto:coxb@battelle.org)_
- Nancy J. Kirkendall:  (202) 287-1706 or _[Nancy. Kirkendall @eia.doe.gov](mailto:Nancy.Kirkendall@eia.doe.gov)_