# Presentation: 429009

## Developing a Survey Program Evaluation Process

- By
- Brenda G. Cox
- Battelle Memorial Institute
- and
- Nancy J. Kirkendall
- Energy Information Administration

## This research is being conducted under a subcontract Battelle has with Z, INC. from the Energy Information Administration (EIA).  The views expressed are the authors’ and do not necessarily reflect the official position of EIA.

## Performance Assessment Rating Tool (PART)

- Encourages rigorous assessment of government programs
- Intended to boost the quality of government programs
- Provided the impetus for this investigation to design templates to evaluate government programs that involve a family of related surveys

## Overview of Study Activities

- Development of a Survey Evaluation Template
- Development of a Survey Program Evaluation Template
- Test of Concept of the Survey Evaluation Template
- Test of Concept of the Survey Program Evaluation Template

## Components of the Survey Evaluation Template

- Survey Name
- Brief Description of the Survey
- Survey Objectives
- Target Population
- Sampling Frame
- Sample Design

## Components of the Survey Evaluation Template (cont.)

- Sample Allocation
- Data Collection
- Editing and Imputation
- Weighting
- Data Analysis
- Summary of Findings

## Program Evaluation for a Family of Surveys:  Step 1

- Includes a summary evaluation of each component survey
- Survey Evaluation Template used to profile each survey

## Program Evaluation for a Family of Surveys:  Step 2

- Document which respondents report to which survey and data requests made of them
- Identify unnecessary burden on survey respondents across survey components and whether specific subpopulations are over burdened
- Determine if the survey program needs restructuring

## Program Evaluation for a Family of Surveys:  Step 2 (cont.)

- Determine how the data across surveys are used in analysis and presented to public
- Delineate the relationship across designs between data reporting and sample designs
- Determine if one or more surveys should be modified to build synergy across surveys

## Program Evaluation for a Family of Surveys:  Step 2 (cont.)

- Determine if some data items are being collected repetitively across surveys
- Examine the relationship between question wording and consistency from EIA’s and the respondents’ points of view
- Decide if there is a better design for forms and instructions

## Program Evaluation for a Family of Surveys:  Step 2 (cont.)

- Ascertain how well confidentiality protection is applied—is confidentiality assured and response encouraged?
- Examine response rates across surveys—are particular subgroups prone to greater nonresponse?
- Determine if data are reasonably timely and understandable to users

## Program Evaluation for a Family of Surveys:  Step 3

- Document strengths of the program and desirable design elements already in place
- Specify areas for improvement and specific recommendations for improvement
- Suggest enhancements that would increase the utility of the data across surveys

## Questions for the Committee

- Are there additional items that should be included in the Survey Evaluation Template?
- Do some items need to be modified?
- Is this overkill for evaluation of an individual survey?
- Any suggestions for important components that should be included in the Program Evaluation Template?