#  ![](media/image1.png){width="5.170833333333333in" height="0.1986111111111111in"}

> **DEVELOPING A SURVEY PROGRAM EVALUATION PROCESS**

Brenda G. Cox

Battelle Memorial Institute

and

Nancy J. Kirkendall

Energy Information Administration

> Presentation for the Fall Meeting of the American Statistical
> Association
>
> Committee on Energy Statistics with the Energy Information
> Administration
>
> October 28 and 29, 2004
>
> Washington, DC

![](media/image1.png){width="5.125in" height="0.19791666666666666in"}

# Number range CHAPTERDeveloping a Survey Program Evaluation Process

Brenda G. Cox and Nancy J. Kirkendall[^1]

The Performance Assessment Rating Tool (PART) was instituted to
encourage rigorous performance assessment to boost the quality of
Federal government programs. For survey programs, an important component
of performance assessment is the evaluation of each of the program's
survey components and then the evaluation of the effectiveness of the
overall program. This paper summarizes the results to date of a study to
develop templates for evaluating individual surveys and to evaluate a
program composed of a family of surveys.

For this investigation, we will develop two templates, one for the
evaluation of an individual survey and one for the evaluation of a
family of surveys that form a survey program. The goal will be to
develop templates that an external group could use as guides to conduct
survey program evaluations. To validate and refine the templates, we
will use the templates to evaluate EIA's Petroleum Marketing Program and
its 11 component surveys. Then, the templates will be updated to create
the penultimate templates for survey and program evaluations.

#  Program Evaluation Template for an Individual Survey

Program Evaluation for a family of surveys must begin with a summary
evaluation of the individual surveys that comprise the survey program.
Understanding the surveys that comprise a program is essential to
assessing the effectiveness of their integration to create the survey
program. The evaluation template for an individual survey will summarize
aspects such as the following:

-   survey objectives, including population subgroups for which separate
    estimation is desired;

-   target population definition, exclusions and inclusions, and
    temporal dimension;

-   sampling frame, including coverage of the target population and
    other quality issues;

-   sample design type and the steps taken to implement it;

-   sample size, amount and how derived, and associated precision;

-   weighting, including nonresponse adjustments and poststratification;

-   response rates and the outcome codes and definitions used for their
    derivation;

-   edit and imputation procedures for missing data items and/or data
    records; and

-   questionnaire content in terms of screening procedures, structure of
    form, and instructions.

Note that the steps in the summary evaluation contain many of the same
steps that would be followed for EIA surveys that are not part of a
\"family of surveys\" and need to be evaluated individually. The summary
review that will be designed as the first step in the program evaluation
will of necessity be less detailed and more descriptive in nature.

# Program Evaluation for a Family of Surveys

The Program Evaluation for a family of surveys will include a summary
evaluation of each survey. The summary evaluation of each survey will
cover the objectives, target population, frame, and sample design as a
minimum, but most likely will comment on the components outlined in the
previous section although not in the detail that would be done were the
survey being evaluated separately. In addition, the Survey Program
Evaluation will examine:

-   which respondents report to which surveys, and the data requests
    made---to identify unnecessary burdens placed on survey respondents
    and whether the survey program needs restructuring;

-   how the data within and across surveys are used in analysis and
    presented to the public and the relationship to the survey
    designs---to determine if one or more of the surveys should be
    modified;

-   the relationship between question wording and consistency from EIA's
    and the respondent's points of view---to decide if there is a better
    design for forms and instructions;

-   how well confidentiality protection is applied---to assure
    confidentiality and encourage response; and

-   response issues and timeliness.

The program evaluation will identify strengths and weaknesses of the
program and will result in recommendations for improvement and
recognition of desirable design elements already in place. In addition
to its use for a particular survey program, the methodology being
developed could also be extended to do evaluations across survey
programs that have related topic areas and populations such as electric
power with natural gas, and petroleum supply with petroleum marketing.

# Task Statement

This project has been partitioned into phases and the activities that
will occur under each phase.

Phase 1 of this task has the goal to produce a survey evaluation
template that can be used as a guide to evaluate individual surveys.
Phase 1 of this task has been completed.

Phase 2 of this task will involve the preparation of a program
evaluation template for conducting the evaluation of a program that
incorporates a family of surveys.

Phase 3 will be devoted to a proof of concept of the survey evaluation
template by evaluating the 11 surveys comprising the petroleum marketing
survey program.

Phase 4 will be devoted to a proof of concept test of the program
evaluation template by conducting an evaluation of EIA\'s Petroleum
Marketing Program. The program evaluation template will be used to
summarize the statistical and survey attributes of the petroleum
marketing survey, highlighting strengths of the program and areas for
improvement.

Phase 5 will involve taking what was learned in evaluating the petroleum
marketing survey program and its 11 component surveys to create
penultimate templates for the survey evaluation and the program
evaluation.

# Questions for the Committee

The survey evaluation template is included in the Appendix. We would
appreciate comments on the template in particular:

-   Are there additional items that should be included?

-   Do some items need to be modified or expanded?

-   Is this overkill for evaluation of an individual survey?

The program evaluation template is still in the preparation phase. We
would appreciate comments on what the committee considers to be the most
important components of evaluating a program that incorporates a family
of surveys.

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# APPENDIX

# 

# 

# Survey Evaluation Template 

# Survey Evaluation Template

Survey Name:

Name of survey and (if applicable) the family of surveys to which it
belongs.

Brief Description of Survey:

Include subject matter, sponsors, and periodicity (one-time, repeated
cross-sectional, or longitudinal.

Survey Objectives:

Describe the primary purpose for the survey. Compile a detailed list of
the uses for which the data are to be input and the research questions
that the data must answer. Indicate whether the survey is primarily
intended for descriptive purposes versus policy research. List reporting
domains for which separate estimation is needed. Indicate when and to
what extent comparisons will be made to other surveys or to previous
cycles of the same survey.

Target Population:

Delineate the entire set of population units for which the survey data
are to be used to make inferences. If business entities are involved,
indicate the level of the business forming the units of the target
population. If inferences are desired for multiple population units,
then document each unit. Be specific so that inclusions and exclusions
from the target population can be completely understood. Specify the
temporal and geographic aspects of the target population units. Specify
the treatment of units whose characteristics change over time. Describe
the treatment of units that enter the population (births), exit it
(deaths), or assume a new identity (movers, ownership changes, mergers).

Sampling Frame:

Describe the list or mechanism used to enumerate population units for
sample selection and data collection purposes. Specify differences
between the units comprising the target population and the units
comprising the frame. Delineate the relationship between frame units and
target population units (one to one, many to one, one to many). Discuss
whether duplicate listings are present and how treated in frame
building. Discuss the extent to which target population members may be
missing from the frame and what is known about the characteristics of
the uncovered units. If appropriate, discuss size measures
available/constructed for the frame and their accuracy.

Sample Design:

Note whether the survey is based upon a sample or a census of population
members. For samples, provide an overview of the design, including
whether selection procedures involve explicit or implicit
stratification, the sample selection method(s), any multiple stages or
phases of selection, definition of strata, use of size measures, and
unique or unusual approaches. Present the rationale for the sample
design chosen.

Sample Allocation:

Summarize the sample size allocation to strata and the methodology
involved. Describe the precision constraints/guidelines used to
determine sample size in terms of completed interviews. List assumptions
made about response and eligibility rates that were used in converting
the sample allocations into the number of cases to be selected from each
strata.

Data Collection:

Summarize steps in data collection. Review outcome codes in terms of
capture of available information about nonrespondents based upon where
nonresponse occurs in the interview process. Review response and
eligibility results for the survey---tabular display best with outcome
codes and formulas given. Screener and questionnaire review in terms of
target population definition, capture of data needed for weighting,
structure of form, and instructions. Review of mailout package materials
if pertinent.

Editing and Imputation:

Review data cleaning and editing methods. Document relative magnitude of
data records changed in processing or set to missing and whether too
much/not enough editing has been done. Review imputation procedures in
terms of method(s) used, their appropriateness, and the extent of
missing data replaced. Document QA procedures implemented to ensure data
quality.

Weighting:

Summarize steps in weighting and treatment of nonresponse/undercoverage.
QA procedures implemented during weighting. Agreement between weight
totals and external data sources.

Data Analysis:

Review reports to determine uses made of the data and domains of
particular importance based upon frequency of use. Review other uses
that are being made of the data, such as using the data as input for
model building. Relate findings back to sample design and data capture.

Summary of Findings:

Summarize the findings of the evaluation with respect to the ways in
which the survey is incorporating appropriate survey methods and/or
creative or innovative procedures. Summarize any areas where the survey
departs from what would be considered best practices and provide
recommendations for improvement.

[^1]: This research is being conducted under a subcontract Battelle has
    with Z, INC. for Task Order 04-EI-40503.000 from the Energy
    Information Administration (EIA). The views expressed are the
    authors' and do not necessarily reflect the official position of the
    Energy Information Administration.