# Survey Quality Assessments at EIA

Friday April 23 9:15 am

Tom Broene, SMG (Statistics & Methods Group at EIA)

Phone: 202-287-1715 email: <tbroene@eia.doe.gov>

Contents:

\- page 1: relation to our strategic plan

\- page 2: recap from meetings last year

\- page 3: progress, findings, and questions for discussion

\- pages 4-7: attachment 1: description of process and objectives

\- page 8: attachment 2: matrix of measures to be collected

\- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
-

Our new strategic plan places an emphasis on data quality, and includes
a measure for frame coverage (discussed yesterday) and a measure on
survey data quality, namely:

Percent of EIA surveys meeting quality targets.

The survey quality assessment is one tool for setting these
survey-specific targets. This is spelled out in our internal action plan
in item 1d:

This is a new measure that will be implemented using the EIA Survey
Quality Assessment Instrument and Quality Measures Matrix. ... **SMG
will help Offices select an appropriate set of output performance
measures for each survey**; the baseline time period will be 2002. SMG
will support Offices in the initial gathering and compilation of
appropriate performance information for the baseline time period and in
determining EIA-wide percentage values. ...

Performance Measure Implementation Schedule:

Percent of EIA surveys meeting quality targets

  ------------------------------------------------------- ---------------
  **Activity**                                            **Due Date**

  SMG meets with survey groups to explain use of the      7/1/04
  instrument and works with Offices to explain the        
  process and collect the information                     

  Offices (with SMG help) collect performance data for    9/30/04
  each quality measure for each survey for the baseline   
  time period. **Offices set out-year performance targets 
  for selected quality measures**. SMG compiles results,  
  forms EIA-wide percentages for the baseline time period 
  and posts results on the EIA Intranet.                  

  Offices complete annual Survey Quality Assessment       11/15/04
  Instrument and Quality Measures Matrix for 2003 data    
  and provide to SMG.                                     

  SMG compiles results, forms percentages and posts       12/1/04
  results on the EIA Intranet.                            

  Annual Survey Quality Assessment Questionnaire and      Annually by
  Quality Measures Matrix to be completed for calendar    11/15
  year ending 11 months before, provided to SMG           

  SMG compiles results and posts on EIA intranet          Annually by
                                                          12/01
  ------------------------------------------------------- ---------------

This topic was outlined at our meetings with the ASA Committee on Energy
Statistics in April 2003, and the summary notes included:

A new initiative will involve survey managers identifying specific
indicators of the survey process, and is intended to improve survey
process and results. This initiative will involve collaboration between
Survey Managers, SMG (Statistics and Methods Group within EIA), and an
EIA contractor.

The Committee agreed that a survey quality initiative would be useful to
EIA program and survey managers, and recommended:

-   Developing an assessment tool by and for survey managers to
    establish targets;

-   Recognizing differences among surveys due to context, frequency,
    scope, budget, and purpose;

-   Developing feedback mechanism for diagnosing and treating problems
    with survey systems;

-   Adding incentives for major and minor improvements in survey
    systems;

-   Providing survey documentation which is available and up-to-date;
    and

-   Emphasizing the importance of conducting independent survey
    assessments.

Progress

We formed a team with representatives from the survey offices, and
received guidance from a consulting firm with experience in this area.
We developed a short description of our goals (attached - pages 4
through 7) and a one-page form for collecting data (page 8.)

So far, we have only collected data for a few surveys.

-   We did an extensive "evaluation" of one survey and collected more
    data than what is being sought for the assessment.

-   Had a group interview with survey managers in one area.

-   Have some notes from one interview conducted by a survey manager
    with staff.

Findings so far are

-   Good participation so far.

-   It was suggested that we expand to capture all processes. We will
    need to interview more than just the survey managers, but also
    support staff, such as data operations. Have not discussed
    interviewing contracting staff.

-   It is difficult to summarize interviews and determine whether or not
    a major problem exists. Are we developing a concise summary for
    management, or are we updating our documentation?

-   Not sure that we will really share the best practices. One person
    had suggested that the survey managers get together and complete the
    form in one meeting, or maybe a workday.

-   Need a procedure to use findings or suggestions.

Questions for the committee

-   Anything to add or subtract from our matrix form or description?

-   How to make sure findings are used within the office and shared
    among offices?

-   Any suggestions for implementing?

**attachment 1: Description of Process and Objectives**

The EIA Strategic Plan identifies several levels of measures, which
together, define the quality of EIA products and services. At the top
level, measures address mission outcomes, and are directly measured by
the users and customers of EIA information products. At the second
level, measures address EIA outputs, which are measured by product
attributes, such as relevance, reliability, and consistency with
industry structures, timeliness, and quality. At the third level,
measures address process quality, particularly accuracy of the basic
data.

Assessing survey process quality requires a good survey quality
assessment instrument, and a procedure for using the instrument. A good
instrument will have the following elements:

1.  It will *measure the right things*.

2.  It will *ask the right questions.*

3.  It will help guide survey personnel to determine if changes are
    needed and if so, where they are needed.

The right questions will lead to the right quality measures. The
following is a proposal for questions and measures that may be
appropriate for assessing the quality of an EIA survey process. For this
procedure, one should consider what measures are currently available,
how these were selected, and how they are being used. You might also
consider what new measures you would find useful if resources were
available to provide the measures.

## Survey Quality Assessment Instrument

Availability of performance measures. How easy is it to see and use
performance measures for this survey?

> a\. Please describe the status of performance measures for this
> survey - pick the statement(s) that best describes your situation.
>
> i\. Information to support computation of measures not captured by
> processing system.
>
> ii\. Information available, but preparation of reports requires manual
> effort.
>
> iii\. Information available, some automatic reporting of measures
> available.

iv. Information available, automatic report concerning all required
    performance measures is prepared regularly.

> b\. What work is needed to make survey performance measures readily
> available for use? Do you have a planned date when the work will be
> done? Do you need additional resources for this work?

1.  Frames and respondent lists. How current, representative and
    accurate is your frame? (coverage error).

To what extent does the frame cover the target population of the survey?
What is missing from the frame and can you quantify the impact of the
missing part? (For example, if you only survey companies above some size
threshold.)

Quality measures

a.  Coverage rates (where measurable)

b.  Frame/respondent list volatility (births and deaths, changes of
    ownership, both as counts and weighted---because impact of such
    changes in big companies is potentially greater.) What is the impact
    of those changes on data quality. Can you quantify what is/would be
    lost with less frequent frame updates?

c.  Describe your steps taken to review and update frame/respondent
    lists.

d.  Do you have a coordinated frame list for all surveys in your
    division? Is it regularly updated with information from surveys
    based on that list?

```{=html}
<!-- -->
```
2.  For sample surveys only. Sampling Error. What is the magnitude of
    the sampling errors?

Quality measures

a.  What was the target CV or sampling error used in survey design?

b.  What is the achieved CV, sampling error, or relative standard error
    for key survey variables?

c.  How do you use the measures of CV, sampling error or relative
    standard error as part of your quality control activities?

```{=html}
<!-- -->
```
3.  Nonresponse error. What is our response rate and imputation rate?

Quality measures

a.  Survey response rate (counts)

b.  Survey response rate for key variables (volumetric)

c.  Percent imputed for key variables

```{=html}
<!-- -->
```
4.  Measurement error. Can respondents provide the data you ask for?

    a.  Has EIA conducted usability testing of the survey with
        respondents? Describe the procedure and give number of
        individuals tested and dates. Do you have measures on reporting
        error and its effect?

    b.  What approaches do you use to ensure data requested on the
        survey form can and is being provided correctly by the
        respondents?

    c.  Are there questions on the survey that respondents seem to have
        particular difficulties in filling out (resulting in many failed
        edits, or call backs?) What plans do you have to fix these
        problems?

    d.  When did you last consult with respondents concerning their
        ability to provide the information requested?

5.  Processing error. Is our editing procedure maintaining a good
    cost-benefit balance?

Quality measures --

a.  False positives (the number of time data fail edits, but turn out to
    be correct.)

b.  Percentage of forms failing at least one edit.

c.  Percentage of forms requiring a follow-up phone call or email in
    response to failed edits.

d.  What is the net impact of editing on one or more data items? Can we
    estimate what the impact would be with more or less time spent
    editing?

```{=html}
<!-- -->
```
6.  Overall assessments of quality. Do the survey results agree with
    other surveys that try to measure the same or similar things?

    a.  Describe comparisons to external data, and what was learned
        about data quality.

    b.  Describe any input you have received from customers (either from
        user surveys, or by direct message) concerning the quality
        (timeliness or accuracy) of the data.

7.  Use of data collected from other sources, either compiled and
    provided by a third party such as States or purchased from private
    vendors. Answer these questions if you use data from external
    sources.

    a.  Describe any quality control activities you use to make sure the
        data are of sufficiently high quality.

    b.  Describe any manipulations of the data used in the preparation
        of the official "EIA" estimate.

    c.  Are these descriptions available to users of the data?

8.  Do we measure the trade-off between timely data and revisions?

Quality measures

a.  Difference between planned date of publication and actual date, for
    both first release of data and the final data.

b.  Revision between first release of data and final data.

c.  Please explanation any large delays or revisions?

```{=html}
<!-- -->
```
9.  Do we have good measures for how our resources are used?

Quality measures

a.  If more resources were available, what area would you improve and
    what impact would this have?

b.  If no additional resources were available, what re-allocation of
    current resources within the survey area would have a positive
    impact on the data that you can quantify?

c.  Can you measure individual productivity on a weekly or daily basis?

```{=html}
<!-- -->
```
10. Are data accessible to users of the data?

Quality measures --

a.  What information is available from this survey on EIA's website?

    i.  Purpose of survey

    ii. Sources of information,

    iii. How to interpret numerical values,

    iv. Data quality measures? Yes or No

```{=html}
<!-- -->
```
11. Are some of the data items collected considered confidential? If so
    describe the steps taken to protect the data.

    a.  Describe steps taken to make sure unauthorized individuals do
        not have access to individually identifiable data.

    b.  Describe steps taken to protect confidential data in tables or
        public use microdata products.

Based upon the information in the quality measures and the targets or
desired results for the survey, do we need to make any process changes
to improve the quality measures? If so what are the targets for
improvement and how do we plan to make the needed improvements?

**Attachment 2: Quality Measures Matrix**

+--------------------------+------------+-----+----------+-----------+
| Survey Measure           | Availabity | Lat | Target   | Needs     |
|                          | of measure | est |          | Im        |
|                          | A          | va  | (if      | provement |
|                          | =automated | lue | app      |           |
|                          |            |     | licable) | Y or N    |
|                          | M=manual   |     |          |           |
|                          |            |     |          |           |
|                          | U=u        |     |          |           |
|                          | navailable |     |          |           |
+--------------------------+------------+-----+----------+-----------+
| Coverage rates           |            |     |          |           |
+--------------------------+------------+-----+----------+-----------+
| Frame births and deaths  |            |     |          |           |
+--------------------------+------------+-----+----------+-----------+
| Relative Standard Error  |            |     |          |           |
+--------------------------+------------+-----+----------+-----------+
| Number of failed edits   |            |     |          |           |
| which turn out to be     |            |     |          |           |
| correct                  |            |     |          |           |
+--------------------------+------------+-----+----------+-----------+
| \% of forms failing at   |            |     |          |           |
| least 1 edit             |            |     |          |           |
+--------------------------+------------+-----+----------+-----------+
| Survey response rate     |            |     |          |           |
| (counts)                 |            |     |          |           |
+--------------------------+------------+-----+----------+-----------+
| Survey response rate     |            |     |          |           |
| (volumetric)             |            |     |          |           |
+--------------------------+------------+-----+----------+-----------+
| Revision error for key   |            |     |          |           |
| variables                |            |     |          |           |
+--------------------------+------------+-----+----------+-----------+
| Percent imputed for key  |            |     |          |           |
| variables                |            |     |          |           |
+--------------------------+------------+-----+----------+-----------+
| Difference between       |            |     |          |           |
| release date of data and |            |     |          |           |
| last day of the          |            |     |          |           |
| reference period         |            |     |          |           |
+--------------------------+------------+-----+----------+-----------+
| Is there a planned       |            |     |          |           |
| publication date? Y or N |            |     |          |           |
|                          |            |     |          |           |
| Percent of times met     |            |     |          |           |
| planned date(s)?         |            |     |          |           |
+--------------------------+------------+-----+----------+-----------+
| Comparisons to other     |            |     |          |           |
| data                     |            |     |          |           |
+--------------------------+------------+-----+----------+-----------+
| Is source of information |            |     |          |           |
| and interpretation of    |            |     |          |           |
| numerical values         |            |     |          |           |
| available to users? Y or |            |     |          |           |
| N                        |            |     |          |           |
+--------------------------+------------+-----+----------+-----------+
| Has EIA received         |            |     |          |           |
| complaints from          |            |     |          |           |
| customers about products |            |     |          |           |
| related to this survey?  |            |     |          |           |
| Y or N                   |            |     |          |           |
+--------------------------+------------+-----+----------+-----------+