Title: Evaluating the Quality of EIA Analyses Using a Survey of
Independent Expert Reviewers -- Research Design

**Date: October 9, 2003**

**Presenter: Bill Weinig, (202) 287-1709; william.weinig@eia.doe.gov**

**Contributors: Howard Bradsher-Fredrick, Tom Broene, Stan Freedman,
Alethea Jennings (editorial) and Herb Miller**

# Background

EIA is presently in the process of developing a strategic plan for the
five-year period 2004 -- 2008. The plan is under development and is not
available at the time of this writing. As such, the plan and this work
should be understood as "work in progress," i.e., an early attempt to
seek advice from the Committee.

In the draft plan, the Agency is intent on:

-   Becoming a performance-based organization,

-   Using a systematic agency-wide process to review all on-going work
    activities, and

-   Involving customers and stakeholders.

EIA has been conducting customer surveys and employing other
methodologies to assess the quality, timeliness and other attributes of
EIA products for a number of years. For example, EIA has surveyed
incoming callers to the National Energy Information Center as a way to
quantifiably evaluate the level of customer satisfaction on EIA's
products and information services.

However, a systematic quantitative assessment of attributes related to
EIA analytical products has not been conducted.

This work-in-progress paper is about an effort to quantifiably assess
the quality of EIA's analytical tools and analyses.

# Scope

Considering the limited resources available to assess the quality of EIA
analytical products, it is important for EIA to work within existing
institutional structures in order to attain any associated quality
measurement goals. Possible sources of information and data are from the
Independent Expert Reviewers (IERs) and outside technical experts who
conduct Category I reviews on behalf of the Statistics and Methods
Group. These persons are knowledgeable about EIA products in general.
EIA draft products are sent to them for review, and they can be easily
and readily contacted. However, the number of such external experts
called upon to conduct reviews and provide advice to the Agency is very
limited. For example, in FY 2003 six products were independently
reviewed through the IER Program by a total of 19 reviewers.

Independent Expert Reviewers are typically Nationally recognized
subject-matter experts in specific energy-related fields. They are
frequently but not always academics. EIA seeks and pays for their advice
with competitively awarded, fixed-price contracts and single-source
honoraria. Lists of subject-matter experts are maintained, as specific
competitive lists are drawn from these lists and developed cooperatively
between the program office and the Statistics and Methods Group. The IER
contracts are managed within SMG. These contracts are more recently
co-funded between the sponsoring office and SMG. The number of
Independent Expert Reviewers and the EIA products reviewed is shown in
Table 1.

Table 1: Reviews and EIA Products reviewed (Conducted in Fiscal Year
2003)

  --------------- -------------------------------------------------------
  Number of       Title of Materials Reviewed
  Reviewers       

  4               EIA\'s Assumptions of the Costs of New Domestic Nuclear
                  Power Plant Construction Based on Outside, Industry
                  Data

  3               EIA\'s Projections of Natural Gas Supply in the Oil and
                  Gas Simulation Model

  2               National Coal Production, Distribution and Price in the
                  Coal Market Model

  5               Draft Analysis of the McCain-Lieberman requested
                  Analysis of Carbon Reduction

  3               Analysis of Hale\'s Electricity Transmission Study

  2               Regional Coal Production in the U.S. Southwest, with
                  Focus on Supply, Transportation and Price
  --------------- -------------------------------------------------------

A summary of Category I reviews conducted by two EIA contractors follows
in Table 2.

(Continued)

Table 2: EIA Products Reviewed Under the Category I Review Process
(Conducted in Fiscal Year 2003)

+----------------+-----------------------------------------------------+
| FY 2003 Months | Category I Materials Reviewed by Professors Robert  |
| and Number of  | Trost and Fred Joutz, George Washington University, |
| Products       | Washington, D.C.                                    |
| Reviewed       |                                                     |
|                | Note: Does not include reviews done by SMG Staff.   |
+----------------+-----------------------------------------------------+
| October (0)    | None                                                |
+----------------+-----------------------------------------------------+
| November (1)   | Annual Energy Outlook including graphs              |
+----------------+-----------------------------------------------------+
| December       | Market Trends, Forecast Comparison sections and     |
|                | Assumptions Appendix of the Annual Energy Outlook   |
| \(1\)          |                                                     |
+----------------+-----------------------------------------------------+
| January (0)    | None                                                |
+----------------+-----------------------------------------------------+
| February (0)   | None                                                |
+----------------+-----------------------------------------------------+
| March (0)      | None                                                |
+----------------+-----------------------------------------------------+
| April (1)      | International Energy Outlook                        |
+----------------+-----------------------------------------------------+
| May (0)        | None                                                |
+----------------+-----------------------------------------------------+
| June           | Annual Energy Outlook 2003 Domestic Natural Gas     |
|                | Projections                                         |
| \(3\)          |                                                     |
|                | Analysis of McCain Lieberman Bill, S139             |
|                |                                                     |
|                | Coal Market Module Documentation                    |
+----------------+-----------------------------------------------------+
| July (0)       | None                                                |
+----------------+-----------------------------------------------------+
| August         | Price Responsiveness in the NEMS Building Sector    |
|                | Models (2003)                                       |
| \(2\)          |                                                     |
|                | Clear Skies/Carper Analysis: Service Report on      |
|                | Multi-Pollutant Legislation requested by the Senate |
+----------------+-----------------------------------------------------+
| September      | Comparative Evaluation of Two Methods to Estimate   |
|                | Natural Gas production in Texas                     |
| \(2\)          |                                                     |
|                | Annual Energy Outlook 2004 Tables                   |
+----------------+-----------------------------------------------------+

Note: Data from Inder Kundra, SMG, EIA.

Thus, in Fiscal Year 2003, ten products were reviewed by outside
consultants in the Category I review process.

**[Objectives]{.underline}**

The following are the research objectives of this study:

-   To determine to what extent the EIA analysis and forecasting
    products are relevant and reliable.

-   To determine to what extent the EIA analysis and forecasting
    products are consistent with changing industry structures.

-   To determine the quality of EIA analytical and forecasting products

-   To determine the timeliness and usefulness of EIA analytical and
    forecasting products.

We believe these objectives are compatible with EIA's emerging strategic
plan.

## Methodology

We propose to ask Independent Expert Reviewers to complete a short set
of closed-ended questions. A 5-point Likert scale could be employed. It
is important that the selected wording used in the survey be similar to
that used in the strategic plan. This will provide a mechanism in which
to quantitatively measure the various quality attributes of EIA analysis
and forecasting products. The questions (in a Word format) may be found
in Attachment 1

Considering the relatively small number of IERs available to be
surveyed, a very small number of selected independent reviewers should
be used for the purposes of pretesting the survey instrument. Our
current thinking is that 2 or 3 independent reviewers should be used for
the pretest. The remaining reviewers would then be available to serve as
respondents for the regular testing. We think a survey that can be
completed by a respondent in no more than 15 minutes would be
appropriate. If this works well, we would include the questionnaire in
subsequent IER contracts as a deliverable once the reviews were
completed.

The reviewer responses are to be sent to a disinterested party so that
there will be no connection between the respondents' evaluations and
either the EIA program office whose materials are under review or the
IER Program Manager.

A difficulty arises in that Independent Reviewers may not be familiar
with the quality of the end product actually disseminated by EIA, or
that there may be a substantial delay between the returned "review" and
the publication of the analysis or use of the reviewed model. The
individual Independent Expert Reviewer may (1) have very
product-specific knowledge based on conducting the review, and also have
(2) an excellent overview of the general quality of EIA products beyond
those which s/he have been asked to review. This might mean that EIA
should consider asking the reviewer to evaluate EIA products "in
general' rather than the specific product sent them. Thus, the
capability of IERs to respond to the survey requires careful
consideration.

Use of Independent Expert Reviews

Typically IERs are conducted on:

(1) analytical work plans and study assumptions,

(2) draft study outlines,

(3) early drafts,

(4) final drafts

(5) model descriptions for model characteristics (not the adequacy of
    the documentation), and

(6) model assumptions.

(Continued)

Sometimes reviewers are asked to do multiple reviews, which might
include items (1) through item (4), above, as a study develops. A series
of reviews such as this might take months and include multiple reports
and one or more discussion meetings either in Washington or by telephone
with EIA management and staff.

Other outside experts are used to augment the SMG staff in reviewing
products. These reviewers participate in what EIA calls the Category I
Review process. Note Table 2 above. Our current thinking is to include
Dr. Robert Trost and Dr. Fred Joutz who augment the SMG reviewing staff
in completing the survey.

Questions to the Committee:

1.  Are EIA's proposed survey questions balanced and comprehensive given
    the goal?

2.  Are the Independent Expert Reviewers likely to be the appropriate
    survey audience?

3.  Is the pretest group of two or three reviewers, described earlier,
    sufficient?

4.  Are there other questions we should be asking?

(See next page for Attachment)

Attachment

Recently you conducted an Independent Expert Review of the
\_\_\_\_\_\_\_\_\_ under contract to the Energy Information
Administration (EIA). We would appreciate you completing the 7-question
survey and returning it to Mr. Tom Broene of EIA's Statistics and
Methods Group at either:

> Fax: (202) 287-1705, or by
>
> Email at <tbroene@eia.doe.gov>

We are intentionally processing your survey response separately from
your connection with the materials you reviewed and the Independent
Expert Review Program. We are doing this to keep your responses from
influencing additional requests for your helpful advice. So, tell us
what you think. No need to be nice here!

We believe answering these questions should take no more than 15
minutes. When coupled with others it will help us evaluate the quality
of EIA's analytical tools and products.

Thank you in advance for your support and your timely response.

1.  How "relevant" was the information in the (*material you reviewed)*
    in meeting EIA's customers' needs?

    a.  *Relevance -- "The relevance of the information reflects the
        degree to which it meets the real needs of clients."*

> ***Use the scale from 1 to 5 with 5 being very satisfied and 1 being
> very dissatisfied**. (DK = Don't Know)*

*1 2 3 4 5 6(DK)*

> *     *

2.  How "reliable" was the information in the (*material you reviewed)*
    in meeting EIA's customers' needs?

    a.  *Reliability -- "The reliability of the information reflects the
        degree to which it is seen as dependable by EIA's clients ."*

> ***Use the scale from 1 to 5 with 5 being very satisfied and 1 being
> very dissatisfied**. (DK = Don't Know)*

*1 2 3 4 5 6(DK)*

> *     *

*Continued*

3.  How "accurate" was the information in the *(materials you reviewed)*
    in meeting EIA's customers' needs?

    a.  *Accuracy -- "Accuracy of statistical information is the degree
        to which the information is complete and correctly describes the
        concept it was designed or issue it was designed to measure."*

> ***Use the scale from 1 to 5 with 5 being very satisfied and 1 being
> very dissatisfied**. (DK = Don't Know)*

*1 2 3 4 5 6(DK)*

> *     *

4.  How satisfied were you with the "usefulness" of the information in
    the *(materials you reviewed)* in meeting EIA's customers' needs?

    a.  *Usefulness -- "Usefulness is­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­ the degree to which EIA's clients
        find the material beneficial or serviceable."*

> ***Use the scale from 1 to 5 with 5 being very satisfied and 1 being
> very dissatisfied**. (DK = Don't Know)*

*1 2 3 4 5 6(DK)*

> *     *

5.  How satisfied were you with the "timeliness" of the information in
    the *Impact of the Kyoto Protocol* in meeting EIA's customers'
    needs?

    a.  *Timeliness -- "Timeliness of information refers to the delay
        between the reference point (or the end of the reference period)
        to which the information pertains, and the date on which the
        information becomes available."*

> ***Use the scale from 1 to 5 with 5 being very satisfied and 1 being
> very dissatisfied**. (DK = Don't Know)*

*1 2 3 4 5 6(DK)*

> *     *

6\. How "consistent" is the information in *(the materials you
reviewed)* with changing industry structures?

b.  *Changing industry structure -- "The extent to which EIA's
    analytical products correspond to the various components of the
    domestic and international energy system."*

> ***Use the scale from 1 to 5 with 5 being very satisfied and 1 being
> very dissatisfied**. (DK = Don't Know)*

*1 2 3 4 5 6(DK)*

> *     *

*Continued*

7.  How "coherent" is the information in the *(materials you reviewed)?*

    a.  *"Coherent" -- "Coherence of statistical information reflects
        the degree to which it can be successfully brought together with
        other statistical information within a broad analytic
        framework."*

> ***Use the scale from 1 to 5 with 5 being very satisfied and 1 being
> very dissatisfied**. (DK = Don't Know)*

*1 2 3 4 5 6(DK)*

> *     *

Thanks for your cooperation in completing and returning this survey in a
timely manner.