Information Quality Guidelines Completed. What's Next.

Jay Casselberry

Statistics and Methods Group

Energy Information Administration

**[Introduction]{.underline}**

In response to section 515 of the Treasury and General Government
Appropriations Act for Fiscal Year 2001 (Public Law 106-554), the Office
of Management and Budget (OMB) on September 30, 2001, issued information
quality guidelines and directed Federal agencies to issue their own
guidelines for ensuring and maximizing the quality of information they
disseminate.  (OMB later reissued their guidelines with revisions on
February 22, 2002 (67 FR 8452) as the [final information quality
guidelines](http://www.whitehouse.gov/omb/fedreg/reproducible.html).)
OMB's deadline for agencies to have their own information quality
guidelines completed and publicly available is October 1, 2002.

OMB's issuance of their guidelines in September 2001 was the impetus for
a project to establish EIA-specific information quality guidelines. In
conjunction with developing these guidelines, EIA undertook a project to
update its standards, most of which were last updated in 1988.

The ASA Committee on Energy Statistics was informed of this project in
the Fall 2001 meeting and was provided with an update in the Spring 2002
meeting. In both meetings the Committee members gave advice on related
topics (e.g., the quality of energy information collected by third
parties that EIA disseminates).

EIA has now finished its Information Quality Guidelines and a new
Standards Manual.

-   The [Information Quality
    Guidelines](http://www.eia.doe.gov/smg/EIA-IQ-Guidelines.html)
    inform the public of EIA's commitment and activities to ensure the
    quality (i.e., objectivity, utility, and integrity), transparency,
    and reproducibility of disseminated information. (Guidelines to be
    provided after Administrator Caruso approves.)

-   The Standards Manual ([PDF](http://www.eia.doe.gov/smg/Standard.pdf)
    or [Word](http://www.eia.doe.gov/smg/Standard.doc) format) is
    directed to EIA staff and contractors with specific required actions
    designed to help EIA achieve the goals set forth in the Information
    Quality Guidelines.

The Guidelines and Standards Manual will be disseminated to EIA
staff/contractors and are publicly available on EIA's Internet site at
the [Quality](http://www.eia.doe.gov/neic/aboutEIA/guidelines.html)
page. Beginning October 1, the public has an administrative process to
request correction of information disseminated by EIA that does not
adhere to information quality guidelines issued by
[OMB](http://www.whitehouse.gov/omb/fedreg/reproducible.html),
[DOE](http://cio.doe.gov/informationquality/finalinfoqualityguidelines.pdf),
or [EIA](http://www.eia.doe.gov/smg/EIA-IQ-Guidelines.html).

EIA will now focus additional resources on activities affecting the
quality of disseminated information. The three components of a quality
system are: (1) quality control which is comprised of the quality
efforts (including implementation of the new EIA guidelines and
standards) by the program staff, (2) quality assurance (including
independent reviews of a program staff's quality efforts), and (3)
process improvement (including performance measurement and feedback
mechanisms where the program staff learn what is working and what is
not). Within the scope of its quality system, EIA has initiated efforts
in five quality projects areas.

1.  Number range CHAPTERDeveloping an EIA-wide survey managers' quality
    process to include self-assessments, selecting annual targets for
    improvement, and monitoring progress toward them.

2.  Auditing EIA data systems (the first systems to be audited will be
    coal, standard energy processing system (StEPS) used by the Office
    of Oil and Gas, and electric power).

3.  Assessing EIA model documentation.

4.  Evaluating the status of documentation and back-up computer systems
    necessary to assure that EIA could operate essential functions
    identified in the Continuity of Operations Plan in the event of an
    emergency.

5.  Collecting and tracking core data quality measures with an initial
    focus on response and imputation rates.

EIA is requesting the Committee's:

-   Suggestions for the specific quality projects mentioned.

-   Advice on issues where EIA should focus its quality resources.

-   Recommendations on quality projects that would be expected to yield
    higher benefits relative to costs.

### Quality Project Areas

1.  **Survey Quality Process**

The goal of this project is to establish a process within EIA to improve
surveys by using measurement, monitoring, and inter-office
collaboration. The process includes:

-   Implement survey data quality measures and work to establish the use
    of such measures for on-going survey improvement.

-   Design and implement annual survey assessments by survey managers.

-   Identify survey-specific targets for improvement, projects to
    address target areas, and measurement systems to track
    accomplishments.

The Statistics and Methods Group (SMG) will work with survey and quality
staff within the program offices to tailor the quality process to the
specifics of each survey. SMG will take the lead on training EIA staff
on quality tools and their utilization, developing self-assessment
materials, and coordinating these efforts across EIA. For each survey,
the focus should be on a limited number of key areas and there must also
be understanding and acceptance by the survey staff of the goals and
methods.

An initial effort has already begun with EIA's Office of Coal, Nuclear,
Electric, and Alternate Fuels which has established a team (including
SMG representatives) focusing on quality control (QC). That Team is
developing a list of quality-related questions that pertain to each step
of the survey process. Areas to be addressed include: (1) Survey Design;
(2) Data Collection; (3) Data Processing; (4) Data Analysis; and (5)
Data Dissemination. The questions are intended to assist CNEAF survey
managers in the design of a system-specific QC procedure/checklist that
will control the overall quality of their work. The new CNEAF QC
procedures are intended to integrate smoothly into a broader EIA quality
management system that will also include independent quality assurance
(QA) audits of the QC procedures and process improvement activities.

2.  **Data System Audits**

This quality assurance function directed by SMG will include an
examination of selected EIA survey systems to: (1) determine compliance
with EIA standards, (2) assess whether data quality measures are
collected, maintained, and utilized, (3) evaluate whether survey and
information dissemination operations are conducted in an efficient and
effective manner, and (4) assess whether the systems' operations have
up-to-date, complete documentation. (These audits will be similar to
audits conducted by EIA from 1981 -- 1990.)

During the audit process, the Statistics and Methods Group and its
contractor will work closely with the survey staff to address any
quality issues. The final audit product will be a report identifying
future improvements that could be made in the system, as well as system
operations that are not in compliance with EIA standards and actions
that should be taken.

3.  **Model Documentation Assessments**

The documentation for each active EIA model will be evaluated to verify
that:

-   It is clear, consistent, and concise

-   The design makes economic and statistical sense

-   The assumptions are explicit and reasonable

-   Equations are error free and understandable

-   Estimates accompany the required statistical measures

-   Data references and citations are provided

-   Documentation is in a stand-alone mode

-   The model can be conceptually replicated

-   Tables and graphs are understandable and referenced appropriately

-   Flow diagrams make sense

-   It is free of editorial inconsistencies

    Documentation for the following models have been reviewed:

```{=html}
<!-- -->
```
-   Transportation Sector Demand (TRAN)

-   Electricity Market Module (EMM)

-   International Energy Module (IEM)

-   Macroeconomic Activity Module (MAM)

-   DRI-WEFA Model of the U.S. Economy (DRI)

-   Commercial Sector Demand Module (CSDM)

The program office has taken care of our comments regarding EMM model,
and has promised to consider and incorporate our comments in the
forthcoming issues.

The final product for a model will be a report on the documentation, an
assessment of its reliability and validity, and possible new and
experimental methods for model reliability and validity procedures.

4.  **Documentation and Back-up Computer Systems for Continuity of
    Operations Plan Systems**

In its Continuity of Operations Plan (COOP), EIA has identified
immediate, short-term, mid-term, and long-term essential functions
critical that EIA be prepared to restore in case of emergency or
catastrophe. (These functions are specified in the Attachment.) To carry
out these functions, EIA has identified 43 core capabilities (e.g., data
collection/processing/dissemination, modeling, quality assurance) that
must be available.

For the 43 core capabilities, EIA is collecting information needed to
assess its current level of preparedness. For the 43 capabilities, EIA
is determining:

-   Does documentation exist, how complete it is, and is it up-to-date?

-   What personnel are essential and the methods to contact those
    personnel?

-   What are the information technology requirements?

-   Are there on-site and/or off-site backup capabilities?

The final product will be a report identifying potential problems in
documentation and back-up capabilities that could impact EIA's ability
to restore operations. In addition, the report will serve as a reference
document on personnel and IT requirements in the event of the need to
restore operations.

5.  **Core Data Quality Measures**

In Standard 2002-11 EIA establishes seven categories of core data
quality measures that each EIA survey should have or, if a measure is
not currently available, the survey staff should develop a plan with
milestones for collecting the measure(s). (This work has previously been
discussed with the Committee in April 2001 and October 2001.) The core
measures are:

1.  Frames - Frame size, volatility (e.g., births, deaths, mergers), and
    estimated coverage.

2.  Response Rates - Unweighted unit response rate and response coverage
    rate (as defined in Standard 2002-8).

3.  Revisions of Disseminated Data - Percent difference between first
    disseminated data and the final disseminated data for key data
    series.

4.  Timeliness - Time from the close of the survey reference period
    until the dissemination of key data series.

5.  Data Edits - For key edits (as identified by the survey staff), the
    number of survey responses with that edit flag and the percentage of
    flagged responses that are revised.

6.  Sample Surveys - Sampling variances (or confidence intervals or
    relative standard errors).

7.  Imputation - Item imputation rates for key data items.

The survey staff have been asked to provide SMG with information about
the measures, including what measures are currently available, and plans
for collecting those that are not currently available. SMG will
initially focus on the response rates and imputation rates measures and
summarize these measures in reports to senior management which will help
identify survey specific or general issues.

Repeating EIA's request is for the Committee's:

-   Suggestions for the specific quality projects mentioned.

-   Advice on issues where EIA should focus its quality resources.

-   Recommendations on quality projects that would be expected to yield
    higher benefits relative to costs.

*(as of October 8, 2002)*

Attachment

EIA Essential Functions

**Immediate (Within 2 hours)**

Begin the process of providing public access to the EIA web site
(completion goal -- 48 hours)

**Short-Term (Within 7 days)**

-   Bring systems on line to publish on the EIA web site the information
    contained in the following publications:

```{=html}
<!-- -->
```
-   Energy Situation Analysis Report

-   Short-Term Energy Outlook

-   Weekly Petroleum Status Report

-   Weekly Coal Production

-   Weekly On-Highway Diesel Prices

-   U.S. Retail Gasoline Prices

-   Weekly Natural Gas Storage Report

-   Establish a limited capacity to analyze energy markets. (Following
    an event that would cause the plan to be implemented, it is likely
    that energy markets will be disrupted. An analysis of the current
    petroleum, natural gas, electricity, and coal markets would need to
    be done immediately. While some of this information would be
    developed for the Energy Situation Analysis Report, more specific
    analysis would need to be done for each of these energy sources.)

```{=html}
<!-- -->
```
-   Establish a limited mid-term modeling capability

-   Establish a limited inquiry response capability

-   Provide logistics, procurement, and financial management operations
    support

**Mid-Term (within 30 days)**

-   Bring systems on line to publish on the EIA web site the Monthly
    Energy Review, including the petroleum, electricity, and natural gas
    data systems that feed it.

-   Begin the establishment of a core quality assurance function.

**Long-Term (within 90 days)**

-   Bring systems on line to publish on the EIA web site the information
    contained in the following publications:

```{=html}
<!-- -->
```
-   Petroleum Marketing Monthly

-   Petroleum Supply Monthly

-   Monthly Oxygenate Telephone Report

-   Company Level Imports

-   Prime Supplier Report

-   International Petroleum Monthly

-   Natural Gas Monthly

-   Electric Power Monthly

-   Quarterly Coal Report

-   Domestic Uranium Production Report

-   Electric Power Annual

-   Inventory of Electric Utility Power Plants in the US

-   International Energy Outlook

-   Inventory of Nonutility Power Plants in the US

-   International Energy Annual

-   Natural Gas Productive Capacity for the Lower 48 States

-   Renewable Energy Annual

-   Petroleum Supply Annual

-   Uranium Industry Annual

-   Foreign Direct Investment in the United States

-   Annual Energy Review

-   Petroleum Marketing Annual

-   Fuels Oil and Kerosene Sales

-   Electric Sales and Revenues

-   Natural Gas Annual

-   Emissions of Greenhouses Gases

-   Financial Statistics of Publicly Owned Electric Utilities

-   Annual Coal Report

-   Annual Energy Outlook

-   State Energy Data Report

-   State Energy Price and Expenditure Report

-   Performance Profile of Major Energy Producers

-   US Crude Oil, Natural Gas, and Natural Gas Liquids Reserves

-   State Electricity Profiles

```{=html}
<!-- -->
```
-   Bring full mid-term modeling capability on line

-   Complete the establishment of a core quality assurance function

-   Provide personnel operations support