**Follow-up on Frames Team Activities**

Presenter: Howard Bradsher-Fredrick

**April 5, 2005**

# I. Introduction

The EIA Strategic Plan calls for efficiency improvements in EIA's
operations. In particular, the Strategic Information Technology Subgroup
has been chartering teams to study the role information technology can
play in affecting increased efficiencies. As part of this effort, this
Subgroup has chartered several "frames teams" during the course of the
past year. These teams have been charged with well-defined objectives.

The first frames team was charged with providing recommendations
concerning activities in the frames area that will lead EIA toward
saving resources, facilitating commonality in data and software,
improving processes, thus resulting in frames and frame information that
can more readily be shared within EIA. This team met from March 2004 to
May 2004 and developed a set of six recommendations.

The second frames team met from August 2004 to January 2005. The primary
focus of its activities was to assess each unique EIA survey frame for
"sufficiency." The third incarnation of the frames team began meeting in
February 2005. It is specifically charged with following up on the
recommendations put forward by the first incarnation of the team.

This paper will discuss the objectives, methodologies, results, and
future directions of these frames teams.

# II. The First Frames Team

##### [Objectives]{.underline}

An inter-office team[^1] was chartered to identify efficiencies EIA can
pursue regarding frames and the computer systems that support frames.
This "Frames Team" was chartered by the Strategic Information Technology
Subgroup to serve for approximately two months. During this time, the
team was charged with "providing recommendations concerning activities
in the frames area that will lead EIA toward saving resources,
facilitating more commonality in data and software, improving processes,
and resulting in frames that can be easily shared within EIA." The team
was initially notified of its objectives on March 3, 2004 and requested
to prepare a priority-ordered list of recommendations by May 1, 2004.
The team met on a weekly basis with frequent "off-line" activities being
assigned in order to meet the deadline.

# Methodology

The team began its work by identifying a number of topics around which
to arrive at a set of recommendations. Developing such a list was
thought to be useful for the purposes of an organized discussion. These
topics are the following:

-   Communication and commonality among offices,

-   Standards/Best practices/Frames update procedures,

-   Maintenance/Support/Development/Design Flexibility (System),

-   Quality/Performance Measurement/Timeliness,

-   Reorganization/Office to deal specifically with frames.

One of these topics was then discussed each week with recommendations
developed through this discussion. In some cases this discussion led in
the direction of needing to consult and study related documents from EIA
or other statistical agencies. These documents included the following:

-   EIA Strategic Plan 2004-2008,

-   EIA Standard 2002-05 (Frames development and Maintenance),

-   EIA Standard 2002-11 (Data Quality Measures),

-   EIA Standard 2002-11 (Supplemental Materials, Additional Suggested
    Data Quality Measures, and periodic Quality Reviews),

-   Census Bureau Guideline: Quality Checklist for Census Bureau
    Products ((Part IV. Develop and Maintain Survey Sampling Frame(s)
    (Survey and Census Products Only)) (Part V. Select Survey Sample
    (Survey Products Only)),

-   Federal Committee on Statistical Methodology Statistical Working
    Paper No. 17.

# Recommendations

The team agreed upon a set of recommendations for EIA to adopt in order
to achieve the original team objectives. These are as follows:

1.  Complete and maintain a descriptive inventory of EIA frames
    including necessary information, such as:

-   File names

-   Elements and descriptions

-   Last update

-   Frequency of update

-   Surveys used

-   Frame contact name

-   Confidentiality information

2.  Conduct quarterly meetings within and among affected or interested
    offices; these offices could include OIAF, EMEU, and SMG in addition
    to OOG, CNEAF, and OIT. The purposes of these meetings are the
    following:

-   Review frames issues with respect to new surveys, and prior to
    clearance, review established surveys

-   Discuss any activity affecting frames

-   Discuss and become aware of new tools (e.g., IT)

-   Discuss major frames updates (sharing information and/or costs on
    survey anchors (non-IT tools))

-   Recommend subcommittees for conducting special projects related to
    frames

-   Review existing frames

-   Promote commonality of frames and collaboration among offices

-   Ensure the confidentiality of the frames' data is preserved in data
    sharing.

> The office representatives who would attend these quarterly meetings
> should be provided with sufficient resources in order to fulfill the
> mission of the team. Participation as a team member should be more
> than merely attending meetings; it will be necessary for team members
> to prepare pertinent information and identify important issues in
> advance of the quarterly meetings. Representatives would probably be
> the frames managers from the affected offices and should be familiar
> and up-to-date with all frames activities and direction in their
> office.

3.  Develop best practices for system maintenance, system support, and
    flexibility of system design regarding frames databases including:

-   Develop an architecture that uses a data dictionary with common
    terms, has software built around common modules, and has a data
    model that is generic and simple

-   Employ a lifecycle development model

-   Involve major user groups in all stages of the life cycle

-   Promote sharing of frame information for statistical purposes among
    Federal agencies

-   Use commercial off-the-shelf software where possible (e.g., software
    to locate and match records)

-   Strive for cost effective development and maintenance, taking into
    account the costs and effects on the survey and the data

-   Strive toward maximum platform independence of application and the
    database

-   Perform system's reliability and load testing prior to
    implementation in the development process

-   Build in data and metadata/control data integrity edits and edit
    messages

-   Develop an intuitive, user-friendly interface

-   Build in capability to locate or match records beyond exact matches

-   Implement standardization of frequent words and abbreviations

> 4\. Develop IT mechanisms as needed to copy/view other frames across
> EIA.
>
> 5\. Hold ad hoc meetings for new EIA surveys.
>
> When new surveys are to be developed, hold timely meetings in order to
> determine if there are particular frames issues that need to be
> discussed. In particular, frames information from other EIA offices
> may be available and/or modification of an existing frame might
> provide information needed by another office.

6.  Replace EIA Standard 2002-05 (Frames Maintenance and Development)
    with the revised version.

# III. Second Frames Team

**[Objectives]{.underline}**

As part of EIA's performance measurement effort, a team was formed by
the EIA Strategic Planning Team to evaluate EIA's survey frames. The
American Statistical Association -- Energy Committee also recommended
that EIA evaluate its survey frames.

The team was composed of representatives from various EIA offices. All
offices that actually conduct surveys had a representative on the
team[^2]. The team met on a weekly basis from August 2004 to January
2005.

An EIA-wide effort to systematically assess all EIA survey frames was
probably never attempted prior to this effort. Thus, prior efforts were
not readily available to serve as models for this activity.

The second frames team was charged with achieving the following
objectives:

-   To identify surveys having unique frames in EIA and surveys using
    subsets of those frames,

-   To compile information regarding each frame to enable the team to
    make a judgment regarding the frames "sufficiency" or
    "insufficiency" relative to each survey's stated purposes and goals,

-   To render an informed judgment regarding each frame's sufficiency or
    insufficiency,

-   To make recommendations for improvement or identify challenges with
    respect to each frame,

-   To fulfill these objectives within a reasonable time period.

**[Methodology]{.underline}**

Prior to the first meeting of the team, a significant amount of
background material for each frame was compiled. This provided the basis
for the team getting started. The team perceived its primary function as
two-fold: to determine sufficiency and thus provide input into the
performance measurement stated in the EIA strategic plan[^3] and to make
recommendations on improving all frames. In addition to the
recommendations, challenges are also stated; these are defined as
difficulties presently faced by survey managers with respect to frame
quality. Also, any frame comparison activities presently being conducted
in conjunction with another agency (e.g., frame comparisons being
conducted in cooperation with the Census Bureau) were also stated.

Because survey frame "sufficiency" is not well defined, the team
initially agreed upon the criteria to be employed for judging frame
sufficiency. These criteria are:

-   The existence of a systematic approach toward maintaining the frame,

-   The volatility of the frame over time,

-   The existence, availability and use of other EIA and third-party
    comparable frame listings (especially comprehensive lists),

-   The existence of balancing items and the magnitude and stability of
    the balancing item over time,

-   The relative concentration of volumes in a relatively few
    respondents,

-   Changes in the industry, legislation, regulations, and other
    exogenous considerations,

-   An independent assessment by the survey manager of the quality of
    the frame.

While no single one of these criteria led to any specific judgment,
these all played a role in leading to a determination of sufficiency or
insufficiency. In deciding upon sufficiency for any given frame, the
team considered whether or not the frame was sufficient for key purposes
(i.e., relative to the survey's purposes and goals) and not necessarily
sufficient for all potential purposes.

While some survey frame information had already been collected from an
earlier information collection effort on frames, the team deemed it
necessary to obtain further information from survey managers. This was
accomplished using various modes of contact: the e-mail system,
telephone and personal interviews. Thus, survey managers were often
contacted more than once for specific information. Moreover, the team
often required further clarification concerning information provided by
survey managers. This necessitated further contacts with survey managers
in order to obtain the requisite clarification information.

The basic information collected by SMG[^4] prior to the formation of the
team is shown in Appendix A. The following additional survey information
and quantitative data (Appendices B -- D) were collected by the team.
The appendices are the following:

-   Appendix A: Initial Basic Frame Information,

-   Appendix B: Stability (or Volatility of the Frame),

-   Appendix C: Corresponding Useful Listings of Frame Units or
    Available Survey Frames,

-   Appendix D: Balancing Item, Concentration of Volume, and Other.

**[Results]{.underline}**

The team reviewed 34 EIA survey frames; the results of the team's
assessment are shown in Appendix A. As was previously noted, some of
these frames serve as a "master frame" for more than one survey; these
frames and the associated surveys are designated in Appendix A.

Five of the frames reviewed were regarded as not being within the team's
purview for evaluation. This was generally due to the fact that the
survey frame was perceived by the team to be a subset of another EIA
frame or the frame was being managed by an organization other than EIA.
Frame definitions and frame size designations were found to be very
subjective among team members and survey managers.

Of the remaining 29 that were reviewed for sufficiency, 25 were deemed
sufficient for key purposes, while 4 were either thought to be either
"Insufficient" or given a "Don't Know" by the team at the present time.
The rationale for the insufficiency determination were primarily due to
concerns about mechanisms being in place to ensure future frame quality,
or due to expected changes in the structure of the energy market or, due
to changes in laws and regulations. However, merely because a frame is
deemed sufficient does not indicate that there is no room for
improvement or there aren't challenges ahead to be addressed. These
recommendations for improvement and challenges are also listed in the
table.

**[Future Frame Evaluations]{.underline}**

The team believes that this was a useful exercise and an exercise EIA
should conduct once every 3 or 4 years. Since EIA management began
showing particular concern regarding the quality of EIA survey frames a
few years ago, survey managers appear to have been engaged in various
activities with the goal of improving their survey frames. It appears as
though the quality of survey frames might have improved considerably due
to these recent activities.

Therefore, EIA's decision to focus more management attention and
resources to the development of better survey frames appears to have had
the desired effect of improving the survey frames. In order to continue
and extend these improvements, the team recommends that EIA maintain its
management focus by continuing to assign this area a high priority.

Although the team was only assigned the task of determining survey frame
sufficiency, the team noticed that there was considerable variation
among the surveys regarding the degree to which changes to the survey
frame were actively monitored, documented and archived. For example,
some survey managers were unable to provide precise data regarding
recent frame turnover (i.e., the number of new respondents added,
deleted, or merged into other entities, etc.). The formal tracking of
survey frame membership might have long-term benefits both in terms of
maintaining management focus and in terms of providing a baseline
regarding a "normal" historical turnover rate vis-à-vis future turnover
rates.

In critiquing the overall effort, processes and decisions made by the
team, the team worked well together with processes that functioned well.
The team recommends, however, that when a similar team is chartered for
a similar future effort that the team members should be broadened to
include representatives from all fuel groups rather than merely office
representatives. Thus, for example, representatives from the coal
division and natural gas division should be added. This would expand the
team's knowledge base regarding a particular industry's practices and
idiosyncrasies.

**[IV. Third Frames Team]{.underline}**

# Introduction

An inter-office team[^5] was chartered to follow-up on the
recommendations made by the first incarnation of the Frames Team. These
recommendations were made in a report to the Frames Team Champions
(dated 5/5/2004). The first Frames Team was charged with identifying
efficiencies EIA can pursue regarding frames and the computer systems
that support frames. Specifically, the team was charged with "providing
recommendations concerning activities in the frames area that will lead
EIA toward saving resources, facilitating more commonality in data and
software, improving processes, and resulting in frames that can be
easily shared within EIA."

The first activity of this team was to draw-up a team charter[^6]. This
activity was completed and the charter was sent to the Champions on
March 8, 2005. The next task of the team was to review the
recommendations of the first frames team and to decide whether or not we
agree that all of the recommendations are worth pursuing at this time
and to prioritize each recommendation. For each recommendation, a brief
summary of the likely resource commitment is also listed.

The third incarnation of the frames team began meeting in February 2005.
The following are a list of team objectives:

-   To work toward a long-term goal of enhanced commonality in IT.

-   To identify opportunities to share frame information among offices.

-   During 2005 the primary tasks of the team will be to develop plans
    for addressing the highest priority challenges identified during the
    2004 frames assessment exercise.

-   To manage and monitor the quality of EIA frames, recommend needed
    enhancements to frames that have been judged to be deficient or,
    alternatively, recommend that a survey be discontinued or that
    result tables be provided with appropriate caveats.

-   To recommend further studies needed to evaluate frames sufficiency.

-   To prepare preliminary recommendations for consideration in the
    Spring 2005 EIA Senior Staff planning session.

-   To prepare updated recommendations to the Fall 2005 EIA Senior Staff
    plans for the "Frame Sufficiency" performance measure.

The team has developed a team charter and has completed an interim
report, making recommendations concerning the recommendations advanced
by the first frames team. Additionally, the team has begun meeting with
staff members of other Federal statistical agencies in order to
determine how other peer agencies contend with similar problems,
particularly problems with frame sharing among internal offices.

# Recommendations

The following is a list of the recommendations of the third frames team.
This section describes the recommendation of the third frames team on
each of the recommendations of the first frames team.

The team agreed that EIA should follow through with Recommendation #1
from the first frames team. This should also be granted a high priority
due to the fact that it will take a period of time to actually collect
this data and information[^7].

#### Descriptive Inventory of EIA Frames (High Priority)

1.  *Complete and maintain a descriptive inventory of EIA frames
    including necessary information, such as:*

-   *File names*

-   *Elements and descriptions*

-   *Last update*

-   *Frequency of update*

-   *Surveys used*

-   *Frame contact name*

-   *Confidentiality information*

The team noted that it is necessary for the team to define important
terms in order to obtain the proper information. Terms of particular
interest are "frame" and "elements" (of the frame). The team may also
need to further refine the complete list of items to be requested from
the frames managers.

Should this task be undertaken, the resource commitment would probably
include the following: one meeting of the entire frames team to decide
upon a list of elements that would constitute a "descriptive inventory
of EIA frames," a commitment on the part of survey/frames managers in
the EIA offices to take the time to complete the template/questionnaire,
a commitment on the part of and team members to send-out and follow-up
on receiving the responses, a commitment on the part of the team's chair
to organize the responses in a suitable format, and finally, a team
meeting to review the responses.

#### Quarterly Meetings to Discuss Frames & Ad Hoc Meetings to Discuss New Frames (High Priority)

The team agreed that EIA should follow through with Recommendation #2
from the first frames team. Also, Recommendation #5 should be folded
into this recommendation. After combining these recommendations, the new
recommendation should read as follows:

2.  *Conduct quarterly meetings (or special meetings as new surveys are
    developed) within and among affected or interested offices; these
    offices could include OIAF, EMEU, and SMG in addition to OOG, CNEAF,
    and OIT. The purposes of these meetings are the following:*

-   *Review frames issues with respect to new surveys, and prior to
    clearance, review established surveys*

-   *Discuss any activity affecting frames*

-   *Discuss and become aware of new tools (e.g., IT)*

-   *Discuss major frames updates (sharing information and/or costs on
    survey anchors (non-IT tools))*

-   *Recommend subcommittees for conducting special projects related to
    frames*

-   *Review existing frames*

-   *Promote commonality of frames and collaboration among offices*

-   *Ensure the confidentiality of the frames' data is preserved in data
    sharing.*

The office representatives who would attend these quarterly meetings
should be provided with sufficient resources in order to fulfill the
mission of the team. Participation as a team member should be more than
merely attending meetings; it will be necessary for team members to
prepare pertinent information and identify important issues in advance
of the quarterly meetings. Representatives would probably be the frames
managers from the affected offices and should be familiar and up-to-date
with all frames activities and direction in their office.

These meetings should have an agreed upon agenda prior to their taking
place. The size of the committee should be limited to at most 12
participants and preferably fewer. Individual offices should decide who
should attend. OIT needs to be included in these meetings so that
possible IT solutions to frame sharing, confidentiality and commonality
issues can be effectively discussed.

It was noted that of the above listing of objectives and tasks, the
objective of promoting commonality of frames and collaboration among
offices should be emphasized. It might also be useful for the first
meeting to include a discussion involving the definition of the term
"frame." Since this term and other related terms are difficult to
define, it is important that a common vocabulary of key technical terms
be adopted for these meetings.

Should this task be undertaken, in addition to the preparation and
meeting time commitment on the part of attendees, as discussed in the
recommendation above, there will be preparation time on the part of
someone (possibly the frame team chair) to organize the meeting. This
task will include preparing an agenda, reserving a room, and contacting
meeting participants.

**Adopting "Best Practices" with respect to System
Development/Maintenance/ Support (Medium Priority)**

The team decided to revise this recommendation, including folding in the
4^th^ recommendation, as follows:

3.  *\[Develop\] best practices for system maintenance, system support,
    and flexibility of system design regarding frames databases
    including:*

    a.  *Develop an architecture that uses a data dictionary with common
        terms, has software built around common modules, and has a data
        model that is generic and simple*

    b.  *For new systems or major overhauls of present systems, employ a
        lifecycle development model (i.e., system requirements,
        specifications, model development, log maintenance changes,
        etc.)*

    c.  *Involve major user groups in all stages of the life cycle*

    d.  *Use commercial off-the-shelf software where possible (e.g.,
        software to locate and match records)*

    e.  *Strive for cost effective development and maintenance, taking
        into account the costs and effects on the survey and the data*

    f.  *Strive toward maximum platform independence of application and
        the database (Compatibility is desirable and EIA does not want
        to be dependent on particular hardware.)*

    g.  *Perform system's reliability and load testing prior to
        implementation in the development process*

    h.  *Develop an intuitive, user-friendly interface*

    i.  *Develop IT mechanisms as needed to copy/view other frames
        across EIA.*

The team discussed this recommendation at considerable length. It was
noted that this team is probably not going to recommend any major system
development projects, but that these system practices should be employed
for minor system development or major system changes to already existing
systems (should the team recommend any).

Should this task be undertaken, the team will need to meet several times
and possibly do some research apart from the group meetings. The
representative from OIT will probably need to take the lead on this
effort.

#### Ability to Copy/View Frames 

4\. Develop IT mechanisms as needed to copy/view other frames across
EIA.

The team decided that this recommendation should be included in
Recommendation #3. This has been done (as shown in the last bullet
above).

#### Ad Hoc Meetings to Discuss New Frames

5\. Hold ad hoc meetings for new EIA surveys.

*When new surveys are to be developed, hold timely meetings in order to
determine if there are particular frames issues that need to be
discussed. In particular, frames information from other EIA offices may
be available and/or modification of an existing frame might provide
information needed by another office.*

The team decided that this recommendation should be included in
Recommendation #2. It has now been included in that recommendation (See
above.)

#### Frames Maintenance and Development Standard (Medium Priority)

6.  *Replace EIA Standard 2002-05 (Frames Maintenance and Development)
    with the revised version in Attachment A.*

The team will review the original standard and its proposed replacement
and make a recommendation.

Should this task be undertaken, it will be necessary for the team to
meet once in order to discuss the new and old standards and arrive at
consensus on whether the new standard or a revised new standard should
be adopted.

#### Frame Documentation (High Priority)

7.  *The team recommends that the existing standards on frame
    documentation should be reviewed and updated, if necessary. Also, a
    template for the minimum basic frame documentation should be
    developed.*

This is a new recommendation, but one the team thinks is important to
study.

Should this task be undertaken, it will be necessary to conduct some
research apart from the actual team meetings. The chair and his
assistant will need to locate any extant EIA frames documentation
standards and any other necessary background material in preparation for
team meetings on the subject.

# Researching Frames Problems at Peer Federal Agencies

It should be noted that the members of the team plan to meet with
representatives from peer Federal statistical agencies to discover how
these agencies contend with similar frames sharing issues. Indeed
several team members have already met with two separate groups in the
Census Bureau and have shared their experiences with the entire frames
team.

#### V. Questions for the Committee

-   Considering the results of the second frames team, are there any
    suggestions as to how to proceed in determining sufficiency of
    frames in the future?

-   Are there any comments regarding the third frames team's
    recommendations?

# Attachment A

# 

# ENERGY INFORMATION ADMINISTRATION STANDARD 2005-?

**Title:** Frames Development and Maintenance

# Superseded Version: Standard 2002-05

**Purpose:** To promote a systematic process for the development and
maintenance of frames and for monitoring of frames quality while
enabling the sharing of frame information across EIA surveys.

**Applicability**: All EIA surveys.

**Required Actions for Creating New Frames:**

When a new frame is needed for a data program, a frame's construction
plan should be developed and implemented. The plan should document the
following descriptions:

-   Data items necessary to support unique and efficient identification
    of elements in the frame

-   Data items necessary for sampling purposes such as

(a) any area listing information or results that may be used

(b) measures of size or other sampling attributes

(c) information pertaining to how frame elements are organized into
    plants, sites, facilities, plants, outlets, subsidiaries, and
    similar structures

(d) important industrial, technical, legal, or historical relationships
    between or within frame elements

-   Identification, description, and evaluation of any sources,
    including any or all parts of an already existing frame, that may be
    used to construct a new frame

-   Any manual or automated matching/merging/unduplication methods used
    to reconcile independent sources into a coherent non-duplicative
    frame or to find elements existing on the frame

-   Clear and comprehensive definitions of terms and codes

-   The extent to which the new frame is expected to reflect the
    population of interest

-   How the quality of the new frame will be evaluated (under/over
    coverage, accuracy of sampling and identification information,
    timeliness, etc)

-   How other important checks, counts, totals, or statistics will be
    calculated

-   How often or under what triggers the new frame will be updated, the
    methods that will be used, and a projected schedule and resource
    budget for this task (see Required Actions for Maintaining and
    Updating Existing Frames)

-   The frame's major applications and frequency of use.

-   The performance measures that will be produced to measure and
    monitor the frame with respect to quality (see Standard 2002-11 and
    supplement to 2002-11)

**Required Actions for Maintaining and Updating Existing Frames:**

Plans for maintaining and updating existing frames must be documented,
implemented, and revised as needed. Maintenance includes

-   Detecting and adding of new elements (births)

-   Revising for changes in the ownership, name, address, contact, etc.
    of an element

-   Incorporating suggestions to improve frame quality

-   Incorporating cross reference information between surveys or survey
    systems

```{=html}
<!-- -->
```
-   Indicating whether frame elements are in or out of scope

-   Indicating whether frame elements are active or inactive, or other
    status distinctions.

```{=html}
<!-- -->
```
-   Recording the date of changes to the content of an element

-   Resolving conflicts between frame elements and other sources of
    information

Updates may occur as information is discovered or according to the
predetermined schedule as described in the frame plan. Updates on a
predetermined schedule should be comprehensive searches that examine
more current information sources and incorporate it into the frame.
Plans to update under a predetermined schedule should document:

-   the frequency of updates or the triggers for updates

-   how sources of information will be evaluated and used

-   how updates will be incorporated into all appropriate files, mailing
    lists, and

other survey control forms or listings

-   how the updated frame will be evaluated for quality (under/over
    coverage, accuracy of sampling and identification information, what
    performance measures will be produced to measure and monitor with
    respect to quality)

```{=html}
<!-- -->
```
-   a projected schedule and resource budget for an update

To ensure that the most recent information is retained, the record for
each member of the frame must have appropriate transactional variables,
including at least the most current status of the element known and the
last date the record was modified. Before each major update of a frame,
the current version must be preserved in a manner that allows its
recreation with minimal time and effort. Versions of frames so preserved
must retain in accordance with each survey office's record retention
schedule:

-   all respondents whether active or inactive

-   all information about an individual frame member, including all
    transactional and descriptive data needed for sampling or survey
    control.

If the record retention schedule allows for the eventual deletion of
inactive respondents, the survey office should first consult with the
users of the frame to insure deletion results in no major negative
impact. Deletion should proceed after the survey office documents there
are no unacceptable consequences in terms of future frame updates,
conflicting survey files, or in any other terms the users of the frame
may put forth.

**Related Information:** None

**Approval Date**:

**Appendix A: Summary of Frames Team Assessment of Frames Sufficiency**

**\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_**

**Program:** **Petroleum Supply**

**Form Number:** EIA-810 and EIA-820

**Form Name:** Monthly and Annual Refinery Report

**Data Type:** Supply

**Number of Units**[^8]**:** 151

**Sufficient:** Yes

**Rationale:** SMG did considerable work with the Census Bureau to
reconcile frames differences. There were definitional differences
between EIA and Census Bureau "refineries." This appeared to be due to
the consumption of "still gas," and the definition of such facilities.
This provided an opportunity for Census to clean their frame, but EIA's
frame appeared to be in reasonably good shape.

**Recommendations for Improvement:** None

**Challenges:** MECS and EIA-810/820 staff should continue to work
together to improve data consistency.[^9]

**Form Number:** EIA-811

**Form Name:** Monthly Bulk Terminal Report

**Data Type:** Supply

**Number of Units:** 248

**Sufficient:** Yes

**Rationale:** Survey respondents relatively easy to identify due to
their small number, large facility size, and relative industry
stability. Good cross-checking[^10] opportunities with outside sources. 

**Recommendations for Improvement:** Close monitoring may be required
because state legislation on MTBE use and the number of gasoline fuel
blenders (300) is likely to keep the survey frame list in flux over the
next several years. 

**Form Number:** EIA-812

**Form Name:** Monthly Product Pipeline Report

**Data Type:** Supply

**Number of Units:** 83

**Sufficient:** Yes

**Rationale:**  Survey respondents relatively easy to identify due to
their small number and large facility size.   Good cross
checking opportunities with outside sources.

**Recommendations for Improvement:** None

**Form Number:** EIA-813

**Form Name:** Monthly Crude Oil Report

**Data Type:** Supply

**Number of Units:** 147

**Sufficient:** Yes

**Rationale:** Good cross checking opportunities with other petroleum
industry survey frame lists.

**Recommendations for Improvement:** None

**Form Number:** EIA-814

**Form Name:** Monthly Imports Report

**Data Type:** Supply

**Number of Units:** 175

**Sufficient:** Yes

**Rationale:** EIA can compare its imports frame listing with Customs
and Census listings on a company-by-company basis though matching
companies between frames (or listings) presents some difficulties.
However, OOG appears to work hard on this to maintain a sufficient
frame.

**Recommendations for Improvement:** Continue to develop techniques to
better conduct cross checks.

**Form Number:** EIA-815

**Form Name:** Monthly Terminal Blenders Report

**Data Type:** Supply

**Number of Units:** 227

**Sufficient:** Don't Know/No

**Rationale:** It is extremely difficult to identify new motor gasoline
blenders. This difficulty is compounded by the fact that this is a new
survey. Terminals can begin blending operations without readily
providing notice. EIA probably is presently missing a number of blenders
in its frame.

**Recommendations for Improvement:** EIA understands that the EPA is in
the process of requiring blenders to obtain a license in order to
conduct blending operations. Once the EPA has established a viable list,
EIA should attempt to make use of the list for frame maintenance
purposes.

**Form Number:** EIA-816

**Form Name:** Monthly Natural Gas Liquids Report

**Data Type:** Supply

**Number of Units:** 422

**Sufficient:** Yes

**Rationale:** The 816 data is compared with the LPG Almanac.  This
is a more volatile  segment of the  industry so more monitoring may be
needed  than for the other petroleum survey frames.  The potential
survey frame volatility is a result of decline in regional field
production, which has resulted in mergers, closure of facilities, etc.

**Recommendations for Improvement:** None

**Form Number:** EIA-817

**Form Name:** Monthly Tanker and Barge Movement Report

**Data Type:** Supply

**Number of Units:** 42

**Sufficient:** Yes

**Rationale:** The frame is composed almost entirely of large oil
companies. These companies provide the information needed on movement
since they maintain custody of the oil.

**Recommendations for Improvement:** None

**Form Number:** EIA-819

**Form Name:** Monthly Oxygenate Report

**Data Type:** Supply

**Number of Units:** 138

**Sufficient:** Yes

**Rationale:** EIA is doing a reasonably good job at the present time.
However, it may become increasingly difficult to identify small ethanol
producers in the future (e.g., farmer cooperatives). Also, imports are
becoming more important. Potential legislation at the state level
promoting alternative transportation fuel could also play a role in
increasing the number of producers.

**Recommendations for Improvement:** Considering the increase in the
potential number of small producers, be sure to allocate sufficient
resources to maintain a high level of vigilance in identifying new
producers in a variety of states. EIA should also monitor oxygenate
imports which could be volatile over time.

**Program: Petroleum Marketing**

**Form Number:** EIA-863

**Form Name:** Petroleum Product Sales Identification Survey

**Data Type:** Demand and Price

**Number of Units:** 24,400

**Sufficient:** Yes

**Rationale:** While there are approximately 100 sources of frame
information, the various survey samples that use the frame continually
aid in the frame maintenance. However, there is deterioration of the
frame during the course of its four-year run between major frame
updates.

**Recommendations for Improvement:** If the major frame update were done
more often than every four years, the quality of the frame could
improve.

**Form Number:** EIA-856

**Form Name:** Monthly Foreign Crude Oil Acquisition Report

**Data Type:** Price

**Number of Units:** 37

**Sufficient:** Not within our purview

**Rationale:** This frame really borrows frame elements from the EIA-810
and EIA-814. Thus, according to the team's definition of a frame, the
EIA-856 does not qualify.

**Recommendations for Improvement:** None.

**Challenges:** The survey manager believes that resources should be
used to provide for better sharing of data among surveys so that she
could more easily identify respondents for the EIA-856. She is employing
a cut-off sample survey based on frame information from the EIA-814,
810, and 14.

**Program: Natural Gas**

**Form Number:** EIA-176,

**Form Name:** Annual Report of Natural and Supplemental Gas Supply and
Disposition

**Data Type:** Supply, Demand Price

**Number of Units:** 1,944

**Sufficient:** Yes

**Rationale:** This frame appears to be basically good for adequately
surveying natural gas volumes and, additionally, prices for estimating
commercial and residential consumers. However, the EIA-176 historically
being a survey of companies holding custody of natural gas did not
permit easy evolution toward survey coverage of players not involved
with custody (e.g., marketers). Marketers are likely to be difficult to
keep track of over time; marketers enter and leave the market
frequently.

**Recommendations for Improvement:** While it is acknowledged that
resources are very limited with respect to developing and implementing
new surveys, in order to adequately handle the industrial price data
problem, new players in the natural gas arena will need to be surveyed.
One possible action to quantify the adequacy of the EIA-176 to publish
industrial prices is to formalize a review of manufacturing prices
between the EIA-176 and EIA's quadrennial survey of manufacturers
(EIA-846). Note: The latest survey results for the EIA-846 are available
for the 2002 year.

**Form Number:** EIA-910

**Form Name:** Monthly Natural Gas Marketers Survey

**Data Type:** Price

**Number of Units:** 154

**Sufficient:** Don't Know

**Rationale:** This is a new survey that is in a state of flux. New
states are being added as consumer choice programs come into existence.
Some states have comprehensive lists of marketers due to the requirement
for marketers to register with the states. However, in other states
multiple partial lists make it difficult to be sure that all marketers
are being surveyed. However, it appears as though EIA is doing a good
job with developing procedures to ensure as good frame quality as
possible given the circumstances.

**Future Challenges:** It will remain a future challenge to identify new
marketers as they come into existence. This will be particularly true as
new states are added to EIA's frame (e.g., six new states are in the
process of presently being added to the survey frame).

**Form Number:** EIA-895

**Form Name:** Monthly Quantity and Value of Natural Gas Report

**Data Type:** Price/Supply

**Number of Units:** 32

**Sufficient:** Not Applicable.

**Rationale:** The 895 is a subset of the EIA-23 frame. The EIA-895
actually surveys state agencies (a very well-known set of respondents),
but the state agencies are actually surveying a subset of the
respondents to the EIA-23.

**Recommendations for Improvement:** None

**Program: Oil and Gas Reserves**

## Form Number: EIA-23 

**Form Name:** Annual Survey of Domestic Oil and Gas Reserves

**Data Type:** Miscellaneous

**Number of Units:** 21,005

**Sufficient:** Yes

**Rationale:** EIA conducts good cross checking on this frame.  Survey
data are within 1% of Petroleum Supply Annual data and 2% of the Natural
Gas Annual data.  Although small operators are problematic (Cat. III),
they constitute a small proportion of the total reported petroleum
reserves.

**Recommendations for Improvement:** None

**Form Number:** EIA-64A

**Form Name:** Annual Report of the Origin of Natural Gas Liquids
Production

**Data Type:** Supply

**Number of Units:** 527

**Sufficient:** Yes

**Rationale:** This frame has been fairly stable over time and is likely
to continue to be stable. Routine procedures exist for updating the
frame, using petroleum industry publications.

**Recommendations for Improvement:** None

**Program: Alternative Fuels Program**

**Form Number:** EIA-886

**Form Name:** Annual Survey of Alternative Fueled Vehicle Suppliers and
Users

**Data Type:** Miscellaneous

**Number of Units:** 2,491

**Sufficient:** No

**Rationale:** It is very difficult to conduct this survey of
alternative fuel vehicles (AFVs), particularly in an environment
characterized by scarce resources. Some AFV fleets owned by local
governments and private (non-utility) companies are missed in the frame.
This accounts for 60% - 70% of the total AFVs. A number of sources of
information have been investigated with the result being that either the
information source is not workable or overly expensive.

**Recommendations for Improvement:** 1. It might be useful to again have
a look at trying to get at alternative transportation fuel (ATF)
consumption if these are the data of real interest. 2. Resources could
be devoted to developing a frame of local governments to augment the
existing frame. 3. The on-going processing of survey frame information
is lacking; perhaps resources could be devoted in order to have a
dedicated FTE.

**Form Number:** EIA-63A

**Form Name:** Annual Solar Thermal Collector Manufacturers Survey

**Data Type:** Misc.

**Number of Units:** 57

**Sufficient:** Yes

**Rationale:** There are a very small number of companies in a
relatively stable frame. Cross checking this frame with other industry
listings insures frame sufficiency.

**Recommendations for Improvement:** Remain aware of changes in state
legislation affecting renewable portfolio standards.

**Form Number:** EIA-63B

**Form Name:** Annual Photovoltaic Module/Cell Manufacturers Survey

**Data Type:** Misc.

**Number of Units:** 43

**Sufficient:** Yes

**Rationale:** There are a very small number of companies in a
relatively stable frame. Cross checking this frame with other industry
listings insures frame sufficiency. Changes in the new form should
improve the quality of the import data.

**Recommendations for Improvement:** Remain aware of changes in state
legislation affecting renewable portfolio standards.

Form Number: EIA-902

**Form Name:** Annual Geothermal Heat Pump Manufacturers Survey

**Data Type:** Miscellaneous

**Number of Units:** 14

**Sufficient:** Yes

**Rationale:** There are a very small number of firms in the industry
with a high concentration of volume in a few firms. A consortium of
geothermal organizations exists that maintains a list of all firms in
the field (i.e., maintains a frame identical to EIA's).

**Recommendations for Improvement:** None

**Program: Electric Power**

**Form Number**: EIA-860

**Form Name:** Annual Electric Generator Report

**Data Type:** Supply

**Number of Units:** 2,450

**Sufficient:** Yes

**Rationale:** This is a survey frame in which a large proportion of the
generating capacity is concentrated in the largest companies. These
producers are predominantly the fossil fuel facilities or very well
known hydroelectric and nuclear plants. However, there are many small
producers. The problem area in the frame is in locating small renewable
facilities. Alternative sources of such listings are available and used
for updating purposes.

**Recommendations for Improvement:** Since identifying additions to the
frame of renewable plants is problematic and resource intensive, all of
the offices in EIA that collect pertinent frame information should share
that information with EIA-860 staff members. This procedure should be
formalized in order to ensure its proper functioning.

**Form Number**: EIA- 411

**Form Name:** Coordinated Bulk Power Supply Program Report

**Data Type:** Supply

**Number of Units:** 10

**Sufficient:** Not Applicable

**Rationale:** This frame is maintained by the North American
Electricity Reliability Council (NERC) jointly with each of the regional
councils. EIA obtains a single submission from NERC, representing the
aggregate response of the 10 regional councils.

**Recommendations for Improvement:** None

**Form Number**: EIA-861

**Form Name:** Annual Electric Power Industry Report

**Data Type:** Price, Demand, Supply

**Number of Units:** 3,400

**Sufficient:** Yes

**Rationale:** The frame is stable. Routine procedures are in place to
maintain the frame. The identification of power marketers may be the
biggest challenge. Power marketers are registered with FERC, but many of
them do not actually market power. The balancing item is working well
with effective balance in recent years. Because of the heavy
concentration of volume in a few facilities in the frame, survey
resources are disproportionately expended on relatively small frame
units.

**Recommendations for Improvement:** Deregulated power sales are
potentially going to cause survey frame problems. This segment of the
frame should be monitored closely in future years.

**PROGRAM: FINANCIAL REPORTING SYSTEM**

**Form Number**: EIA-28

**Form Name:** Financial Reporting System (FRS)

**Data Type:** Misc.

**Number of Units:** 27

**Sufficient:** Not Applicable

**Rationale:** This survey does not involve a frame. According to the
DOE Organization Act, it appears as though the EIA administrator is
charged with identifying major energy-producing companies that are
involved in the energy industry so that the information collected from
such companies provide a statistically accurate profile of each line of
commerce in the energy industry.

**Recommendations for Improvement:** None

**PROGRAM: COAL**

**Form Number**: EIA-3

**Form Name:** Quarterly Coal Consumption and Quality Report,
Manufacturing Plants

**Data Type:** Demand

**Number of Units:** 562

**Sufficient:** Don't Know

**Rationale:** The balancing item stability and its relatively low
proportion of total volume (i.e., 1%) indicates that frame sufficiency
is probably good. However, the lack of a systematized approach toward
frame updates could lead to problems if a large number of plants begin
using coal. The synfuel plants that were recently added to the frame are
probably all-inclusive. A recent cross-check conducted by CNEAF against
an independent frame showed no differences. The survey manager seems to
have mechanisms in place to remain on top of the synfuel frame
situation.

**Recommendations for Improvement:** Frame quality is dependent upon a
systematic approach toward frame updates. It appears that this frame is
lacking such an approach; appropriate action should be taken to remedy
this. The Bureau of the Census is presently evaluating this frame.

**Form Number**: EIA-5

**Form Name:** Quarterly Coal Consumption and Quality Report, Coke
Plants

**Data Type:** Demand

**Number of Units:** 24

**Sufficient:** Yes

**Rationale:** There are a very limited and stable number of coke plants
being surveyed. Also, a newsletter exists which provides comprehensive
information for updating this survey frame.

**Recommendations for Improvement:** The Bureau of the Census is
presently evaluating this frame.

**Form Number**: EIA-6A

**Form Name:** Coal Distribution Report -- Annual

**Data Type:** Supply, Demand

**Number of Units:** 1,000

**Sufficient:** Yes

**Rationale:** Similar to the EIA-7A, this frame is anchored to the
Mining Safety and Health Adm. (MSHA) frame. The form also collects the
name and address of the other party for sales and purchases, thus
permitting the identification of new distributors. Because the frame is
continually updated by a regulatory agency and thus is essentially
self-maintaining, this frame should have no problems in maintaining its
frame sufficiency in the future.

**Recommendations for Improvement:** None

**Form Number**: EIA-7A

**Form Name:** Coal Production Report

**Data Type:** Supply

**Number of Units:** 1,850

**Sufficient:** Yes

**Rationale:** This frame is quite good due to the fact that the Mining
Safety and Health Adm. (MSHA) requires mines to register and obtain an
identification number prior to production. The primary difficulty is
that some small mining facilities fall in and out of scope because their
annual tonnage varies around the tonnage and manpower threshold levels
as specified by the survey's design. However, this issue can be
addresses by keeping these small parties on the frame list and letting
them self-select in any particular year as to whether their current
status requires their submission of a survey form.

**Recommendations for Improvement:** None.

**PROGRAM: URANIUM**

**Form Number**: EIA-851

**Form Name:** **Domestic Uranium Production Report**

**Data Type:** Supply

**Number of Units:** 28

**Sufficient:** Yes

**Rationale:** This frame is highly concentrated in a few facilities
with only a small frame. Also, the location of uranium mines is well
known.

**Recommendations for Improvement:** None

**Form Number**: EIA-858

**Form Name:** Uranium Industry Annual Survey

**Data Type:** Supply

**Number of Units:** 70

**Sufficient:** Yes

**Rationale:** There are relatively few facilities and these are well
known. New companies and mergers can be readily monitored. Inventories
and transactions are used on a continuous basis to serve as a balance
for the purposes of checking and making corrections.

**Recommendations for Improvement:** None

**PROGRAM: CONSUMPTION SURVEYS**

**Form Number**: EIA-871A/I

**Form Name:** Commercial Buildings Energy Consumption Survey (CBECS)

**Data Type:** Demand/Price

**Frame Size:** Not Applicable because a complete address listing of
commercial buildings does not exist in either public or private sources.

**Sufficient:** Yes

**Rationale:** A list of commercial buildings is not available. Hence,
the Commercial Building Energy Consumption Survey is not sampled from a
complete list frame, but uses a state-of-the-art statistical "area
sampling" approach. The survey is well documented in terms of describing
its frame and sampling methodologies.

**Recommendations for Improvement:** None

**Challenges:** The appropriate selection of individual buildings in
multi-building facilities (e.g., university campuses, hospital
complexes) represents a challenge in the area sampling procedure.

**Form Number**: EIA-457A/G

**Form Name:** Residential Energy Consumption Survey (RECS)

**Data Type:** Demand/Price

**Frame Size:** Not Applicable, because EIA does not have access to a
complete address listing of residential buildings for sampling purposes.

**Sufficient:** Yes

**Rationale:** The Residential Energy Consumption Survey is not a list
frame, but uses a state-of-the-art statistical "area sampling." The
survey is well documented in terms of describing its frame and sampling
methodologies.

**Recommendations for Improvement:** Monitor/review progress of the
United States Postal Service's address listing as a source to develop or
use directly as a sampling frame.

**Form Number**: EIA-846(A,B,C)

**Form Name:** Manufacturing Energy Consumption Survey (MECS)

**Data Type:** Demand/Price

**Frame Size:** 237,000

**Sufficient:** Yes

**Rationale:** The Bureau of the Census maintains an establishment-level
list frame and conducts the survey on behalf of EIA, under an
interagency agreement. The list frame is updated yearly from IRS
records, an annual Company Organization Survey conducted by the Census,
and every 5 years in years ending in \"2\" or \"7\" by the Census\'
Census of Manufacturers.  **Note:** Title 13 of the U.S. Code prevents
EIA from taking possession of the list frame.  Sworn Census agents and
Census employees can view/analyze data onsite at the Census Bureau, but
release of any data is subject to Title 13 compliance.  Currently, EIA
maintains sworn census status for the 2-member MECS team (Survey
Manager - Bob Adler). 

**Recommendations for Improvement:** None

**Challenges:** MECS and EIA-810/820 staff should continue to work
together to improve data consistency.

**PROGRAM: VOLUNTARY REPORTING OF GREENHOUSE GASES**

**Form Number**: EIA-1605B AND EIA-1605EZ

**Form Name:** Voluntary Reporting of Greenhouse Gases

**Data Type:** Misc.

**Number of Respondents in 2002:** 228

**Sufficient:** Not Applicable

**Rationale:** Because the reporting is voluntary, the list of
respondents is not a survey list frame. Note: The program is undergoing
policy changes expected to be implemented by the end of 2005.

**Recommendations for Improvement:** None

[^1]: The team members are Betty Barlow (OIT), Joe Conklin (OOG), Carol
    French (OOG), Marianne Holly (OOG), Tom Leckey (CNEAF), Bill Liggett
    (CNEAF), Tom Murphy (CNEAF), TC Swann (CNEAF), Paula Weir (OOG),
    Grace Sutherland (SMG/Note taker), Howard Bradsher-Fredrick
    (SMG/Facilitator)

[^2]: The team members are as follows: Betty Barlow (OIT), Philip Budzik
    (OIAF), Tom Leckey (CNEAF), Mark Schipper (EMEU), and Casey Tischner
    (OOG). Grace Sutherland (SMG) provided support in assembling
    information tables and facilitated the meetings when the assigned
    facilitator was unavailable. Howard Bradsher-Fredrick (SMG) served
    as facilitator and also provided support in gathering information.

[^3]: The relevant portion of the EIA 2004-2008 Strategic Plan is
    related to Goal #1, elements and performance measures: For relevancy
    and reliability of the EIA information program, the performance
    measure is the percentage of EIA survey frames with sufficient
    coverage to produce reliable supply, demand and price statistics.

[^4]: Grace Sutherland was at the forefront of this earlier effort.

[^5]: The team members are Joe Conklin (OOG), Mark Friedman (OIT),
    William Gifford (EMEU), Marianne Holly (OOG), Tom Schmitz (CNEAF),
    Grace Sutherland (SMG, Ass't to the Chair), Howard Bradsher-Fredrick
    (SMG, Chair)

[^6]: A team's charter normally includes information about the
    following: background of the problem at issue, objectives, expected
    deliverables/deadlines, costs and benefits, resources available,
    performance measures, a list of team champions and a list of team
    members.

[^7]: The original recommendations of the first frames team are listed
    in italics.

[^8]: Different survey managers may have a different perception of unit
    counts, but the team and the SMG staff members who collected data
    attempted to be as consistent as possible.

[^9]: Recommendation of Tom Lorenz at ASA-Energy Committee Meeting
    (October 2004).

[^10]: Cross-checking involves comparing one list of frame elements
    (usually the frame of interest) with an independently-collected list
    of frame elements.