# Presentation: 388518

## Follow-up on Frames Team Activities

- Presentation to the ASA-Energy Committee
- Howard Bradsher-Fredrick, SMG
- April 28, 2005

**Notes:**
Today I’m going to talk about the most recent activities of the frames team.  In the 2004 sessions, some of us have presented information about EIA frames activities.  Also, you heard a presentation from the Census Bureau this morning related to frames comparisons.  

## Outline of Presentation

- Background
- Previous Frames Team Effort
- Frames Sufficiency Criteria
- Results of Frame Evaluation for Sufficiency
- Frames Team New Activities
- Recommendations of Latest Team
- Questions for Committee

**Notes:**
This is the outline of my presentation.  I’ll begin by discussing some background information.  Then, I’ll remind you about previous efforts of the frames team.  After that, I’ll talk about the more recent efforts of the frames team including our work on frames sufficiency and the results of this work.  From there, I’ll talk about possible new directions for the team and finally get to the questions for the committee.

## Background

- EIA Strategic Plan
- Provides the impetus for increased efficiencies in frames activities
- Requires frames performance measures concerning “sufficiency” of frames
- “% of EIA survey frames with sufficient industry coverage to produce reliable supply, demand and price statistics.”

**Notes:**
The EIA strategic plan provides the basis for the frames team’s activities.  It provides the impetus for increasing efficiencies in frames usage and it calls for performance measures concerning sufficiency of frames.

The actual language in the strategic plan is ..... (last bullet).

## Previous Frames Team Effort 

- Frame checking against alternative lists at respondent level
- Data comparisons at an aggregate level
- Examining supply/disposition balances
- For price data, comparing volumes that the price data represent with total volumes
- Dual system methodology

**Notes:**
At both of the meetings last year,there were presentations discussing frame checking activities at the respondent level.  This work included comparing National Renewable Energy Lab plant level data with EIA electric power plant data.  It also included checking Petroleum supply data with Manufacturers Energy Consumption data and comparing EIA power plant data with EIA coal consumption data.   Shawna Waugh discussed the proposed work for conducting data comparisons with the Census Bureau.  You heard an interim report on the results of that work this morning.

Some data comparisons at the aggregated level were also discussed.  We compared petroleum marketing data with consumer price index data. 

The balancing item with respect to natural gas supply versus disposition data was also discussed. 
For price data, we described our comparison of volumes that the price data represent in comparison with total volumes.  This was done with natural gas commercial price data.

At the last fall meeting we discussed the dual system methodology.  As was recommended by the committee, we will be presenting our work on this topic at the Federal Committee on Statistical Methodology Research Conference   this November. 

## Frame Sufficiency Criteria 

- Existence of a systematic maintenance approach
- Frame volatility over time
- Use of comparable frame listings from other sources
- Existence of a balancing item and the magnitude and stability of the balancing item over time
- Relative concentration of volumes in a few respondents
- Changes in the industry, legislation, regs, etc.
- Independent assessment by the survey manager of the frame’s quality

**Notes:**
Now we’ll discuss more recent activities of the frames team.  During the last 6 months of last year we evaluated all of EIA’s master frames for sufficiency.  We employed the criteria shown here.

Read slides..............

## Information Collected

- Initial basic frame information
- Stability (or volatility of the frame)
- Corresponding useful listings of frame units
- Balancing item stability
- Relative concentration of volume in a few respondents (skewed distribution)

**Notes:**
Employing the criteria shown required us to collect a large amount of data for each survey frame.  These data are shown in Appendices A – D in the paper.

Read 2nd – 5th bullets..............

## Results of Sufficiency Evaluation 

- Reviewed 34 frames:
- 5 were deemed out-of-scope
- The team found 29 frames within the team’s purview, with 25 deemed sufficient and 4 deemed insufficient or don’t know.

## Insufficient Frames

- EIA-815 (Monthly Terminal Blenders Report)
- EIA-910 (Monthly Natural Gas Marketers Survey)
- EIA-886 (Annual Survey of Alternative Fueled Vehicle Suppliers and Users)
- EIA-3 (Quarterly Coal Consumption and Quarterly Report, Manufacturing Plants)

**Notes:**
Now to the frames deemed insufficient.

The EIA-815 (Monthly Terminal Blenders Report) is a fairly new frame.  It’s difficult to identify blenders of gasoline and no reliable comprehensive registries currently exist for comparison purposes though the EPA is currently developing one.

The EIA-910 (Monthly Natural Gas Marketers Report) is also a fairly new frame.  Again it’s difficult to identify marketers.  Also, new states are being added so the frame is in a state of flux.

The EIA-886 (Annual Survey of AFV Suppliers and Users) has difficulty in identifying and collecting data from fleets owned by private non-utility companies and local governments.  These fleets could contain substantial numbers of AFVs.  EIA is looking at changes to the law that would facilitate easier and more accurate data collections on the 886.

Concerning the EIA-3 frame (Quarterly Coal Consumption & Quarterly Report, Manufacturing Plants), the balancing item shows this frame to be fairly good, but there were some questions regarding whether or not procedures were in place to maintain quality in the future.  The Census Bureau will shortly be conducting data comparisons to the EIA-3.

## Limitations of Frames Sufficiency Approach

- EIA may want to determine sufficiency by survey rather than by master frame
- Pros:
- Better measure of frame sufficiency (perhaps more useful for EIA)
- Cons:
- 67 surveys versus 29 master frames
- Strategic Plan calls for “frame sufficiency.”

**Notes:**
By the time we completed the frames sufficiency evaluation, it became evident that there may be alternative ways of proceeding with this type of evaluation. 
Subsets of some EIA master frames serve a number of different surveys.  For example, the EIA-860 (The Annual Electric Generator Report) serves as a master frame for a number of surveys including the EIA-920 (The Combined Heat and Power Plant Report).  While the team deemed the EIA-860 sufficient, it may be that the 920 frame would have been more thoroughly scrutinized for sufficiency if it were evaluated separately.  

Begin 2nd bullet.......

## Frames Team New Activities

- The team’s charter states the team’s objectives:
- To work toward a long-term goal of enhanced commonality in IT
- To identify opportunities to share frame information
- To monitor the quality of EIA frames, recommend needed enhancements to frames, recommend surveys to be discontinued or that result tables be provided with appropriate caveats
- To recommend further studies needed to evaluate frames sufficiency
- To prepare updated recommendations for the “Frame Sufficiency” performance measure

## Recommendation on Team’s Activities

- High Priority
- Complete/maintain an inventory of EIA frames
- Conduct quarterly meetings (or special meetings as the need arises) among affected or interested offices
- Study/update frame documentation standards
- Medium Priority
- Develop best practices for system maintenance, support & flexibility of system design
- Look into replacing existing standard on Frames Maintenance & Development with a revised version

## Questions for Committee

- Are there any suggestions as to how to proceed in determining sufficiency of frames in the future?
- Are there any comments or advice regarding the current frames team’s recommendations?