# Presentation: 243845

## October 19, 2007
Washington, DC
Paul Holtberg, Director, 
Demand and Integration Division, 
Office of Integrated Analysis and Forecasting

- Washington, DC
- Paul Holtberg, Director,
- Demand and Integration Division,
- Office of Integrated Analysis and Forecasting

- The Quality Control Process Used for Developing and Releasing the Annual Energy Outlook

*This is a working document prepared by the Energy Information Administration (EIA) in order to solicit advice and comment on statistical matters from the American Statistical Association Committee on Energy Statistics. This topic will be discussed at EIA's Fall 2007 meeting with the Committee to be held October 18 and 19, 2007. ** *

**Notes:**
The Annual Energy Outlook is one of only 4 hardcopy publications still published by EIA.  To understand the process that leads to the release of the AEO, I intend to review the quality control process.
Since the AEO is a major EIA publication, it involves one of the most complete quality control processes of any EIA publication.  

## One of four major “products” produced by OIAF

- The Quality Control Process for Developing and Releasing the Annual Energy Outlook:  October 19, 2007, ASA Committee on Energy Statistics

**Notes:**
The AEO is one of 4 annual products produced by the EIA Office of Integrated Analysis and Forecasting (e.g., AEO, IEO, Voluntary Report on Greenhouse Gases, and Emissions of Greenhouse Gases).

## National Energy Modeling System

- The Quality Control Process for Developing and Releasing the Annual Energy Outlook:  October 19, 2007, ASA Committee on Energy Statistics

**Notes:**
The AEO is produced using the National Energy Modeling System, which is summarized here.
Didn’t want to discuss here, just providing for reference.

## Questions for the Committee

- Is external review process _**sufficient**_?
- Do we need to _**expand**_ number or type of reviewers included in the process?
- Are there ways in which the review process could be _**streamlined**_?
- Are there _**other aspects**_ of the AEO that need to be reviewed?
- How do we _**balance**_ need to provide independent projections and analysis with stakeholder inputs we may not agree with?

- The Quality Control Process for Developing and Releasing the Annual Energy Outlook:  October 19, 2007, ASA Committee on Energy Statistics

**Notes:**
Before I get into the nuts and bolts, I wanted to pose some questions that I would like the committee to keep in mind.
I’m not going to go through these now, but will return to them at the end of the presentation.

## Five Major Objectives for AEO Quality Control Process

- Addresses areas of interest to stakeholders
- Includes input from all of EIA
- Data is consistent with other EIA data
- Models are applied correctly and produce unbiased projections
- Models and projection results/analysis are completely documented

- The Quality Control Process for Developing and Releasing the Annual Energy Outlook:  October 19, 2007, ASA Committee on Energy Statistics

**Notes:**
We try to balance five major objectives in producing the AEO.
First, ensure that the AEO addresses areas of interest to stakeholders. 
This is an EIA product, not OIAF.  So we want to make sure the projections include input from all of EIA. 
The projections include data that EIA reports as historical time series.  We want to make sure that the data reported in the AEO is reasonably consistent with the report EIA historical data.
The models need to be regularly reviewed to make sure they are consistent with current thought, policy, and market trends and that they provide unbiased projections.
Want to make sure that the models and results are completely documented for use as a reference.

## AEO Production Process is Year Round

- January-September: Data/model evaluation and development
- August-October: Projection development and analysis
- October-April: Publications/Documentation

- The Quality Control Process for Developing and Releasing the Annual Energy Outlook:  October 19, 2007, ASA Committee on Energy Statistics

**Notes:**
The AEO production process is a year round process.
Data and model development takes place between January and September.  
Projection development and analysis takes place between August and October, with the projection development focused during the early months and the analysis during the later months.
Publications and documentation is completed between October and April.  This includes multiple documents:
The AEO Retrospective Review is completed between August and January based on the prior year’s AEO.
The AEO Summary publications is completed between October and February
The methodology reports are completed between January and April.

## Quality Control Applied At All Steps in AEO Process

- Reassessment of new historical data (Data)
- Update of model structure and addition of new capability (Modeling)
- Successive drafts of projection (Projection/Analysis)
- Publication/Documentation of projection results and models (Documentation)

- The Quality Control Process for Developing and Releasing the Annual Energy Outlook:  October 19, 2007, ASA Committee on Energy Statistics

**Notes:**
The AEO Quality Control process is applied at all steps of the process.
We do a reassessment of newly available historical data, which is used as input to update the model relationships and to develop assumptions.
We regularly review and revise the models to add additional modeling capability.
The preliminary projection results are regularly reviewed for many weeks before finalization.
All publications and documentation of both the projection results and models is carefully reviewed before release.

## Extensive External and Internal Participation in Quality Control Process

- External Stakeholders
- Formally
- Informally
- Internally
- Office of Integrated Analysis and Forecasting
- Other EIA Offices
- EIA senior management

- The Quality Control Process for Developing and Releasing the Annual Energy Outlook:  October 19, 2007, ASA Committee on Energy Statistics

**Notes:**
The quality control process employs both external and internal stakeholders.
The external reviews are both formal (e.g., surveys, peer review process, working groups, independent expert reviews, and conference presentations) and informal (e.g., questions received from users and independent reviews/articles).
The internal reviews are completed at multiple levels within EIA.  
For example, at the OIAF Office level by staff and management.  
By other EIA Offices through the Category 1 review process, working groups, participation in the development of assumptions and models, review of draft projection results, and review of documentation.  
By EIA senior management through multiple presentations of key assumptions and projection results.

## Timing of Input to Quality Control Process Varies

- Much of external input is “after the fact” for the current AEO.
- However, participation in independent expert reviews and working groups during development process.
- Internal input coincides with process

- The Quality Control Process for Developing and Releasing the Annual Energy Outlook:  October 19, 2007, ASA Committee on Energy Statistics

**Notes:**
Now, the timing of the input to the quality control process varies.
For example, much of the external input, particularly the informal input, is “after the fact.”  Users are reacting to published results.  So this type of review tends to impact the following year’s AEO.
However, that is not always the case.  The external review includes independent expert reviews and working groups, which are completed during the development process.
The internal input tends to coincide with the process and is incorporated in the AEO prior to release.

## Quality Control During Data Update Process Is Predominantly Internal

- Historical data reviewed by OIAF analysts and management
- Third party data reviewed by OIAF staff, working groups, and external experts
- Short Term Energy Outlook data reviewed by OIAF analysts and management

- The Quality Control Process for Developing and Releasing the Annual Energy Outlook:  October 19, 2007, ASA Committee on Energy Statistics

**Notes:**
The next few slides, step through the quality control process for each major step in the AEO, starting with the data update process.
Quality control during data update is predominantly internal.
As new historical data become available, it is reviewed internally by OIAF analysts and management for consistency with prior data series.
Some data, for example data characterizing technologies, is developed through third party studies.  That data is typically reviewed by OIAF staff, working groups, and external experts.
We try to maintain consistency with the EIA Short Term Energy Outlook. The STEO projection is reviewed by OIAF analysts and management for consistency.  This is not just done after the STEO is completed, but involves discussion with STEO analysts during production.

## Quality Control During Data Update Process Is Predominantly Internal (Continued)

- Major exogenous assumptions complemented by a Delphi approach with outside experts

- The Quality Control Process for Developing and Releasing the Annual Energy Outlook:  October 19, 2007, ASA Committee on Energy Statistics

**Notes:**
OPTIONAL
Major exogenous assumptions, such as the world oil price, are complemented by a Delphi approach involving external experts as well as the EIA Offices, management, and staff.  This insures that these major assumptions reflect a broad perspective.

## Model Updates Subject to Extensive Quality Control Review at Multiple Levels

- Model updates identified through Independent Expert Reviews, customer comments, and internal analyst review of trends
- Proposed model updates discussed with working groups and OIAF management
- Internal updates reviewed by analysts, OIAF management, and working groups

- The Quality Control Process for Developing and Releasing the Annual Energy Outlook:  October 19, 2007, ASA Committee on Energy Statistics

**Notes:**
Turning to the model update process.
The need for model updates is identified through independent expert reviews, customer comments received on the previous AEO, and, of course, internal analyst review of energy market trends.
The proposed model updates are discussed with working groups (composed of internal and external stakeholders) at the beginning of the AEO modeling process and with OIAF management.
Once the updates are completed, they are reviewed by the analysts, OIAF management, and working groups

## Model Updates Subject to Extensive Quality Control Review at Multiple Levels (Continued)

- Weekly run review held with analysts
- Contractor updates reviewed by analysts, OIAF management, and working groups
- Deputy Admin reviews major updates
- SMG reviews model updates in documentation
- External peer review includes questions on modeling

- The Quality Control Process for Developing and Releasing the Annual Energy Outlook:  October 19, 2007, ASA Committee on Energy Statistics

**Notes:**
The OIAF managers hold weekly run reviews with the analysts to track modeling progress.  We do these in three subgroups stretched out over about 3 hours once a week. 
Model updates completed by contractors are reviewed by the analysts, OIAF management, and in working group meetings.
Major updates or those that are particularly sensitive include a review by the Deputy Administrator (e.g., work on ethanol).
The model updates are also reviewed by SMG as part of documentation review.
Finally, we conduct an external peer review that includes questions on the modeling, which feed into the next year’s AEO.
Composed 10 big users of AEO who are familiar with the models and projections
Membership of groups changes every year

## Also, Extensive Projection/Analysis Quality Control Process

- OIAF managers meet weekly to discuss projection progress
- Weekly run review held with analysts
- Deputy Administrator briefed on preliminary results for key variables early in process
- Preliminary projection results discussed with working groups

- The Quality Control Process for Developing and Releasing the Annual Energy Outlook:  October 19, 2007, ASA Committee on Energy Statistics

**Notes:**
Next is the projection/analysis quality control process.
Again, the OIAF managers meet weekly to discuss projection progress to identify issues and priorities.
A weekly run review is held with the analysts to discuss draft results, identify issues, and track projection progress.  This is formatted the same as the model update review, over about 3 hours in targeted session held once a week.
Initial meetings are done as one large joint session that allows analysts to provide comments outside their work area.
Joint meetings also alert analysts to changes that may impact their efforts.
Early in the process, the Deputy Administrator is briefed on preliminary results for key variables (e.g., natural gas prices).
The preliminary projection results are discussed with the working groups (again, composed of external and internal  stakeholders) to allow input prior to finalization.

## Also, Extensive Projection/Analysis Quality Control Process (Continued)

- EIA Offices briefed on preliminary projection results
- Administrator and Deputy Administrator briefed on preliminary projection results
- External peer review includes questions on AEO projections
- Publication of Retrospective Review

- The Quality Control Process for Developing and Releasing the Annual Energy Outlook:  October 19, 2007, ASA Committee on Energy Statistics

**Notes:**
All EIA Offices are briefed on preliminary projection results prior to completion to get input.
The Administrator and Deputy Administrator are briefed on the preliminary projection results prior to completion to get input.  This occurs late in the process.
The external peer review includes questions on the AEO projections, which provide input for future AEOs.
The Retrospective Review, which measures the error in the AEO projection over time, is published each year following completion of the AEO.  The Review provides information on the magnitude and direction of error and is used as input when making changes for the next AEO. 

## Quality Control during Writing of AEO Summary Publication is Multifaceted

- Topics developed using input from customers
- Topics approved by OIAF management and Deputy Administrator
- Outlines for all section approved by OIAF management
- Drafts reviewed and approved by Division Director and OIAF management
- Publication professionally edited 4 times
- Drafts reviewed 3 times by EIA Deputy Administrator and 1 time by Administrator

- The Quality Control Process for Developing and Releasing the Annual Energy Outlook:  October 19, 2007, ASA Committee on Energy Statistics

**Notes:**
The quality control process applied during the publication writing process is multifaceted and is probably the most detailed of all.
The major analysis topics included in the Issues In Focus and Legs & Regs sections of the AEO are developed using customer input collected through surveys and other contacts.
The final analysis topics are approved by OIAF management and the Deputy Administrator.
Outlines for all major sections are completed and approved by OIAF management prior to writing.
Drafts of all sections are reviewed and approved by the relevant Division Director and later by OIAF management.
The publication drafts are professionally edited 4 times during the process following comment at different points in the review.
The drafts are reviewed 3 times by the EIA Deputy Administrator and 1 time by the Administrator prior to finalization.

## Quality Control during Writing of AEO Summary Publication is Multifaceted (Continued)

- All sections reviewed or QAed by OIAF staff
- Major publication pieces go through Category 1 review process
- External peer review includes questions on AEO summary publication
- Erratum posted when errors found

- The Quality Control Process for Developing and Releasing the Annual Energy Outlook:  October 19, 2007, ASA Committee on Energy Statistics

**Notes:**
All sections of the AEO are reviewed or QAed by OIAF staff who were not involved in writing the section.  This involves checking all data to ensure that it is correct and reviewing the writing to ensure that it is readable and arguments are logical.
All of the major AEO publication pieces (e.g., Legs & Regs, Issues In Focus, and Market Trends) go through the EIA Category 1 process that involves a detailed review by the other EIA Offices.  Following this review, a Category 1 response memo is prepared that discusses how the comments were addressed.
The external peer review includes questions on the quality and appropriateness of the AEO summary publication.  These comments provide input to the next AEO.
An erratum is prepared and posted online as errors are found during the year.

## Quality Control During Methodology Documentation Completes Process

- Starts from a single template
- Revised based on input from previous year Statistical Methods Group review
- Reviewed by Division Director
- Reviewed and approved by OIAF Director
- Sent to SMG for review

- The Quality Control Process for Developing and Releasing the Annual Energy Outlook:  October 19, 2007, ASA Committee on Energy Statistics

**Notes:**
The last aspect of the quality control process addresses the methodology documentation.
All of the methodology documentation is revised by the modelers to reflect updates and input from SMG reviews from the previous year or years.
The methodology drafts are reviewed by the responsible Division Director.
The draft final methodology documentation is reviewed and approved by OIAF Director prior to release.
The completed methodology documentation is sent to SMG for review on a cycle that results in the review of all sections over a 3 year time period.

## Summary

- AEO quality control process is extensive
- Quality control process
- Involves internal and external participants
- Applied at all production levels
- Provides input to current and future processes
- But, quality control process is time consuming and resource intensive

- The Quality Control Process for Developing and Releasing the Annual Energy Outlook:  October 19, 2007, ASA Committee on Energy Statistics

**Notes:**
In summary, the AEO quality control process is extensive and, we hope, leads to the release of a quality product.
The quality control process:
Involves both internal and external participants
Is applied at all levels of the production process
Provides input to both current and future AEOs.
The one drawback is that the quality control process is time consuming and resource intensive.  
For example, the quality control process for writing the AEO 2007 summary involved over 80 working days. 

## Questions for the Committee

- Is external review process sufficient?
- Do we need to expand number or type of reviewers included in the process?
- Are there ways in which the review process could be streamlined?
- Are there other aspects of the AEO that need to be reviewed?
- How do we balance need to provide independent projections and analysis with stakeholder inputs we may not agree with?

- The Quality Control Process for Developing and Releasing the Annual Energy Outlook:  October 19, 2007, ASA Committee on Energy Statistics

**Notes:**
This last slide once again poses some questions that I would like the committee to consider.
First, is the external review process sufficient?
Do we need to expand number or type of reviewers included in the process?
Are there ways in which the review process could be streamlined?
Are there other aspects of the AEO that need to be reviewed?
How do we balance the need to provide independent projections and analysis with stakeholder inputs we may not agree with?

## Thank you

- Energy Information Administration
- www.eia.doe.gov
- Paul Holtberg
- Paul.Holtberg@eia.doe.gov
- 202-586-1284

- The Quality Control Process for Developing and Releasing the Annual Energy Outlook:  October 19, 2007, ASA Committee on Energy Statistics