AMERICAN STATISTICAL ASSOCIATION (ASA)

MEETING OF THE COMMITTEE ON ENERGY STATISTICS

WITH THE ENERGY INFORMATION ADMINISTRATION (EIA)

Washington, D.C.

Friday, April 29, 2005

COMMITTEE MEMBERS:

NICOLAS HENGARTNER, Chair

Los Alamos National Laboratory

MARK BERNSTEIN

RAND Corporation

CUTLER CLEVELAND

Center for Energy and Environmental Studies

JAE EDMONDS

Pacific Northwest National Laboratory

MOSHE FEDER

Research Triangle Institute

BARBARA FORSYTH

Westat

WALTER HILL

St. Mary\'s College of Maryland

NEHA KHANNA

Binghamton University

NAGARAJ K. NEERCHAL

University of Maryland Baltimore County

SUSAN M. SEREIKA

University of Pittsburgh

DARIUS SINGPURWALLA

LECG

RANDY R. SITTER

Simon Fraser University

ALSO PRESENT:

MARGOT ANDERSON

Energy Information Administration

ALSO PRESENT (CONT\'D):

HENRY BROOKS

Energy Information Administration

BETH CAMPBELL

Energy Information Administration

GUY CARUSO

Energy Information Administration

BRENDA COX

Energy Information Administration Contractor

JOHN PAUL DELEY

Energy Information Administration

HOWARD BRADSHER-FREDRICK

Energy Information Administration

CAROL FRENCH

Energy Information Administration

LYNN GEISERT

Z, Inc.

HOWARD GRUENSPECHT

Energy Information Administration

NANCY KIRKENDALL

Energy Information Administration

LAURIE KRAUSS

Office of Management and Budget

RUEY PYNG LU

Energy Information Administration

RENEE MILLER

Energy Information Administration

KARA NORMAN

Energy Information Administration

ALSO PRESENT (CONT\'D):

IRENE OLSON

Energy Information Administration

JANICE POLING

Energy Information Administration

ERIK RASMUSSEN

Energy Information Administration

KEN VAGTS

Energy Information Administration

BILL WEINIG

Energy Information Administration

MARGIT WHITAKER

Government Accounting Office

JOHN WOOD

Energy Information Administration

BARRY YAFFE

Energy Information Administration

\* \* \* \* \*

C O N T E N T S

AGENDA SESSION: PAGE

Assessments: Presentations and

a Panel Discussion

External Review of Survey

Programs: A Progress Report

External Review of EIA Program

Progress on EIA\'s 914: Response

Rate and Kinds of Challenges

Committee Discussants on EIA\'s

914, EIA\'s 826

\* \* \* \* \*

P R O C E E D I N G S

DR. HENGARTNER: Ladies and gentlemen, welcome back to the second session
of the ASA Committee on Energy Statistics with the EIA. It\'s a pleasure
to be back. Nancy will talk to us this morning about assessments,
presentations, and panel discussions.

MS. KIRKENDALL: This is an update session but we\'ve made a lot of
progress since last time too. Just to remind you, last fall we had our
first PART assessment. That\'s a program assessment rating tool, that\'s
an Office of Management Budget program, and they rated us poorly in two
areas. The questions were regular independent evaluations used to
support program improvement, assess effectiveness and relevance, and
they said no, we didn\'t have any evidence of that. And the second one
was do we have independent evaluations that indicate a program is
effective and achieving results, and they said we had a limited amount
of that.

And so we have done some work to try to get started on doing more with
external assessment and this will be to tell you where we are. And there
are two words in here that had occurred to me when I was looking at them
this morning that we need to pay attention to and what we may not have
is what you\'ll hear about today is we\'ve been looking at evaluations
to support program improvement and look at our relevance but I\'m not
sure that we have them looking at effectiveness. So that\'s something
for the team who are sitting back there to think about. I think we need
to take another look at these words and see where we\'re going with it.

So the last meeting was very similar to this meeting. We had Brenda Cox,
who told us about our plans to look at a program assessment of a family
of surveys. At that point she told you about the template that she was
using for evaluating a single survey.

Well, now she\'s developed a template for evaluating a program of
surveys and she tried it out on the petroleum marketing package. So
she\'s going to give you an update of where she is with that. Last time
Doug Hill did a discussion about external reviews that was supposed to
look at modeling and forecasting but you guys came back with the idea
for a high-powered review team and so we have taken that and are running
with it and John Paul Deley is going to tell about our progress with
that.

For this session we only have like an hour and five minutes and so the
way we thought we\'d set it up is that Brenda will talk for 10 minutes.
We\'ll have discussion among the committee about her talk. That\'s the
evaluation for a survey program. Then John Paul will talk for 10 minutes
telling you where we are with the external study team and we\'ll have
another discussion for 10 minutes and then trying to assemble the
pieces.

So this is just open discussion, really any ideas you have about
external program evaluations, but the other word in the items that we
didn\'t do so well on with PART was that we have this on a regular basis
and so then the thought is how do we put this all together in a sensible
program. Maybe the high-powered review team is every five years,
something that might feed into strategic planning, the overview of EIA,
maybe combined with individual program- specific evaluations during the
interim.

Anyway, that would be an open discussion. Any idea is welcome, throw
things out. Well, those are my questions, what I just said, so why
don\'t we just start with Brenda and go on from there?

MS. COX: You\'re queuing up my talk?

MS. KIRKENDALL: I\'m trying.

MS. COX: Okay, I\'ll start. Well, the last time we spoke for this
external review of survey programs I should tell the newcomers that the
idea behind this project was to develop the procedures. So the process
of an external review of a family of surveys, that was our goal. But we
decided that there\'s no way that you can develop a final process
without trying it out and seeing how it works and evaluating it so that
is what I\'m speaking on today.

This sort of research, by the way, is being done by Battelle under a
subcontract we have from Z, Inc., and they\'ve been very helpful. Henry
Brooks of Z, Inc., has also assisted me on this project.

When I was here before I actually presented the survey template which
was a guide to review on an individual survey. Since then we\'ve
developed the program evaluation template. We\'ve also tested the survey
evaluation template on six surveys. We\'ve tested the program evaluation
template. Both of these were done for the petroleum marketing family of
surveys. The next step is to produce these penultimate templates using
the experience we\'ve gained to say what do we think these survey and
program templates should be like and then to document the findings.

Now, the new thing for this committee is the program evaluation template
and that was included with the materials that were given to the
committee. The template has an overall description of the program, the
program\'s objectives, target populations, sampling frames, program
design, and for an individual survey it\'d be a sample design but for a
program it\'s the overall design, the conceptual design, of the whole
program, data collection and processing, data analysis products and
documentations, and then summary of findings.

We\'ve actually tested both templates now. Only publicly available
documentation was used for the evaluation. That would include the OMB
package, explanatory notes and reports, the actual forms that the
respondents complete, and anything posted on EIA\'s website.

We had limited interactions with program staff at EIA so it\'s pretty
much done independently. And then for both the surveys and the program
we produced a summary of findings which were shared with program staff
to obtain their comments and corrections.

For the survey template we actually evaluated six petroleum marketing
surveys. The survey methods were summarized and critiqued for each item
on the template. Recommendations or endorsements were provided.

Endorsements are when the survey is doing something right you need to
say so. In other words you need to both talk about what\'s good as well
as recommendations for improvement.

And then the results were documented for EIA staff review. These
summaries for the survey were from seven to ten pages so they\'re fairly
long, detailed descriptions of what the survey is doing and what they
might be doing.

The results, which I was very pleased with, is that the whole survey
evaluation process worked. We did discover that originally in the
template I presented to the committee last time we had the
recommendations at the end of the survey. That didn\'t work because it
was like repetition. You\'re discussing a problem and you wait until the
end to talk about what to do about it. So we felt like it was better to
have the recommendations presented and labeled at the time of the
discussion. So if you have a recommendation for target population it\'d
be presented in the target population section.

We found that design attributes and flaws were repeated across surveys
so that suggests in some ways that there will be a synergy possible
through doing these evaluations because not only these surveys but
perhaps other EIA surveys can benefit from the comments.

Then we found what I would say is a serious lack of survey methods
reports which did impede the evaluation. I would say EIA is not unique
in this regard as federal agencies go, it\'s probably more like the
norm, but I think this is something that needs to be worked on. The
typical findings for an individual survey were that the survey
objectives need to be more completely defined. Typically we took the
survey objectives out of the OMB package.

The target population definitions needed to be made more specific.
Sometimes I couldn\'t tell what was being included geographically in the
target population, for instance. Some include the territories and
possessions and some it wasn\'t clear if they were or were not being
included, for instance. We found that coverage and frame updating
procedures need to be specified and sometimes they weren\'t.

We found that insufficient information was provided to evaluate the
editing procedures. We found that the mailout package tends to be very
well designed, might need a little tweaking in minor ways, so this was
not a major criticism. We were just finding little things that could be
tweaked about the mailout package but it has a nice design overall.

For the program itself, which was done after evaluating six of the
surveys and reviewing the rest, we found there was no overall program
documentation at that point in time. We found the survey evaluations
were more important input than we originally thought. When we started
this process we thought the survey evaluations would be done just to get
the reviewer, and that would be me, accustomed to what the whole family
was like and then turn to the program.

Well, we found those were just more important than we had conceived. We
found quite a bit of variations across surveys that were both
intentional and due to happenstance and that in part is because the
program wasn\'t designed from scratch is the best way to put it.

MR. FEDER: Brenda, a variation in what sense?

MS. COX: For instance, the definition of a firm would vary across
surveys. Some surveys were extremely precise and I liked what they did
in the way they defined the survey. Others were not as precise. In fact
it looked like they just let the company decide who should report for
that company or whether multiple reports should be done.

What other examples would I give? The target population does differ
across surveys. It starts out including the territories and possessions.
At some point it changes and it\'s only the 50 states and DC so that\'s
an example of intentional. It\'s not really presented so that you can
clearly see that a change is occurring.

The due to happenstance comes about just because these were derived over
such a long period of time. In fact many or most of these surveys were
inherited by EIA from other agencies. What I\'m saying here is this
program was not designed as a program and then the surveys evolved from
the program. Rather the surveys came about and the program has just been
the compilation of the surveys in some respect.

For the program findings we found that there was a program level
conceptual design needed that didn\'t exist, at least documented as a
public document. I should say EIA being the way EIA is there\'s already
a draft of that started since I did this review. I found the population
definitions and reporting conventions need to be standardized across the
surveys, that because of the way the surveys were created this is
something that\'s needed because often the same companies are responding
to these surveys and it could even be the same person so you need a
certain commonality across surveys and that methods documentation was
needed for the program and the surveys.

Now, for this contract the remaining activities are to use the
information that we\'ve gained to revise the two templates to create
what will be the penultimate template based upon the results of this
actual experiment and then document the templates and their use.

I have these questions for the committee. You\'ve never seen the program
templates so I asked are there additional items that you think should be
included in the program template, do some program template items need to
be modified or expanded, and are there program template items that
should be deleted. So basically I thought that you might want to comment
on the

template itself.

That\'s it. Oh, and I should say the committee was also given a copy of
the template as well as the actual executed version for the petroleum
marketing family.

DR. HENGARTNER: Comments, anyone?

MR. FEDER: More a question than a comment, Nancy, when you were, as you
said, rated not so great on that particular issue was it more at the
program level or the survey level?

MS. KIRKENDALL: It was program level and that comes up with the question
of what\'s a program. I mean, one view is that well, EIA is a program,
we are an energy information program, and so that\'s the topic. Getting
a review at that level is what John Paul will talk about. But we also
have other programs and I think right now we bundle our OMB clearance
packages and they actually do define programs so we have a petroleum
marketing program.

The surveys do fit together in a sensible way and can be evaluated by
themselves. Do we do a good job covering petroleum marketing area, the
prices and things that we\'re supposed to collect in that area?
Petroleum supply might be in another area or maybe those are related and
should be looked at together but you can break down what we do in terms
of programs to see if the answer to the OMB questions could be provided
for those too.

MS. COX: And I would suspect just as we found that you have to look at
the surveys to talk about the petroleum marketing program you probably
have to look at the individual programs then to talk about the larger
EIA program. So it\'s like looking at the building blocks as you go up
to talk about what you\'re doing, why you\'re doing it, and is it being
done in the best way it could be done.

DR. HENGARTNER: It occurs to me that this overall review is a very good
thing but it might miss maybe a third of the work you\'re doing, namely,
every special report that\'s mandated by Congress. How are you going to
evaluate these and how are you going to show Congress or the OMB that
this is valuable and well done given that they\'re one-time shocks?

MS. KIRKENDALL: That\'s a good question. One of the things we\'ve tried
to do for this external review team is to prepare documentation saying
what is EIA. As Brenda said, if you take our little programs and you put
them together and that\'s EIA. But when you look from the top down at
what we do that should be the overall EIA program. And one reason why we
started with Brenda looking at surveys is that, at least for me, it\'s
easier to think about a program of surveys they build up to try to
measure some.

But we also have other programs. We have our forecasting programs.
You\'ve heard about STEO. Margot has lots of different programs under
her jurisdiction. She has the short-term forecasting. She has the
country analysis group. I\'m not sure what you\'d say her program is
because it\'s so diverse. We have the NEMS forecasting. Is that one
program? It\'s easy to say that in one breath so is that one program?
And then there are the service reports and the people in the forecasting
office do a lot of those. So you\'re right. We need to have those
included too somehow.

MR. SINGPURWALLA: How many surveys were included in your initial setup
review?

MS. KIRKENDALL: I think Brenda looked at six but the petroleum marketing
package has, I think, eleven.

MS. COX: Eleven.

MS. KIRKENDALL: So hopefully we\'ll have a little more funding and can
actually finish off the whole package.

DR. HENGARTNER: So, going back to your answer, how many programs does
EIA have?

MS. KIRKENDALL: Well, we have to figure out how to count them.

DR. HENGARTNER: I mean, it\'s very interesting what you\'re showing us,
Brenda, but as a statistician I want to know what\'s the population. And
it might be helpful if we know what the population is just by numbers,
for example, be able to identify if we can help you.

MS. KIRKENDALL: If you\'re talking about survey programs in oil and gas
you\'ve got petroleum marketing. And this is how we\'re organized, too,
which is why I say it this way because it\'s easy to break it down.
Petroleum marketing, petroleum supply, natural gas, and then reserves
and production are separate divisions and those are separate forms,
clearances packages, that go to OMB. In coal you have the electric power
surveys, the coal surveys, and then some miscellaneous surveys like
renewables, alternative fuel vehicles. Margot has the financial
reporting system and the consumption surveys.

And consumption surveys are a big program but they\'re only once every
four years. I would say STEO is a separate program. I\'d say NEMS is a
separate program. And after that it gets hard to say are there other
separate programs. And then we do analysis which is interspersed
everywhere and how do you capture that?

MS. COX: Sounds like the answer is 15 to 20.

MS. KIRKENDALL: And maybe they should be pared down.

MR. BERNSTEIN: The only thing that you said that bothered me was the no
interaction with EIA staff or limited interaction. That bothers me a
lot. I mean, you can do an independent evaluation and still have
interaction with staff.

MS. COX: Oh, that was the way the staff requested so that was the reason
we did it that way. We met with the staff after the evaluations were
complete.

MS. KIRKENDALL: It\'s true you get a different review if you have
interactions with the staff. You can go in for a lot more detail and
things that are missing from the documentation you can talk about it. So
this particular example of a review is a very good review of the
documentation we provide to the public.

The other thing that came about partly because of Brenda\'s review is
that now we have a nicer summary of what is the petroleum marketing
package and what does it try to do. And we didn\'t have that before
although it certainly existed in many minds in EIA. So I think there is
an advantage to doing just a documentation review but I don\'t think
that\'s the whole thing. I think something that involves interaction
with staff is useful too.

MS. COX: I would agree. I think actual documentation review as a first
step is a good idea because you\'re starting with an independent
reviewer who won\'t know the program and the pieces and so doing draft
evaluations based upon the information that\'s available is a way of
educating yourself into the nuances of the surveys. It\'s really hard to
wrap your mind around 11 surveys and what they\'re doing and then how
they fit into an overall program.

So actually doing a draft evaluation is a good way in some ways of both
evaluating the survey as well as educating yourself. But probably there
should be a

second step where then you meet with staff to fill in the holes because
there were things that I didn\'t evaluate because there wasn\'t
information available on it.

MS. KIRKENDALL: Why don\'t we move on and let John Paul talk and we\'ll
probably get more into some of these issues at the end too. Give it up
to my expert.

MR. DELEY: Good morning. The topic of the next presentation is the
External Review Program of the EIA and the title of the study is
Challenges, Choices, and Changes for the US Energy Information Program.

I\'m making this presentation this morning on behalf of a team of
colleagues who collectively are the support team for the external study
and I\'d like to acknowledge those that are here in the room, Erik
Rasmussen, from the Office of the Administrator, Janice Poling, who is
the EIA Presidential Management Fellow, Barry Yaffe from the Office of
Oil and Gas, and not with us today are Colleen Blessing, NEIC, Cathy
Cavanaugh, OOG, Doug Hill, Statistics and Methods Group, and, of course,
Nancy Kirkendall, Statistics and Methods Group.

This team will identify the study team leader, provide study team
questions and background materials, provide any additional information
requested by the team members throughout the study, provide logistical
support for the team meetings here at EIA, support discussions at the
fall meeting providing information and/or briefings requested by the
team, provide drafting support for the final report as requested by the
study team, and respond to the team recommendations.

The purpose of the study is to address the OMB requirements that were
referenced earlier. The external study team will look at these
requirements and EIA senior managers will use the results to help
determine how EIA should focus its resources in the next five years to
provide relevant energy information to policy makers, the marketplace,
and to the public. The results of the study will provide EIA\'s
budgeting and strategic planning process with the information it needs
to make decisions.

The principal study question and the principal objectives of the team
are to provide consensus advice to EIA concerning this overarching
question: Given continuing tight budgets and other external drivers is
EIA doing the right things? What are the right things that EIA should do
over the next five years. The final report to the EIA administrator will
contain the team\'s recommendations and rationale along with the
discussion of important issues. And this will be a public document.

The external study team leader has been identified as Dr. A. Denny
Ellerman, the Director of the Massachusetts Institute of Technology\'s
Energy and Environment Policy Studies Center. Dr. Ellerman\'s
responsibilities will be to work with EIA to select the other four
members of the study team, to work with EIA to clarify the purpose of
the study, the number of meetings needed, the list of questions to be
addressed, and the background information to be provided.

He\'ll help us clarify issues with other team members, conduct research
to answer the study questions, prepare an individual response to study
questions before the fall meeting here at EIA. He\'ll chair the
discussions of the fall meeting. He\'ll direct EIA support staff in
providing drafting services for the final report as desired by the team
and he\'ll provide a set of team consensus recommendations to EIA.

EIA is currently working with Dr. Ellerman to identify additional team
members and we hope to have this completed by the end of May. During the
summer and early fall the team will convene to address the primary study
question, review the background material provided, and solicit any
additional information needed for the study from EIA staff. In September
and/or October it is currently envisioned that the team will reconvene,
develop consensus on recommendations, and begin deliberations on the
form and content of the final report, which we anticipate will be
completed in January of \'06.

Members will work primarily at their own locations. The date and length
of time will accommodate the team member schedules and the work to be
done. EIA estimates that about 100 hours of work time would be required
from each member not including travel and that the leader\'s work would
be about 125 hours over a period of about eight months.

The team is independent of EIA. It will provide its recommendations to
the administrator of EIA in a final report. Individual team members will
compare and synthesize their responses to the study questions at a
meeting in September or October here in Washington, DC. The final report
will be the team\'s consensus recommendations with supporting
documentation and rationale.

The study will focus on EIA\'s outputs, its products and services, its
potential outputs, and its specific program areas and the relationship
to its key external drivers. EIA\'s internal business processes,
management practices, and systems are not within the scope of the study.
The planning horizon is five years. Some of the external drivers will
include energy laws affecting EIA, EIA budgets, emerging energy issues,
changes in the energy sector, and advances in information technology.

I\'m going to go through each of the nine questions that are going to be
proposed. The following are the nine questions from a longer list of
questions that were submitted by EIA staff and refined by the study
support team in cooperation with senior staff. The first question, what
will be the major energy issues facing the United States in the next
five years, how will EIA\'s current activities produce useful
information related to those issues, and what better information should
EIA provide on upcoming issues?

The second question, what kinds of questions should EIA prepare itself
to answer in the next five years? To what extent is EIA fulfilling its
legislative mandate to maintain a comprehensive energy information
system? The team will be provided with a comprehensive list of
EIA-related legislation and summaries as part of their background
package.

Question four, overall is EIA adequately covering key energy subjects
and current issues? What subject areas should be given more or less
attention in the future? Some of the subject areas to be considered
include but are not limited to fossil fuels, electricity, renewables,
nuclear, environment, economics of finance, consumption, production,
distribution, reserves, prices, and the relationship between domestic
and international programs.

Question number five, what should be the proper balance between data
analysis, forecasting, education, and documentation in the future?
Question number six, are there data series and/or analysis on
forecasting efforts that should be changed or discontinued? Question
number seven, what level and type of analytical work should EI do to
maintain the quality and relevance of its data and what alternatives
could EIA pursue?

Question number eight, are EIA\'s dissemination methods comprehensive
and accessible enough for our primary users? What if any changes do they
recommend? The intended audiences include but are not limited to
government policy makers, other governmental users, business and
industry, research and consulting, the media, and, of course, the
general public.

Question number nine, to what degree are the quality attributes of our
products adequate to meet our customer needs? Quality attributes have
been defined as accuracy, completeness, integrity, relevance, and
timeliness.

At this time we\'d like to open the floor for discussion of our plans.
The nine questions presented will be applied by the team to EIA products
and services, data analysis, forecast, education, and documentation.
There are three questions we would like you to consider during this
morning\'s presentation. I\'ll review all three of the questions and put
up a summary slide.

The first question is what sorts of background materials should be
provided by EIA to assist external reviewers more completely and
productively answer the study questions? What experiences have you had
in conducting and participating in external reviews that might assist
EIA prepare for a successful study? And, third, are there any concerns
you can share relating to the use or misuse of review recommendations
that should be considered in the project plan?

So the three questions are what kind of stuff should we put in the
background materials, your experiences with reviews, and any concerns
for our use or misuse of the recommendations. There\'s one other
document that I\'d like to pass out. This is a current draft of what we
planned to put in the background packages and it\'s just a copy of the
table of contents. So it gives you an idea of where we\'re heading with
preparing materials.

So at this time we can open the floor for discussion and address any of
those three questions.

MR. BERNSTEIN: I have a question different than your question. Have you
set out some criteria for what the rest of the committee would look like
other than Denny Ellerman?

MR. DELEY: One of the questions we omitted was about the mix. We
obviously don\'t want five oil guys from Texas. We want a certain spread
throughout the commission \--

MR. GRUENSPECHT: I\'ll say a little about that. I mean, actually we are
because it does need to be independent of EIA to satisfy OMB. Well,
obviously, you can\'t have something completely independent or else it

never happens. Denny was identified as the chairman but in fact he is
going to pick the other members. We had discussed with him the need to
think of different types of users of EIA. We certainly indicated to him
that we wouldn\'t want users that let\'s say all have expertise in the
same sector or all are concerned about the modeling or all are concerned
about oil data or all are from the industry side versus the Hill side or
the NGO community. So I think what he\'s been asked to do is look for
other committee members that would come from a variety of different
backgrounds so industry, the Hill, other executive branches, academics,
so there is a wide group of people.

And the financial community uses a lot of our data plus different
sectoral backgrounds. So in each of those areas you could have oil, gas,
coal, plus different interests. Obviously if you cut it up all these
different ways then we need five people from the Hill with different
coal, oil. It doesn\'t work that way. So hopefully pretty broad but,
again, we can\'t have it both ways in that we can\'t hand pick the
members and then say it\'s independent.

MR. BERNSTEIN: No, but you could set guidelines to help the diversity in
the group.

MR. GRUENSPECHT: And we have. I mean, those kinds of issues have been
discussed with him but on the other hand if we\'re going to start
vetoing or saying oh, gee, this person is not good or this person is
good in some sense, I mean, John Paul is too polite to say it but we\'ve
been asked to have a real external review. Often you would go to the
National Academy to do something like this. It would probably cost a lot
more. We know it would cost a lot more, to be fair. We probably will
need down the line to do something like that, build something into our
budget.

Certainly this is a what part, what are we doing. When we get to the how
are we doing it, which is another thing that can be externally reviewed,
I imagine we have a lot of surveys so we have a lot of statisticians on
that phase. So different phase we\'ll do different things but we\'re
trying to get something as independent as if you just cut a check to the
National Academy.

So therefore we can\'t have it both ways and have a lot of involvement
with hand picking the people although we can have the kinds of
discussions that I\'m trying to lay out. Now, you raised the issue. Do
you think there are other dimensions? I mean, we\'re thinking of
modeling data. We\'re thinking of the different sectors of data and
expertise with it. We\'re thinking of the different places that people
are coming from, be it academia, the Hill, the administration. I mean,
you raised the issue. Are there other dimensions that we might be
missing?

MR. BERNSTEIN: You mentioned all the importants. How many people are
going to be on the committee?

MR. GRUENSPECHT: Only five, that\'s all, so if you drew up the entire
matrix of what sector are you interested in and there were five sectors
and where are you coming from and there were five places that you could
be coming from, well, to fill all those boxes in that cube you\'d have
more people on it than this.

MR. BERNSTEIN: Which is going to make it very difficult, because, I
mean, you do have the policy. You have a reflect the policy. You got to
reflect Denny\'s already an academic so you don\'t have to add another
one. You have to have an industry perspective. Usually you have to have
an industry \--

MR. GRUENSPECHT: I mean, the reason of picking Denny, obviously you want
to pick someone who we knew would follow through and do very well. But a
good example, I mean, I guess we could have handed out his vita but in
addition a bunch of things beforehand. He was in the Department of
Energy at various points in the \'70s. He was actually a budget examiner
at OMB for a little period of time. He was the Deputy Director of the
National Coal Association. He\'s the executive director of this MIT
Policy Center and another policy center that deals with global climate
change. So he\'s the executive director of two centers. The idea is
we\'re trying to find people. Well, we can\'t fill in each box. We\'re
hopefully going to get people and he knows that. Given all the boxes
he\'s got to find people who can fill five of these boxes and that\'s
really the trick but we can\'t say independent and then micro \--

MR. BERNSTEIN: The one area to make sure is, and we\'ve discussed this
in this committee once in a while, is the increasing impact that EIA\'s
numbers are having or potentially having on markets and the implications
of that or is that really happening. And so somebody who has a sense of
that would be really important for that.

MR. GRUENSPECHT: Right. No, I think that\'s right and so in a broad way
the answer is the kind of discussion we\'re having is exactly the
discussion we had with him at the same time because it needs to be
really independent we\'re not going to be saying why don\'t you have
this person, why do you have that person. So it\'s a tricky business.
Given that he has all this long time in this area, our hope is that he
really has the ties that will help him corral some of these people in. I
mean, in the end it\'s not going to be the honorarium that make these
people do it. It\'s not going to be enough but some of these people
actually care a lot about EIA and it sounds to me that, like we were
hoping, he\'s going to be very aggressive in putting this together.

In some ways sometimes when you go to the National Academy you might get
a report essentially written by the staff sometimes and you\'re not
really getting these high-level people in. My sense is that he will
actually get something. I mean, it sounds hackneyed but ideally it would
be both cheaper, get it sooner, and have it be better because the level
of people that he would involve in this thing would ultimately be
higher.

MR. DELEY: We\'ve developed a list of 30 or 40 people that do cross.
We\'ll be sharing that with Dr. Ellerman and we have two or three pages
of criteria on helping him make a mix but one of the points that I
wanted to back Howard up on was the idea that these five folks are going
to be operating as a team and we\'ll need to reach a consensus. And so
obviously he\'s going to work with folks that he knows he can develop
that consensus with. This won\'t be an adversarial process. Any other
questions?

MS. KIRKENDALL: Cards up mean that they have something to say.

MR. DELEY: Oh, didn\'t know that.

MR. CLEVELAND: I think Denny is a good choice and I think he\'ll do a
great job.

MR. GRUENSPECHT: Well, it\'s the only pick we really get to make so
it\'s pretty important.

MR. CLEVELAND: In terms of your questions there I have done independent
reviews on a couple of occasions for EIA, one related to the annual
energy outlook and other work related to the oil and gas supply module
in NEMS. And my experience has been mixed and it speaks to the
connection you\'re not wanting to make but which you\'re going to be
forced to deal with. This is how EIA works internally as opposed to what
you should be doing.

And I say that because on the independent reviews that I\'ve been
involved in there have been some very great things that you\'re doing
and overall I\'m a huge fan and user but some real serious problems in
terms of being up to industry standards on how you do science and
econometrics and things like that largely due in my opinion to

long-held relationships with contractors, who were just a little too
cozy.

So we went through these reviews, pointed out some pretty important
problems, the shortcomings that could be addressed, and it remains to be
seen what comes of those. I haven\'t seen in the stuff that I\'ve been
involved in a whole lot of follow-up done in pushing these contractors
to using the types of methods and analyses that should be done.

MR. DELEY: Is there a specific thing you\'d like to see us put in the
background package that would help the committee begin looking at those
kinds of issues?

MR. CLEVELAND: There\'s not a specific document. It\'s a concern about
follow-up.

MS. KIRKENDALL: I think what he wants is for us to make use of the
recommendations we get.

MR. CLEVELAND: I think they\'ll come up with probably something you can
use effectively on the Hill to say you\'re not doing X, Y, and Z, you
should be doing X, Y, and Z, but if you do X, Y, and Z you\'re going to
have to give up A, B, and C. That\'s my guess. And so you\'re going to
need more support financially and otherwise but that still doesn\'t
address a lot of substantive issues about the actual analyses you do.

MR. GRUENSPECHT: This is more the what you do phase because one of the
things when we went to OMB initially we pointed out that we have a lot
of reviews of individual things but they said no, has anyone really
reviewed your whole program so this is an effort to get a review of the
whole program.

MR. CLEVELAND: Part of that review is also going to be are you using the
best science, the best economics, the best statistics, the best
econometrics, and the answer will be yes in some cases and no in other
cases and so it would be nice to know that other than using this as a
tool to try and get more resources for EIA, which it desperately needs,
there is actually going to be improvement made in what you do.

DR. SITTER: Because the answer can be quite different. We can say well,
if you continue to do it the way you\'re doing it then no, don\'t do it.

MR. CLEVELAND: Look at the website. The website issue is an important
one and this might be a point to talk about this. It is the most
important thing in terms of your interface with the public. It will
become increasingly so. I was impressed by percentage of hits that come
from outside of the US, for example, which will continue to grow as
broadband capacity grows outside the US.

And the change that\'s going on now that we looked at is really an
incremental change. You\'re fixing something that\'s fundamentally
broken, which is the website, and it\'s a reflection of, I think, the
internal culture of EIA where everyone has their report for underground
natural gas storage for last week. And that\'s the most important thing
that EIA does.

And so when you go to presenting the integrated whole of what EIA does
at the website it\'s this scattershot approach and what you really need
is a web design team who has some autonomy and decision making authority
about what goes in and there are probably hundreds of thousands of pages
at the website. Every new report just gets slapped on the website as a
new page somewhere.

There are whole different, modern ways of doing dynamic programming with
data so you can update it instantaneously. It\'s just a scattershot
approach which is reflective, I think, of how the EIA does things. So,
again, this really important tool that you have, which is the website,
in part what you do is also connected to how you\'re doing it. So there
are these interconnections, I think, which are going to be unavoidable
to deal with.

MR. DELEY: So that\'s a recommendation for us to look at the way we do
dissemination and the relationship between dissemination and the actual
work?

MR. CLEVELAND: Yes, when you\'re looking at the website to me all these
\--

MR. DELEY: Your website has become a document data repository ad hoc.

MR. CLEVELAND: It\'s not oriented to users. It\'s oriented to you. It\'s
oriented to the inside of the EIA, not to the outside world. If you
really were serious about building a website that was user- focused it
would look a lot better. The changes you\'re making are positive but
they\'re really incremental. I think you\'re going to want to
------------------ yourselves on really revolutionizing what you do in
the website when really you could be doing yourself a big favor in the
long run by making this information easier for all your major
constituencies.

MR. DELEY: Thank you. Sir, did you have a question over here?

MR. WOOD: I\'ve a recommendation. John Wood, EIA. Probably it might even
be appropriate to only have three more guys from Texas rather than five
but the kind of credentials I would like to see them bring to it would
be to be able to muster up real experience in data and analysis on how
fast new technologies can penetrate the market. And we always talk about
that in our long-term and even some in our mid-term models but that talk
is pretty fuzzy and not terribly well supported. And just one very
specific reason why you might want to know, there\'s a lot of discussion
that the world oil supply has already peaked and the EIA\'s position is
that peak just from a resource question is 30 years out.

Well, I don\'t feel like we need much help in assessing the resource but
what we always say is what if that peak did sneak up on us there are
terrible repercussions and some people think it would take 30 years for
another technology to fully penetrate the market and substitute for the
liquid hydrocarbons as we have it now so to be able to have someone
speak with authority and quantify those kinds of questions and perhaps
also have some insight into what type of technologies those might be but
specifically how long it might take them to penetrate the market. What
if we make a decision light water nuclear reactors is the way to go to
replace most of nuclear power and electricity generation in the next 25
years? Is that a feasible thing?

MR. DELEY: So the recommendation is that someone on the committee be
able to look at emerging technologies and their role on the marketplace?

MR. WOOD: Not just look them but how fast can a technology penetrate the
marketplace and then it needs to be under what circumstances. In a war
you can do certain things in two years. Without a war you can do things
in 22 years. That type of thing ought to be addressed.

MR. DELEY: Thank you. Sir.

DR. NEERCHAL: I think I want to support what Cutler was saying. I think
that it\'s not possible to really do this kind of very high level review
without looking at some of the details and, given that, I think the
number of hours you put together there, and these are very busy people,
they will not be able to do that kind of very detailed review. So
that\'s why when you are providing something like this whatever internal
review you have taken you have done for individual surveys, programs,
provide that documentation in addition to this as appendices perhaps or
past reviews, past recommendations.

Usually they have some executive summaries, one or two page thing that
are attached, so they can actually at least browse those documents
quickly and get a feel for some of the past recommendations without
actually doing the review themselves. So I would say that these
subtitles look fine but I think what they say in it will be much more
crucial.

MR. DELEY: And again, to back up the things that Nancy and Brenda said,
Nancy was talking about the fact that we really haven\'t identified what
we mean by program and Brenda was talking about the fact that
documentation doesn\'t always exist. So in some cases if you can\'t list
the programs and you can\'t find the documentation then it\'s going to
be hard to provide that information in a background package.

DR. NEERCHAL: The survey we used that Brenda was talking about, that is
really helping you go towards this. I think whatever findings there you
and I think that\'s very detailed but it will be useful to these people.
They will be able to identify problems spots so experienced so I think
provide more than less.

MR. DELEY: Thank you. Sir.

MR. BERNSTEIN: I have a specific question. Following up on Cutler\'s
statement on the website issue, in your question on communication you
talk about dissemination, that bit. You might actually say in there
well, what about our website or what we get from another website. Be
very specific. The other thing Cutler said is that what they\'re
probably going to do is say okay, we think you should add this stuff and
take away this stuff. But there was nothing in the questions that guide
the committee to thinking about well, if we suggest you should do
additional things then we need to trade that off knowing something about
budget constraints.

I\'ve been involved in some staffing high level committees in the White
House and other places. What happens at these things a lot is they say
okay, you should be doing this and this and this and this, and they
never say well, there\'s no way government is going to pay for all that
so we also have to say you got to get rid of this and this and this to
balance it. So somewhere in the questions or in the guidelines to the
committee you should say think about the trade-offs that we\'re going to
have to make budget-wise.

Too many times I\'ve been in these reports and they say well, we should
do all this stuff and then on the flip side of that on question six it
says how should series and analysis
------------------------------------------------ be changed or
discontinued but there\'s nowhere that I could see what you say well,
what else should we be doing, other things that we\'re not doing that we
should be doing.

MR. DELEY: That ties in with that trade-off thing. Something goes away,
something gets added.

MR. BERNSTEIN: So we\'re structuring the question so that you\'re not
just going to get okay, we need to be doing this and this and this and
well, you can figure about this one thing \--

MR. GRUENSPECHT: And we definitely don\'t want that. I mean, that
doesn\'t help.

MS. KIRKENDALL: The trade-off is

really what we need the advice on.

MR. DELEY: Reword the question to focus on the trade-off issue.

MR. GRUENSPECHT: I\'m relatively new to EIA but my sense is that for a
long period of time we did only add things and never subtracted things
and we just got thinner and thinner and I think the discussions we had
here maybe a year and a half ago, I mean, some of you are new, some of
you are not new. There was an issue of quality. This silly putty, do you
keep on just trying to spread it thinner and thinner or when you have to
pick up something do you drop something else and I think this group the
strong impression I got I think also about six months ago was this
notion of don\'t give up quality, if you have to make choices you\'re
better off making choices rather than giving up quality.

And in fact when we needed to do what we\'ll hear about, I guess, from
John Wood, about this Natural Gas Production Survey, we in fact
explicitly gave something up. We gave up the survey of public power
entities. In that case it had been a long time since I think EIA had
added lots of things but hadn\'t dropped anything.

There\'s this whole notion of thinking of things in terms not everything
can be the same priority and they\'re going to have to have a way of
weighing that because someone else will be making the decision about the
level of resources. So I think just saying we need to do this, this, and
this but somehow assuming that additional resources are going to be
provided when in fact it\'s a very tough environment for additional
resources.

MR. BERNSTEIN: You\'ve got different ways to cut that. You\'ve got the
data job, you\'ve got the analysis job and the forecasting job, but then
you have different things you\'re looking at and at different levels of
depth and so this committee ought to look at this thing as a big matrix
and help you think about strategically what are the most important
element of the matrix that have to be there, what are the elements in
the matrix that you don\'t need.

MR. GRUENSPECHT: So you think we ought to have recommendations where
anyone who recommends adding something has to recommend dropping
something of equal \--

MR. EDMONDS: Just in thinking about the problem that you\'re posing to
Denny and the committee that he forms, you\'re giving him a pretty short
fuse. You\'ve got 100 hours down there for each committee member
exclusive of travel and Denny\'s going to spend 125 over the course of
this period, which is a pretty compressed time line, and one of the
things you could do to help them in that context is you\'ve got a
mission and a set of charters that go all the way back to the original
foundation of this organization. And in fact producing a summary of that
which is what are the things mandated by Congress which is ultimately
where this organization derives.

And then you\'ve got your products and the things that you\'re
producing. And what I see here in this outline is a core dump. Give you
all the legislation that\'s come out and we\'ll give you all these
products and it\'s out there on the table but, given that you want them
to do this pretty darn fast, I think you also need to give them a map.
This is how we read the legislation as identifying the tasks that we are
to undertake and the mission that we\'re to accomplish. And here are the
products that we produce and here\'s the crosswalk between those as we
see it.

We see this response that we\'re making and that could actually help you
a lot too in just going through that crosswalk and it will certainly
help Denny and his committee as they think about how do they give you
advice about have you missed something when they go back and they look
at ------------------------------------ do we see something that\'s out
of line with the way you see it and then as we look at the crosswalk do
we see something that\'s not being done or things that are being done at
a level of detail but probably more than you need to accomplish the goal
that was really established for you. So that\'s my advice.

MR. GRUENSPECHT: So you\'re saying don\'t give them just raw material?

MR. EDMONDS: Right.

MR. GRUENSPECHT: I think that\'s a good idea. I\'m really concerned
about any implication that we\'re trying to pre-process or push them in
a particular direction. First of all my sense is that, knowing Danny,
he\'s not a rubber-stamp kind of guy. I know the level of people that
he\'s looking for but I agree with you. In some way giving them this big
sack of wheat stalks and say go find a wheel someplace and grind it up
and make it all yourself, it\'s a tricky thing.

MR. EDMONDS: The value that I see in putting that crosswalk together is
that it provides them with an additional piece of information which is
how do you think about the problem. They\'re independent and you have a
clear charter when you go back to OMB but in fact what you say is we
told them how we think about this problem and their charter then isn\'t
to construct this whole crosswalk and then ------------------------ the
ideal crosswalk and then do this comparison. Here are your products and
\--

MR. GRUENSPECHT: You\'re saying give them at least a local maximum?

MR. EDMONDS: Exactly, give them a point of departure because Danny can
put himself in the role of what if I were the administrator of the EIA
and how would I look at it because he\'s got a lot of experience and I
think he\'s just a first rate choice as chair and he could put himself
in that position and say how would I do it and what would I do to change
these things.

MR. GRUENSPECHT: Right, so give him a starting material that
------------------------ where you\'re starting from.

MR. EDMONDS: Exactly. How are we thinking about this today and how do we
see this mapping?

MR. DELEY: Thank you very much. We have two more questions and then I
think we want to take a break.

MS. KHANNA: I think there\'s a fundamental question that we haven\'t
asked here which is what is the EIA trying to get out of this review. Is
it to satisfy OMB and say all right, now we\'re
------------------------------------ all your standards or is it really
to use this as a means to actually self-evaluate to see what are we
about and where are we going? Those are two very different things and I
think the

two kinds of comments we saw here actually addressed it from sides of
that same point.

What you said, Cutler, is much more dynamic in a sense in that maybe our
mission is changing, use the website as ------------------------ whereas
what you propose more was static. Here is our mission and here is how we
have been doing it. So I think that\'s a question maybe the EIA really
needs to do a little bit of soul searching. I mean, you\'re going to
spend a lot of time and effort on this.

What\'s the purpose? I mean, we talked about this in a different context
in an earlier meeting when we were talking about how the financial
markets are now using EIA data and we jokingly but half seriously also
talked about maybe asking them to pay for it. If they want the data
before 8:00 a.m. pay for it; 8:01 you get it free. It\'s a whole
different thing. EIA has never done it.

MR. BROOKS: The SEC does it that way.

MS. KHANNA: Right, and it may be a whole different change in style at
work in which it\'s been working or even in what you see is the mission
statement but I think those are important questions at a time when
budgets are getting tighter and more is being expected of you. I think
it\'s really a time to sit and do that kind of soul searching.

MR. DELEY: Thank you. Sir.

DR. SITTER: Now this is to follow up on all the comments but at least in
Canadian institutions we have departmental reviews so that\'s my
reference point. You\'ve got a very formal setup that\'s been worked and
that is we have to do a self-study. We have to give the reviewers a
self-study, which is basically what you\'re asking for, and a five- year
plan. That\'s the exercise we have to go through. We give them a
complete package of what we think of ourselves and where we think we are
going and then they as an independent review picked not by us take a
look at what we think we\'re doing and they come and look and say well,
what do we think you should be doing and how do we think you\'re doing
yourself, realistic or not realistic, should you be doing it that way.

That\'s really what Jae is suggesting and it\'s really the only way you
can do it. How can you go in as an external review to an organization,
have all experiences and so forth, but basically in a few hours you have
to determine stuff that you guys have been thinking about for a long
time and
------------------------------------------------------------------

MR. DELEY: And certainly part of the goal in addition to meeting the OMB
requirement is also to inform the strategic planning process and the
budget processes. Both budget materials and strategic planning materials
will be presented to them so they\'ll have a sense of where some of our
internal committees have ------------------------------

DR. SITTER: Well, my philosophy on external reviews is the same. Even
when our department was being externally reviewed I just throw out the
notion of satisfying the committee or satisfying the university and just
ask really what do we think we should be doing and just be completely
honest about it because it\'s the only way to do that kind of
------------------------------------ I just don\'t see any other way to
do it.

MR. DELEY: Thank you. Anyone else who has a comment, question,
suggestion?

MR. CLEVELAND: Why isn\'t EIA doing a self-study as part of this? For
those of us in academia, which is a different kettle of fish obviously,
that\'s pretty standard. An external review always starts with a
self-study.

DR. NEERCHAL: This is a self-study document. At least it should be.

MR. CLEVELAND: No, this is a list of publications.

DR. NEERCHAL: ------------------------------ legal framework, I think
that those are the points that Jae was talking about.

MR. CLEVELAND: But it\'s not the same thing as a self-study. I mean, it
should be, should be.

MR. BERNSTEIN: No, this is the appendix to the self-study.

MR. GRUENSPECHT: This is the wheat and the thing and they want us to
take it to the mill and bake all of this. I think that\'s fair. I think
in some ways in terms of what we should be doing we do a self-study
every year. It\'s called the budget process, and, again, we address for
better or for worse this issue of trade-off between quality and trying
to do everything where at least on the margin we decided to drop
something, we added something. On the 2006 budget, which is on the Hill
now, we dropped a couple of relatively small things but I think it\'s
fair to say we do it but we do it incrementally starting from \--

MR. CLEVELAND: I have to interrupt but that is way too bureaucratic a
view of it. You have the charge of studying one of the most fundamental
forces in the world economically, environmentally, geo- politically, and
will only become increasingly so. The types of stuff that you mentioned
which I know dominate your existence most of the time are secondary from
my perspective because you should be worrying about are we studying the
right things. Are we collecting the right kind of information that is
going to answer these larger questions?

MR. GRUENSPECHT: I don\'t want to argue. I agree with you but it\'s like
this classic thing of think of EIA as a sailing ship. You can sail the
boat and you can pull the boat out of the water and contemplate should
we change the keel, should we do this, should we do that, but the real
problem that was really hard is to sail the boat and contemplate its
keel at the same time because really we can\'t say gee, we are not going
to put out oil data for the next four weeks because the people who are
putting out the weekly oil data have been pulled aside to do a study of
is the oil data structured. So I hear what you are saying but at some
level we are a little bit like a hamster on a treadmill. I don\'t know
how to describe it any other way because if the stuff that\'s supposed
to come out at 10:30 isn\'t out by 10:31 we know about it.

MR. CLEVELAND: Well, then this is doomed
------------------------------------ why bother?

MR. GRUENSPECHT: First of all I thought this was a very good discussion
but we do sometimes think about these issues at least incrementally but
in terms of deeply thinking, again, a strategic plan would be our
five-year a but strategic plan does tend to be more general. A strategic
plan does not identify we are going to add this, this, and this, and we
are going to drop this, this, and this.

So it\'s not at the level of detail but we do have a long run process,
we do have an incremental process, I think what\'s been suggested here
in several places from Randy, from Jae, that in some sense we might form
our own committee in some sense to give our self-evaluation along the
lines that we are asking these five people to do \--

MR. BERNSTEIN: And you might do it in a small series of facilitated
sessions amongst yourselves, take people off for a few hours or a day. I
mean, not weeks but take different groups out and you have an outside
------------------------------

MR. GRUENSPECHT: We could probably take five people. If this group is
five people we could take five internal people well selected. I mean,
people have a lot of background, have them put in the amounts of time
that we\'re thinking about having this committee put in, and that\'s
certainly doable and give that in some sense to the committee. I think
that is doable but there is this issue of we are torn in some sense
between sailing the ship and making the ship better and you can do both
if you could pull the thing out of the water and spend yourself for a
while but we are not in a position to do that a whole lot and it\'s
tricky.

DR. SITTER: I think the thing to remember, though, is this that the
reason it even works in an academic thing is because it\'s been going on
for a long time. I mean, this is like a first time thing but eventually
you\'ve got like the past external reviews and the first thing you do in
your next external self-study is to look at the past one and see if
things have changed and have you addressed some of the issues that were
raised and you\'ve got this framework in place.

A self-study when it\'s the first time you\'ve ever thought about
yourself in that global way is, of course, daunting but the process
exists and it\'s not as difficult as it sounds.

MS. KIRKENDALL: Actually I think that\'s a good way of looking at it. I
think if we go through with this one and get an external review then the
next one we can do the self-study and look at what we\'ve had from this
and what we\'ve done in the past, and then get into more of the regular
cycle like ------------------------------

DR. SITTER: Then usually it takes knowledge.

MR. BEINSTEIN: That\'s not what Randy is suggesting at all. He\'s just
saying the next self-study will be easier. He didn\'t suggest not do a
self-study.

DR. SITTER: You don\'t have to do a self-study each time but it is much
easier to focus. I mean, there are things that are very stable within
your organization. You\'re quite aware of that. Your opinions on that
haven\'t changed. You\'ve got the things you haven\'t changed and then
you spend more time identifying the things that should change. So
therefore it just becomes a lot easier and you also have some knowledge
about how you look at yourself related to how an external review might
look at you. They are quite different.

Having that from the past history changes the way you look at yourself
too and so it makes the process a lot easier. I remember even as a new
faculty member the first time you are involved it just seems like
totally overwhelming just evaluating yourself and what you think of your
department. I\'ve been through a few of them. You also learn like you
think of yourself. You look at yourself.

DR. HENGARTNER: This self- evaluation is every five years or what is the
schedule? Obviously they want to have it on a regular basis, correct?

MS. KIRKENDALL: Yes, that was my other question. How do we build a
regular program and this the external review looking at the overall EIA
thing needs to be part of it. I\'m not sure you need to do it more than
once every five years. It needs to feed into strategic planning. We need
to use the results and act on them. And I think there need to be other
reviews or other kinds of things that are done in the intermediate years
so I don\'t think this is the only thing but this is the kind of thing
that EIA has never done before.

DR. HENGARTNER: I hear everything everybody is saying and I see a real
dichotomy. On the one side is what is EIA doing right now, let\'s look
from inside out, and that\'s where self-assessment and how it\'s tied to
the budget and what we were doing is important and what Cutler is saying
is let\'s take the outside view, let\'s see where we could go. And
somehow those two forces I don\'t know how to reconcile but I see both
of them having value and I\'m not sure if the purpose of the external
review is like outside guidance and the self-assessment to provide
inside guidance.

DR. SITTER: Well, I think that it\'s very difficult even as an
individual or anything to take outside criticism if you haven\'t really
looked at yourself well.

MR. EDMONDS: I don\'t think it\'s an either/or. It\'s a both. I agree
with Randy. Until you\'ve actually taken a look at yourself and put
yourself in the position that you want the outside reviewer to look at
it\'ll be hard to really appreciate what the outside reviewers are
coming back to you saying, and I think the point is that actually you
are trying to create an environment in which you\'re best able to take
on board information that\'s coming down the pike \--

DR. SITTER: It really ties to that last question, which is what you do
with the review. I think it\'s almost impossible if you haven\'t thought
about it yourself.

MR. EDMONDS: The other point you

made, and maybe I\'ll just make it a little bit more explicitly, was
that this needs to be a regular event and that there\'s a lot of benefit
the first time but the second time you do it not only is it easier but
also you are in a better position to take on board this external
perspective. And I think that five years may actually be too long a
period between these kinds of activities. I would think you don\'t want
to do it every year because then it becomes almost a box checking
exercise. If you were thinking more like every three years that\'s
enough time so that you have an opportunity to do some implementation
but not so long that the experience is almost starting all over again.

DR. HENGARTNER: But, for example, do we need to have the same schedule
for self-evaluation and self-assessment and the external evaluation? For
example, I could imagine doing the self-evaluation every two years and
the external review every four years.

MR. EDMONDS: There are details to work out but I think five years is
probably too long.

MR. CLEVELAND: One more point in the self evaluation, when I look at the
time frame in which you want this done and what you budgeted for it and
the reluctance which is understandable, perhaps, do a self- assessment
it just worries me that you\'ll do this review and say we did that, we
did an outside review. Some of the work, the internal review I\'ve done
for NEMS, I felt that way a little bit like we know there\'s a problem
with this finding weight function in the oil and gas. It\'s a very
controversial thing, how much oil we find when we drill, and there are
some issues here so we are going to look at it, and we looked at it very
carefully and ------------------------------------------ happened.

They could say look, we were objective, we did an outside independent
review, so it just gives a little bit of the impression when I look at
this that it\'s going to be a little too almost superficial, not
superficial. That\'s a wrong word but that will be not taken seriously
enough internally in changing what you do and how you do it.

MR. DELEY: Thank you.

DR. HENGARTNER: Anything else?

MR. DELEY: Thank you all very much.

DR. HENGARTNER: Guy, do you want to respond or \--

MR. CARUSO: No, I think many of those comments are not only well stated
but I think that very much reflect the internal discussions we\'ve had.
The main thing\'s not to have ------------------------------ we\'re not
doing this to just ------------------------------------ Laurie over
there because we decided we could have structured it differently that we
could have checked off the box. But what we said was we want this to
actually be something that we can use to influence our action plan
------------------------------------------ so we\'re very much moving in
the spirit of what your last comment was
------------------------------------------------------

MR. CLEVELAND: I think he\'s a good choice, an excellent choice. If you
give him the right stuff he\'ll do a job so that was a good choice.

MR. GRUENSPECHT: We were lucky to get him. I mean, the issue is the
caliber of people that you want. It\'s hard to get them give you the
time but also a lot of people have a lot of good feelings towards EIA.
So in some sense I think we\'re going to get more than we paid for. I
hope it\'s a little more than we paid for.

MR. CARUSO: And he\'s already gotten two acceptances but he probably
doesn\'t want to make them public yet but the first two are top people
\--

MR. GRUENSPECHT: Very, very high level, a very high level.

DR. HENGARTNER: Before we break I\'d like the members of the audience
who haven\'t introduced themselves yesterday come up to the microphone
for the record.

MR. WEINING: Just give me your name and organization, please.

MR. RASMUSSEN: I\'m Eric Rasmussen with EIA.

MS. POLING: Janice Poling, EIA.

MS. GEISERT: Lynn Geisert, Z, Inc.

MS. FENCH: Carol French, EIA.

MR. YAFFEE: Barry Yaffe, EIA.

MR. BROOKS: Henry Brooks, Z, Inc. and EIA.

MS. ANDERSON: Margot Anderson, EIA.

MS. KRAUSS: Laurie Krauss, OMB.

MR. WOOD: John Wood, EIA.

MS. MILLER: Renee Miller, EIA.

MS. WHITAKER: Margit Whitaker, GAO.

MS. OLSON: Irene Olson, EIA office of oil and gas.

MR. LU: Ruey Pyng Lu, EIA.

MR. VAGTS: Ken Vagts, EIA.

MR. BRADSHER-FREDRICK: Howard Bradsher-Fredrick, EIA.

MR. HILL: Walter Hill, ASA Committee.

DR. HENGARTNER: Let\'s take a break and resume as soon as possible.

(Recess)

DR. HENGARTNER: Ladies and gentlemen, let\'s start and I hope that Jae
and Cutler will join in soon ------------------------------------ Spoken
to our committee several times now and he has set up
------------------------------------------ I would say more natural gas
stories but ------------------------------ different types of
------------------------------ hi, John.

MR. WOOD: Hi. I\'d like to briefly run through the presentation points,
first a brief history of the EIA 914, then a discussion of our response
statistics, data quality issues, problems with the Texas data, and
questions for the ASA committee. This, I think, whole process of the 914
survey went very well and I think we\'ll get to with the discussions
with problems with Texas data that it was clearly a just in time survey
that is ready to go now and it is very good that the data from the 914
survey is available at this time.

EIA had a history of persistent criticism of its natural gas production
data. Most of the time EIA was actually correct within a few percent of
the total natural gas production as verified later by final production.
Usually that final production data was not available for two to three
years and the normal debates would be occurring just a few months after
a production history. Again, there were stories in the trade press
frequently that EIA was low by 8 or 10 or 12 percent and getting worse
and as it turns out we were usually within 2 percent or so with lower 48
production level.

In any case there was a number of things that made the problem even
worse. There was a period, a year and a half, where there was no
production data even available from the Mineral Management Service for
one of the largest producing areas in the country, the offshore Gulf of
Mexico. I think the Natural Gas Division and Ken Vagts, Director of the
Office of Oil and Gas, pushed very hard, I think, to have an EIA survey
where EIA controlled the process, controlled the data. This became if
not EIA\'s highest property certainly in the top two and it certainly
was at the top of the list even if that might have been alphabetical.
The process was directed by a steering committee and that steering
committee made it very clear that it was going to be done well and they
were expected to make sure it got done well.

The objectives were to substantially improve both the quality and time
lines of our monthly natural gas production estimate for key states and
the national level and our target is to have releasable numbers 60 days
after the close of a report month with the sampling area at the national
level within 1 percent and within 1 to 5 percent for six areas, Texas,
the Federal Gulf of Mexico, Oklahoma, Wyoming, New Mexico, and
Louisiana.

The collection data for January 2005, the first survey month, is nearly
complete. No data have been published yet and the EIA plans to release
January and February estimates at the end of May.

In the short history, again in 2004 January we began drafting the OMB
request for clearance for the 914. That chore was handled by Barry Yaffe
and very well and I think Ken and Nancy Kirkendall did the final
negotiation with OMB and that clearance went through very well. Again I
think this whole process in the development of the statistical data
series and methods to approach estimating has gone very well, a number
of offices participating all the time as well as the ASA Committee.

We also are depending more and more on effective contractor-federal
employee teams and Z, Inc., Lynn Geisert in particular, handling those
tasks provided a lot of the manpower and expertise to put the system
into its data system and to help with the survey collection process and
there were many discussions with this ASA Committee and specific members
of it during the development process. We did conduct site visits to nine
companies for cognitive interviews and I think especially the SMG group
got a very good feel for the nature of the data and how the companies
could provide and what they could provide. We got in October 2004 OMB
clearance and in November we selected a cutoff sample of 286 and all
these steps were again discussed with ASA Committee members.

In December of 2004, which really wasn\'t that long ago, we initialized
the steps which is our survey processing system that\'s going to be
utilized for the 914 and the first mail link to operators to alert them
that this survey was coming and what the general procedure would be and
in coming through this I think, again, comments we got from everywhere,
in particular the comment from the industry trade groups. It was
constructive and helpful and in the end we asked for a lot less
information. And, speaking of things we asked for, Guy and Howard did
not give us this much money as we asked for. They actually gave us more
money than we said we had to have and in this particular day and age and
time that was good.

In January 2005 we started calling the operators who had not responded
to the first contact and I might mention in the letter there were ten
points made of how to avoid hearing from us and the first one was to get
your data in on time and the tenth one was to get your data in on time.
And we\'ve done pretty well. By February we had had ten responses and,
of course, only one of them was correct. And by March though we had 240
respondents out of a target of 280 and again just arranging and having
an adequate frame was a significant problem and there were enough
mergers between the cut-off and the actual mailing so that instead of
286 we had 280 companies.

As of April 25th all but two small companies had reported and the
production weighted response rate is better than 99 percent. We\'re
proud of that. This is the response statistics. We\'re tracking two
statistics series and, for example, Nancy Kirkendall and John would
agree that the production weighted response rate, the red line, is the
most important one and the blue diamonds are the actual companies that
have actually reported.

And this is actually rather interesting. There were more of the smaller
companies reporting early and the production weighted response rate was
lagging it. We had talked to all of these operators and they said, the
large ones in particular, that they were going to turn their data in on
the due date, not the day before and not the day after, but on the due
date. So we assumed that they would and they did. And so as you approach
the due date for that data on January production there was a very large
jump in the production weighted number and it crossed the company
response rate curve.

There were no real problems in any area and in particular the Gulf of
Mexico where we had had the most systematic problem with getting data we
have 100 percent response rate. And we are actually sampling companies
that account for more than 98 percent of the production. So this will be
the source of production data for the Gulf of Mexico and it will be
relatively quick and it will be very accurate.

DR. SITTER: Can I ask where on that curve would you have to actually
release data?

MR. WOOD: Back here. Now, we have set the analysis system that makes the
estimates up and, again, I think most of the people in this room have
seen description of how the data behave and the fact that in the
calibration year if you have 90 percent coverage those operators over
the next two years represent a smaller and smaller percentage of the
total production and so if you want to be in a 1 or 2 percent accuracy
range you have to have a relatively sophisticated estimation process.
And we have one but that system now is operating in SAS and we can
change the number of operators we want to use in the estimation process
and turn it all around again in a couple of

minutes. And so we have a lot of flexibility and we have a system to
estimate for non- responding operators. We currently aren\'t going to
have to depend on it too much but we are going to change from a 60-day
to a 40-day system which will be interesting to see what happens to the
response statistics when we do that.

Plotted here are both the January production data, January response
statistics, and the February statistics and as you approach the due date
for this data, which was today, we had a week ago about 65 percent of
the companies responding and the production weighted response rate is 49
percent. Here we\'re going to track the response rate production
weighted of successive months to make sure that we\'re on the same
curve. The blue line is the production weighted lower 48 statistics and
again we got up to as of Monday 99.4 percent. We were up at about 90
percent by the due date and since we can process this data very fast we
can afford to wait and in fact we\'ve been doing successive runs on the
data for January and there are little changes when you get greater
coverage but the basic results you can get like, an 80 percent response
rate.

Now, how do we get the high response rates? We work hard to develop good
working relationships with the operators and the Reserves and Production
Division has a 30-year history of working directly with the personnel
and the producing companies in surveying thousands of the oil and gas
operators. And we found in the crude reserve survey that it really is
incredibly effective and useful to have a personal working relationship
with the people that are going to report the data to you.

They know you understand the data. Often they like to talk to us because
we uncover problems they have with their own data submission. And this
is so far working out as we planned. We tried to make reporting as easy
as possible for the survey of natural gas producers and we call up and
walk them through the process especially the first one, two, three,
four, five times they try to submit the data.

We have assertively and repeatedly called and followed up with the
respondents. The staff that made those calls was trained. Rhonda Green,
who is the technical monitor in running the operations for the survey,
had six people they could call and, remember, there are only 280 so
it\'s a quite manageable thing. Just two did most of the calling.
Periodically we went through other levels of the company to make sure
that people realize that their obvious highest priority was reporting on
the national gas survey.

And then a very strong point is

that the staff that called includes people with a lot of industry
experience that are very familiar with the production data and company
operation and that\'s just always useful. How is the data quality? At
first one could have been worried a little bit. Nine of the first ten
had errors and the tenth one correctly reported all zeroes. A large
number of the respondents made unit errors and this was discussed a lot
before we actually designed the form and in retrospect most of these
errors could probably have been avoided if we had asked the operators to
submit this in the same units that they report to the regulating body
that they have to report to, the Texas Railroad Commission in Texas, the
Mineral Management Service in the Federal Offshore Gulf of Mexico, the
uniformity of data across all states and certain publication logic and
wanting to perhaps show that this is the data reported to us and exactly
what was reported to us and here is the answer influence that some but I
think in the future we probably ought to assume that we can add and
multiply and understand the data and if we get it in one pressure base
we can convert it to a standard pressure base, if we get it in thousands
we can publish it in millions or billions of cubic feet per day. And
especially on this first cycle and the next one and the next one
resubmissions were asked for until the data was correctly submitted by
the operator.

How do the production estimates look? Obviously pretty good except for
Texas and we did the following. There\'s a seven- year history of our
final gross gas production that EIA publishes or estimates and the first
estimates are made in the Natural Gas Monthly. They\'re based on
preliminary state data. At the end of the year the Natural Gas Annual
revises that data and during the last few years there\'s been a special
effort made to revise the final data in subsequent ------------------
past years and to do it again. So some of final data have been revised
several times and in a state like Wyoming the data are clearly accurate
to within 1 percent.

Well, obviously this is what we expect. This is Wyoming. It is a major
producing state and it\'s been a state that the production has been
changing rapidly and going up. The average absolute error in the data
first published in the natural gas monthly was 13 percent over this
period of time. The red square is the January estimate made a couple of
weeks ago and the error bar is drawn at plus or minus 13 percent, which
is the historical average of the first production estimate.

This is the Gulf of Mexico and again the Gulf of Mexico, very dynamic,
again here there were two separate occasions when there was a long gap
where there was no official data available and EIA again in a
presentation the ASA Committee saw started taking the individual well
level data and put together estimates from the partial data and back in
this period of time we used other methods. The hurricanes have a
dramatic effect on production and with all that we still had a 4 percent
average absolute error over this seven-year period on the monthly data
even with the hurricane.

DR. HENGARTNER: If you have final revised data how can you calculate the
error you\'re doing?

MR. WOOD: Because we compare it to the first published estimate, which
was in the natural gas monthly so again this is a direct comparison,
what was the final, final, final number and some of the MMS data was
changing and it\'s still changing. But from 2003 back it looks solid and
we just subtracted the value published or we probably published marketed
but the equivalent gross gas production, which is subtracted from the
final number and divided by the final number and that\'s the percent
we\'re talking about.

And since this is something that I hope is a meaningful bound for
discussing what quality of data you should have again seven years the
average absolute error was 4 percent. This number is probably more
reliable than the estimates we were making with preliminary data.

Oklahoma, the error band was 6 percent and we\'re certainly in the
ballpark here and again a state like Oklahoma periodically it is
fluctuating plus or minus 2 percent 48 months after the production month
so it\'s okay. In Louisiana the absolute average error was 6 percent and
this number we estimated two weeks ago is certainly in the right range.
Again New Mexico, it looks dead on the trend.

Now, we did some general statistics of these first set of estimates and
again they\'ll be revised slightly and revisions to the January date are
coming in steadily with the February data and the operators were asked
only to report if they were off by more than 4 percent but a large
number of the operators is reporting at 1 percent. They just want to
know that they can get it right. And what we estimated was the average
production rate for the fourth quarter of 2004 and we compared it to the
estimate that we\'re making for January and in Texas there was a 1.23
percent difference. I\'ll just tell you the average absolute error was
about 2 percent in Texas and that\'s in this column over here. In the
Gulf of Mexico the average error was about 4 percent and the difference
between the average rate in the fourth quarter and our January estimate
was a negative .8 percent, about 1 percent, in Oklahoma the difference
between January estimate and the fourth quarter 3 percent. Louisiana had
the highest number here, almost 5 percent difference, and then in New
Mexico and Wyoming it was like .2 percent for both of them. Especially
the Wyoming data is pretty good by now to estimate that number. So these
are very close to the trend of the data.

Now, EIA 914 percent difference, the area target for sampling error was
1 to 5 percent. Actually the formal sampling errors are extremely small.
I think in reality errors from all sources will fall easily in this 1 to
5 percent range. Again, you can\'t get too excited here because there\'s
still a significant uncertainty in the fourth quarter estimates and then
I like to see this. Three of the differences were positive and three of
them were negative. In fact when you look at the lower 48 states the
error was quite small.

And then again we\'re looking for things to assess how the areas are
comparing. This is last week\'s effort. The relative comfort zone ratio
where you just divide a difference and then you take that difference and
divide it by the seven-year absolute error percent and, given the
quality of the data in the past, how do these look and near zero is
good, over 1 is bad, and smaller than Texas is pretty good.

This is something that we certainly wouldn\'t have planned to have
happened at this time but it did. Effective February 1, 2005, Texas
changed its production forms and electronic procedures for gathering
data. That meant the January report and revisions to all subsequent data
were impacted.

So the last quarter of 2004 and the first month or two and January look
like they\'ve been impacted by this change in a major way. Texas
reports, and I love the language under a direct request from us, their
current data has more deficiencies and delinquencies. It\'s quite
alliterative.

Now, AIA\'s current process again for Texas grew out of discussions and
reports with the ASA and Randy Sitter\'s student Crystal had a nice,
straightforward procedure and we implemented it and have been using it.
But as all of the similar types of procedures this method is giving very
low estimates right now and it\'s inherent in the process. It\'s because
it depends on a relatively stable time series from the preliminary data
reports and that time series is no longer stable.

And I\'m going to show a couple of graphics, the blue line is the final
data through 2003 with the current method. The orange line is the
Crystal method for 2004 and January 2005. There are three short curves
in here, simple empirical estimates based on month to month changes
given the pattern of 2001, the pattern of 2002, and the pattern of 2003
and remembering that the average absolute error from the preliminary EIA
published number to the final one is about 2 percent.

And you look at this and, again, this is the production history. If we
were to estimate coming off of the June, July, August average and the
pattern of monthly changes for 2003 then our estimate made two weeks ago
and we did it again maybe last Wednesday, is just under that. If we use
the 2001 pattern that\'s a little lower, the 2002 pattern is a little
lower still, and if we used our current standard process then you would
come down to here and this is exceptionally unlikely to have occurred.
The natural gas prices have stayed very high. Usually if there\'s a drop
in production it\'s because there\'s a drop in demand. The drilling for
natural gas is setting all time records. There\'s just no indication
that this behavior would take place either from resource constraints or
demand changes.

Again this is a blow-up of the current period and here\'s the estimate
we made last week using the 914 data. The response rate production
weighted is 98 percent or 99 percent. And then here are three simple
empirical estimates and, by the way, if you go to August, and the reason
we picked that June, July, and August, we\'re pretty confident that
August we probably know within 1 percent, so if you do a one-year
projection that line goes over the 914 estimate. You do a two-year
projection, I think the line comes slightly under it. The six-year
production was even higher. This would appear to be in the right rank.
We overrode this estimate in December and actually estimated a number
about here.

And I think we will look one other time at the source of this, that the
process that we were using before we adopted the Crystal method again
depended on analysis of time series. And again our nomenclature is P1 as
the first preliminary number you get from Texas on a specific month, P2
is the second, P3 is the third, P7 is the seventh. And we take it out
and as you get to P24 you\'re tending to get to the final number. It\'s
getting pretty flat.

As long as this P3 is relatively consistent then you just get a
historical factor that you can multiply that number which is low and you
get a good estimate of the final number but that depends on a stable
time series and you can just see that P7 looks like the data we\'re
confident in and so you go to the final extent of the P3 data and it\'s
looking a little low and you come down to here and this is the P2
estimate available for Texas in January 2005 and it shows an 11 percent
drop. Now, the largest previous drop in seven years in the final data
from that June-July-August average to January the following year is 3
percent.

The percent drop using the Crystal method and the new Texas data series
from June-July-August to January is 10 percent. And then the average
drop from June-July- August to December the same year was about 1
percent. The average drop from June-July- August to January of the
following year is effectively zero for seven years and so this data due
to the change in the underlying data series is just not a reliable
estimator any more.

On the other hand we have a reliable estimator, the 914 estimate, and
sometimes things look better if you have the right perspective. This is
a zero-based plot and this estimate is obviously on trend with the Texas
data.

We have some questions for the ASA Committee. The first one is 100
percent response rate for the Gulf of Mexico good enough. We\'re proud
of these response rates. Will the high response rates
------------------------------------ confidence in the 914-based natural
gas production estimates are good? Is a comparison with the historical
data series and that average absolute error band a good way to evaluate
the EI 914-based estimates? And what additional data could be shown to
increase the confidence in the 914 estimates?

DR. HENGARTNER: Thank you, John.

Questions. Moshe?

MR. FEDER: I just noticed that the last series that looks very reliable
it\'s possibly because the y-axis there starts from zero which shrinks
the other volatility in that series.

MR. WOOD: Well, I\'m not sure which graph. This one?

MR. FEDER: In this graph you start the y-axis from zero and the others
just focus on the time series.

MR. WOOD: At 14 but it\'s how much scale you want to be able to see.

MR. FEDER: I know. All these show it very well but the last slide does
not.

MR. WOOD: No, the last slide was to show that obviously we\'re on trend
and inside the error bounds and we wouldn\'t actually publish that
anywhere but here.

DR. SITTER: Now I\'m assuming that if you hadn\'t done this you probably
still would have looked pretty carefully at that aspect?

MR. WOOD: Our current method is fairly broad in its description and it
includes the option of analyst override. There was I would say a very
careful review of comparisons of using different estimating methods to
get an estimate from historical data series. In the end as with what we
did in the Gulf of Mexico, for example, if you understand the general
driving trends on the data, et cetera, you can estimate reasonably well
where they\'re going to go.

For many years we actually depended mostly on these similar time series
projections. In fact the number we had turned in for December was turned
in about an hour after the steering community review of the 914 project.
Carol was at that meeting. There was some question of what we\'d
actually done. Gary Long said I think maybe we already sent them. And we
sent a number that was 16.1 or -2 in here and it was basically an
average of these last three years trend. And you don\'t absolutely know
how it\'s going to go within a few percent. Within a few percent I think
you do.

This year natural gas storage, which EIA knows quite well now, showed
that we were way ahead of the average remaining storage level. In
January and February there might not have been as much pressure to
produce but on the other hand the natural gas prices stayed up very
high. On the other hand natural gas prices correlate very well with oil
prices and the oil prices were setting records.

So it\'s not absolutely clear. What was clear to us is that the natural
gas productive capacity, the physical ability to produce gas, in Texas
was going up during this period. And so if there was no indication in
the price structure which we know immediately that demand was cratering
and there is no evidence that your resource constraint or production
capacity restraint under no particular reason to expect you to change
and again over a seven-year period were some very dramatic things
happened, like a change from \$2 an MCF to \$9 an MCF, the biggest
change from June-July-August to the following January was 3 percent. And
yes, we would have and in fact we did about 6:30 the night before we
promised we\'d deliver them to the Natural Gas Division we put out the
number 16.1-something and it was \--

DR. SITTER: When you say you\'re going to drop your due date to 40 days
it\'s very difficult to know what your responses are because a lot of
your big companies are just going to hit that deadline
------------------------------------

MR. WOOD: Yes, that is our presumption and if they don\'t even though
I\'m sure they love talking to us we do chat about it, in fact just say
and mean and tell them in the phone calls, et cetera, that we really are
trying to make this as easy as possible. Now, there is only a couple of
numbers you have to give us usually and we\'ll take them practically any
way you can get them here.

And we expect as the learning curve

goes on that people will find this easier and easier to do. The two
operators who have not reported, we\'ve talked to them a lot. One of
them swears their computer system has been down for three months and
we\'re basically offering to fix it for them and there are various basic
English responses that we\'re discussing with them.

The other company says bring the marshal and I\'ll give it to you. We
had a lot of those in the beginning and after we carefully explained how
important a national issue this was that their data be correct and in on
time some of them that were the most vociferous and threatening, I gave
a lot of money to my congressmen and he\'s going to have your ass in
trouble if I have to fill out this report, and that\'s a verbatim and
that guy actually reported correctly and on time.

And so it is nice to talk to them but you have to be kind and gentle and
authoritative and it escalates. And I assume eventually we\'ll bring the
marshal. We know people that know these people and we will try that.

DR. SITTER: So when you have the 40-day deadline when does your estimate
go out?

MR. WOOD: About 20 days after that.

DR. SITTER: Twenty days, sixty days.

MR. WOOD: Of course, this is where the key is going to be. Well, any of
these. Obviously this isn\'t quite good enough.

DR. SITTER: No, but that\'s not realistic.

MR. WOOD: But hey, that\'s the first month and the second month, and
then again in the annual report, the EIA 23 Survey, we moved the
deadline at least a month. It might have been more than that. Six weeks?
And we got the exact same shape curve. It had the same response rate on
the due date as the 15 years before when we had them report two months
later. Kara?

MS. NORMAN: The other thing I was going to say as far as the response
rates are concerned is that the respondents know that the response due
date is going to be moved up so this isn\'t going to be thrown out at
the last minute. They\'ve been aware of it in the beginning.

MR. WOOD: The steering committee and it was primarily the SMG and oil
and gas spent a lot of time thinking about this. And the letters that
went out to them told them that this is the way it\'s going to be and to
expect the change and we\'re already planning how we announce this to
them, et cetera, and our expectation is that most of the data will be in
by the due date. And if it isn\'t then we can make the last round the
day before we publish the data.

DR. SITTER: But you also have a lot more information to adjust for
nonresponse in this case because you know who didn\'t respond. Before
you just had these state estimates. You had absolutely no information
about number of firms that had responded.

MR. WOOD: Oh, you mean when we\'re dealing with Texas, yes, it\'s blind.
And the one that I keep remembering because I kept trying to deal with
it, how could we use the available data historically and get on the
publication time scale that we want, and that would\'ve been using the
P-1 and the P-1 data compared to the final data varied from 5 to 15
percent low. And so the only things that worked, and in fact we\'ve got
one and it runs, that just says you can\'t have a 10 percent change, you
can only have a 2 percent change. And so it just won\'t allow that type
of thing to happen.

On the other hand it\'s unsettling and right now I would expect in the
trade press within a month to see some wailing that the world has ended
and, just like a well know energy spokesman said three years ago that
oh, I hate to say this but Texas production\'s going to decline 15
percent; well, it went up the next year but now he can come back and say
well, I was a little bad on the timing but it just dropped 10 percent in
three months.

DR. HENGARTNER: Yes.

MS. CAMPBELL: Beth Campbell, Natural Gas Division. When you do move the
schedule forward to the ------------------------------ schedule you
probably will have more nonresponse. I expect it to be certainly better
than the 65 percent you have right now but it\'s still somewhere
nonresponsive. Is it your plan to impute for nonresponse on an
individual company basis or is it your plan to impute on some other
approach? What is your plan at that point in time? Imputation is a major
challenge to deal with nonresponse.

MR. WOOD: It\'s always best to answer a question with a question if you
want to think about it. Kara, do you remember what date it is that we go
from 60 to 40 days? When do we change from a 60-day to a 40-day?

MR. VAGTS: It\'s after three months.

MS. NORMAN: It\'s after the first three months. We give them the first
three months in order to get all their systems straightened up.

MR. WOOD: What we have in place right now, again, we have, let\'s say,
90 percent coverage in a state. We are going to take out the top five
operators which typically is the whole first quartile. I mean, it\'s a
very skewed distribution. So the top five operators in Texas have about
29 percent of the production. So then we have maybe 50 percent of the
production represented by the rest of the operators we\'re surveying. If
we have a nonresponse in that group then we have the statistical average
change of everyone else that we\'re going to use to estimate with and we
assume that group changes the same way.

After six months we ran tests on how well we could estimate the
individual response rates and again that was actually presented to the
ASA Committee and even when we had a 10 percent nonresponse rate the
error in 22,000 tests was in the tenth of a percent range. And that was
projecting using six months of data and making a projection of every
individual company. And to do that you have to have six months of data
so we would go to that process in about six months. Until then we use
the statistical behavior of the operators that did report and assume
that the operators that did not report had the same behavior.

MS. CAMPBELL: So essentially it\'s a group imputation for the first six
months and then after that it will be based on the individual history of
the respondent?

MR. WOOD: Correct, and there\'s the possibility if you allowed it to
have major errors occur in the process because of a unit\'s problem. If
you\'re used to reporting in thousands and this report assumes you are
reporting in millions then any operator who actually gets their data
through the edit system with a three order of magnitude error is going
to torque the whole system.

Now, there are edits that usually start before the data even go into our
data collection module because a lot of this comes in on e-mail and the
people managing that process look at it and if it looks erroneous they
don\'t even put it in the data collection module. They call the operator
and say how come your number is three times as large as the number we
expected.

If it gets into the system then there\'s an automatic edit that says are
you within 10 percent of the number we expected. And so far all order of
magnitude data units problems have been captured and would have been
captured in the automated system.

DR. HENGARTNER: Any more questions?

MR. VAGTS: I have a question for the committee. I\'m Ken Vagts from Oil
and Gas. We\'re very interested in particularly questions 3 and 4 here
because part of this is public confidence in the information so if you
have any suggestions, I mean, is 3 a good way to approach this? Do you
have any other suggestions? When we put this out the first time we like
to present some analysis with it, with the first data, but then over a
period of time we want to present additional analyses and so any
suggestions you have for us on questions 3 and 4 would be greatly
appreciated.

MR. WOOD: For example, when we do put it out we will have two months and
the plan would be to release at the end of May and release January and
February at the same time so you can see the trend and there are many
industry analysts that could take two points and be totally confident
that they got the trend nailed probably for the next decade.

But in all seriousness there will be people who say given the trends in
the Texas data unless they read about it first in the paper say don\'t
bother to worry, Texas production data is plummeting. When we had a
natural gas meeting that you chaired that question in fact Howard, I
believe, very pointedly asked me had I actually gotten the report from
Texas lately if I was so sure where the production was going. And it was
one of those times when production preliminary data was dropping much
faster than the actual data ended up dropping and so people look at that
and they\'re going to start worrying about it. It\'s like anything
you\'ve seen today ought to be published at the same time to say why we
think these are very good estimates.

MR. CLEVELAND: Which system is this replacing? Initially, at least, when
you make this new data available people who have been following gas
production will want to know what the difference is and why you made it,
why you made a change.

MR. WOOD: We can present that \--

MR. CLEVELAND: Well, what is it? I don\'t know \--

MR. WOOD: It\'s clearly a matter of public record available on the
website and I\'m sure everyone has read it if they could find it.

MR. CLEVELAND: So what is it? It\'s a new instrument, right?

MR. WOOD: The way it is done now for the major states is to take the
time series analysis over let\'s say a six-year period and say, for
example, after the third preliminary report for Texas you\'ve got 96
percent of the data and divide by .96 and that\'s you final estimate and
then we continue that process. So when we publish the annual for January
you\'ve got like P-22 or something and the expected difference between
the reported number from Texas and the final number is tenths of a
percent and we revise at that time and that is the general process.

On the states that is not done with there are some like Wyoming that now
I think we\'re getting and interpreting the data pretty well and after
three to four months their preliminary data are pretty close to final
and then the rest of them, Beth, you want to comment on some of the
small states?

MS. CAMPBELL: On an annual basis we have about 32 producing states and
right now what we have operating essentially is a survey that we get the
states to contribute their data reports for those states.

MR. CLEVELAND: So you\'re not surveying the operators?

MS. CAMPBELL: No, no, we have no operator survey to date at all and so
as a result the data has lagged. It\'s not even asked for from 120 days
later and many of the states can\'t provide it on 120 days so what we
have to make the data concurrent with the other data series we\'re
trying a 60-day reporting basis. We\'ve essentially been estimating at
the national level only but then you go back to 120 days and you\'ve got
some state report dating, you\'ve got some estimated imputed data
essentially to deal with this, so it\'s a cobbled together report even
at the 120-day level.

What John is speaking of is in fact that for some of the states where
they appear to have data, for instance, Texas, in fact he knows from
having analyzed it closely that what we could pick up there from their
website in fact is not going to be a good measure of their final
estimate. So he has created over time these estimation methodologies,
Texas being a leading state, of course, to try to put together as early
as possible a more credible estimate even though it is different from
what they have put on the website. So there\'s been a lot of analysis of
what the states could provide to us and what basic problems they faced
in putting this together.

We\'ve been pulling this together for years and years and trying to
accommodate to this and so finally we\'ve reached a point where we said
it\'s time to get an operator- based survey. It\'s a fairly skewed
population and we\'ve got to be able to get reasonable estimates
together with a reasonably sized survey so that\'s where we are now.
It\'s just finally coming to the acknowledgement that we think we can do
this better on an operator basis than we can dealing with all the other
methods ------------------------ ------------------ does that answer
your question?

MR. CARUSO: And another point that John\'s alluded to a couple of times,
that there are these public estimates out there from mainly financial
analysts who use 10-Q data and the 10-Q data is only for those companies
that are required by the SEC to report 10-Q data. I don\'t remember. I
think the coverage in that is at least less than 50 percent or maybe 60?

MR. WOOD: They probably represent more than 50 percent of the production
but not only do they behave differently than averages, again, we\'ve
showed in presentations here but there are all these mergers and
acquisitions going on all the time and they say they try to account for
it and may have accounted for it but they obviously usually have not.

MR. CARUSO: So what you typically get is a story that comes out whenever
the 10-Q data is available that says Cutler Cleveland from Merrill-Lynch
says production of natural gas in the US is down 4 percent in the first
quarter of 2005 and EIA is \--

MR. CLEVELAND: Out to lunch.

MR. CARUSO: Wrong again just when our budget\'s going
------------------------------

MR. CLEVELAND: Well, I think in terms of point 4 in terms of building
confidence, I mean, what I just heard builds confidence that you\'re in
essence replacing an older system with one that\'s better and that\'s
operator-based. The question is with the changes and all the mergers and
acquisitions how do you keep up with the changing population of
operators?

MR. WOOD: Because we asked them and one of the things that if someone
turns in a production report and it\'s 40 percent more than they did the
month before the person that calls them in this case for something like
that would be someone who had 25 years of experience with Mobil and
Exxon/ Mobil and Halliburton. Why did it change? And the usual answer is
we acquired another company.

So we also find some company that has gone out of existence or dropped
40 percent or 50 percent or 70 percent. Well, why did that happen?
Because we sold a bunch of these properties. And we always did that on
an annual basis in maintaining the EIA 23 approved reserves database and
frame and again there is nominally about 15,000 operators. You get 90
percent coverage on the production with 208 and 98 in several states and
the Gulf of Mexico.

So we would get direct information. Any time there\'s a significant
change we ask why and, again, in the complimentary reserve survey any
time there\'s a change at the field level, and a large company might
have 300 fields, we ask them why and that\'s actually a standard
question. They submit footnotes that say so. If the footnote doesn\'t
explain it to us then we call and ask and often go visit. If it really
doesn\'t look right you want to explain it to us and periodically
they\'ve done it wrong themselves.

There was a great deal of our people corrected the company\'s
submissions this time. So again the short answer is we track any change
inside of a company that is significant at the company level and ask
them to explain it and explain it now and so we get immediate
------------------------------ again I would say that the one thing that
is a systematic potential problem is we don\'t know yet whether there
will be any biases in reporting the data early, whether the companies
will systematically be low or high, and what we\'ve asked them to do if
they are missing some of their data to estimate it and give us their
best estimate of their company total as opposed to reporting only what
they have on their books. And it looks like for the most part we\'ve
gotten what we asked for and we verified it with the operators and it\'s
about right.

In fact is that a good point to make, obviously, that we\'ve actually
discussed these numbers at the individual company level and they both
appeared to us and the operator to be correct?

MR. CLEVELAND: Still that\'s superior to a firm was using a sample of 50
or 60 percent ------------------------------ Wall Street does it.
They\'re using these Q-forms where they only have X percent of the
operators and you can say that we\'re getting a much large percentage
and talk to ------------------------------------ certainly that will
build confidence in your people\'s estimates of gas production.

MR. WOOD: So just the point that 90 percent or more of the production
represented in the sample \--

MS. KHANNA: That and the fact that you follow up with every company,
that to me is much more important in some way. You can give me 90
percent coverage but I have no idea what the quality of that data are
whereas the fact that you follow up with each company whenever you see a
slightly out of the ordinary change is important for me to know. I
believe you did a lot more once I know that.

MR. WOOD: That\'s good because that\'s the point that we would like to
make and that will be done.

MS. KHANNA: I can say this from experience because I\'ve worked with
some EPA data which is collected at the plant level and they don\'t
follow up and you never know why it\'s changing so much at the company
level until you spend hours yourself digging back.

DR. HENGARTNER: Any more questions or can we wrap it up? Okay, thank you
very much, gentlemen, for a very nice discussion.

Although we had a break scheduled we\'re slightly running late and I\'d
like if possible to actually add an item to the agenda; namely, the
people who had this interesting web experiment done yesterday would like
to provide us feedback or get a way ------------------------------------
are they here actually?

MS. KIRKENDALL: I don\'t see them. Is she going to come back and talk to
them?

DR. HENGARTNER: Well, I thought you told me they wanted to \--

MS. KIRKENDALL: Oh, no, no, it was Cutler who wanted to say something
but I think he already got it out in the last session.

DR. HENGARTNER: Then we can go on to the next item. In this case I\'d
like us to summarize the two break-out sessions that we just had.
Nagaraj is still missing? Well, Randy, are you ready?

DR. SITTER: Ready for what?

DR. HENGARTNER: The response rate and the kind of challenges, Randy
Sitter, Discussant, ASA.

DR. SITTER: Yes, I\'m ready. This session was talking about the new EIA
914 monthly natural gas production survey which is just being
implemented.

MR. CLEVELAND: Aren\'t we missing some of our members?

DR. SITTER: Oh, you want to wait?

MR. CLEVELAND: Are we summarizing it for them?

DR. HENGARTNER: They came in and then they left?

DR. SITTER: Well, there\'s a break.

MR. CLEVELAND: Right, they\'re on breaks.

MR. GRUENSPECHT: Yeah, what kind of response rate do we need before we
go forward?

MR. CLEVELAND: So you\'ve been getting a lot of heat for your natural
gas numbers?

MR. GRUENSPECHT: I\'ll put it this way. If you have a number where
people live or die whether it went up 1 or down 1 percent and it\'s like
your number\'s plus or minus 3 percent it\'s not too helpful. Or it may
have gone up or it may have gone down. We can\'t really tell you. And
the other thing is the timing; 120 days after the report month is the
best we\'re able to do using the same stuff.

And ideally when this gets down to a 40-day report then 20 days after
that would be 60 days so we\'d be able to go for 120 days, after the end
of the report month 60 days, so it\'s a combination of being able to get
more accurate data and to be able to report it much quicker. But
there\'s just very little data in the natural gas area. People go nuts
on this weekly storage number, which is a big deal. And then other than
that like, if during the winter you\'re saying basically well, we\'ll
tell you how things went production-wise in January in early June it
doesn\'t really help them so much because natural gas the fact that we
couldn\'t even really put a sign on it given our own --------------- two
years ago, I guess was 2003, there was a \-- really drew a lot of gas
out of storage and people were going very crazy about this whole issue
of is it going up, is it going down.

MR. CLEVELAND: What\'s the time lag with the oil?

MR. GRUENSPECHT: Well, in a way we care less about domestic, I mean,
some ways.

MR. CLEVELAND: So what\'s the reporting? So we\'re down to 40 days?
Well, it\'s actually 60 days then for gas? What is it for oil
production?

MR. GRUENSPECHT: Oil, well, about the same but we actually
--------------------------------- maybe one option is one could think of
a parallel type of arrangement for oil but again we had to give
something up to do this.

MR. CLEVELAND: No, this seems to be excellent improvement so I\'m not
criticizing. I\'m just curious.

MR. WOOD: But, see, the same problems exist that existed or do exist
with our gas data since we depend on the states or MMS for the data and
in any crisis mode like Hurricane Ivan MMS actually depended on us for
baseline production and we talked just about every day in a crisis
period.

Obviously Ms. Howard brilliantly suggested that we did something like
the 914 for ------------------------------------------------ but the oil
data comes in a little faster. Even like in Texas it reaches a given
percentage coverage, preliminary data, earlier than
------------------------------------

MR. CLEVELAND: This stuff is done by paper?

MR. WOOD: We either have them e-mail to us, phone it to us, or we
------------ ------------------------ websites the world changes and
take it straight off their website.

MR. CLEVELAND: Any thought of doing an electronic submission where
they\'d go to the EIA website and get a form right there and do it?

MR. WOOD: At some point in time you just see on that Texas data right
now if you don\'t have someone that\'s in the estimation process
directly to discuss what the numbers mean, the source of the numbers,
you are going to encounter huge problems. And Randy might have wondered
why we weren\'t going to use his system. Well, because all of the
assumptions that underlay its validity no longer hold and we know that.
An increase in delinquency is a deficiency.

DR. HENGARTNER: Okay, now that Nagaraj is here we can continue. He was
the last one to walk in. I\'d like Randy to summarize the session on
natural gas we just had.

DR. SITTER: Well, there\'s been a lot of talk in this committee about
natural gas production and estimating it over my time here and certainly
I have been involved. In the past EIA\'s really depended on state
published numbers of just gross natural gas production. They never went
to the operators and collected it themselves and there\'s quite bit of
lag in the reporting of that so one month after a production month you
might have

quite a low percentage of what would eventually be the final production
for that state as your preliminary estimate and that would increase
monthly as they reported each month and even three or four months after
a production month it still might not be that close to the front number
in some cases.

So there was a lot of problems with this in trying to estimate, say,
after one month or two months what the final number would be based on
the historical time series comparing first and second month preliminary
reports or third month to what the final estimate might be from each
state. And both John and myself and a student worked using the inherent
stability of the time series over the previous nine months or twelve
months or six months depending on what you would like to try and
basically boost up the preliminary reports in some reasonable fashion to
get a better estimate of what the final report might be so that your
preliminary reports might in principle hopefully be released.

So in any case the decision was made to introduce the EIA 914, which is
a monthly survey of the actual gas producers, which certainly is a
better thing to do if you can do it, and this presentation was on the
implementation of that over the last number of months. So this would be
a monthly survey of the natural gas producers rated to producer level
and the focus of the discussion was on the response rate since that was,
of course, the big problem and the response rates through a lot of
effort and contact with the producers themselves and follow-ups seemed
to be quite impressive and it looks very good at this time.

They were given 60 days after the production month to report but that\'s
going to be moved back to 40 days and hopefully the response rates will
be maintained but even if they were to deteriorate somewhat having the
producer level data will give you much better ability to adjust for
nonresponse.

Coincidentally Texas is the biggest producing state and it changed its
collection process early in 2005 and that change seems to have impacted
their time series of what they would have released if you were depending
on their data now tremendously and therefore any estimates based on
preliminary data that would have given by Texas would have been
dramatically impacted using either the methods that we suggested or the
methods that John was using before that, no doubt, because they were all
based on a table time series which obviously took a hit because of this.

So coincidentally 914 is able to give an alternative estimate based on
going to the producers which seems much more likely to be true and no
doubt will turn out to be. And so some questions were asked at the ASA
committee, I think four. One is is 100 percent response rate for the
Gulf of Mexico good enough? Generally the ASA Committee, though they
didn\'t specifically say so, seemed to be fairly satisfied with that.

Will the high response rate inspire public confidence in the EIA 914
based natural gas production estimate? I think that\'s certainly true.
It\'s not the only thing that would inspire confidence. Even if the
response rates weren\'t as high as they are, which is like 99 percent or
98 percent, which is very good, but even if, I mean, the fact that the
data quality, the actual numbers that are responding are followed up,
was commented on this that the quality of the responses you do get will
be better if you could let them know just how much effort is being put
in to make sure that numbers you are getting from the respondents are in
fact better will also inspire confidence.

And, of course, having the producer level data will allow you to adjust
for non- response. Certainly you\'ll have a lot more data to work with
and you should be able to do a much better job even if those response
rates can\'t be maintained.

The other one, is comparison with historical data series and the average
absolute errors associated with the current methodology a good way to
evaluate the EIA 914-based estimates, certainly I think that most of us
would agree that there is nothing wrong with that. That is, look at what
the estimates are then eventually compare that to the long range series
which is what they did and it seems sensible.

And what additional data could be shown to increase confidence in the
EIA 914 estimates? And I think that the previous comment on making known
just how much effort is being put into keeping the response rates high
and the quality of the data actually coming in on a producer by producer
level, certainly will inspire confidence and so I think that\'s \--

MR. CLEVELAND: Anyone who lacks confidence we can send them over to
John\'s office for a pep talk.

DR. HENGARTNER: And to man the phone banks. Excellent. Thank you very
much, Randy. I\'d also like to have a report from the break-up session
from Nagaraj.

DR. NEERCHAL: For our break-out session Nancy reported on an ongoing
project. It is something that Nancy and Joe Sedransk are involved in and
this is a collaborative project between CNEAF and SMG. This was actually
something that they reported on last meeting also.

So this is like a statistician\'s dream in a way. It\'s a beautiful data
set and I think that\'s the thing Nancy and Joe are doing, playing with
it, so I think that\'s fun, right? Basically the setup is like this.
There are two surveys. The surveys deal with regulated and unregulated
companies that sell electric power to end users. Essentially both
surveys deal with the same population roughly. One survey, 826, is a
sample survey undertaken every month. It\'s a monthly survey of a
sample; 861, the other survey, is an annual census of all companies so
that\'s the scenario you have. One survey is collecting data annually on
everybody, another survey is collecting monthly data on a subset, and
the idea is that if we can somehow learn from the monthly data and come
up with the full data set on a monthly basis so you want to estimate the
reported number of, for example, state level sales for the nonsampled
companies so it\'s a pretty straightforward stated problem. Maybe
that\'s all that\'s straightforward about it.

So the idea, they are doing a regression model, so they use a regression
model on the seen part of the data, the sample part of the data, and
extend that regression over to the unseen part and the regression is not
very straightforward. That is one of the complications. I think it\'s
regression with changing heteroscadastic variance and how you estimate
the heteroscadasticity. There are many ways to do that and so they have
some approaches. I think they are happy with the kind of approaches they
are using for doing that so essentially there we have a regression model
to do that.

So to see how this works they decided that they are going to run an
experiment on our annual data so they looked at 2002 and 2003 so you
have two data sets, one data for all the companies for 2002, another set
of data for all the companies for 2003, and they looked at the sampled
companies only and did a regression and extrapolate it by that
heteroscadastic regression to the unseen part and now since you have for
this play data set the actual values you can compare and see.

And one of the questions that is asked of the committee is exactly how
do you compare predicted versus observed. They are trying different
tuning parameters, which model is doing better. In the write-up I think
you have in your stack, they list four or five different measures. One
of the questions that was asked of the committee was to see which one do
you like the best and so on, and I think on this one the committee did
have a concrete recommendation. They felt that the coefficient of
variation thing, per person prediction error expressed as a percentage
of some sort of a mean seemed to capture the feature they are looking at
specifically in the outlier factor and things like that. That seemed to
capture that more so. That is the one the committee liked the best and I
think for reporting purposes the group there felt that it\'s also useful
to provide another overall error of the model, an overall plus error, so
we chose two columns from this, something to pare down the presentation
also, to eliminate some of the things that seem to be saying the same
thing.

I guess that\'s the only concrete recommendation we had, right?

MS. KIRKENDALL: We supported of the simulation study.

DR. NEERCHAL: Yeah, we supported the simulation study; however, we felt
that the final objective has something to do with the monthly data so
learning how to do stuff with the annual data may not quite translate
because monthly data has a lot more wrinkles to it, a lot more features
to it, so we recommended it but they already started doing this idea of
cross-validation. So look at the data set you already have, the sample
data set. Set aside some for learning, get the model and evaluate it on
the set aside value.

So the cross validation is probably more informative than playing with
the annual data. Annual may be good for stabilizing your estimation
techniques and things like that to learn what is happening to the
estimation techniques but for the purposes of evaluating how good
you\'re doing, how bad you\'re doing, probably you should focus only on
monthly data because that seemed to help.

And there\'s a related issue, I think. Like I said, there is a
regression model. The slope, for example, can depend on the season you
are looking at or the region. If you divide the data set into several
regions you might have different behavior within regions and so on.

So there is a grouping issue. Now, how do you stratify this whole
population in a way that is useful for your prediction purpose, and I
think that is something that Nancy has spent a good bit of time on and
did you give this other handout to the committee? No? They were really
beautiful.

MS. KIRKENDALL: I have them here if anybody \--

DR. NEERCHAL: I think a good statistician can always do better than a
good statistical method. I think this is a real proof for it. I think
they first tried a clustering technique and I think Nancy didn\'t like
it but then she looked at the data. She has looked at the data for a
long time, I bet, and came up with a very intuitive way of grouping
these profiles, if you like.

For example, take two years of data. You really have eight quarters of
data. So for every company you really have a profile of eight quarters
so you have sales for a specific dimension like residential sales. So
essentially you have an 8 by 1 vector for every company so any time you
aggregate you have a 8 by 1 vector.

So she specifically looked at the February data. That seemed to be a
very key number because it\'s the winter sales and so she sorted things
and grouped them according to the February data and then went further
and looked at the difference between the summer versus winter peaks and
grouped them accordingly. So she came up with a grouping and you can see
that it picks up the grouping. Whatever she has done really picks up the
profiles, the groups, the states, especially the state level data,
groups the profile very well.

This is not easy to do. Those who have done with the clustering thing,
you take any one of those matrix, they are not doing what they tell you
they are doing. This seems to be doing and I don\'t think committee had
any suggestions. I\'m sorry we were out of our depth on that one, I
think, didn\'t have any kind of matrix that will capture this kind of
grouping and that will be very useful in the long run because John and
Nancy are thinking about grouping not only by the region, maybe moving
states from one group to another or even within a state a particular
company might be servicing an area that is straddling two states or
close to it, boundary and so on, so that company may better belong in
the neighboring state rather than the state where it is located.

So lot of issues and I think that\'s a challenging issue. I think I will
stop by saying that it\'s a challenging issue. I think that overall
it\'s a good project and the committee likes the project, definitely,
and I think we gave you useful information in terms of instead of
looking at five or six things focus on one or two matrices for your
model comparison purposes.

And the simulation study, I think, maybe the committee\'s
recommendations focus on the monthly data. Do whatever you can with the
monthly data. That\'s better than doing fantastic things with your
annual data.

Any suggestions on how best to package the results of this study to
convince managers to implement changes, I think you should throw it to
everybody open here because that\'s the case with any study.

MS. KIRKENDALL: Actually it occurred to me that since they were in the
audience that wasn\'t a bad way to do it, just have a presentation at
ASA and then invite them to come.

DR. HENGARTNER: Anybody else wants to add to this very thorough
discussion?

Well, at this stage I\'d like to open the mic to the general public. If
anybody wants to discuss any issue including issues you want to bring
forth among the committee this is the time. We have to schedule the free
for all. Anybody, anyone?

DR. SITTER: Let\'s talk about scheduling.

DR. HENGARTNER: Talk about scheduling, very good, yes. First before I
continue let me welcome Walter Hill to the committee. He unfortunately
missed yesterday\'s session but welcome. Just for your information,
there is a free lunch this session. It\'s an internal joke. Forget it.

Beyond that I also would like to announce that thanks to Bill Weinig\'s
efforts there is already a date set for the next meeting; however, we
will not announce it until he clears it with his boss so you should be
receiving in an e-mail shortly some official dates that agree with
everybody\'s choices. So thank you very much, Bill, for taking that away
from our lunchtime discussion. Anything else?

Well, it seems that we are really in luck. We\'re 15 minutes ahead of
schedule and it\'s my pleasure to adjourn this session until next fall.

(Whereupon, at 11:45 a.m., the

PROCEEDINGS were adjourned.)

\* \* \* \* \*