AMERICAN STATISTICAL ASSOCIATION

\+ + + + +

COMMITTEE ON ENERGY STATISTICS

\+ + + + +

FALL MEETING

\+ + + + +

THURSDAY,

OCTOBER 16, 2003

The Committee met in Room 8E089 in the

Forrestal Building, 1000 Independence Avenue, S.W.,

Washington, D.C., at 8:30 a.m., Jay Breidt, Chair,

presiding.

[PRESENT]{.underline}:

F. JAY BREIDT Chair

NICOLAS HENGARTNER Vice Chair

JOHNNY BLAIR Committee Member

MARK BURTON Committee Member

JAE EDMONDS Committee Member

MOSHE FEDER Committee Member

NEHA KHANNA Committee Member

NAGARAJ K. NEERCHAL Committee Member

POLLY PHIPPS Committee Member

[ALSO PRESENT]{.underline}:

GUY CARUSO Administrator, EIA

HOWARD GRUENSPECHT Deputy Administrator, EIA

NANCY J. KIRKENDALL Designated Federal Official

CALVIN KENT Invited Guest

BILL WEINIG EIA

COLLEEN BLESSING EIA

SHAWNA WAUGH

JOHN WOOD

A G E N D A

OPEN THE MEETING

Jay Breidt, ASA Committee Chair 5

EIA ADMINISTRATOR\'S REMARKS: WHAT\'S UP AT EIA?

Guy F. Caruso, Administrator, EIA 11

PRESENTATIONS OF APPRECIATION TO JAMES K. HAMMITT, Ph.D. and POLLY A.
PHIPPS, Ph.D., Retiring Committee Members 20

EIA RESPONSES TO COMMITTEE ADVICE SINCE THE SPRING MEETING

Nancy Kirkendall, Director, SMG 20

EIA\'S STRATEGIC PLANNING AND PERFORMANCE GOALS FOR 2004-2008:
INTRODUCTION TO THE STRATEGIC PLAN: OVERVIEW

Guy F. Caruso, EIA Administrator 27

OVERVIEW OF GOAL 1

Nancy Kirkendall, Director

Statistics and Methods Group, EIA 44

QUANTIFYING ANALYTICAL QUALITY THROUGH ASSESSMENTS BY

INDEPENDENT EXPERT REVIEWERS

Bill Weinig, SMG, EIA

ASA Discussion: General Membership 66

BREAK 92

SURVEY ASSESSMENT: WHERE WE ARE, WHERE WE ARE GOING, and THE MISSING
PIECES

Nancy Kirkendall, Tome Broene 92

FRAMES

Nancy Kirkendall, Howard Bradsher-Frederick,

SMG, EIA 107

CUSTOMER SURVEYS AND OUTREACH ACTIVITIES

Nancy Kirkendall, SMG

Colleen Blessing, National Energy Information

Center, EIA 120

STEWARDSHIP

Howard Gruenspecht, Deputy Administrator,

EIA 144

PUBLIC COMMENTS AND QUESTIONS 163

LUNCH 163

DATA INITIATIVES IN NATURAL GAS

Beth Campbell, Office of Oil and Gas, EIA 163

BREAKOUT SESSIONS: CRITERIA TO SELECT AND

IMPLEMENT ESTIMATION PROCEDURES:

A COMPARISON OF TEXAS PRODUCTION

METHODOLOGIES 172

SUMMARY OF ASA ADVICE BY ASA LEADS

CRITERIA TO SELECT AND IMPLEMENT ESTIMATION

PROCEDURES

Nagaraj K. Neerchal 220

INDUSTRIAL NATURAL GAS PRICES: AN ALTERNATIVE

APPROACH

Jae Edmonds 225

SECOND BREAK-OUT SESSION: NATURAL GAS PRODUCTION

ESTIMATION IN THE GULF OF MEXICO 231

SUMMARY OF ASA ADVICE BY ASA LEADS

NATURAL GAS PRODUCTION ESTIMATION IN THE GULF OF

MEXICO

Nicolas Hengartner 283

RESIDENTIAL AND COMMERCIAL PRICES FOR NATURAL

GAS DATA

Polly A. Phipps 285

PUBLIC COMMENTS AND QUESTIONS 291

MEETING CLOSE 291

P-R-O-C-E-E-D-I-N-G-S

(8:38 a.m.)

MR. BREIDT:  Welcome to the Fall Meeting of the American Statistical
Association\'s Committee on Energy Statistics. This meeting is being
held under the provision of the Federal Advisory Committee Act. This is
an ASA, not an EIA Committee, which periodically provides advice to EIA.
The meeting is open to the public, and public comments are welcome. Time
will be set aside for comments at the end of each morning in the
afternoon session. Written comments are welcome and may be sent to
either ASA or EIA.

All attendees, including guests and EIA employees, should sign the
register in the hall, and include their email addresses. Rest rooms are
at the end of the hall toward the back of this room. Fountains in the
same corridor on the way. Telephones in this room share a single number,
which is (202) 586-3071.

Tara Stull with the ASA Meetings Department is here and available for
Committee Member questions on expense reimbursements, and she\'s always
very helpful.

In commenting, each participant is asked to speak toward a microphone.
The transcriber will appreciate it. And in particular, Committee Members
at the head table need to speak clearly and into a microphone. They say
that the microphones are reasonably sensitive, so you shouldn\'t have to
lean too far in, see how that goes.

Speakers, there is a lapel microphone, and there\'s also this microphone
up here. Bill Weinig can help you out with the lapel microphone if you
need it. So now we\'re going to go around and introduce ourselves. I\'d
like you to ?? that will include everyone in the audience, as well as
everyone at the head table. We\'ll need your name and your affiliation,
so I\'ll start out.

My names is Jay Breidt. I\'m a Professor of Statistics at Colorado State
University.

MR. McDOWNEY:  Preston McDowney, EIA Statistics.

MR. BURTON:  I\'m Mark Burton. I\'m an economist from Marshall
University.

MR. HAMMITT:  I\'m Jim Hammitt, and among other things now, Director of
the Harvard Center for Risk Analysis.

MR. EDMONDS:  Jae Edmonds, Chief Scientist, Pacific Northwest National
Laboratory.

MS. KHANNA:  Neha Khanna, State University of New York at Binghampton.

MS. PHIPPS:  Polly Phipps. I\'m with Washington State Institute for
Public Policy, Research Analyst.

MR. CARUSO:  Guy Caruso, Administrator of the EIA.

MS. KIRKENDALL:  Nancy Kirkendall, Statistics and Methods Group, EIA.

MR. WEINIG:  Bill Weinig, Statistics and Methods Group, EIA.

MR. BREIDT:  Thanks. Now back to the audience, and please use the
microphone.

MR. TAYLOR:  Mike Taylor, University of Michigan, and summer intern at
Statistics and Methods Group, EIA.

MS. MILLER:  Renee Miller, EIA.

MR. BRADSHER-FREDERICK:  Howard Bradsher-Frederick, SMG, EIA.

MR. CARROLL:  Joe Carroll, Bloomberg News.

MS. WAUGH:  Shawna Waugh, EIA.

MR. COLE:  Stacey Cole, Bureau of Census.

MR. HOUGH:  Rich Hough, Bureau of Census.

MR. TSENG:  Phillip Tseng, EIA.

MS. HOLTE:  Susan Holte, EIA.

MS. BLESSING:  Colleen Blessing, EIA.

MR. RUTCHIK:  Bob Rutchik, Statistics and Methods Group, EIA.

MR. MILLER:  Herb Miller, EIA.

MR. FREEDMAN:  Stan Freedman, EIA.

MS. JENNINGS:  Aletha Jennings, EIA.

MR. SMITH:  Antoine Smith, EIA.

MS. CAVANAUGH:  Kathleen Cavanaugh, EIA.

MR. SEDRANSK:  Joe Sedransk, EIA and CWRU.

MR. YAFFE:  Barry Yaffe, EIA.

MR. BREIDT:  Great. Thanks very much. Okay. It\'s now time for the
highlight of the meeting, which is the following announcement. For your
information, Nancy J. Kirkendall is the Designated Federal Officer for
the Advisory Committee. In this capacity, Dr. Kirkendall may chair, but
must attend each meeting, and she is authorized to adjourn the meeting
if she determines this to be in the public interest. She must approve
all meetings of the Advisory Committee, and every agenda. Also, she may
designate a substitute in her absence.

Okay. We\'ve got an interesting agenda today. It\'s a fairly full
agenda. We have multiple break-out sessions. We\'re going to be pretty
busy. We\'ll try to keep on schedule. Of course, we\'re already off
schedule, but hopefully we can make up some time down the road.

The first section of this morning\'s meeting will begin with a briefing
by EIA\'s Administrator, Guy Caruso, followed by presentation to two
Committee Members, Polly Phipps and Jim Hammitt are retiring from the
Committee. Then Nancy Kirkendall, the Director of SMG, will comment on
EIA work since the Spring Meeting. And after Nancy\'s comments, we\'ll
begin with the morning session on EIA\'s Strategic Planning and
Performance Goals for 2004 through 2008.

Lunch for the Committee and Invited Guests will be in the usual place on
the first floor at 12:30. And this evening, we have a reservation for
dinner at the Lee Garden in Washington\'s China Town. It has a varied
memo, and we\'ll take a look at that during lunch. Bill claims that
it\'s quite nice. It sounds really nice, so it is at 629 H Street, N.W.
We can take Yellow or the Green Line to China Town or take a cab, so
we\'ll probably meet in the hotel lobby at 5:45 because our reservations
are at 6. So can I have a show of hands for Committee Members who will
be attending the dinner tonight?

Okay. And tomorrow morning, there will be breakfast for the Committee
beginning at about 8, so we can meet at about 7:45 to walk over. And our
meeting will start up at 8:30 tomorrow morning.

So just a couple of other announcements and then I\'ll turn things over.
First is, Cal Kent has accepted a promotion at Marshall University.
He\'s now Vice President for Technology Commercialization, and he will
be here this afternoon. Tomorrow he will be a Discussant for Jay
Casselberry\'s stuff on CIPSEA. We expect two new members for the Spring
2004 Meeting, Barbara Forsythe from West Stat, and Susan Sereka from the
University of Pittsburgh.

And then finally, the usual announcement about discussions. If you have
something to say, if you could turn your name tent vertically to
indicate that you have something to say, that would be helpful. Okay. So
I guess that\'s it as far as announcements and logistical kinds of
things, so it\'s my pleasure to recognize Guy Caruso, Administrator of
EIA, for his presentation.

MR. CARUSO:  Thank you, Jay. And good morning, Members of the Committee,
and fellow attendees to the Fall Meeting of the ASA Advisory Committee
once again. I\'d like to thank Jay, and Polly, and Jim, who will be
leaving, but we\'ll recognize them later. But I wanted to thank all of
you for coming here and the advice we\'ve gotten from you over the past
years has been extremely useful. And I know this will be my, I guess,
third meeting as Administrator, and I continue to value the information
that comes from these meetings. And so I wanted to just take a few
minutes to briefly bring you up-to-date on where EIA has been devoting
its efforts, and where we think we will be heading in the coming months,
but it\'s been an extremely busy six months since we last met, as usual
driven often by events beyond our control, which is the ?? just before
we met last time, we were just coming out of the Iraqi war. And then
after we met, there was some continuing problems in Nigeria which
affected the oil market, and the events of last summer with respect to
natural gas. So these are some of the things that have driven sort of
the ad hoc work of EIA over the last six months, which is, of course, in
addition to the core work we\'ve done over the years, and that you are
very familiar with.

Clearly, one of the issues that we worked on was gasoline prices. And,
in fact, there was a big run up in California gasoline prices last
winter, and we got asked by Congressman Ose to do a service report on
that, which we delivered about two weeks ago. There was testimony
involved, so again, the hot button issue of gasoline prices continues to
be an important driver for some of the work we do on oil and in the oil
division.

One of the factors involved in gasoline price run-ups is the elimination
of MTBE as an oxygenate in California, and in coming days, in New York,
Connecticut and other states, so we\'ve worked on that issue, as well.

During the winter, we had a spike in natural gas prices, a severe
drawn-down of natural gas inventories leading to not only Congressional
hearings and the Secretary making natural gas a major initiative of his,
leading to a ?? what we call the Natural Gas Summit in June. And I think
the relevant issue there for EIA is the upgrading of data quality and
natural gas, which we\'ve talked about with you and this Committee. And
this is not only ongoing, but accelerated as a result of the winter\'s
experience, and the Secretary\'s initiative, and I\'ll mention a bit in
the budgetary aspect of that, as well, so we\'re working to improve the
?? not only the data quality, but the amount of data and timeliness we
collect on natural gas directly.

As Beth Campbell has talked to you previously, we rely on state data for
production, and we\'re now looking at other ways of collecting that more
timely and directly by EIA.

LNG has become a focus in that overall natural gas analysis, and we\'re
going to be collecting ?? we have gotten ?? we have a [Federal
Register]{.underline} notice out about a new forum for collecting LNG
storage, and that is well underway. And we expect that will be part of
this initiative, as well.

Again, the other event that was ?? that you\'re very familiar with was
the blackout this past summer, and EIA, of course, played an important
role in that in terms of supporting the emergency operations within the
Department, the Office of Energy Assurance and the ongoing investigation
headed by Jimmy Gladfelty in the Secretary\'s Office. So EIA is, of
course, playing an important role there.

These are just a few of the outcomes that those events have led to us at
Congressional hearings. We had the great honor to be part of the same
hearing that Chairman Greenspan, the Federal Reserve. And when he
decided that natural gas issue was important for our economy, I think a
lot of people listened, as opposed to many of us have been saying that
for some time, but that really gave it a lot of impetus. And as I said,
we had a summit in conjunction with the National Petroleum Council. The
National Petroleum Council has issued a new report on the outlook for
natural gas supply and demand over the next 20 years, and we worked
mainly as an observer to that study.

I mentioned the MTBE removal from gasoline, continues to be an important
issue, and is one of the issues that\'s being debated this very day in
the Energy Bill before the Senate and Congress Conference Committee.

Regional interest continues to focus on heating fuels, natural gas,
heating oil and propane, and we are frequently briefing the Members of
Congress and their staffs. We most recently, just last week, did that
for the Northeast and Midwest Coalition, and we released our winter
fuels outlook as part of that process.

Again, I\'ve mentioned in the past how we\'re being asked to do a lot
more with respect to analytical work directly related to legislation and
policy issues. Some examples are the work we\'ve done for Congressman
Tauzin on the preparation of the Energy Policy Act, which is currently
in Conference Committee. Senator Dorgan also asked us from the Senate
side to look at what the different provisions of that bill would mean to
supply, demand, and price. And on the, I would say the environment and
climate change side of the business, continue to be asked to work on
that by Congressmen and Senators. And most recently, we did two service
reports from Senator Inhofe on what we call the Three Pollution
Analysis. And then the McCain/Lieberman Bill, which was a comprehensive
look at the carbon restrictions, as well as other proposals that they
have in their bill.

The renewable portfolio standard analysis, another important component
of the debate in the Senate and House this year, as well as other issues
related to renewal energy. So as we\'ve talked in the past, EIA keeps
being asked to do even more and more in the area of analysis related to
both legislation and policy.

Jay Casselberry, later on this meeting, will talk about the new
confidentiality law which was passed last year, CIPSEA. We are now
implementing that, and Jay will go into more detail. But at least
initially, we\'re going to put 10 of our surveys under CIPSEA for 2004.
And as you know, we have about 55 surveys, so that Jay will explain the
decision making, the process we went through to come up with that, the
use \-- only some of our surveys under CIPSEA, and others continue
outside of that, continue to be heavily in demand in terms of EIA\'s
products that are on the website. More than 1 million hits a month, and
I must say that everywhere I go, it never fails that someone comes up to
me and says oh, yeah, I know who you are. I use your website all the
time, and please tell all your people how much we appreciate the work
that\'s on that website. So it\'s clearly become one of the most
important products that EIA does, is the dissemination of material
through the website.

Finally, where we are on the budget, the ?? we\'re under continuing
resolution for fiscal year 2004, but the House and Senate Interior
Appropriations Committees are meeting in conference this week to try to
complete the Interior Bill, under which EIA\'s budget is. And the House
mark for EIA for 2004 is \$82 million, 82.1, and the Senate is \$80
million, so we\'re hoping that the House version prevails. And if it
does, a substantial part of that \$2 million increment that we would get
would be used to fund the Natural Gas Data Initiative that I mentioned,
and that we\'ve talked about in this Committee, to perhaps develop a
production survey or some other methods to better collect natural gas
data.

This is the story we always say, we use that we\'re on a ?? in real
terms we\'ve been on a downward trend in terms of our budget; yet, being
asked to do more and more, so we\'re certainly hoping that the
initiatives that are underway would turn that line in the other
direction.

With that I\'d be happy to answer any questions you have, sort of what
EIA has been doing, or where you think the Committee would be
interested.

MR. BREIDT:  Any questions from the Committee?

MR. CARUSO:  Okay. I think we\'re going to go into a little ?? present
some awards of appreciation to our departing Committee Members. Polly
and Jim, would you mind coming up. What we have here are two things \--
letters from Secretary Abraham, expressing his appreciation on our
behalf for the work that Polly Phipps and Jim Hammitt have done over the
number of years. I know six years for Polly. How long ?? about the same.
Nancy has the Certificates of Appreciation from EIA.

MS. KIRKENDALL:  Yes. Signed by Guy and myself.

Okay. Well, I\'d like to add my welcome to Committee Members and
Visitors. We do have a full agenda today. And one of the things I wanted
to remind you is that what we\'re trying to do is to present projects to
you early in their cycle so that we can actually get advice before
we\'ve made decisions. And that has good things and bad things about
trying to do that. We might actually be able to take your advice and
implement, so you can have an impact on what we do.

However, the bad thing is that it means you\'re going to get things a
little on the late side, so my apologies to that. I think we finally got
most of the information to you last week. On Friday at lunch, you can
give me your opinion on how well it works, and if that was ?? it\'s
really difficult to get things more timely. Bill spends a lot of time
trying to get the papers together and on the web.

I wanted to mention, we have a few new faces in EIA ?? well, we\'ve got
lots of new faces in EIA, but I will mention the ones on the Statistics
and Methods Group, mostly because you\'ll probably be seeing them quite
a bit. Joe Sedransk is a professor at Case Western University, and he\'s
on sabbatical this year. We were fortunate to get him to join us as an
expert consultant. He\'ll be working with us about half-time.

He was on the Energy Committee in the late 80s and early 90s. He was on
the committee for six years, and was the chair for three years, so that
gave him a wonderful background, so that he can come in and work with us
very quickly. So having his statistical expertise available to us
through the next year will be invaluable.

We also have Philip Tseng. He is an OR analyst. He\'s got a Ph.D. Philip
is here \-- a Ph.D. in Economics. He worked in EIA many years ago on
modeling and analytical issues. Then he circulated around the
department. He was in the policy office. He was in the Office of Energy
Efficiency and Renewable Energy. One of the interesting things he worked
on that we struggle with a lot is measures under the Government
Performance and Results Act, how do you measure the outcome of
information collections. And I think we\'ve talked to the Committee
about that. I don\'t think we\'ve ever come up with really good ways of
doing that, so he may have some good insights on that.

We also have ?? Kara Norman is a New Math Stat. She\'s starting her
Master\'s at George Washington. We\'ve already got her fully engaged.
She\'s leading one of the discussion in a break-out session this
afternoon, so at least half of you will get to meet Kara this afternoon.
She\'s going to be talking about how we officially made the decision
within EIA to use Crystal Linkletter\'s method for estimating natural
gas production in the State of Texas.

And then last but not least is Mike Taylor. He was an intern with us
last summer, and we lured him to come back because he started a nice
little analysis project with the industrial natural gas price data, so
he and Ruey-Pyng Lu will be talking about that also later today.

So an update on your recommendations from last year, last meeting \--
not quite last year yet. On the Survey Quality Initiative, you
recommended developing an assessment tool, letting survey managers
establish targets for measures, having a process that recognizing a
flexible process, basically. You also wanted us to establish good
documentation. I\'m not sure we\'re there yet, but we do know that it\'s
important.

Later today you\'ll be hearing about where we are on that project. It
has been rolled into the Strategic Plan. We haven\'t made quite as much
progress as we\'d like on it, but we have drafted an assessment tool,
and we\'ll be talking about that later.

Jim Joosten talked about the quality activities in the office of Coal,
Nuclear, Electricity and Alternative Fuels, and he\'s continuing with
that. He\'s developing guidelines. He\'s trying to come up with
guidelines that will merge into the overall EIA assessment process. We
may ask him to come back at a future meeting.

You did recommend that we not proceed looking for ISO 9000
Certification, and the office has decided not to do that; although, they
will make use of whatever good ideas they can steal from the process.

We talked to you about performance-based survey contracting, and we
didn\'t have any questions for you, but Tom Murphy is going to ?? what
we\'ve decided to do on that project is to have a little team draft
statements of work that would describe how you would run a contract
using performance-based contracting metrics. This would be for running a
survey, where you have a regular process, you have forms coming in. You
process the forms, you clean the data, you get the reports out. That
process you should be able to come up with standard language for
running. And we do have several surveys that we contract out to process,
so we ought to be able to do a good job with that.

We talked about CIPSEA, and that\'s the Confidentiality and Information
Protection Act, and we\'ll be talking about that tomorrow, as Guy
mentioned. We talked about natural gas production, Crystal\'s method in
a break-out session, and you recommended that we adopt it. We are going
to be talking about the process we went through to do that, and this
time, the purpose of the discussion is to get you to get your input on
the process that we used to make the decision. But in the meantime,
it\'ll also tell you the things we looked at, and how we convinced
ourselves that we had to take your advice.

On the transmission study, Doug talked about what he was doing last
time, Doug Hale. With the blackout in August, of course, there\'s a lot
more interest in transmission data, so you\'ll have another briefing on
transmission also tomorrow.

Redesign of the 906 Power Plant Report \-- the Committee wasn\'t sure
that it was a good idea to get the combined heat and power plants to
estimate the breakdown the fuel used to produce electricity, had some
reservations. But in our cognitive testing, they discovered that
companies either have the data and are willing to provide it; and that
is, the data on the fuel use to generate electricity, so that\'s one
segment. Or they think they can make pretty good estimates, or they
haven\'t got a clue. So we decided to go ahead and ?? but they can\'t,
many of them can\'t estimate, you know, useful thermal output either, so
the approach we decided to take was to add a column asking them to
estimate the fuel used to generate electricity. And I would encourage
you to talk to Stan Freedman, Bob Rutchik, and Stan Kaplan about the
details of the cognitive testing on that.

For industrial natural gas prices, you\'re going to be receiving an
update in the break-out session by Ruey-Pyng Lu and Mike Taylor. And
this one, I guess at the last session we talked about data analysis.
This time they\'ve gone into modeling, so we\'re taking a slightly
different approach. And that is my update on the last time. Does anybody
have any questions, comments? Okay.

MR. BREIDT:  According to the agenda, it\'s back to Guy.

MS. KIRKENDALL: Back to Guy, that\'s right. We\'re alternating.

MR. CARUSO:  Thank you, Jay. And thanks, Nancy, for that update. Since
we met last time, we further developed the draft Strategic Plan that I
mentioned that we were working on, and have been for, think it\'s a good
nine months, more than nine months. And what I said at our last meeting
was that what we wanted to do once the plan was in good shape, to get
your feedback on that. And so I know you\'ve now had a chance to ??
it\'s been sent out in draft form, and so we wanted to just introduce
that, and we\'ll go into more detail on the specific Goal 1 after this,
when Nancy comes back.

We\'ve posted it, the draft, on our website, on our internal website for
our EIA employees to review and to get feedback on it. And that\'s been
about a week, I think, and so far, we\'re very ?? we\'ve gotten some
very interesting comments. And I know we\'ll be working with those
comments, with your comments, with the comments from the Union, the
NTEU. And we\'ve picked some selected customers who are intimately
involved with EIA and our products, and are asking for their feedback at
this point.

Once we\'ve had a chance to absorb that and the Committee will meet
again, I\'d say in about two weeks. We have ?? the way the process in
which we are ?? have developed this, is we are doing a facilitation of
our Strategic Plan in-house with a small team of people. And I think
almost every one of the team is here today, Barry Affee, Kathy
Cavanaugh, Paul Stoller, Colleen Blessing, Tom Broene, and John
Stanley-Miller, so that clearly, if you have any questions or comments,
the people are in the room this morning. And, of course, Nancy has
really taken charge of Goal 1, and she\'ll go into more detail on that
in a few minutes.

I don\'t think I have to belabor what EIA\'s mission is. You\'re
intimately familiar with what we do, and why we do it. But clearly,
it\'s an important part of the strategic goal to focus on that mission,
and to tie those ?? the strategic goals to that mission. And within the
Department, the overall departmental Strategic Plan is also in the
process of being updated and finalized, and that\'s been out for
comment. So clearly, there is a movement afoot within the federal
government to tie budgetary issues to strategic goals and performance,
so that this probably has become even more important under this
administration. But we think that it\'s important as a guiding plan for
EIA, as we go forward.

And I think the ?? you know, the mandate that we face hasn\'t changed in
its core, but the things EIA is being asked to do, the scope, and as I
mentioned in the opening remarks, the more policy analysis, more
legislative analysis, that has changed. And I think it makes it even
more important that the surveys we conduct, the data quality issue that
we work with you so closely on, have become even more important. And the
analytical work is being used more than ever. So all that having been
said, we also have as an important part of our strategic plan, the
recognition that EIA was formed to be an independent statistical and
analytical arm of the department, and we continue to, of course, value
that and guard that very carefully. And that\'s an important part of
this Strategic Plan, as well.

I think our customers are pretty clear, but as I mentioned earlier, the
use of the website, the electronic dissemination of our products has
vastly expanded the first customer group. Public, I think, use of EIA
has just mushroomed in the last 10 years, and I know that was one of Jay
Hakes\' ?? one of his goals was to make EIA more accessible, make EIA
more visible. And I think we\'ve succeeded, or you all have succeeded,
the EIA has succeeded beyond even Jay\'s wildest dreams.

But in one sense, it also is extremely important in the things that we
do in this group, and the advice you give us; and that is, the quality
and the need to continue to maintain that high level of quality, and
timeliness, because of the pressure that we get from all of these
customers to ?? someone said how formal a review group do you have? And
I said, well, we have, of course, the American Statistical Association
Energy Committee, and the other aspect of that is the peer ?? in effect,
peer pressure from this group of customers that are always ?? we\'re
always getting feedback on a daily basis to question why we\'re doing
things the way we\'re doing them. And so, in a way, this feedback from
our customers, not only on the Strategic Plan, as I mentioned, but on
the things we do on a daily basis, is a vital part of our thinking that
went into developing these goals.

Clearly, things have changed since EIA was formed in 1977, and I think
the first one I mentioned was how much more publicly visible, and use
our work is, particularly on our website. But fundamental to the issues
that we work with you on, are the changing structure of the industry,
the changing nature of the way our company, the companies that supply
the data to us, that do business makes it, in some cases, much more
difficult in terms of respondent relations.

The one example that I talked about earlier on natural gas is that the
natural gas industry has changed dramatically, even in the last 15
years. And our respondents have changed, so we need to recognize that in
terms of the data quality aspect. And I was asked at a recent talk
whether respondent, I guess, cooperation has gotten better or worse in
my time here. And it\'s awfully difficult to answer that question
because, you know, in some cases it\'s better, in some cases it\'s
worse. But one of the things that the changing structure of the industry
has done has, I think, made some ?? clearly made many of our respondents
much more cost conscious, and they\'re aware of the burden. And there\'s
a lot of resistance in some areas to filling out some of these forms,
and so we do have to ?? I think respondent feedback has become a far
more important part of that, and that comes into play in achieving the
goals that we\'re going to talk about.

The other one was the ?? in Goal 2, where we talk about adequate
funding. And I showed you that declining real budget picture that we\'ve
been facing here for a number of years, at the same time trying to deal
with increasing requests for our reports, our analysis, and improvement
in data quality. So what I\'d like to focus on, and this, of course,
isn\'t so much for this group, but in terms of adequate funding, is
somehow to increase the understanding of the linkage between what our
products of EIA do, and what our data is used for, and the funding,
because I think there\'s an under-appreciation of that, and that, I
guess, would be part of how we try to deal with Goal 2 in terms of
adequate funding.

And there\'s always pressure in terms of tightening up the way we
manage, no matter which administration there is. And we\'re now, I
guess, in sixth or seventh administration since EIA has been formed. And
there\'s always new initiatives, but ?? and they all are designed to
improve the efficiency with which we use our federal money. And clearly,
EIA, I think has adhered to that, and we will continue to do that in
terms of the funding aspect of this, which is related to Goal 2.

And then the need to not only work harder, but smarter, has been fairly
obvious to me, even in one year here. And EIA has a tremendous staff and
dedicated group of people here, and I think I recognized that, of
course, coming in. And this is clearly recognized in the strategic goal.
We need to keep staff motivated and keep them happy in the work that
they\'re doing, feeling good about what they do, because it\'s so
important to the customer group that I showed earlier on. And we get, as
I mentioned, feedback every day on that.

So we\'ve got these three strategic goals, and the idea is to try to
keep it as straightforward and simple as possible. And more importantly,
not only to develop these goals in a way that we\'re going about right
now, but also to ensure that there is follow-up, that we don\'t just ??
I mean, we\'ve been doing strategic plans in the department and in EIA
well before my time. And I think the feedback I get, both
department-wide and from EIA, is that what we need to do is make sure
that there\'s an action plan that follows the strategic goals, and that
there\'s follow-up. And in particular, that the leadership of EIA, and I
know Howard Gruenspecht and I are committed to doing that. And these are
the three goals we will be focusing on.

I know for this group, Goal 1 is what we want to really hone in on, and
talk about. But I think it\'s important that if you have any feedback on
the others, we\'d certainly welcome that as well, as we develop the
final plan.

I don\'t have to go into any detail about these, but the first one is
the one we want to, as I mentioned, really spend a lot of time on with
you. And we recognize the changing nature of not only the industry, but
the business that we do. And so, therefore, maintaining data quality and
maintaining the reliability and consistency that we ?? of our products
is extremely important, and we would value any comments you would have
on those.

We need a resource base to accomplish that, and our ?? as I say, our
most important asset, the EIA staff, has to be motivated and continue to
?? well, what we say is rate themselves high in productivity. I think
our experience ?? my experience just in one year here is that EIA
employees are extremely motivated and productive, and we want to
maintain that, and even improve it.

This is the one that we will go into a bit more detail on. We want to
maintain the relevant and reliable information program, keep our product
quality high that we\'ve talked about with you on so many occasions.
Performance-based organization \-- this is, as I mentioned, the linkage
between budget and performance is one of the key initiatives of the
President\'s Management Initiative, and we\'ll focus on that in the
coming budgetary cycle, in the fiscal \`05 budgetary cycle.

And clearly, we need to continue to involve the customers and the
stakeholders in maintaining our quality. I mentioned that we have ?? our
most important review process is our peers, not only the ASA Committee,
but others in the community, in the Economics OR Statistical community,
and others. So we\'re doing that on a regular basis, but we need to
continue to do that, and maybe strengthen that.

And we are also going to institute a more systematic agency-wide process
in reviewing all of our products. We\'re looking at one of the issues,
what the management books say is the stovepiping problem. We want to try
to do more to look at products on a cross-cutting basis, and to perhaps
stimulate more movement across offices among personnel, as well.

This, again, just repeats what I showed you in the opening remarks, is
that we need to make sure that the appreciation for the product that EIA
does, the products that EIA produces, are recognized in the need for
budgetary ?? linking the budget to that kind of performance, which I
think EIA would fare very well if we could get that message to those who
are responsible for our budgets. And we\'ll be working on that as part
of this Strategic Plan.

Finally, as I mentioned, the need to keep EIA employees not only
motivated, but recognition \-- provide recognition to them for the
highly productive and, I think, high quality work that\'s being done
here. And I ?? certainly, we\'ll be working on strategies to do that.
And as I mentioned, the feedback we\'ll be getting from the employees
will certainly ?? I\'m sure most of it will relate to Goal 3. So our
follow-up will be to ?? we\'re going to have some further discussion
here in this meeting on the analytical work being done by independent
energy expert reviewers. Bill Weinig is in charge of that.

We\'re doing an extensive survey assessment in Nancy\'s office, the
Statistical Methods Group, and Nancy and Tom Broene are taking the lead
on that. As well, Nancy is also leading a review of the frames with
Howard Bradsher-Frederick. And we have a number of customer surveys and
outreach activities, and this again relates to this linkage of what we
do, and the feedback from our customers, which I think is critical to
keep us current. And Howard Gruenspecht, the Deputy of EIA, is going to
take the main lead from the front office on following up.

As I mentioned, it\'s important that we don\'t just have a Strategic
Plan, and spend a lot of time of our own valuable advisors like you all,
and then just leave it on the shelf, and so we\'re committed to a
follow-up program and action plans to implement these goals. And as I
mentioned, in some ways, we may not have the luxury of even putting it
on the shelf because of the need to make that linkage between our goals,
and our performance, and our budget. So I think this ?? hopefully, this
Strategic Plan will be part of that process.

We have a number of questions for the Committee that we\'d like to
discuss this morning. They\'re listed here. I don\'t know if I need to
read them, but clearly, this could frame the discussion that we have
today on the ?? on your views of the Committee ?? of the Strategic Plan.
Does everyone have these questions, Nancy, do you know? I\'m not sure.
Well, just in case you don\'t, I\'ll certainly make ?? we\'ll make sure
that they\'re available. But clearly, what reaction would this Committee
have for EIA to provide you with all the information we have on the
quality of a particular survey or group of surveys and sending it to the
Committee for review. And does the Committee have any other ideas for
either their involvement, or involvement of other stakeholders? And it
mentioned the importance of the feedback in terms of the things we do,
such as NEMS, and the Winter Fields Conference. And then anything ?? any
other further ideas or areas where the Committee would suggest that we
could use to further evaluate and improve our quality which, of course,
is our main focus, to prepare the quality profiles.

With that, Jay, we could either take questions now or we could go into
more focus on Goal 1, which I think is really more germane to this
group.

MR. BREIDT:  Could I ask just one quick question?

MR. CARUSO:  Oh, sure.

MR. BREIDT:  You collect these ?? first of all, you produce high quality
data products, and you get it out in the world, and it has a big impact.
It shows up all over the news, and our Congressmen are referring to it
all the time. And then you collect these measures of the quality of the
products, and then how does that get transferred to your actual budget
proposal. How does it go ?? how do you propose a budget, and how do you
?? because \$2 million is not even a drop in the bucket. It\'s, you
know, a molecule in the bucket and it\'s ?? to all of us in this room,
I\'m sure it seems give you the guys the money. It\'s well worth it, but
why ?? where\'s the disconnect? Where ?? you have all this information
here, and now you have a budget that you need to justify. How does that
actually happen?

MR. CARUSO:  Well, we have to go through the internal budget process
within DOE, and we submit on an activity-by-activity basis how we\'re
going to spend all of that \$80 million. And again, we do our best. In
my view, I think we did a very good job of justifying and explaining how
we use that money, and what the products are. And as I ?? the problem is
that, of course, it\'s a zero sum gain. We owe ?? as a department, we
have a fixed amount of money, and everybody is competing for that, so
that\'s the process we go through.

And then, of course, OMB ?? from the department it goes to OMB, then
ultimately to Congress, so there\'s a lot of opportunities there for,
you know, shaving it. So we have our wish list, and we think, you know,
when we submit a budget, it\'s obviously ?? our proposal is that we want
to ?? we think we could justify doing a lot more and we ask for more.
And, in fact, last year we submitted a budget, and the budget that\'s
currently being debated in the House and Senate Conference Committee was
about \$84 million. And as is pointed out, we actually sent up a request
for 80, with the incremental 4 we are taking out of monies that we had,
in effect, had de-obligated from previous years. So we think we\'re
running, at least this year, for FY 04, we\'re going to be working on
getting products out that cost \$84 million for an \$80 million budget,
because of the ?? and you can\'t keep that up forever. That\'s, you
know, kind of a one time only, because of the pressures to do more on
natural gas and other things that came up on an ad hoc basis. We\'ve had
to, in effect, do that for this particular year. I think ?? that\'s
really what I think Goal 2 is about, somehow making that linkage with
the products and the amount of money.

Anyone else have any questions on general Strategic Plan and the way we
went about it? As I mentioned, the facilitators are all here. They could
certainly respond, as well. But other than that, we can move on to more
detail on Goal 1 that Nancy is going to present. Thank you.

MS. KIRKENDALL:  I draw your attention to Tab 2 in your notebook,
that\'s the detail of the Goal 1 Action Plan. There\'s lots of stuff in
there. And, of course, you\'ve seen this before. It was on one of Guy\'s
slides. It just says, you know, our goal is that we\'re going to have
relevant, reliable data. It\'s going to be consistent with industry
structures, and we\'re going to have high quality, timely information,
of course. We\'re trying to do that.

Go on to the ?? oh, I can do that, can\'t I? If I can figure out ??
okay. On the activities, Guy also went over these. We\'re going to
become an activity-based ?? a performance-based organization. We\'ve had
performance measures for a number of years. We haven\'t always been
quite as good at using them for management, and so that\'s going to be
something we have to learn how to do. And that is one of the activities
in the goal.

One of the things that we thought we would try to do is to get this
systematic agency-wide process together. Always before, things were done
sort of negotiations between the Deputy Administrator and the Office
Director, instead of getting everybody together to make decisions
together. So we are actually trying to change, or as part of
implementing this, we\'ll try to change how we actually manage. And then
the third activity will be to involve customers and stakeholders more
systematically.

All right. This little chart is ?? I went too far. Okay. This is
basically to tell you what I\'m going to talk about. First, we\'ll give
you the little input/output chart. It\'s on page 3, Tab 2. You probably
won\'t be able to read it on the screen, but it talks about the flow.

The next thing will be the Table of Measures that you definitely won\'t
be able to read, so I\'ll definitely refer you to the book. And I\'ll
talk a little bit about the measures that have targets. These are
measures we\'ve been collecting for a number of years, and so I\'ll just
go over those fairly quickly.

The measures without targets are new, and so that\'s sort of going to be
a point key to the rest of the discussion this morning, so I\'ll go over
those, and give you a mapping of how they\'re listed in that table, to
how we\'re going to talk about them this morning.

This is a little input/output chart. You get your resources data, we
process it. We put it out. We can measure outputs pretty well. We can
measure intermediate outputs, those that are listed. We\'ve got web
hits, media citations, information requests, things that indicate that
people are interested in our information.

We still ?? along with other statistical agencies, we aren\'t very good
at actually measuring mission outcomes. How do you measure the outcome
of an information program? And so that\'s something that we haven\'t
really addressed. And I actually think that\'s key to your question \--
how do you influence the department and make sure that they understand
that we\'re important? So we\'re really not quite there yet. You know,
we can do the intermediate stuff, and that\'s what we\'re going to talk
about this morning, is our plans to do the intermediate stuff, so
that\'s kind of an area for further work. And if you have some ideas on
how do you let people know how important you are \-- why is information
important? What impact does your data have? You can do it with little
stories, other systematic measures, so that\'s sort of an open question.

Here is our unreadable chart. You\'ll notice that this is a Goal 1
element, the big box down the side. We\'ve got relevancy and reliability
first, and then some goals under that. The first three website visits,
number of information products requested by policy makers and media
citations. Those are measures we\'ve been collecting through the years.
We have pretty good data. We\'re going to continue to monitor it. If
you\'ve got questions, you can talk to Tom Broene. He knows a lot about
all of those things.

The last one is a new one, percent of independent expert reviewers
giving a rating to our analysis products, so that one is new, and we\'ll
talk about that more.

The second item is consistency of our program with industry structures.
The reason why that\'s important to us is that two of our major energy
industries have been changing over the years. Natural gas, with
deregulation the structure has changed completely. We now have to survey
people that we didn\'t used to have to survey. And the same is happening
with electric power, so there\'s huge impacts on our data programs, and
we need to make sure that we\'re looking at those, and capturing them
properly.

Quality of information products \-- you\'ll notice that we talk about
surveys that meet output performance targets, and that ties back to our
survey assessment program. What we\'re trying to do is to get surveys to
have targets for improvement. And it\'s because not all surveys have the
same problems and the same challenges, so we want them to establish
their own goals for improvement, and then work towards those.

Key variables in short term energy outlook \-- again, we get our
independent expert reviewers, customer satisfaction. Timeliness measures
are fairly easy, but we\'re going to move towards asking each product to
say what their schedule is, and then measure how well they adhere to the
schedule. That way you can say, you know, my publication is coming out
on such and such a date, and you can decide whether you made it or not.

I think that some of those will be easy to do. We just haven\'t done
things that way before. And then the last one is customer and
stakeholder involvement in the planning process. And this is sort of
what do you count as one of the big questions? We have here the number
of formal customer stakeholder events, so we have two meetings with the
ASA Committee every year. Is that a major stakeholder event? Well,
maybe. You know, it\'s not the only one. And, of course, by itself it
wouldn\'t be enough, but it\'s certainly a way we get input.

And now I think I\'m going to talk about the next slide while this thing
is still up here. For the rest of the morning, I\'ve organized ?? you
know, these are all sort of all organized around the statement and the
goal that we want for this relevant, reliable consistency with industry
structures, quality, timeliness, and customer and stakeholder input. So
what I\'m going to do is to map that to how we\'re going to collect the
data.

So, for example, we want to know about what the independent expert
reviewers tell us. We have a rating, an analysis of where programs are
relevant and reliable. They show up here again with rating that is
consistent with industry structures. And down here, is high quality. All
right. So Bill Weinig is going to talk about all of those things,
because they all have to do with getting some information from our
independent expert reviewers, so that\'s just the detail in the
questionnaire.

Similarly, things that will come out of our survey assessment program
would be the survey frames with the coverage. Actually, that might be
independent, but we have meeting performance targets, meeting timeliness
targets and so on. I\'ll go to the next slide.

All right. So while our table talks about what to measure, the rest of
the session we\'re going to talk about how to measure it. And the topics
really parallel the schedule for this morning, and we\'ve got the IER
reviewers. That will be Bill Weinig \-- surveys meeting output
performance and timeliness targets, that\'s our Survey Assessment
Program. And you actually heard about that last time.

Survey Frames, we\'ll have a quick talk, because we haven\'t done very
much with that. Customer Satisfaction \-- that\'s not me. It\'s Colleen
Blessing. She\'s going to tell you what she\'s proposing to do on
customer satisfaction.

STEO targets \-- I have one slide on it, and we probably won\'t go into
too much more detail. WE\'ll talk about it a little bit later. And then
customer and stakeholder involvement \-- I\'ll go through that quickly
too.

All right. The assessments by independent expert reviewers \-- this is
Bill\'s talk. And it\'s an attempt to get input on the quality of our
analysis and forecasting products. We have not had much luck in the past
in terms of doing that. How do you measure quality of analysis? How do
you measure quality of forecasting? And so this is one way. Our
Independent Expert Review Program is mostly used by our Office of
Information ?? Analysis and Forecasting, Integrated Analysis and
Forecasting. And they will send to these reviewers their latest work or
model documentation, and they\'ll get comments.

These are outside experts. They\'re chosen because of their stature in
the community. They review whatever we give them. They write up their
report. They come in and they talk to EIA, and then EIA either decides
to take their advice or not. But they do make changes to the products as
a result of the IER program.

We think this is a very valuable program because it\'s one of the ways
we get outside input on our products, and so we thought that that might
be a good way to get an outside assessment of our analytical and
forecasting products too. It does raise some questions. Bill is going to
give you an update on where we are on that. But issues that we haven\'t
come to closure on, and so I wanted to put them up here. We don\'t want
to have this review or assessment influence who we pick to do
independent expert reviews. We still want to get our best reviewers,
even if they may not like us that much. Sometimes it\'s good to go to
the people who don\'t like you so much.

And we would really like an assessment of the final product, not
necessarily an assessment of the preliminary stuff that we send them, so
there\'s some little details that go into how we might actually
implement this. So that\'s just for you to keep in mind.

Okay. Surveys meeting output targets \-- and as I\'ve mentioned before,
this is part of the survey assessment process you\'ve heard about.
We\'ll give you the update later. We did you a draft of the assessment
form, it should be in your notebook. This is really much bigger than
just setting survey-specific targets. We\'re trying to establish a
collaborative process, open discussions among all of the survey people,
sharing information, finding out what are best practices, and getting
people to balance quality and resources in a reasonable way. So these
are all things that we want to have this process do, in addition to just
providing these output targets.

The information we provided to you on the survey frames is a list of EIA
frames, and it\'s a draft. We think it\'s pretty good. There may be a
couple that we\'ll add later. We\'ll talk about it later. Some of the
issues about this that are listed in the action plan, we need to select
key frames for purposes of measurement, but then the harder problem is
?? what I\'d like to do for everything in the frame is to come up with
assessments of frame coverage. And so we would be interested in creative
ways of looking at that, and we\'ll talk more about some of the ideas
we\'ve got when we get to that section a little bit later.

Customer satisfaction \-- we\'ve had customer surveys through the years,
and we\'ve done pretty well, 98 percent satisfied or very satisfied.
It\'s a little hard to think you might be better than that, because
there\'s always some grumpy person out there that wants something
different.

I think the focus this time is to try to get assessments of specific
projects or products, rather than sort of overall EIA, so that at least
that way it might be more actionable. And so that\'s really what Colleen
will be talking about later.

Since we need to have quality measures for all EIA products, this is the
short term energy outlook, and it does forecasts for, you know, one
month out to something like two years. But for that ?? they have been
monitoring the relationship between their forecasts and the data as they
come in on a regular basis. They\'ve been doing it for years. And we
actually have information, the measures for most of their key products.
What we haven\'t done yet is to collaborate with STEO staff on figuring
out which variables we would like to monitor for reporting in the
Strategic Plan, and to select key target accuracy limits. But we have
historical data, and so this one is one that we think should be fairly
straightforward. And the STEO staff actually have looked at their
accuracy. Even 15 years ago, they used to do an accuracy document every
year, and they\'d look at it and decide what they needed to fix for the
next iteration of the model. So they\'ve had a history of looking at
their quality, and using that information to guide their analysis. So
this is another one that you may have some input on.

The Performance Measures Committee is charged with figuring out what to
count for purposes of customer and stakeholder involvement. Our initial
thoughts are that we could include focus groups or other focus
discussions with people outside of EIA. We do focus discussions,
particularly when we\'re redesigning forms to try to find out if
there\'s data that we need to add to the forms.

Whenever we have new survey packages, we put out [Federal
Register]{.underline} notices with all of the changes in it, and we do
get comments through that. How do we want to count that, is one
question. Just putting out a [Federal Register]{.underline} notice is
not a particularly interesting measure, and doesn\'t mean you did
anything with it.

We do some conferences and workshops, and these are good outreach
activities. Our National Energy Modeling System Conference, we just
recently had the Winter Fields Conference, and then there are ASA Energy
Committee meetings. And then I would be interested on any other ideas
that you may have on the kinds of events that we would either count, or
that we should establish in order to get better input from the customers
and stakeholders.

So my question on this session is, in this quick review of the measures,
did you notice that there are any that we have missed, particularly some
aspect of quality that we\'re missing. Are there any other obvious
measures that aren\'t on our list that you think that we should include?
And then any ideas on mission outcomes.

As I mentioned, that\'s kind of a missing point. That was one we\'ve
been struggling with, so ideas on that would be welcome.

MR. NEERCHAL:  Some of your data are actually used by other departments
\-- for example, Transportation uses some of your data. And I was
wondering, in your Table One, where will ?? there is obviously a clear
indication of quality, someone else is using your data for doing some
information product of their own. And I was wondering whether ?? where
you count them in your Table One?

MS. KIRKENDALL:  Those will show up there. Thank you. That\'s a good
idea. There\'s another place. The Census Bureau just put out a product,
I think the end of September. It\'s a poverty-related book, but they
used our Residential Energy Consumption Survey to provide some
information for that product, so that would be something that we should
highlight, that our data were good enough to be used in some of these
other publications.

MR. BURTON:  I have a couple of comments and a question. First of all, I
think if your budget is based on relative productivity, you all would be
swimming in dollars. You should be very proud of the work that you do.

Secondly, I think from the standpoint of customers or stakeholders, we
?? these meetings are important. I view our role not only as providing
guidance where we\'re able, but also representing the people that use
these data, so I think that this is an important part of that function.

And lastly, my question is \-- and it goes to relevance \-- you monitor
the number of web hits. Do you know among the various stakeholder or
customer groups who the larger web users are, whether it\'s the academic
community or business, or who\'s hitting the web?

MS. KIRKENDALL:  Paul, have you got any input on that?

MS. BLESSING:  I\'m Colleen Blessing. I\'m the Customer Focus Advocate.
We do a customer survey pretty much every year. We did a routine survey
between \`95 and 2000. It was a telephone survey, and we asked them if
they used the web and who they were. In 2002, we did a big web survey.
It was only up for two weeks, but we got 4,500 responses. And every
single person we asked what type of organization are you \-- business,
academic, media, finance. So we have a pie chart that\'s actually fairly
?? there\'s lot of big slices. It\'s not like 90 percent is government,
2 percent is academic. But it\'s government, and research consulting,
media, finance, academic. So we do have ?? periodically we monitor that.

MR. BURTON:  And it\'s fairly evenly distributed, not dominated by one
group or another?

MS. BLESSING:  Not dominated by one group. I mean, media is smaller,
finance is smaller, like maybe libraries are smaller, but there\'s
several big hunks that are, you know, 25 to 30 percent on the pie chart.

MR. FEDERER:  From the international perspective, and Canadian in
particular, is Statistics Canada your Canadian counterpart in terms of
production of energy statistics? And how does the ASA stake up against
that in terms of what they do with the funding and so on?

MS. KIRKENDALL:  I don\'t know whether ?? you know, we certainly look at
Statistics Canada as a model for general statistical agencies. They do
have ?? they do everything like the Census Bureau and the Bureau of
Economic Analysis and all that, but I don\'t know for sure about energy
statistics. Some of that may be collected by an other government agency
in Canada. I should find out. You are right.

MR. FEDERER:  One argument you could use if you could find if they get
more funding, is to ??

MS. KIRKENDALL:  What do they do?

MR. FEDERER:  Yeah. And also, in terms of international hits to your
website, get a lot of Canadian, other hits.

MS. BLESSING:  In this survey that we did, I didn\'t ask people if they
were domestic or international. But I did a sample of about 2,000 of the
4,500 users that came into the survey, and it was a laborious process,
but I just looked to see if they were .DE, .JN, and I calculated on
those weeks that 19 percent of the users were international. That\'s
what it looked like to me. I didn\'t look ?? actually, it was 81
countries. There were countries that I\'ve never even heard of, that
were ?? there were some islands and things. I didn\'t even know they had
Internet connections there. And I did have a breakdown of ?? and Canada
was one of the bigger countries. I mean, there were large numbers from
them, but there were many, many, many other countries, as well.

MS. KIRKENDALL:  Nick.

MR. HENGARTNER:  One thing else that I found missing in the
stakeholders, or maybe a way again to get more funding, is the
realization that energy sector is probably one of the most important
sectors for Homeland Security issues. And I know people are working to
assess Homeland Security and infrastructure reliability and those kinds
of issues, which is not exactly what you\'re doing. But the work you\'re
doing is extremely important for people who model those infrastructures,
and particularly, the energy sector. And so that would maybe be a way to
recast or motivate why Congress should pay more attention, more emphasis
on your job, because it serves as an input for the models that they\'re
doing to assess Homeland Security, and essentially, the resilience of
our sector. I mean, if electricity goes down just a few days, all hell
breaks loose. And the other parts of the energy sectors are extremely
important also, so there\'s just a suggestion that there might be a
slant, or a way to recast some of your products \-- not changing them,
but just recasting \-- say it is important not just for our customers,
but also for the whole nation. Here\'s one reason why it\'s important,
and everybody is aware of it. Just a suggestion.

MS. KIRKENDALL:  Thank you. Yes.

MS. KHANNA:  In relation to mission outcomes, it seems that if you could
get independent statements of how important EIA has been to independent
researchers, independent users\' own work, I think that would be a
really good support for you in picking up ?? there are other federal
agencies. How much do they rely on the work that EIA does to do their
own work? That\'s an obvious source of goodwill.

And the other is if you\'re running surveys and you have some idea of
who your major users are, maybe survey them, have an online survey. When
you go to use the EIA data site, fill out a survey, what do you think
about ?? you know, how important is EIA to your work, or how do you see
EIA\'s performance changed over time, you may get some good input. Maybe
extending the survey a little more about quality assessment right in
there is a possibility.

MR. NEERCHAL:  Each time you download data you would have to fill out a
survey form?

MS. KHANNA:  Maybe interested users would. I mean, I think as a user, I
would want to know why ?? actually wanted to give that kind of feedback.
This is improved or, you know, we need more data like this. And I think
statements like that coming from independent, even anonymous users,
could be very powerful.

MR. NEERCHAL:  This is in relation to the last item in your table \--
customer and stakeholder enrollment. Now since you have so many web hits
going on, I think you were saying your customer-base seemed to be very
web savvy. And so I was wondering, you know, you want to think about
some of your customer enrollment through the web, you know, through a
discussion board perhaps, or something that they can ?? you know,
especially in the international community that you are serving might be
?? it\'s easier to do a conference on web, perhaps. Less expensive, and
you can have a registration form when you are downloading the data.

MS. KIRKENDALL:  Jay.

MR. BREIDT:  You were talking about some of these stakeholders meetings
that you have. And maybe you do something like the following \-- maybe
it\'s a dumb idea, but it seems to me that some of your stakeholders
meetings are mostly for specialists. I mean, I wonder if it would make
any sense to have a meeting for ?? at a general level, kind of like
energy outlook for 2004, where you would invite Congressional staffers
and the media, and kind of have this broad overview at a general level
of what you have ?? what\'s your forecast for the year, what\'s coming
down the pike or something. It seems like if you can influence in
particular Congressional staffers or something, it might be more
influential than collecting a bunch of measures.

MS. KIRKENDALL:  Thank you. Johnny.

MR. BLAIR:  Yes. I was just going to come back for a moment to the idea
of collecting information from people who download, at the point that
they do. Certainly, to do that every time for everyone is burdensome
and, obviously, wouldn\'t work, I think. But, you know, you can
certainly look at developing sampling schemes so that you\'re not doing
this for every download, for every person, but that you\'re doing some
sort of a random sample. It would probably take some time to get, you
know, the design right and so forth. But conceptually, I think that you
could develop an ongoing sampling procedure that pretty early-on could
be automated so that, you know, the cost isn\'t that much once you get
the sample design done, so I think that that is something that you could
pursue. You just can\'t take a broad brush approach to it, and just say
everybody that downloads every time will provide information. But there
are other ways to get that same data.

MS. KIRKENDALL:  Thank you. That\'s a good idea.

MR. BURTON:  I\'d just like to underscore something that Neha said, and
that is, I think that you all need to collect information and highlight
the various policy arenas where your work is used. For example, we did
work on Mountaintop Mining that eventually wound up before the U.S.
Supreme Court, that would not have been possible without EIA data. And
to the extent that you can catalogue the various policy debates that
you\'ve helped inform, I think that it would be a very impressive list.

MS. KIRKENDALL:  You send us documentation about how you used EIA data
in a Supreme Court case. That\'s a nice little quote that can\'t hurt.
Okay. Anything else? I think our time is about up, if not more than up.
Thank you.

I now will give you Bill Weinig to talk about his IER.

MR. WEINIG:  Good morning. This talk is about evaluating the quality of
EIA analysis using a survey of independent expert reviewers. I\'m Bill
Weinig, and I\'m the front person of five others who contributed to this
paper, and I appreciated it a lot.

EIA is presently developing a Strategic Plan for the five-year period
2004 to 2008. The plan was not available for reference or quoting at the
time of this writing, but we had some insights on its content. EIA\'s
Strategic Plan and this paper are work in progress, and your comments
will be very helpful.

As you know, this is an early attempt to seek Committee advice. The
agency is intent on, as you saw in Nancy\'s slide, becoming a
performance-based organization, using a systematic agency-wide process
to review all ongoing work activities, and involving customers and
stakeholders.

While we have been conducting customer surveys of EIA products and
customers for a number of years, which Colleen mentioned, EIA has not,
until now, attempted to conduct a systematic, quantitative assessment of
its analytical products. This work is about EIA\'s proposed effort to
assess the quality of EIA\'s analytical tools and analysis.

Work is being done within the existing institutional structure to attain
associated quality measurement goals. That is to say, that resources for
this survey were limited to in-place staff, and no funding. Possible
sources of data are from the independent expert reviews mentioned
earlier by name, and outside technical experts who conducted Category 1
reviews for Nancy\'s office, the Statistics and Methods Group.

Let me talk for a moment about independent reviewers. Independent Expert
Reviewers are nationally recognized subject matter experts in specific
energy-related fields. EIA seeks and pays for their advice with
competitively awarded contracts, and sometimes single source honoraria.
These reviewers are considered knowledgeable about EIA\'s energy-related
work and products.

EIA draft products are sent to them for review, and the reviewers may be
easily and readily contacted based on this relationship. Still, the
number of such reviews is very limited.

In fiscal year 2003, six products were independently reviewed by these
experts, and 19 reviewers were used. Howard has a handout. Did you pass
that out already?

MR. GRUENSPECHT:  I did.

MR. WEINIG:  Thank you, Howard. Table 1 shows the number of reviewers
that we used, and the products that they were used on. In the second
page of the handout is a summary of Category 1 reviews conducted by two
EIA consultants under contract to Nancy\'s office. So in fiscal year
2003, 10 analytical products shown in Table 2 were reviewed by outside
consultants as part of the Category 1 review process. I hasten to point
out, we also do Category 1 reviews within Nancy\'s office by EIA staff.
Our thought to use these contractors in Nancy\'s office was to ask them
to evaluate EIA\'s analytical tools and products on an annual basis.

Continued methodology. We\'re still on slide two. Typically, EIA\'s
independent expert reviewers may be conducted on an analytical work
plans and study assumptions, draft study outlines, early drafts, final
drafts, model descriptions for model characteristics, not for the
adequacy of the model documentation, and model assumptions.

Our current approach is to ask the independent expert reviewers to
complete seven closed-end questions in the survey. A five point scale
could be used. We intend that the questions\' wording be similar to that
used in the Strategic Plan. We will get to those questions in a minute.

Now on to pre-test. Under methodology for the pre-test, considering the
small number of outside reviewers available, we intend to pre-test the
survey with two or three independent reviewers. The remaining reviewers
would be used then and available to serve as respondents for the regular
testing. We think respondents can complete the questionnaire in 15
minutes or less. My plan is to ask them to confirm that estimate.

If this works well, we would test the remaining reviewers for FY 2003
results, and include the questionnaire in future independent expert
review contracts as a deliverable once the reviews were completed.

Under returning the questionnaire response \-- the reviewer responses
are to be sent to a disinterested party so that there will be no
connection between the respondents\' evaluations and either the IER
Program Office, whose material is being under review, or your\'s truly,
the IER Program Manager.

Under difficulties at the last \-- a difficulty arises in that
independent reviewers will not be familiar with the quality of the end
product at the of their review. There may be a substantial delay between
the return review and the publication of the analysis, or use of the
reviewed model.

The individual independent expert reviewers may have very
product-specific knowledge based on conducting the review, perhaps a
bias. In the beginning of work on this paper, we were considering asking
reviewers to comment on the materials they reviewed, but now we are
leaning more toward asking the reviewers to evaluate the broader range
of EIA\'s analytical tools and products.

In some cases, independent expert reviewers are asked to do more than
one review of a study, as the study unfolds. This could be included in
multiple reviews on the same product, part of the list I read you a
minute ago. Analytical work plans and study assumptions, followed by
draft study outlines, followed by early drafts, and followed by final
drafts. We\'ve used that at least twice last year, and it\'s been
helpful to get the reviewers involved early, and to maintain a
continuity of their advice through the final product. I don\'t know that
the plan mentioned that, but we have used outside reviewers for that
purpose most recently.

Such a series of reviews might take months and include multiple reports,
and one or more discussion meetings with EIA management and staff either
in person or by telephone. As mentioned before, other outside experts
are used to augment the SMG staff in reviewing products. These reviewers
participate in what\'s called a Category 1 review process, and this is
typically a process that involves one-time studies or major revisions to
EIA\'s ongoing work.

Our current thinking is to include Dr. Robert Trost and Dr. Fred Joutz,
who do these outside reviews on a once a year basis, as to their
reactions to our EIA quality. Still, the reviewer may have an excellent
sense of EIA\'s product quality beyond those which he or she has been
asked to review, so what we\'re doing is trying to balance the distance
between product-specific reviews and EIA\'s analytical tools and
forecasting analyses in general.

The study objectives are to determine the quality of EIA\'s forecasting
tools and analytical products in the following areas. There are three
research objectives in this study, to determine to what extent EIA\'s
forecasting tools and analytical products are relevant and reliable, to
determine to what extent EIA\'s forecasting tools and analytical
products are consistent with changing industry structure, and to
determine the timeliness and usefulness of EIA\'s forecasting tools and
analytical products. These objectives are compatible with EIA\'s
emerging Strategic Plan.

We\'re proposing seven questions, which I\'d like to go through next.
The first question has to do with relevance. How relevant is EIA\'s
information in meeting EIA\'s customer needs? How reliable is EIA\'s
information in meeting EIA\'s customer needs? How accurate is EIA\'s
information in meeting EIA\'s customer needs? How satisfied were you
with the usefulness of EIA\'s information in meeting EIA\'s customer
needs? How satisfied were you that the timeliness of EIA\'s forecasting
tools and analytical products to meet EIA\'s customer needs? How
consistent is EIA information with changing industry structure? And
finally, how coherent is EIA\'s information?

Our four questions you might take in sequence. WE\'ve got more than 10,
but less than 15 minutes. I\'d like to suggest we take them one at a
time, if you don\'t mind. The first, are EIA\'s proposed survey
questions balanced, comprehensive, given the survey goals? Any thoughts
on that?

MR. BLAIR:  I\'ll take a little bit of a stab at it. I don\'t ??

MR. WEINIG:  Please.

MR. BLAIR:  They seem to, I mean, in terms of balance and
comprehensiveness to cover areas that are of interest, and that are
important, but each one seems to be fairly complex and open to, I think,
some range of interpretation. But by trying to keep the number of
questions limited which, you know, obvious reasons for that, I think
you\'ve kind of packed in a lot of potential ?? a lot of cognitive work,
I guess I would put, within each question. And it allows for a lot of
variability in interpretation, and in what people will come up with.
And, you know, if you\'re looking at this as a survey, you know, that\'s
not a good thing that you want to do, but whether this is a survey the
way we typically think of it, you know, I\'m not sure. I mean, usually I
think of a notion, you know, you have a sampling. I think you talked
about some relatively small number of reviewers, but I would think of a
sample as taken from some larger pool that that sample then represents
in some fashion. And I\'m not sure what that larger pool is, or what it
is that you are ?? or if you\'re proposing \-- I may be misunderstanding
\-- that this sample of experts is representing. So taking, you know,
what is, I think, an appropriate and reasonable notion of expert review
and putting it in this notion of a survey, I\'m not sure is going to get
you reliable data with those kinds of questions.

I\'m being vague here, because I haven\'t really looked at this closely,
but I feel, you know, when I see these questions and I see that, you
know, a definition was put in for terms, you know, like usefulness and
some of these other terms, that definition suggests that, you know,
you\'re dealing with something that\'s complex, and that, you know, just
may not be amenable to a single survey question. I mean, this is a
common problem in surveys where you have something that is fairly
complex, and it sometimes just isn\'t amenable to being measured with a
single question without paying a price in terms of the, you know,
validity of it in the sense of what you\'re measuring, so ?? well, let
me just stop there.

MR. WEINIG:  May I ask you a question?

MR. BLAIR:  Sure.

MR. WEINIG:  In this reaction, are you thinking about the pre-test, or
the survey, or both?

MR. BLAIR:  I was thinking about the survey, but I may be
misunderstanding what you\'re ??

MR. WEINIG:  I don\'t think so. I think you\'re really on point. In the
pre-test, we were thinking about two or three people because we were
looking to increase the numbers of surveys that we could send others,
and you\'re right. It\'s pretty small.

We had 19 independent expert reviewers, and we had two Category 1
reviewers under contract in Nancy\'s office, so we\'re really only
talking about a total for fiscal year 2003 of 21 people. If you pull off
2 of them, or 3 of them for the pre-test, you\'re down to around 18.

MR. BLAIR:  Yeah. I mean, if you think of this in the traditional sort
of way of surveys, you know, you\'ve got an N of 19 or 20, you think
about the variance, again, of 19 or 20. You know, if you get a different
sample and, you know, 3 or 4 of those people are different, you know,
your results could, you know, obviously fluctuate, you know, quite a
bit. So the combination of the, you know, small sample size, the small
number of experts; which again, you know, is not inappropriate for an
expert review, but if you\'re thinking of it as a survey, the
combination of the small number of observations with the difficulty,
potential difficulty with the questions, I think could give you some
fairly unstable results out of this kind of process.

MS. KIRKENDALL:  Let me just say something about ?? I wouldn\'t call
this a survey, because it\'s an assessment form that would be ?? I mean,
these guys all have a contract with EIA, and so it would be part of
their contract that at the end they could give us this little
assessment. That\'s how we get out of having to have a form that\'s
cleared by OMB, for one thing. But what they represent is themselves,
you know, so it\'s not really a sample. It\'s all of the independent
expert reviewers, because there aren\'t very many of them, including the
two outside experts also.

MR. BLAIR:  Right. And as I say, you know, the notion of expert review
is fine, and I would then though not look at, you know, this list of
seven questions and this kind of way of getting ?? you\'re not going to
get the best information you can out of experts with a format like this,
I think. That, you know, you\'re basically ?? you know, you\'re going to
get something that\'s easy to tabulate, but that\'s not very rich in
information. And that\'s not going to tell you a whole lot. I mean, if
you imagine that you have the results from 20 people or something for
these 7 questions and, you know, you do some tally and, you know, I
think you just haven\'t made a good use of expert ?? of the expertise
available to you by having them compress their summaries in that
fashion.

MS. KIRKENDALL:  So would it be better to do open-ended questions and
have them write a letter or something around these topic areas, or
something?

MR. BLAIR:  The other extreme would be open-end ?? the other is to see
if you can take some of these concepts, you know, usefulness and some of
the other things that were mentioned and, you know, kind of unpack them
in a sense, you know, to get more detail about what kinds of things
might constitute usefulness to different experts, different types of
users. So I guess what I\'m suggesting is that you go in the direction
that I think you\'re describing, thinking of it as an expert review. But
then, you know, to have an expert review that\'s summarized in this
limited fashion, I think you\'re just not going to get as much
information as you could.

MS. PHIPPS:  I think one thing that might be more useful ?? I mean, I
know there\'s sensitivity around the expert review process. But I mean,
they do have a contract, and it seems like these questions \-- similar
things should be integrated in the review that they send you back, you
know, in a more detailed manner, like Johnny was talking about. So, you
know, these are certain ?? this is part of the independent review
process to address these issues in it. And then what you could do is
take the reviews and have, you know, independent people code the reviews
as to, you know, what they said, and how well they may have met these
objectives, so you could have some kind of ?? use the independent review
as your data, instead of this piece after that. And then you might be
able to get more ?? you\'d probably have more information about these
issues that might be useful in changing the product too.

MR. BLAIR:  Yeah. I mean, I think ?? if I can for just a second, that
there\'s a tensioner between wanting to have, you know, a kind of
quantitative result, but that you\'ve really got a situation and group
of people that\'s probably more suited to more qualitative kinds of
methods of summarizing their judgments.

MR. WEINIG:  One thing that I may not have been very good at getting
across is that if we take the independent reviewers\' report of
findings, for example, and cull through that for the kinds of responses
to our survey, at least the ones that we have in our head, by design,
we\'re going to these people early; which is why I went over a couple of
times how early we can do independent reviews in the outline stage, in
the assumptions, in the early drafts, and so forth. And that\'s where we
thought we were having the problem.

By getting in early and reviewing early that we were having products
that even we felt needed help, that the kinds of feedback that we would
be getting would be critical, fairly so. So it\'s something that we\'re
wrestling with, and I appreciate your thoughts on it. I think, Neha,
were you up a minute ago?

MS. KHANNA:  Yeah. I think Johnny articulated very well the concerns
that I was going to raise. I mean, some of these questions are very
difficult to answer for a reviewer. For instance, how accurate was the
information? If I don\'t know how the information was actually gathered
and the analysis that goes behind the final version, I\'m not going to
know how accurate it was. Or how timely was it? Timely from whose
perspective? You know, from the EIA\'s perspective of getting it out
within, you know, whatever your target was, or from the user\'s
perspective of having it available when I thought I needed it, which is
going to vary from user to user. So I think, again, the points that
Polly and Johnny made, I think are really crucial, because questions
like this, I don\'t think you\'re going to get accurate answers.

MR. NEERCHAL:  I think it sounded to me as this is more like an exit
interview after having reviewed a product. And I also think considering
the number of people involved, and also I thought I heard Bill say that,
you know, try to get ?? try to make it anonymous somewhat so that they
can be very ?? they can speak their mind. It may be hard to do that
because the products are so specialized. If they start talking about a
product, you essentially know who is talking about it, so I\'m not sure
exactly how that can work.

MR. BREIDT:  If you did want some kind of anonymous independent
reviewer, one possible model would be to have your independent reviewers
act something like associate editors for a journal, where they would
send it out to anonymous referees, and they would collect it and get it
back to you.

MR. HENGARTNER:  Jay, but I am not sure ?? I mean, if you\'re an expert,
do you really want to be anonymous on positions? I mean, let\'s be
honest. This is a case in which I think anonymous doesn\'t pay much. I
mean, for surveys yes, but this is not a survey. We\'re talking about
experts, so I don\'t think ?? well, maybe it is an issue, but I don\'t
see it as one.

MR. BREIDT:  Well, there\'s two different goals that apparently this is
trying to serve. One is the actual review, and then the other is some
kind of tabular summary of the IER\'s view of the quality of these
analytical products. And so the actual review of the product you don\'t
want to be anonymous. You want to show your expertise and all that, but
to somehow summarize, it seems to me that you want to sort of summarize.
I think you do have kind of a survey in mind in a sense that these
IER\'s are serving as a sample of experts. What do experts think about
your analytical products? And maybe that\'s not what you\'re doing, but
it seems that that concept is kind of running around, and that\'s what
you would want to summarize these responses, and use them as
representative of kind of a larger pool of experts. And presumably,
these experts would know the other experts, so they could get at the
pool. Maybe that\'s the level at which you would want some anonymity.
But I agree with you, at the initial level ??

MR. HENGARTNER:  There is another issue of timeliness. I mean, I think
we always have this tension on what we can realistically achieve,
getting good quality data out there, and what the customer wants. And
the customer wants it yesterday. And if you ask the customer when did
you want it, he\'s going to say well, I needed it before. You know, the
faster the better, and he\'s always going to say that. And at some
point, it\'s like Heisenberg\'s Uncertainty Principle, you know, if you
make it faster, the quality will go down. And it\'s always going to be
there. And the fact is, at some point, it\'s not the customer that\'s
going to have to decide what the ?? well, unless you give them a chance
to evaluate it, and then they\'re going to complain about it also. I
know how it is. It\'s going to be more a matter of us saying if we
really want a quality product, I mean, this is the best we can do. And
at that stage, asking ?? again, the customer will give us a skewed view.
It\'s going to be ?? that looks bad on us, because they will say well,
bring it up a week earlier. You have the data. You say well, almost, you
know. And so it\'s hard to make that line. And I think this Committee
here should emphasize our role in making sure that the quality of the
data is there. I mean, that\'s the role of this Committee, and we should
somehow be instrumental in saying if you do it faster, you\'re going to
lose something, even if the customer says something else. Now I don\'t
know how that can be factored in, but that\'s my view.

MR. WEINIG:  Jay, you were an independent expert reviewer.

(Laughter.)

MR. EDMONDS:  Just my reaction looking at those questions is pretty
similar, in that what we seem to be getting out there is do I like it or
not? That\'s kind of what you\'re getting back, and I think if you have
experts, you could get so much more. And what you ?? particularly, if
the goal is to have a better product in the next iteration. Looking at
these questions, I don\'t see that you\'re getting much information that
would help you do something different. And since you\'ve got experts,
these are presumably people that could give you some very specific
information. And so, you know, for example, the accuracy question \-- I
mean, you could just ?? that could be quantified, or you could push them
to quantify it. And then push them to explain well, if it\'s inaccurate,
you know, could you do better? There may be some inherent limits on
accuracy but, you know, these are experts. You want to get everything
you can from them.

MR. WEINIG:  I\'m feeling two tensions. I\'ve only asked you one of the
four questions.

(Laughter.)

MR. WEINIG:  And in my effort to help Jay keep the meeting on track,
what I\'d like to do is take just a couple of more minutes, but see if
there are any burning bushes out there. Okay?

Are the independent expert reviewers and Category 1 reviewers likely to
be the appropriate survey audience, or do you have some other survey
audience that you would suggest?

MR. BREIDT:  I think we\'ve all been talking around this issue; which
is, would the IER seem exactly the right people to take a specific
product and give you an independent expert review, a detailed review
without some summary comments at the end? If you want some kind of ?? I
guess it\'s not clear to me, if what you\'re seeking from this summary
is kind of an overall, a broad overview of the quality of your
analytical products, then we get into all the issues we discussed here.
If you just want a specific review of a specific product, the model you
already have in place works for that, and you don\'t need to really
summarize it. But if you want to have some broader overview of the
quality of your analytical products, then I\'m not sure this is the best
way to go without much more detailed questions, and maybe a closer look
at what that actual audience of experts is, and how you might get
representatives from that population.

MR. WEINIG:  Any other thoughts from the Committee that add to or
disagree with that? All right. If we proceeded on with a very small
sample in a very limited way, which I hear you\'re not crazy about,
would 2 or 3 in the pre-test be sufficient? Johnny.

MR. BLAIR:  Well, it partly depends on what it is you\'re doing in the
pre-test. Let me suggest something, rather than going back and beating
up on the questions again, is that if you imagine taking this set of
questions that you have, and doing a ?? you know, 2 or 3 people and
doing something like cognitive interviewing, whether it was a
think-aloud or some other kind of cognitive testing procedure, I think
that one of the things that you would get would be a lot of information
of the components, for lack of a better word, that go into the summary
judgment for each of those questions. And that might suggest to you how
you could break those questions down into things that, you know, if you
want to get a quantitative kind of, you know, summary, that would be
more useful, that you could ?? in the course of having someone do a
think-aloud with these questions, to find out what are the cognitive
tasks that they\'re going through, what are the things that they\'re
considering in coming up with the summary assessment. And that it\'s
probably those things that they\'re considering that you want to find
some way to ask more directly, rather than just getting the summary,
because I think that as you go across, you know, your 2 or 3 experts,
you\'ll find, you know, that they have different things they\'re
considering, but moreover, they\'re giving different weight to the
different components that they\'re considering. So I would think of
using a pre-test in a way to almost develop a second version of your
instrument. And for that purpose, to answer your question, I think that,
you know, you could do some small number of people and get a lot of
useful data out of it.

MR. WEINIG:  Okay. And then finally ?? I\'m sorry.

MS. PHIPPS:  One thing that you might want to think about if you\'re
going to go ahead with that is to do people at different points during
?? how close they are to have completed a review, so when they\'ve just
done the review, you might want to get somebody who\'s just done one,
and you might want to get somebody six month later, you know, to just
see, you know, the different problems they\'re having and the thoughts
they have, because it may be very different, depending on how close they
are to when they\'ve actually completed their review.

MR. WEINIG:  And then finally, are there any other study or survey
questions that you think we should be asking? Thank you very much for
this discussion. Nancy told me that this was going to be interesting,
and to me it has been. Thank you.

MR. BREIDT:  Okay. Before we break, if we could have the people who have
come in since the morning introductions, introduce themselves, starting
at the front table. So your name and your affiliation, please, starting
with you. I think, Nick, you slipped in.

MR. HENGARTNER:  Yes, I\'m sorry. My name is Nick Hengartner, and I\'m
at Los Alamos.

MR. BLAIR:  I\'m Johnny Blair, and I\'m with Abt Associates.

MR. NEERCHAL:  I\'m Nagaraj Neerchal of University of Maryland in
Baltimore County.

MR. BREIDT:  And then in the audience, if you came in since the morning
introductions, would you please introduce yourself at the microphone.

MS. NORMAN:  I\'m Karen Norman. I\'m with SMG.

MR. LONG:  Gary Long with EIA in Dallas.

MR. WOOD:  John Wood, EIA Reserve Production Division in Dallas.

LULU:  I\'m LULU from Office of Integrated Analysis and Forecasting,
EIA.

MR. SITZER:  Scott Sitzer, EIA.

MR. SEDRANSK:  Joe Sedransk, SMG.

MR. KING:  Bob King, RPD in Washington.

MS. NORTHRUP:  Mary Northrup, Financial Analysis Team.

MR. BREIDT:  EIA?

MS. NORTHRUP:  Yes.

MR. BREIDT:  Is that it? Okay. Thank you. So why don\'t we take a 10
minute break, and reconvene at 10 minutes before the hour.

(Whereupon, the proceedings in the above-entitled matter went off the
record at 10:40 a.m. and went back on the record at 10:59 a.m.)

MS. KIRKENDALL:  Okay. Well, we should get started again, I guess, since
we\'re only what \-- 15 minutes late. Hey. This is a session on the ??
it\'s really essentially an update on the session we had at the last
meeting that we called \"Overview of EIA\'s Survey Quality Initiative.\"
We\'re trying to come up with a process by which we could get a
self-assessment to be done by each of the survey managers, and
coordinate that with developing measures and looking at things in a more
coordinated way.

So under, is it Tab 4 in your book? Thank you. All I wrote down was
\"Tab\", which is not very informative. This is basically where we are
right now. There\'s a summary sheet. The team came up with ?? I think I
have a few people from the team here. But the team came up with
something that is a reasonable looking assessment form, although we\'d
certainly ?? it\'s drafty enough that we\'re still open to comments, but
we thought that it captured most of the information you\'d like to have
in an annual assessment. And it begins on page 2 of the instrument.

And the only problem is it\'s kind of open-ended questions, which means
that it\'s going to be really hard to just tabulate. One of the ?? so
there were two comments. Tom Broene, when he first looked at it, said
you know, this is so long, nobody is going to fill it out. The other
comment was, you know, well, it\'s all open-ended questions, so actually
tabulating the results is going to be a nightmare. But we had somebody
from the team who took this assessment and went through the questions
with her person who ? here she is, in fact \-- with the person who runs
her survey, and she said that it was a useful dialogue around these
questions, so they learned something by going through the process.

So the current thought is that maybe the form stimulates interesting
discussion, and that instead of having them fill it out, we could have a
discussion with them around the questions in the form, so SMG could
interview either a survey manager, if it\'s a really small staff on the
survey, or have a facilitated discussion, if it\'s a group of surveys.
The petroleum surveys tend to go in clusters, so you have more people
involved in those discussions, perhaps. And then we could use it to come
up with assessments that we would give back to them to say yeah, that\'s
what I heard too. So that\'s sort of the current thought on the
assessment instrument. And anybody who wants to pipe up and say
something about it.

The other piece of it is the last page is sort of a table, and this is
much draftier. I think there are too many items on here, but we had not
fine-tuned this and narrowed it. What we\'d like to have is have them do
something with filling in a table that would give you some measures, and
establish targets. So that part of the process is a little more drafty.
So basically, the questions for you are, what do you think of the survey
assessment form? Are we capturing the right kinds of information? Do you
think that a process like that would work, or any suggestions for
changing the process? And then any comments about the table in the back,
the identification of measures and setting of targets.

So this time my part of the presentation is much shorter, and I\'ll
leave it to you to look at Tab 4. Johnny.

MR. BLAIR:  You just ?? is the notion that at the end of a survey, that
the survey manager would complete answers to these questions?

MS. KIRKENDALL:  Right now the proposal would be since they\'re so
open-ended, is that we would have a facilitated discussion with a survey
manager once a year, sort of at the end of the cycle, and around this
questionnaire. So we send the questionnaire to them before, the
assessment form to them before so they\'d know what we\'d talk about.
And then we would come and have a discussion with them, and record.

MR. BLAIR:  I think this is a little bit of an aside. I mean, given the
nature of some of these things, sampling and the magnitude of sampling
error in some of these other things, that unless someone had this, you
know, very early on in the survey process, that this would be, you know,
quite a shock at the end to get this and be invited to a session. I
think this is something that almost has to build over the course of
doing the planning, and doing the survey. And then I think, you know,
the kind of thing you\'re suggesting could work. But I don\'t think that
getting all of this at the end, if it wasn\'t planned for, that people
would be or could easily come up with some of this information. And, I
mean, just looking at sampling on a complex survey, I mean, that\'s ??
you know, obviously you will ?? the rating and computation of sampling
variances and so forth and, you know, how those are presented and, you
know, is it an average over a number of variables or, you know, exactly
what is it? You know, just that one item, if you start to look at it,
there\'s a lot that\'s involved with it, so I think both getting this to
people early, and then probably having to say a little bit more about
some of these items like sampling error, as to what exactly it is that
you want, or I think people could show up with very different kinds of
information, you know, and it would be hard to have the kind of
discussion that you\'re suggesting.

MS. KIRKENDALL:  Yeah. That\'s a good point. We\'ll need more
information about what we mean by some of these things. I think when we
were doing it, we had in mind that we would pick ?? each survey would
have key variables that they would monitor for things like the sampling
error, or response coverage, and that sort of thing. We have a comment
from Census.

SPEAKER:  Speaking as a survey manager of surveys, it would be very
useful to know your expectations for these things, so that we can
monitor it as the survey is being processed. And if we can\'t meet the
expectations, then the discussion should go towards why.

MS. KIRKENDALL:  Okay. Actually, that\'s a wonderful point. He\'s from
the Census Bureau. He\'s the Survey Manager for Manufacturing Energy
Consumption Survey that is conducted by the Census. I mean, that is the
kind of thing we\'d like to do. It\'s supposed to be a communication
device, you know, what our expectations ?? I mean, even inside we need
to know what is it we\'re expecting for response rates, and what is it
we\'re expecting for accuracy, so it\'s communication. John Wood.

MR. WOOD:  Yes. I mean, you might carry that just a little farther. And
if this would be built into the initiation of every survey cycle, and
then followed. And then assess it at the end. And do you know what your
targets are, etcetera.

MS. KIRKENDALL:  Okay. Well, that would certainly be the easiest way to
do it. But, of course, what we\'re going to try to do is ?? and it\'s
not going to be perfect the first time we do it, is to try to start out
?? because we have many, many ongoing surveys. You know, they\'re not in
the initial design stage. They\'ve been going on for years, and so we
need to sort of leap in and maybe get a benchmark on those. So there
will be some start-up problems with some of the ongoing surveys. It\'s
certainly going to work best for something that\'s just being designed.
You can have your plan targets and work towards them. Johnny.

MR. BLAIR:  And is this ?? I mean, sure, for something that\'s just
being designed, obviously, this is useful so that people are looking
across all ranges, but I think it\'s also very interesting for something
that\'s ongoing, because you\'ve got real data as to sort of where
you\'re at now. And then if, as was suggested, targets are specified,
you have a sense of sort of, you know, how big is the gap between where
you\'re at and what the target is. So I would think for both ongoing and
for new surveys, it would be useful.

MS. KIRKENDALL:  The next step on this is once ?? I haven\'t actually
gotten the Committee together to talk about it, and agree that this is a
good starting point. You know, nothing is going to be final for a while,
but the next step would be to go down and talk to individual groups
within the offices, the individual survey managers to see what kind of a
process they think would work with them. It just ?? it seemed like
having SMG involved with a discussion with them might make it easier,
because we can help guide the discussion. Mary.

MS. NORTHRUP:  You probably know better than I, but how many of EIA\'s
surveys are census versus samplings. And are there some of the census
surveys that are going to sampling, and should that figure into the
composition of groups if you have facilitated discussions? A person
who\'s got a census might be very, very concerned about response rate,
but would fall asleep in the discussion of sampling, or vice versa.

MS. KIRKENDALL:  We don\'t have a whole lot of samples, we have some.
But yeah, that\'ll be ?? that\'s the kind of thing I think we\'re going
to have to work out with each office, as the right way to organize the
surveys to have these discussions. I think Guy told you today we have
something like 55 surveys. I think the list I looked at was 70, so it\'s
somewhere in there. But petroleum supply, for example, has 5 or 6 census
surveys that go out monthly, and then they have 5 or 6 parallel weekly
surveys so, you know, they\'re ?? and each one goes to a small targeted
population. But they all feed into the petroleum supply monthly, and the
petroleum supply weekly, so you count up a whole bunch of surveys if you
look there. But their processing is similar, so that\'s probably a
natural family of surveys.

MS. NORTHRUP:  Small population surveys are each a census?

MS. KIRKENDALL:  For the monthly they are. Okay. Well, that made up a
lot of time.

(Laughter.)

MS. KIRKENDALL:  Okay. Well, I\'m sure you\'ll hear more about this one
as we go down and talk to people at the lower levels. If you look at
this and you have other comments, please send them to me. Johnny.

MR. BLAIR:  Let me just mention one other thing, and that is, in
starting out with something like this, I think how it\'s framed in terms
of the purpose for the managers or, you know, whomever is going to be
involved in this \-- as look at some of these things, I would not be
surprised \-- I mean I don\'t know these details \-- but that for some
of the ongoing surveys, someone might not know the answer, you know, to
some of these. I mean, you know, some of the questions about frame
coverage or some of the other things, and so it could be embarrassing
coming to the kind of meeting, you know, and not be able to answer some
of these. I think how this is framed in terms of the purpose of it and,
you know, not just where it\'s starting, but where it\'s going would be
important in getting people to, you know, react to it the way that you
want.

MS. KIRKENDALL:  The other thing that might be interesting, I mean ??
that\'s a good point as to ?? they also are going to have ?? at the
beginning will have to know what the questions are so they can get the
right people there. Because you\'re right, there are going to be some of
the people who run a survey who don\'t know what\'s done on frames
maintenance. Joe.

MR. SEDRANSK:  You might add documentation of analytical methodology.
And possibly, evaluation of it.

MS. KIRKENDALL:  Certainly, something we need to know about, where do we
have good documentation, and how good is it. Okay. With that, I guess I
can go right onto the next topic. Maybe just to be different, I\'ll sit
down over here.

Okay. This section is even draftier. We wanted to do something that
would help us to come up with an evaluation of our frames, and it\'s ??
one of the things we\'re going to have to do is to sort of balance
resources and figure out what we can do that makes some sense. So what
I\'ve given you \-- is that Tab 5? Look at that, Tab 5 is a very short
discussion of what the frames project is, and the list of what we
identified as our current frame surveys. Since we have a number of
census surveys where there is no sampling, we\'ve listed all of those
census surveys here, complete with the number of units on the frame, and
a description of the respondents.

So what we would like to have for the Strategic Plan is take a look at
these frames and see how good they are. So we\'d like to know where do
we need to put more effort into improving frames, where do we think
we\'ve got pretty frames and we\'re happy with it? So I guess what I\'d
like to do with this is to start with this little table, and maybe add
some columns that would have information from the program offices on how
often they update the frame, the volatility of the frame.

I mean, one of them is an actual frame survey, and that is used as the
basis. Which one is that? That is the ?? the one on the bottom of the
second page, the 863 is a petroleum product sales identification survey.
That really is a frame survey, and it\'s used to select all of the
samples that are used in that office, so that one ?? and that one is
done every four years. So clearly for that, you\'d have when it was last
conducted. You might like to have some information about how badly it
deteriorates over time. Because at one time that was done every three
years, it was shifted to four years for financial reasons, and so what
have we lost by not doing it quite as often? That might be a good
question to ask.

MR. HENGARTNER:  So you\'re saying you don\'t know how many changes
there are each time you update the frames?

MS. KIRKENDALL:  I don\'t, but I\'m sure the program office does.

MR. HENGARTNER:  Because that would be useful information, when it was
last done, and how many changes there were from the frames before, and
trying to get a ?? I mean, this goes back to volatility or however you
want to call it, but I\'m looking there down the page, financial reports
and that\'s major energy companies, and there are 30 listed, you know.
If I ask the same question in 5 years, are the same 30 going to be there
or not?

MS. KIRKENDALL:  Mary can talk to that one.

MS. NORTHRUP:  Well, you happened to pick an odd duck for EIA, because
maybe some of you can tell me, we don\'t actually know whether our frame
or our survey is a certainty sample, or a universe or a census. But
basically, what the law says is that the administrator of EIA is
supposed to decide who the major energy companies are, certain kinds of
analysis can be done, and a certain type of profile blind business can
cover.

Technically he could say I like this company and this company. We have a
set of criteria we use, and then we have less ?? the changes we\'ve had
recently have been intra-sampling \-- excuse me \-- intra-respondent
list because there\'s a consolidation, except for bringing in a bunch of
independent refiners who, when the traditional integrated majors took
their money and went overseas and divested their refining assets,
independents grew, and now the independents are emerging amongst
themselves. So my question to you is, we certainly know a lot about
that, and we publish who\'s in and who\'s out over a time series. But
how does this information help you, or help ?? I mean, we tell people
who\'s in and who\'s out, and what the criteria are.

MR. HENGARTNER:  I mean, in this case ??

MS. NORTHRUP:  Is that a certainty sample, by the way? If you hit the
criteria you\'re in, unless we change our mind, and you\'re not. Like
some people fall in, but we know they\'re going to divest something in
two years and fall out, so it being a big burden, we don\'t drag them
in. So it\'s a certainty sample with judgment.

MS. KIRKENDALL:  That one is one we\'re not going to spend a lot of time
worrying about.

(Laughter.)

MR. HENGARTNER:  A chosen bad point. It was just the smallest number on
the list, and I figured, you know, if you have 30, there might be
changes.

MS. KIRKENDALL:  How often do you update it, every year?

MS. NORTHRUP:  Well, every three years for sure, but in times of big
restructuring like this, we look at it every year to see if we should
change it.

MR. HENGARTNER:  When you do merge it do you ?? I mean, I see this like
Dow Jones, by the way.

MS. KIRKENDALL:  That\'s interesting. Okay.

MR. HENGARTNER:  And the Dow Jones, each time there is this split or
merger, they change the rates. The company stays there possibly, but
they say well, whoops, the rate from that company now changes within the
??

MS. NORTHRUP:  We do. Let\'s take Chevron/Texaco as an example. We
publish our numbers in aggregate, so we didn\'t really lose any asset
base. But to the extent that there\'s a major acquisition that pulls
somebody in from outside, or major divestiture that goes the other way,
we report the numbers, and then we put a little note saying guess what,
the step function you\'re seeing would otherwise be had it not been for
this big change that happened this year. I really don\'t know how else
to handle it.

MS. KIRKENDALL:  But some of these frames, like the 810 for example,
we\'ve got 335 units. They\'re refineries and blending plants, so two
very different groups. We think that the list of refineries is probably
pretty complete. I don\'t think they\'ve been building refineries for a
long time, so they\'re not too hard to verify. But the blending plants
we may not have quite as complete a list of the blending plants. So one
question is, are there some nice easy ways, or relatively easier ways of
coming up with assessments of a frame?

One thing that we\'ve talked about, we\'ve been working with the Census
Bureau because they collect our manufacturing energy consumption survey,
and refineries are, of course, manufacturing industry, so they have a
list of refineries. We have a list of refineries, and we could share our
frame with them and come back with some assessments.

Now that may not be all that inexpensive. We have to work out the
details on that, but that could be valuable for both Census and for us,
so we are looking at some of those things. Johnny.

MR. BLAIR:  Yeah. One of the things that I always want to know about a
frame is how is it constructed and how is it updated? That is, one
assessment of a frame, it seems to me, is the process for keeping it
complete; that is, that everyone is on it that should be, and that no
one is on it that should not be. So what I would ?? if you were adding
columns to this, what I would be interested in seeing for each of these
is sort of what\'s the process for updating the frame and making
corrections in it. Is it just updated by the fact that when you do the
survey you find errors in it, in which case, that\'s not probably, you
know, the greatest way of doing it.

And if you look at the process that\'s used for each frame that, I
think, right away would ?? you know, someone who knows about these types
of companies and whatever would right away, I think, be able to make
suggestions about things that you can do to improve the frame, once you
know something about that process.

MR. NEERCHAL:  It sounds to me that to really keep them all current all
the time may be a very expensive task. Some of the frames are very large
and, you know, to audit them once will take a lot more effort. So in
terms of getting some kind of a priority list, and also to answer your
question \-- what kind of columns to add to this table \-- might be the
corresponding product. You know, this frame goes with this product. You
probably know it, but I don\'t see it here. And that way you know the
important product that you really need to keep, you know, shining. And
so those are the real important frames.

MR. HAMMITT:  How do you identify units that aren\'t in the frame when
you update it? It seems like a principal, a very hard question to find
the things that you don\'t know of. And is there some possibility of ??
obviously, you could look at other people\'s frames in Census. That is
one idea. Is there enough overlap among some of these that people in one
frame either ought to be in another, or would be related to companies
that should be in another? Somehow you can use one frame to better
understand a different sector that you also try to survey?

MS. KIRKENDALL:  We can do some comparisons across frames. The
refineries, for example, also report in the electric power survey, the
906, so we can always do a check there.

In terms of finding missing units with our project with the Census
Bureau, we won\'t find specific units that are missing because they
can\'t give that to us with the confidentiality pledge, but they can
give us assessments of coverage. They can say, you know, you are 99
percent of electricity generation, and we\'ll just have to work with
them to find out if you can get down to something more actionable than
that. But we can certainly get general assessments of frames through
that kind of collaborative effort.

MR. HAMMITT:  So that would at least tell you which frames you might
want to work on, the ones with poorer coverage.

MS. KIRKENDALL:  Right. Some frames are really easy. When we have
regulated utilities or, for example, you go to FERC, you get the list.
We would probably do that fairly regularly, and we probably aren\'t
missing anybody that\'s regulated. Marketers are much harder, and my
guess is we are missing marketers. You know, we ask people who they sell
to. We ask utilities who they sell to or something, so you get some
information like that. But how complete is it?

Now what I\'d like ?? ultimately, what we\'ll have to do is to look at
this and see what is our general assessment of all these frames, and
where would we like to put more resources to improve things? Do we see a
place where we need to improve things. Johnny.

MR. BLAIR:  I don\'t know anything about these types of companies, but
in general in businesses, people that are on the frame know of other
people that are in that same business, and so one way of checking a
frame, particularly with some of these smaller ones, is to run it by,
you know, the people that are on the frame and they\'ll say gee, you\'re
missing so and so that, you know, was just formed last year or
something.

MS. KIRKENDALL:  John, I think you were first.

MR. WOOD:  On natural gas liquids that come from a natural gas plant,
EIA runs a monthly survey and an annual census where we go to everyone,
and those two frames are bumped against each other and reviewed at the
monthly, which is updated and kept in sync.

A much more massive job where you\'re talking about 60 or 80,000 people
in a frame are the EIA 23 crude reserves report frame, and the most
efficient way to keep that up-to-date is to bump it against all the
producing states who tax the company, and so we keep a frame of 22 to
24,000, which the last time we checked made returns over 20 percent a
year by just bumping it against the state comparable list of operation.
And it\'s rather efficient.

MS. KIRKENDALL:  Mary.

MS. NORTHRUP:  Combining what these two generally were saying, a column
for the process by which you assess your frame, just writing it down is
not, to me, of interest. But I\'m wondering if there\'s a role for the
expert reviewers here, particularly some of these people with
statistical expertise were available for the expert reviewer. If you had
the process documented and you could identify those surveys, as you
said, you said you can always improve a frame, but there\'s a cost to
doing it.

I\'m thinking of an expert reviewer who would look at the methodology,
look at the upgraded data, look at how EIA\'s data is particularly used,
or if there\'s a law or something like that, and assess how the frame
could be better, but the cost benefit for doing it. And that, to me, you
can always make a frame better, as Nancy said, but if you\'re going to
make it ?? if it\'s 80 percent good now, and you\'re going to make it 10
percent better for \$900,000 a year on an ongoing basis, I would think
most consultants would say given EIA\'s situation we shouldn\'t do that,
but what should we do? Should we flag certain data items? And, you know,
if it were free through the independent expert reviewer thing, I would
love to ask them to do that for my survey.

MS. KIRKENDALL:  You know, we have used the IER Program to do much with
surveys, but there\'s nothing that keeps us from doing that.

MS. NORTHRUP:  If you\'ve got the right kind of talent lined up.

MS. KIRKENDALL:  Yeah, we\'d have to find the right talent, but we can
get ?? I mean, Bill is really quite good at that. We always do a good
job finding the right talent to fit on this fence.

MS. NORTHRUP:  So if you have another column in addition to the
methodology for frames update that said, you know, what is your program
office sense of certainty on your sample, and SMG went through these
various columns and said, you know, this looks like one that could
benefit from an expert review. Let\'s ask the program office if they\'re
interested. I mean, then you would get into the sensitivity decisions
with a lone certainty, really want to be reviewed for fear that they\'re
not going to get any money to fix it. You know, I mean, you get into all
the politics of it. But it\'s still an interesting avenue.

MS. KIRKENDALL:  Yeah. It is an interesting avenue. We need to make sure
we have good documentation, and then we could do it.

MS. NORTHRUP:  By why put program offices that are already strained to
the trouble of doing documentation if, in fact, there\'s not an end use,
a benefit to which you guys are going to put it? I don\'t believe in
documentation just to feed the system, because we\'re all too burdened
as it is.

MR. HENGARTNER:  I\'d like to bring up another point with those frames.
Suppose that we know that the frame isn\'t perfect. Well, there are ways
of using even incomplete frames for sampling purposes. And there are
notions of sampling hard to get populations, like HIV users and so
forth. And some of the ideas, snowball sampling and all kinds of very
cultic needs that are turned to deadlock right now that say well, I
start with knowing someone, and I\'m going to ask him who they know. And
I might also sample these guys.

Now what you\'re doing is you take off the burden of getting a complete
frame, and putting it on the sampling strategy. But if that\'s an option
for you, then having a column saying the quality of the frame very
complete, moderately complete, incomplete might still be useful. What
I\'m saying is that yes, we should get as close as possible to have a
complete frame. But in the absence of a complete frame, they\'re not
useless. And putting in some thoughts on the way the sampling is done
for incomplete frames or frames that are less than complete might in
alternative ways, essentially shifting the burden onto someone else\'s
shoulder. But nonetheless, it\'s an option that we should keep in mind.
And so when you talk about cost benefit, that\'s exactly what we\'re
talking about, trying to do the best use of the information we have.

MR. BURTON:  From an analytical standpoint, if you\'re missing some
firms, the harm that that does to my ability to use that information
directly depends on whether there is something systemic in who\'s being
missed. In other words, are you missing all small or a
disproportionately large number of small firms, or firms from a
particular region. If that\'s not the case, if there\'s nothing
systemic, then I can still do an awful lot with that data. And
consequently, I think the benefits of trying to perfect the frame are
greatest when the firms that are currently being missed fall into some
particular category.

MS. KIRKENDALL:  Shawna.

MS. WAUGH:  In the current document, it lists the number of units. And I
would add a column that says are these companies or establishments,
because that would be significantly different. And even with perhaps
providing a little more information to someone reviewing these frames,
how they define an establishment, what the boundaries are for that
establishment, because that could also impact the number of units.

MS. KIRKENDALL:  That was supposed to get into description of
respondents, but it may not be complete. Actually, you know, this time
it\'s Colleen who\'s up again.

MR. BREIDT:  Okay.

MR. BLAIR:  Can I make a quick comment?

MS. KIRKENDALL:  Sure.

MR. BLAIR:  And that is, as someone mentioned, the cost of updating a
frame or making some marginal improvement in it, and sort of whether
that\'s worth it. I think it\'s important to keep in mind for some of
the things that were discussed earlier, as well, in terms of ??
essentially, you\'ve got this limited budget that everyone has been
talking about, and you\'ve got all of these different things that you
can spend it on, updating a frame, improving your response rates and so
forth. And really, whether you do it formally or not, you want to think
in terms of some sort of total error model; that is, what are these
different sources of non-sampling error and sampling error, for that
matter, contributing to the total error for the survey, and using that
notion, again whether informal or not, as a basis for deciding, you
know, where do you put your resources, where are they going to do the
most good. As long as you look at these in isolation, you know, the
frame issue is over here and the response rate issues over here, it\'s
going to be hard to make those kinds of judgments.

MS. KIRKENDALL:  Yeah. This project needs to fit in with our survey
assessment project too, because that\'s where you take the bigger look
at things. It\'s just ?? this one was a better well-defined piece of it,
but we could do some on maybe before we get to the general assessment.
Thank you.

MS. BLESSING:  Thanks. As you all probably know, EIA is really active in
listening to our customers and there\'s a whole bunch of ways that we
get feedback. The first one is, if you\'ve been to our Home Page,
there\'s a button that says answer a couple of questions. And also on
the second level pages, there\'s an invitation to send us some feedback,
and we get 40, 50, 60, 70 comments a month that I and other people read.
We get a lot of web mail, and there\'s a summary of the web mail that I
and other people take a look at.

We have information specialists. You guys probably already know this.
There\'s a whole bunch of people that are on the phone all day long
answering questions. And every month, they give me a summary of the
kinds of things that are bugging people, or that aren\'t working, or
sometimes things that are working.

And the fourth way that we get customer feedback is through customer
surveys, and that\'s what I\'m going to talk about today. I just wanted
to make sure that you guys knew that the customer survey, since I\'m
going to say we only do one once a year, it kind of sounds like there
might be this, you know, 51 week gap where we\'re not doing anything.
We\'re really listening all the time.

As I said at the microphone a couple of hours ago, EIA tries to do about
one customer survey a year. And in 2002, early in 2002, we did a ?? what
I call a pop under survey of our main website. A pop under survey is one
in which if you come to our site and you leave our site, and you close
your browser, there will be one of those windows waiting for you, you
know, do you want to buy this secret camera, you want to go to Mexico,
you know those ones that you close, close, close. Well, there would be
an invitation, a very clean, very simple invitation. And that\'s the one
where I said we got 4,500 responses in two weeks.

Now in 2003, we decided to do a survey of our kid\'s page. For those of
you who may no know, it\'s an up-and-comer at EIA. Of the million users
a month that we have on our main page, a million or more, up to 100,000
of them a month are on the kid\'s page, so it\'s the fastest growing
area of our site. And, obviously, even in real numbers, it\'s a whole
lot of people coming to our kid\'s page and actually talking to people.
Sometimes people \-- adults will say it\'s the only place I can go where
I can really understand what\'s going on, so it\'s not just students
that are going to the kid\'s page.

Okay. Now we\'re going to go to the video tape. I\'m going to talk for
just a minute about how we conducted this survey, and the kinds of ??
and what we did with it, so you can idea of, I think, the richness of
what we get. And that we don\'t actually just say okay, we did the
survey, and move on.

The kid\'s page survey was also the dreaded pop under. And we used that
format because we had experience with the one the previous year. There
were basically six questions on the survey, and they were kind of like
who are you? If you\'re a student, what grade are you in? Is this your
first time here? What were you looking for, and did you find it?
Anything else you want to tell me? Very standard kind of questions we
ask at EIA.

The survey was live for two weeks, kind of right before Spring break. We
really had to time it before the private schools got out, and before
Spring break, and after Christmas break, so we did do a lot of thinking
about when we were going to get the most kids. In two weeks, we got 705
responses as a side light, and a number of those responses were more
interesting than I\'ve ever gotten from adults, in that they were either
silly, nonsense, or X-rated. So kids, I don\'t know, they\'re more used
to the medium, and they\'re not thinking anyone is going to catch me, so
we got about 650 legitimate responses.

Okay. You can go to the next slide, Preston. I hope people can kind of
see this. The theme ?? the mascot for the kid\'s page is who we call
Energy Ant, and there he is right there. And this is the invitation.
This was the dreaded pop under. When they close their browser there
would be this cute screen \-- I thought it was pretty cute. And Energy
Ant says, \"Will you take my survey? It\'s six short questions. It\'ll
only take a minute.\" Okay. So I thought that was pretty not offensive.

Okay. And then the next slide shows the actual kid\'s page survey. And
there\'s Energy Ant kind of hanging out by the different questions, so
Energy Ant was in all the questions. So it says, \"Are you a student
teacher? What grade?\" And then at the end, you know, Energy Ant, I
think after you ?? after you submitted the survey, then a screen came up
with Energy Ant saying, \"Thank you for taking my survey.\"

Okay. So what did we find out? Results of the survey. Probably not
surprisingly, 70 percent were students, 16 percent teachers, parents are
5 percent, and the 8 percent other was things like museum curators, and
people who were resource specialists, who didn\'t fall in ?? or tutors
who didn\'t put themselves in any of the other categories.

Interestingly, 76 percent of the users were first time users, 48 percent
of the students were fourth to sixth grade, but that still leaves half
of the people in lower grades and higher grades.

A possibly disturbing finding is that 59 percent of the people found
what they were looking for. That\'s not unusual for a web statistic, but
it still horrifies me. That means that 41 percent of the people did not
find what they\'re looking for.

Okay. So the next slide \-- why didn\'t they find what they were looking
for? And we can tell, when we asked them what were you looking for, did
you find it? You can look at what they wrote. We knew going into this
survey that the kid\'s page, the navigation wasn\'t as intuitive as we
would have liked it to have been, and the survey confirmed that. People
said I couldn\'t find it, you know, I had to scroll too much.

A number of people came to the site and thought this was the main energy
site. Hey, it\'s energy, you know, and so they said I can\'t believe you
don\'t have tables on this, that, and the other thing. Well, of course,
we\'ve got ?? you know, you guys have got 100,000 HTML tables on the
main site, but they didn\'t know that there was a main site, so there
was a disconnect in where they thought they were, and what we ?? the
vast holding that we have.

And also, a smaller percentage of people wanted information that wasn\'t
relevant. They wanted like exercise energy, or title, and energy means
more things, or calories can be energy. So there were certain things
that\'s like get over it. We don\'t do that, but maybe we didn\'t make
it clear. The search engine may have put them here, but we don\'t do
that. The major problem was that the navigation was not intuitive.

Okay. Now you can stop for a second here. We\'re going to go off the
video tape. After we did the survey, we also ?? and for people who are
on ASA, you know that EIA is real big on doing usability testing, so we
decided that we were going to do some actual hands-on with kids. We
tested, myself and some other people in this room on the Usability
Committee, and also the Kid\'s Page Committee. We tested 43 people.
There was 37 students and 6 teachers.

Usability Testing in one sentence is, I would sit down with a student
and I ask them a series of questions \-- what state produces the most
coal? What do you think you\'d get if you click this button? You need to
define electricity. You know, can you find a definition, and I would
watch them work. And I would listen to what they were telling me. So 43
people is a whole lot more, it\'s a larger sample than we normally do in
our Usability Testing. And we noticed ?? actually, it was interesting.
We did the first testing on \"Take Your Sons and Daughters To Work
Day\", you know that day, so the building was crawling with kids, with
parents who didn\'t know what to do with their kids the whole day, so
they gave their kids to us for an hour. And we had a huge basket of
candy that was visible to the kids while they were doing the test, and
they knew when they were done they got to pick like three of these \--
not little Halloween candies \-- big candies, so that worked very well.

Basically, we were testing things like the links names, navigation, the
terminology. And I\'m not going to go into a lot of detail about a lot
of results of what we\'re doing, but just some interesting findings. One
of the links on the Home Page was Milestones, clever word. Sixth grade
and under didn\'t know what Milestones meant. When you would ask a
child, \"What does Milestones mean to you?\" \-- what do you think a
fourth grader said? \"What\'s a Milestone? It\'s a rock. It\'s a
stone.\" So if they were looking for a timeline or history, they didn\'t
know where to go.

Another one was that we had a button, a very enticing button called
\"Fun Facts.\" Well, \"Fun\" \-- kids like fun, and \"Facts\" \-- when
we asked them what state produces the most oil, they said well, I\'ll go
where the facts are. Well, \"Fun Facts\" was neither fun nor facts, and
we just ?? we learned that kids would go and they would be very
frustrated. The place where all the meat was, was not intuitive, and so
a lot of kids missed it. And we sort of knew that already. We knew that
in our heart. We knew that from the survey, and then we sort of
documented with the Usability Testing.

So what we\'re doing now with those results is, I was in a meeting
yesterday. We are actually ?? actually, I have a paper prototype today
of a new screen with \"Milestones\" \-- my kids even said did you get
rid of \"Milestones\", because they didn\'t like that word either. New
link names, more intuitive, shorter. And we\'re going to do a paper
prototype just asking kids where they would click, where they would
start to click, to see if the navigation even at the first level is
working.

And then we\'re going to work on the second level pages. People said
there\'s too much scrolling, they couldn\'t find stuff. Instead of
having things like in a sequence where there\'s 10 or 15 pages of
scrolling, if you have links at the top that show the different
categories, and they can ?? you know, science projects. You can jump to
the science projects rather than having to scroll to them, which is what
they have to do now.

So we ?? I don\'t know, I think we take it real seriously. And we\'re
actually doing a lot of changing. We don\'t just go oh, yeah, we did the
survey and I can\'t find it. I talked to somebody recently who said that
someone in their organization said well, the people are just stupid that
are coming here, you know. I mean, some people have that impression so
we\'re not ?? we don\'t want to do that. If it\'s not intuitive, even if
we think it is, if people are telling us it\'s not, it\'s not. It\'s
that simple. We have to fix it. Okay. So we\'re working on that one.

In 2004, in order to support the Strategic Plan, I\'m planning to do
another global survey, the one at our main website. Now I\'d like to
just say well, we\'re going to do a survey of all of our customers, but
a paper survey and a telephone survey in this day and age just can\'t
reach the numbers, and I don\'t think we\'re really reaching ?? our main
customers are web people, so I\'d like to do a web survey.

Now I have ?? since I was here all morning, I\'ve gotten some feedback
already. Okay. You can put the last slide up. My questions to you are ??
I like the pop under. Now I know Johnny suggested maybe doing a ??
sampling every Nth person, not sampling every person. I guess we decided
when we were doing this before that we were going to sample \-- not
sample \-- we were going to ask every person, but only for a very
limited amount of time. I don\'t remember why we decided to do that.
I\'m not the one that codes the survey, so I\'m not sure if there was a
technical reason why we didn\'t do that.

I like the pop under. I think, you know, if it\'s go to Mexico, you
know, those do bug you. But our customers \-- I know Neha said that, you
know, it\'s like she likes to have the opportunity to say something
good. There are a lot of customers out there that really appreciate
having the opportunity to respond.

And also, when we did a sample survey, but we were developing a frame
from which to draw samples, a number of people said, wait a minute. I
want to be called. I don\'t want to be in a sample where I might or
might not be called. I want you to call me. So they said please do an
electronic survey so that we can all be included. So, I don\'t know,
I\'m kind of justifying this pop under thing, but it works and I like
it.

But another idea, instead of a pop under, is to have some kind of a
banner, no flashing, no moving text, but something that would be on a
page that would invite them in a more active way, than say the little
button at the bottom. Maybe for a week or two, invite them to come do a
survey. They would still be in their face, but it wouldn\'t be the pop
under \-- so that\'s the first question.

I\'m going to ask all the questions. You all can answer any of them. The
second one is, what should we ask? And one of the questions ?? one of
the things that I thought was interesting is that you guys this morning
said we might want to ask how important is the EIA information to your
work?

The problem with that is, if you\'re getting third graders or the
general public who\'s coming in for the first time, that statistic might
not be as high as you might ?? you know, you might only say 40 percent
of the people say it\'s important for their work, so we might have to
think about that question.

In previous times ?? in previous surveys on the web \-- one that we did
in \`97 when we didn\'t really know too much about what we were doing
\-- we asked, is it easy to use, and does it meet your needs? And they
kind of said yes, no, no, yes, yes, no. Well, so if a bunch of people
say it doesn\'t meet their needs or it\'s not easy to use, what the heck
do you do now? So that\'s not actionable.

What we want to do, I want to have actionable questions. And I think
what were you looking for, and did you find it or not \-- if they say I
was looking for oil something or other, and no, I didn\'t find it. And
if 50 people say that, you know, it\'s going to start pinging you that
there\'s a pattern here. But anything else that you think we should ask
would be interesting.

Now from the statistical side of it, I have heard so much criticism of
web surveys, and how they\'re not valid samples, and they\'re
self-selected, and I know you guys might be familiar with the American
Customer Satisfaction Index, which is a survey ?? it\'s a whole system
that\'s run out of the University of Michigan. And they get a lot of
publicity, a lot of government agencies, including a lot of the
statistical agencies are participating in ACSI. They get their names in
the newspapers, they get their index score. I have other people in
statistical organizations that say I can\'t believe anyone would do that
because a web survey is not a statistical sample.

My feeling is, if I get 4,500 responses, I\'m hoping that the weight of
the numbers and the themes that come out of the numbers overwhelms the
fact that I didn\'t ?? that I don\'t have a true statistical sample,
comments on that.

Then the fourth question \-- I\'m sorry, I\'ve got a lot of questions
\-- that I just thought of today is, fortunately or unfortunately, on
the survey of the web that we did in 2002, 98 percent of the customers
said they were satisfied or very satisfied with what we had on the web,
98 percent. We are trying to come up with a target for the Strategic
Plan.

I\'m afraid to go with 98, because I think that\'s setting the bar high.
Say there\'s 10 grouchy people, and we could come up with 97, we\'re
going to get pinged. We\'re going to get an orange light, or a yellow
light, or you know, some color of light in the President\'s Management
Initiative isn\'t a good thing.

(Laughter.)

MS. BLESSING:  So I had decided that maybe 90 percent would be a better
target to go for, even though we got 98 before. A number of people are
saying you\'re trying to make more ?? your goal is to make more
customers unhappy? Why would you go with anything below 98? Anything 98
or above sort of scares me, so comments on where you go in the future
with a target when you\'ve already got almost the universe happy.

Okay. Any answers to any of those questions? Polly is first.

MS. PHIPPS:  First, did you have any sense of what the denominator is on
your pop unders and, you know, how you\'re getting 4,500 but, you know,
how many pop ups ?? you know, how many people are you ??

MS. BLESSING:  The denominator is ?? if we\'re getting a million a
month, take a million divided by 30, and that\'s ?? you know, the
denominator is really large.

MS. PHIPPS:  I mean, I think the problem with that, without using some
kind of sampling, is you just don\'t have a ?? you know, you don\'t have
a sense of what your problem is. And in these kind of customer
satisfaction surveys, they tend to be everybody is satisfied so, you
know, it\'s ?? you know, I would argue that some kind of sampling scheme
would probably be more useful in just getting at what you might want to
know about satisfaction and dissatisfaction. I would strongly agree with
asking those questions on, you know, did you find what you want \--
because I think the general satisfaction questions are just so biased
towards positive. But if you don\'t ?? I mean, you\'re probably getting
the people who answer those, you know, are likely the people who are
more satisfied.

MS. BLESSING:  See, I would have thought that ?? some people argue that
the people that answer are the people that are grouchy and mad. Who am I
sampling then? If I just do every Nth person, I\'m still getting a ??
I\'m getting a sample of still the random people that come to the site.
Who would I sample? I don\'t know how to find the ??

MR. BLAIR:  Well, one important thing you want to keep in mind \-- if
you start doing some sort of sample, and that is that what you\'re
sampling at the first level there is visits, not people; that is, that
the probability of a person being selected is proportionate to the
number of visits that they make to the site, if you\'ve got some kind of
random selection procedure. So if you want to translate those visits
into people, you\'ve got to get some kind of an estimate of frequency of
visit, or some way to weight those data to turn it from a sample of
visits into a sample of people. So one of the things to think about, you
know, fundamentally, is what is it that you want as your unit of
analysis. Is it visits, or is it people?

MS. BLESSING:  Okay.

MR. FEDERER:  You could ask them, actually.

MR. BLAIR:  Yeah, sure. I mean, that\'s typically the way it\'s done.
That\'s subject, of course, to some measurement error \-- how well
people can answer how often they visited a site.

MS. BLESSING:  Well, I did ask them, is this your first time? Do you
come more than once a week, more than once a month, you know, every so
often, or this is your first time. I did ask them that, but then I
didn\'t really correlate the data back to the ??

MR. EDMONDS:  You can just use a cookie to tag the machine.

MS. BLESSING:  Can\'t do that.

MR. EDMONDS:  Oh, you\'re not allowed to use cookies?

MS. BLESSING:  No. I don\'t know. Is there anybody here that\'s from IT?
The government is not allowed to cookie people.

MR. EDMONDS:  Oh.

MS. BLESSING:  My understanding is that ??

(Laughter.)

MS. BLESSING:  And I don\'t think that\'s negotiable. Isn\'t that right?
I don\'t see any IT people here. I mean, that\'s a policy. It\'s not
just EIA. It\'s the whole government.

MS. KHANNA:  I have a question about your pop under. Does it come out
only when you close the browser, or does it come out when you open the
EIA website?

MS. BLESSING:  It comes out when you close your browser. And the reason
for that is, I didn\'t want to have the invitation to the survey come in
the minute they opened the EIA website, because for the people who
haven\'t been there for a while, or it\'s their first time, how are they
going to answer unless they\'ve had at least one experience going
through the site, so it comes up after they\'ve closed. Now I might know
where you\'re going.

They could come to EIA, and EPA, and whatever, and three hours later
when it\'s time for lunch, they close their browser and there\'s this ??
you know, they\'re thinking what site was that, you know, if they were
hopping around. It\'s possible that they didn\'t even remember which
site they were answering for.

MS. KHANNA:  Well, I\'m one of the people that keep the browser open all
day long, and the only time I shut it off is when I turn down my
machine, and that\'s when I\'m going home. I\'m not going to fill out a
survey.

MS. BLESSING:  Yes.

MS. KHANNA:  But then on the other hand, they may be ?? I don\'t know
what fraction of Internet users actually open and close browsers and
keep them open. That\'s ?? I\'m just pointing out one group that you
might miss.

MS. BLESSING:  Right. Another critique that a friend of mine, who works
in Justice statistics, who\'s very adamantly against the
non-statisticalness of the web surveys, said that what I might want to
do is \-- we have listservs. We have email lists that go out to targeted
customers, and I could sample them. But then I think I\'m already
getting ?? I think I\'m already biasing the survey, because those are
already our valued customers. They might be able to answer some
different kinds of questions, but asking them if they\'re satisfied \--
hey, I already know they are.

I want to get, you know, Joe Q. Public out in Kansas that I haven\'t ??
and I don\'t know how to reach those people except random ?? sort of
randomly. I know which of you guys is next.

MR. FEDERER:  One way to improve response rate is use incentives. I
don\'t know if you can send them some printed material if they bother to
respond, but it\'s still not ?? I mean, it probably raised the issue of
the denominator. I think you never get something that will satisfy us
statisticians.

MS. BLESSING:  Uh-huh. I was afraid you\'d say that. I was just going to
say responding, if I had to send out 4,500 Hershey bars or something
like that, it wouldn\'t work.

MR. FEDERER:  Energy Bars.

(Laughter.)

MS. BLESSING:  Hey, I like that. In theory, that\'s a good idea. On the
telephone surveys, we did send them out a thank you packet if they asked
for it. In the telephone survey, we were only talking about 100, 200
respondents, not 4,500. Yes?

MR. BURTON:  I have two questions, one for you, and one for the people
that know a lot more about sampling than I do. First of all, do I
understand correctly that you gave candy to a bunch of kids, then
returned them immediately to their parents?

(Laughter.)

MS. BLESSING:  Yes, actually we did.

MR. BURTON:  With regard to the question about the web survey, does the
fact that a high percentage of their users access the information that
you provide through the web lessen sort of the onerous nature of web
surveys?

MS. BLESSING:  Is there a yes on that one?

MR. HAMMITT:  My comment was that what you\'re doing is surveying people
who visited your web site. And subject to the point here about people
versus visits, that seems like a reasonable population, if you\'re
trying to see does that website work. You don\'t want a random sample of
people to be less. You want people who looked at it.

MS. BLESSING:  People who come. And actually, all kinds of people, our
valued customers, our business customers, our public customers. And I
think if I picked a sample, I would be already targeting people that I
know, and I already know what I think their answers will be. Oh, I
don\'t know. I\'m sorry.

MR. HENGARTNER:  One comment, in terms of usability. There\'s another
way really to answer these questions. Google, you type the keyword in
Google. It searches them by the number of times a person actually ??
they give you a list and you click. Each time you click it reports that,
and then if all at once everybody wants that site, it moves up the
ladder.

MS. BLESSING:  Uh-huh.

MR. HENGARTNER:  Now usability, putting the words in Google and see
where your website comes up. That shows you how relevant your website is
to specific keywords. Now it\'s an indirect measure, but it\'s a help.
It\'s a good way to get a good crowd without ?? they don\'t use cookies.
That would be an interesting way to see how relevant EIA products are
with response to certain keywords.

MS. BLESSING:  We do ?? I actually do that just for fun sometimes.

MR. HENGARTNER:  Okay. I thought it was unique.

MS. BLESSING:  I think Google is also looking at ?? I mean, the search
engine optimization, people will tell you it\'s looking for key words on
the page and things, so it\'s not just the popularity of the site, even
if it\'s crummy. I think Google is also looking for the words in
context, but you\'re right. And I look at that a lot. I want to make
sure we\'re popping up. Thanks.

MR. BREIDT:  All right. Next we have Howard Gruenspecht, Deputy
Administrator, who will be talking about Stewardship.

MR. GRUENSPECHT:  Good morning, or should I say good afternoon? I guess
I\'ll say good afternoon. I\'m Howard Gruenspecht. I\'m the Deputy
Administrator at EIA. I guess the EIA is sort of unique, in that there
are not a lot of political appointees, there\'s only one \-- so Guy is
the Administrator, and he\'s a political appointee, and I\'m the Deputy.
And I\'m a career person. In part, that means I\'ll be here hopefully a
long time, unless I do something really bad. And Strategic Plans are
inherently long-lived activities, so it\'s quite natural that, you know,
I would play a key role in stewarding a Strategic Plan.

And I guess what I want to discuss with you today is give you a little
bit of overview of some of the activities we have going on, share some
thoughts with you, and pick your brains a little bit. You know, to
steward is to watch over, advocate, assure the continued well-being of
and, you know, the history of Strategic Plans is often they don\'t
really have ?? they\'re not maintained in a very healthy state. They
often kind of sit on the shelf. They don\'t get much exercise. They
often die alone and unnoticed.

You know, the other definition of steward is to bring the wine to the
table, which I don\'t plan to do, so I really view this first role of
stewardship, you know, of watching over, advocating, and assuring the
continued well-being of the Strategic Plan, as the role I want to
undertake.

Here is a little bit of a schedule of sort of ideas. At this critical
phase, we\'re really just trying to launch the Strategic Plan. As most
schedules are done, in fact, we\'re already a little bit off, but we\'re
off in sort of a good way, actually. You know, the beginning of this
plan, and there\'s been a lot of time spent internal to EIA on this, but
it is very important to reach out. And I know one of the things that\'s
being done here, especially with this group, is the focus on Goal 1, and
talking about some of the metrics we would use.

The plan, the schedule here indicates that we\'re reaching out to the
employees and to selected customers and stakeholders. In fact, the good
news is we\'ve reached out to the employees, and we\'ve gotten such
interesting responses to date from the employees that, in fact, I think
some of that input is going to be incorporated in the plan before we,
you know, reach out more fully. Obviously, we\'re taking advantage of
the opportunity of the ASA to reach out to you in the areas of, you
know, your particular expertise, so we\'re not quite out to everybody on
10/9, but we started the process, and we\'re going to sort of move ahead
with it.

Again, there\'s a ?? you know, I don\'t have to talk in detail about
each of the steps. You know, a very important part of the Strategic Plan
process is developing the action plans. I mean, a Strategic Plan without
action plans is really an empty shell. And we are already hard at work
in developing those plans.

In some areas of the Strategic Plan, and I think I gave you the overview
of the goals. You know, Goal 1, you know, you\'ve heard a lot about
today. Goal 2, the Resource Question, is a very sort of tricky question.
And exactly how to proceed with those, you know, plans \-- we know where
we want to end up, but laying out your end point at the beginning of
what is, in Washington, a very complex process, is probably not the way
to get to your end point. So we have some challenges in sort of, you
know, working on the action plans, and how much information we\'re going
to include in those plans. We may have sort of internal and external
versions of those action plans, but the idea is basically to, you know,
move from the initial Strategic Plan, get comments from you, from the
employees, from other stakeholders, move to the action plans at the
corporate level, bring those action plans then down to the individual
office level, you know, have the stewardship information system put in
place, and then really get going, if you will, with our Strategic Plan.

I think the different goals in the plan probably require different kinds
of stewardship on my part, and that\'s something, I guess, we could talk
a little bit more when we talk about some of the strategies. I think in
Goal 1, many of the ?? Goals 1 and 3, much of the activity, or almost
all of the activity in terms of making the Strategic Plan live, will
really have to take place, you know, in the offices of EIA that do the
work. And, you know, what we really need there is a mechanism for the
offices to interact with each other, and interact with the
Administrator\'s office, to sort of make that happen, and to make the
cross-office activities that are necessary to implement the plan happen.

In Goal 2, we are going to need a lot of work, you know, in the offices.
Particularly, the work there will be helping, I think, our customers and
stakeholders really understand what the value of, you know, the EIA
activity is. I think that\'s something that is really not well
understood.

When somebody wants a number, they\'ll call us. I mean, that\'s
understood. But beyond that, in terms of really what it takes to produce
reliable numbers, and how the difficulty of producing those numbers has
been affected by changes in the industries in the energy sector, is
something that I think is not well understood. And I think sort of, you
know, understanding where the numbers come from and what it takes to
make them reliable is something that our customers and stakeholders
don\'t have. And I think that\'s critical to, you know, success of
reaching out in Goal 2, and getting the resources that we need. But, you
know, I think Goal 2 will take a little bit more than just coordinating
the work of the offices because, you know, fundamentally it will involve
reaching out to the people who can make a difference, you know, in the
resource end of things. And that\'s something, for better or for worse.
I don\'t think you can have, you know, tens of people running around
town doing that.

I think that area is a place where stewardship will actually involve
sort of taking on the responsibility of doing more in that area than EIA
has historically done. I think it\'s a very tight environment that we
face. I think it\'s a very tight environment for federal discretionary
funds. It\'s only going to get tighter, given the budget situation.

I guess the good news is that EIA is a relatively small activity, and
even in a context of a very tight budget environment, I guess we believe
that if we\'re good enough at making the case, we\'ll be able to get
some additional resources.

So again, the different goals I think will require different kinds of
stewardship. And, you know, we\'re going to make significant efforts to
do that. Could you move on to the next one?

This is an interesting picture. You know, process is very boring, but I
think it\'s the process that really makes or breaks, you know, a
Strategic Plan, you know, and have it be a living document, versus one
that sits up on a shelf and gathers dust. I think there are really three
types of strategies that one can use, and this is an illustration of one
of them; which is, to tie the Strategic Plan directly to activities that
managers and staff pay attention to.

And what this sort of complex diagram shows is that if you look at the
two red stars, you know, we envision sort of two times in the year where
it will be sort of important to sort of tie the Strategic Plan to the
activities that go on at EIA. And those are sort of the two regularly,
what we will consider to be regularly scheduled places where the
Strategic Plan, you know, is pursued and monitored.

One is at the end of the year. I mean, those of you who have federal
experience know that there\'s a periodic at the end of fiscal year
review process. The individual office, you know, directors would be
reviewed on the performance of their offices. That would tend to be sort
of the individual contracts between the office and the Administrator\'s
office. But it turns out that in those reviews, we suspect that some
issues will arise that are inherently cross-cutting across the offices.
And traditionally, there\'s really been no place for those issues to
come up.

Another thing that occurs, supposed to occur before the end of September
but rarely does, is that you figure out what your budget is for the
upcoming year. I mean, the tradition, you know, is that they pass all
the appropriations bills but, in fact, they don\'t pass all the
appropriations bills. You get a continuing resolution. But we, in fact,
expect that in most years, sometime around December, we would have a
handle on what our appropriation would actually be. So we have two
things that we have in December really. We have the results of the
reviews of the individual offices and those, you know, will identify
some cross-cutting issues that may come up. And we have the, you know,
the budget that we\'re getting back, which may or may not, and often is
not the same as the budget we submitted. So there are really important
decisions at a corporate level to be made and those, you know, either in
addressing the sort of cross-cutting issues, or in dealing with budget
surprises, both positive and negative.

And the idea of sort of bringing the Strategic Plan, you know, into
force and meeting in December will help address, you know, apply the
Strategic Plan to addressing those cross-cutting issues, and in
addressing the allocation of resources that\'s likely to be somewhat
different than what we expected.

The other time in the year that the Strategic Plan will be important is
when we\'re proposing our budgets. I mean, the question is, you know,
traditionally you have each office coming forward to the front office
and kind of saying these are the things I\'d like to do, these are the
things I would do if I had more money. And these are handled
individually, not as part of a corporate process.

What we\'d like to do is have a discussion among the offices, again
guided by the Strategic Plan, to see if we can work out a more
corporate, you know, approach to prioritizing projects that would carry
out the strategic plan. So, you know, we have these two focal points of
tying the strategic plan to the budget process which, you know, my
experience in the federal government tells me is a process that people
pay attention to.

You know, in addition to this kind of approach, we would have other
meetings throughout the year, as appropriate, as issues come up. But we
want the benefit that would bear on the Strategic Plan, but the notion
is, schedule things so that they tie to the budget process, have
unscheduled activities, as required, to deal with the corporate
cross-cutting issues. So that\'s sort of the process, if you will, in
terms of, you know, getting the group together, so to speak.

The other thing we\'re going to do that\'s a little unusual, is we\'re
going to have staff in the Administrator\'s office that has a continuing
responsibility of following the Strategic Plan. That\'s not something
that has historically, you know, been done. And we think that is
important to keeping the Strategic Plan alive \-- someone who is
responsible on a periodic basis for taking it off the shelf, opening it
up, dusting it off, if you will, so it doesn\'t end up in the role of
the traditional Strategic Plan.

A third sort of strategy that I have in mind, you know, for stewardship
of the Strategic Plan is to take advantage of the demographic
transition, you know, in EIA. In some respects, that\'s a bad thing,
demographic transition at EIA. We\'re going to lose people who have a
lot of experience and knowledge about their areas, and replacing them is
a real challenge.

On the other hand, you know, you have an opportunity, when you\'re
looking at hiring and promotion decisions, to sort of take account of
the fact of, you know, who takes sort of the goals of the Strategic Plan
seriously and, you know, wants to move them forward, versus who is sort
of very, you know, narrowly focused on a sort of existing program or
mission. So my hope is that by focusing, you know, by making it clear
that hiring and promotion decisions in some sense, that one of the
things we\'re going to be looking for is commitment to the Strategic
Plan, you know, we\'ll be able to get a lot of buy-in from the ?? you
know, the middle management level that in many respects on a day-to-day
basis really makes EIA go.

So again, it\'s a combination of some kind of process, of a dedicated
resource to follow the issues, you know, and this emphasis on the
Strategic Plan in looking at, you know, decisions that we make on
personnel in EIA that will hopefully, you know, make this Strategic Plan
different from all others.

Is it really going to happen? You know, well, we\'ll know in five years
or four years. Some of you will be here, some of you won\'t. You know, I
am committed to try to make that happen. Again, I have a pretty long
horizon, as I said early-on. And, you know, we look forward to working
with you, particularly in the Goal 1 area, but in other areas, as well,
where you have expertise. And, you know, we\'re very open to any
suggestions that you have, you know, maybe from planning experience in
your organizations as to how to make this work. Thank you very much.

MR. HENGARTNER:  How is the Strategic Plan looking into changes in the
energy market, changes of demand? Are you going to look at it
retrospectively, or are you going to try to look into a crystal ball and
try to see what\'s going to happen in the future?

MR. GRUENSPECHT:  I think that\'s a good ?? you know, in many respects,
we are catching up with some of the changes that have occurred, you
know, in the past. You know, I guess your point would be that we
wouldn\'t be in that position, so to speak, you know, if we were looking
a little bit more forward. You know, our thought is, I guess under Goal
1, you know, improved contacts with customers and stakeholders,
hopefully would enable us to get off the mark a little bit more quickly
in some areas than we have. Our reaction time ought to be improved, I
guess is what we\'re trying to say.

We\'re probably in a very tight, you know ?? I guess the challenge is
how much resource in a tight situation do you devote to kind of
speculative thinking about possible future changes, versus reacting to
the situations that we find ourselves in a number of sectors; like
natural gas, for instance, where our industrial ?? you know, I\'m sure
you\'ve heard about this in many meetings that coverage industrial
natural gas price data, or natural gas storage \-- something that we ??
we didn\'t have a weekly natural gas storage survey. It\'s very
important to the market. The American Gas Association decided to drop
their survey, you know, we ended up picking it up \-- something that the
markets follow, you know, religiously, I would say \-- get an
illustration of the importance of EIA\'s work.

But again, a reactive decision, you know, where someone else in the
market made a decision to drop something. We ended up picking it up at
the Secretary\'s request that we do some other things on natural gas.

Again, I\'m not sure we can always predict or project what these
challenges will be. Should we devote some effort to it? Probably, yes.
How much? You know, that\'s an interesting question. You know, I don\'t
know the answer.

MR. EDMONDS:  Howard, you know, in the 15 minutes that they gave you to
describe the plan and how it\'s going to get implemented ??

MR. GRUENSPECHT:  Oh, I didn\'t realize I was supposed to describe the
plan. I thought Guy did.

(Laughter.)

MR. GRUENSPECHT:  I\'m in deep trouble here. Did Guy go over the plan?

MR. EDMONDS:  I was wondering if you would highlight the things that you
see as being the most important elements that are new and breaking new
ground in the Strategic Plan.

MR. GRUENSPECHT:  Well, I think EIA has probably, you know ?? we\'ve
always focused on information program that\'s relevant, reliable, you
know, consistent with changing industry structure, high quality, and
timeliness. So that ?? we want to do a better job. That I don\'t think
is really new, you know. Obviously, with something we\'ve been doing
before, we\'d like to do it better.

The focus of the Strategic Plan, you know, specifically calling out the
resource space as being sufficient to accomplish its mission, I think is
not something that EIA has called out, if you will, you know, as one of
the top tier elements in a plan before. We\'ve kind of sort of rested on
our laurels a little bit and, you know, EIA was well-funded in the 70s,
I think, when it was set up. You know, went through a couple of bumps,
but we\'ve basically been pretty quiet in terms of on the resource side,
and not really pursued aggressively, you know, matching the resources to
the mission. And we\'ve tended to sort of take on the mission,
continuing everything we\'ve done in the past, plus picking up new
things as requested, like the gas surveys, like additional gas work
we\'re going to do, like, you know, whatever it is, like the
restructuring in the industry. But we\'ve always sort of added more and
never really worried about subtraction.

Ultimately, the resource base we can\'t control. We can only make our
case, and we\'ll get what we get. And we\'re in a tough environment. But
I think we can control matching the resource base, you know, to the
work. And I think as we take things on, we have to consider dropping
other things off. And that\'s not something we\'ve done. And that\'s
something that I think is relatively new in this regard.

I don\'t know if I can talk about it. I think for the first time, the
bunch of packages that went over to OMB explicitly indicate that at
certain funding levels, there are certain surveys that we\'ve done that
were ?? you know, we don\'t think we can do at a certain budget level.
So I think the idea of, you know, defining ?? pursing a resource base
that we think is appropriate, but at the same time recognizing that
whatever resource base we get, we just ?? it\'s not silly putty. We just
can\'t keep on spreading it thinner and thinner across, you know, and
try to do everything. And then, inevitably, you hurt yourselves on
quality, you hurt yourselves on timeliness.

At some point, the idea of the resource base and the mission matching up
is, I think, something pretty new. I would say it\'s very different than
what we\'ve done in the past, where we just kind of said yes, we\'ll
take on additional, we\'ll take on additional, we\'ll take on
additional.

You know, I guess the EIA employees rating EIA high in the areas of
leadership, management, meaningful work, rating ourselves high in
motivation and productivity, I\'m sure we\'ve had goals similar to that
in the past. I think now they\'re probably taking on more importance,
you know, at a time when EIA does have this demographic transition going
on. I mean, you know, for inside baseball people, you know, a lot of EIA
staff have been here a long time, since the inception of the
organization. A lot of them are involved in a retirement system that has
?? might be said to involve golden handcuffs. You know, people are going
to stay and do their work. They\'re well-skilled, but their happiness or
not happiness \-- it always mattered.

It\'s always good to be in a happy work place, but in some sense, you
know, we did not have to attract as many people as we, you know, need to
attract now. And I think in the modern, you know, the new systems and
all that, I think, you know, how well people feel fulfilled in their
work is going to be more important in retaining the people that we need.
So I don\'t think it\'s different. It\'s not like we weren\'t for these
goals, or didn\'t pursue goals like this in the past. But I think these
goals matter more in the environment we\'re headed into. I guess that\'s
what I would say.

MR. BREIDT:  Okay. All right. So we\'re right on time now to request any
public comments. Yes.

MS. WAUGH:  Shawna Waugh with EIA. My comment relates to stakeholders.
And in the Strategic Plan, two of the major stakeholders seem to be
suppliers of data, the respondents themselves, and also the contractors
who do a lot of the work here at EIA. And I didn\'t really see anything
in the Strategic Plan that addressed those stakeholders as important
people in terms of our providing reliable and accurate data.

MR. BREIDT:  Thank you. Does anyone have a follow-up comment to that, or
any other comments? Okay. I guess we\'ll break for lunch then. Thank
you.

(Whereupon, the proceedings in the above-entitled matter went off the
record at 12:23 p.m. and went back on the record at 1:39 p.m.)

MR. BREIDT:  All right. One break-out group will select and implement
estimation procedures, will stay in this room, and the industrial
natural gas prices will be in Room 5E069.

MS. CAMPBELL:  Good afternoon. My name is Elizabeth Campbell, Beth
Campbell, and I\'m the Director of the Natural Gas Division. And my
purpose here this afternoon is primarily to give you some lead-in and
some background for this afternoon\'s break-out sessions.

In addition, I\'m going to talk very briefly about some of the new
initiatives which have been proposed by the Secretary and the National
Petroleum Council, which I think will provide also some relevance for
the discussions this afternoon, talk a bit about the work we have
underway and move on.

For those of you who are not familiar with the natural types of data and
frequency of data available to discuss the natural gas industry, I\'ve
provided this overview table. Along the top, you can see the frequency
with which data are available, and along the left-hand side you can see
the types of data that are provided.

There are annual, monthly, and weekly data. As you can see, there\'s
only one weekly data series, which is the underground storage data
series, which helps to explain the amount of attention it receives. And
then there are a number of annual and monthly data sources. Not all of
these sources are within the Natural Gas Division. Some of them are
derived from other parts of EIA, and others are derived from outside of
EIA. For instance, the import/export information comes from the Office
of Fossil Energy, a piece of the price data comes from the Federal
Energy Regulatory Commission. There are just a variety of data sources
implied here.

In addition, I\'ve said annual and monthly. You will see some data form
numbers appearing in both places. In many and, in fact, most cases there
are actually annual and monthly schedules that distinguish these two
types of frequencies of data collection.

Today\'s presentations will be focusing on the first of these rows \--
production, the last of the rows \-- prices.

This past spring, Secretary Abraham announced a Natural Gas Data
Initiative. Now I think most of you are aware of the fact that the
natural gas industry has been experiencing great change in the last
decade or so. This has been a time of deregulation and market
restructuring, relatively strong demand for natural gas. I\'m choosing
this word carefully \"flat supply\". Some people saying rising, some
people say falling, I\'m just saying flat supply, at least, and volatile
prices. And there has been a considerable concern about understanding
the overall gas picture.

When Secretary Abraham announced the initiative, he focused on a few
particular program areas. One of them is the issue of understanding
natural gas production. You\'ll hear something about the work that\'s
underway in that area. A second is, he asked that the Office of Fossil
Energy investigate the option of more timely import/export data. While
we receive monthly data estimates, we receive it only after the close of
a quarter. So, for instance, we are ?? right now we have not yet
received the file for the April data. They just released it to the
public, and they\'ll ship it to us as a file shortly.

And the next one was to initiate a survey of on-system liquified natural
gas storage, which has become, I think, increasingly important in
certain areas for peak demand. The third area was that he asked us to
continue to work on improving our end-use price series. And you\'ll hear
something today about residential, commercial and industrial price
series. And then finally, we were charged to do something to regionalize
the forecast in the short term energy outlook, in part because I think
the feeling is that the natural gas markets, and perhaps some other fuel
markets are geographically separate enough that describing them from a
national perspective only isn\'t a complete picture.

In addition, the Natural Petroleum Council has recently completed a
first draft of its report to the Secretary, and it included a section on
recommendations to promote efficiency of the natural gas markets. And
these were recommendations that were included in that report that are
relevant to EIA\'s Natural Gas Program, not necessarily just the
division, because as I\'ve said earlier, some portions of the Natural
Gas Program are performed in other parts of EIA.

Most of these you can see here, production and consumption data more
timely, more complete storage data, more timely storage data, fuel
switching data, so these are other elements of concern that the Natural
Petroleum Council has identified recently.

So these are some of the things that we have been doing in the past
year. You\'re going to hear something about the work we\'ve done in the
production area shortly, when it refers to a marketed production, but
there\'s a second issue that has been begun, and that\'s, we have
initiated a new survey section in the natural gas liquids production
report. This is actually part of the petroleum reporting supply system,
to determine whether or not we can get monthly estimates of extraction
loss, which would enhance the accuracy of our dry production numbers by
using this new data source.

We also have seen this [Federal Register]{.underline} notice released by
the Office of Fossil Energy with regards to the examination of using
monthly reporting, rather than quarterly. We haven\'t heard the outcome
yet.

We have published a [Federal Register]{.underline} notice requesting
comments on the new LNG survey. We have implemented a new processing
system for the monthly and annual underground storage surveys. We have
also made the alignment last spring with respect to natural gas
historical data series, aligning it to other data series on industrial
sector and electric power generation contents.

We have made changes in our price series affecting the residential and
commercial sector. That\'s a break-out session this afternoon on the
910. We have also implemented the use of the new EIA 176 subject that
you\'ve heard about in earlier briefings here.

We still have work underway in production that you\'re about to hear
about. We do have work underway in reviewing the estimation methodology
for the weekly storage survey. We still have to resolve the issue of
clearance and implementation of the LNG survey.

In the next slide, I provide a little bit of perspective, additional
perspective on this proposal. That comment period ends November 17^th^.
If any of you are interested in providing comments, we would be glad to
receive them.

We plan, under this, to collect data from active LNG storage facilities
in the United States on those particular series presented here. Based on
the data that we have seen thus far, we cannot present as much detail on
the LNG market as people might like, because of some confidentiality
disclosure issues, the relative structure of the market.

We also have some further work underway with respect to understanding
our basic monthly sample-based consumption survey, the one from which we
get end-use sector prices and deliveries to end-use sectors. We want to
try to move that into our standardized processing system this year, and
we need to make some basic analyses about how that survey works before
we make decisions about moving into the new processing system.

We are also investigating the idea of adding possible more states to the
marketers\' survey. That\'s one of the break-out sessions this
afternoon, and we are looking at the issue of options for industrial
sector price estimation again this afternoon.

For those of you for whom I was talking in jargon, here\'s a brief
analysis of what those forms are. Okay? Any questions?

MR. BREIDT:  Okay. So we are now ready for the break-outs, and you can
go ahead and head off to your sessions, those of you on the Committee,
or anyone interested in going to the industrial natural gas prices, it
will again be meeting in Room 5069.

(Whereupon, the proceedings in the above-entitled matter went off the
record at 1:49 p.m. and went back on the record at 1:51 p.m.)

B-R-E-A-K-O-U-T S-E-S-S-I-O-N

(1:40 p.m.)

MS. NORMAN: Criteria to Select and Implement Estimation Procedures. A
subtitle of that is \"Comparative Evaluation of two methods to estimate
natural gas production in Texas\".

MR. NEERCHAL: Could I ask you one question?

MS. NORMAN: Sure.

MR. NEERCHAL: Do we have a copy of your slides?

MS. NORMAN: The paper should be available. The slides, no. Okay. We\'ll
start off with a little bit of background summary. At a prior ASA
Committee Meeting, the natural gas production estimation method that is
currently, or was currently at the time, utilized by Dallas was
discussed. Linkletter, Sitter and Vetter proposed an alternative model.
Some people might remember that as the Crystal model; it was Crystal
Linkletter that helped to propose that.

The models were compared using an evaluation of methods, and a model was
selected. Today we are here to examine the criteria of that evaluation
of methods for both credibility and future usability. Basically, we are
here to evaluate the evaluation.

A summary of the methods, and then I will go on and go through the
evaluation of methods, and give a little bit of feedback as to how that
was implemented. But again, just try to keep in mind throughout the
discussion that we\'re basically trying to focus on the evaluation
method itself, despite the fact that I\'m going to give you some
background just to sort of catch you up on how it was used.

The Energy Information Administration publishes state-level natural gas
production estimates, both monthly and annually. Texas is the largest
producing state. In 2001, 27 percent of the U.S. total was based out of
Texas, so timely information is important to EIA and its customers.

The principal source of Texas natural gas production data, measured as
gross withdrawals, is the Texas Railroad Commission. Natural gas
production values are posted on the TRC website, between 45 and 60 days
after the close of a report month. The initial values are revised
monthly for about 24 months, and even occasionally thereafter. The
posted production values or the P sub i, start out and approach their
final values over those two years.

Figure 1 on the following slide will illustrate this reporting pattern
for the month of May of 2001. The diamonds indicate sequential reports
from the initial through to the final values.

Along the X axis, you can see the reporting months, and along the Y is
the reported production values in billions of cubic feet per day. As
time increases, the initial values are revised until the \"final\"
values are reported around 24 months after the close of the month in
question. Comparing the final value with the initial value, one can
produce a weight to aid in the estimation process.

For EIA\'s purpose, the publication in the [Natural Gas
Monthly]{.underline}, we would use the third value \--the initial plus 2
revisions \-- as the starting point for estimation of the final value.
Both methodologies, both the one that was formerly used by Dallas, and
the other one that we were comparing, use the same data, and require the
same data preparation as previously described. And both methodologies
estimate the weight, this W3, in making an estimate.

The reported volume data approached the final values according to a
relatively stable pattern or curve, as you saw before, over those 24
months of data. This model\'s historical pattern is applied to recent
production months for which reports may have been revised up to 23
times, or every month. It attempts to account for changes in the
relationships between the preliminary and final data over time. This
model produces estimates that are usually within 1 to 2 percent of the
final production value. A detailed discussion of the current model is in
Appendix 2 of the associated paper that was handed out.

The alternative method that was proposed, the multinomial model, is a
statistically rigorous multinomial model. It is based on a distribution
model of the reporting patterns observed in the data. Similar
multinomial models are used both in AIDS research and warranty claims.

The model assumes that all gas produced in a month will be reported in
one of the following 24 months. The model also assumes that the
reporting patterns remain constant over the most recent end months. The
model produced more accurate results with M equal to 6, although we did
also try M equal to 9. A detailed discussion of that model is in
Appendix 3 of that paper.

Again, these are the questions that I have, and I kind of want you to
keep in mind as I go through. On the following slide, I will go through
the guidelines for the evaluation of methods, but these are the
questions that I kind of want you to keep in the back of your head as I
go through these. Are the outlined criteria adequate and appropriate?
Was the Evaluation of Methods applied correctly and sufficiently? Would
\-- or more likely, how should \--the criteria change, based on the
comparisons of different methodologies? And also, sort of as a side
note, although this isn\'t directly related to the evaluation of
methods, it is related to the model that we chose based on the
evaluation of methods. And so now that that multinomial method has been
selected, we are seeking your guidance on what improvements could be
made to the chosen estimation procedure.

As I continue further on, you\'ll see that we think that we might have a
little problem with bias, and if you have any suggestions for how we
could improve that, that would be very helpful.

All right. On to the evaluation of methods. These are the guidelines.
EIA\'s information quality guidelines state that all estimation methods
are to be transparent, reproducible, and provide both timely and
accurate results. The goal of the evaluation of methods is to determine
which of any two competing methods more closely meets these
requirements. Ultimately, the evaluation of methods would be used in the
future to compare new or improved methods as they are proposed.

Reviewers will be asked to examine model documentation and Appendices 2
and 3 for this particular document to assess the transparency and
reproducibility of the models. Once adopted, the same methodology will
be used to produce estimates of natural gas production for publication
in the [Natural Gas Monthly]{.underline}, unless a new methodology is
adopted using this procedure, the evaluation of methods. Methodology
will be documented for each estimate, and the code and data used to
generate these estimates will be archived.

Estimates for state level data should be completed within a set number
of days after the close of reference month to allow timely publication
in the NGM. Accuracy will be assessed by comparing volumes estimated for
a month to be best final monthly data available. Estimates published in
the NGM and NGA, the [Natural Gas Annual]{.underline}, will also be
compared to best final data available for the same three-year period.

These comparisons will help EIA to assess current methods, as well as
illuminate discussions of revision policy for natural gas production.

The following criteria will be used to assess accuracy. The volumes
estimated for a month will be computed using only data that would have
been available at that particular time. If it is impossible to use only
the data available at that time the estimate would have been prepared,
then all parties will agree to alternative data sources in advance.

Both estimation methods will be run using exactly the same data sets
each month from January of \`99 through December of 2001 for this
particular example, and predictions of natural gas production will be
computed for each month.

For each method, the monthly estimates will be compared to the final
monthly values. The error will be computed as the final value minus the
estimate. The percent error is the error multiplied by 100, and divided
by the final value. And the following summary statistics will be
computed: error by year, percent error by year, error for three years
from \`99 through 2001. And percent error for those three years for each
of those four statistics will be calculating the average mean, absolute
deviation, mean squared error, and the maximum and minimum values. Time
series plots comparing the final data with estimates will also be
prepared, and comparison of these statistics will allow reviewers to
assess which method produces more accurate results.

The multinomial method was compared to the parametric method, the
alternative one was compared to the former one, to assess improvements
in accuracy and predictability during the month of August, 2003. The key
results of the comparison are summarized in Table S-1 and Figure S-1,
and the details are provided in Appendix 4 of that paper that was handed
out.

This is one of the summary tables outlining the resulting statistics
that were requested in the evaluation of methods. You can see there the
four main statistics that I listed before, your error, percent error,
error over three years, and the percent error over those three years,
and the values for both the multinomial model and the current parametric
model. And then this was a graph displaying the percent error over time
for both methods. The pink is the multinomial, and the parametric is in
the darker blue.

The accuracy and quality can be assessed by examining the summary
statistics for both methods. Transparency can be assessed by reviewing
the descriptions of the two methods for clarity, and reproducibility is
achieved by maintaining archived versions of the exact code and input
data used to produce the estimates. Using the same procedure regularly
without manual intervention enhances the reproducibility.

Both models provide natural gas production estimates for gross
withdrawals that are usually accurate to within 1 percent. For the 36
months from \`99 to 2001, the multinomial model produced 36 estimates
with errors less than 2 percent, 28 of them with errors less than 1
percent. In contrast, the parametric model only had 19 estimates with
errors less than 1 percent, and 35 with errors less than 2

The results show the multinomial model produces a lower mean squared
error, a lower mean absolute deviation, smaller magnitude of the largest
error. In addition, the multinomial model provides mathematical theory
for the reporting pattern, which allows for the estimation of prediction
intervals. These estimates showed that all 36 estimates for the
multinomial model were within the 90 percent prediction intervals.

The smaller estimates of variation, mean absolute deviation, and mean
squared error indicate the multinomial model is more accurate.
Furthermore, the multinomial model can be executed in about 5 minutes,
and is not expected to require the application of expert judgment.

Prior to August of 2003, the use of the current or parametric model
required the setting of model parameters, and about an hour for
execution. Setting model parameters required expert judgment, and could
produce different results depending on the parameters chosen. However,
the multinomial model appears to have a potential tendency for a slight
negative average error of about negative .72 percent. It is suspected
this small currently statistically insignificant bias is due to the
assumption that reporting probabilities stay constant over six months.
The data clearly showed that there are increasing delays in production
reporting from the State of Texas.

In the future, EIA will investigate alternative methods to reduce the
bias. It is hoped that a relatively simple enhancement to the
multinomial method can be developed to remove the bias from the
estimates.

These were the recommendations based on the evaluation of methods. The
Office of Oil and Gas recommends implementation of the alternative or
multinomial method to estimate Texas monthly natural gas production
beginning in August of 2003, with the [Natural Gas Monthly]{.underline}.
We plan to examine methods to minimize the bias of the multinomial
model, with a proposal pending approval based on an evaluation of
methods by January of 2004, or sooner, if the bias becomes statistically
significant.

An annual evaluation of the model performance in conjunction with the
preparation of the [Natural Gas Annual]{.underline} will also be
required. Any resulting model changes will be approved according to the
evaluation of methods guidelines which we have gone over earlier. If the
model results for a particular month appear adversely affected by
unexpected events, such as significant data errors, missing data or
revisions, or most importantly, changes in Texas reporting procedures,
any estimate modifications will require the approval of a review team
prior to publication.

And again, these are the questions that were mainly here. Again, we\'re
focusing on the evaluation of methods itself, and the guidelines
therein. Are the outlined criteria adequate and appropriate? Was the
valuation of methods applied correctly and sufficiently? Would, or how
should, the criteria change based on the comparisons of different
methodologies? And now that the multinomial method has been selected,
what improvements could be made to the chosen estimation procedure?
Again, kind of pinpointing the bias issue, but any other issues that you
might see.

MR. NEERCHAL: Since I\'m the lead, I think I want to ask you a couple of
questions. We saw earlier in the review list that this was one of the
reports that was actually sent out for IER. And I was wondering whether
you could kind of ??

MS. KIRKENDALL: Actually, not IER. It\'s Category 1.

MR. NEERCHAL: Category 1.

MS. NORMAN: Uh-huh. And the evaluation of methods was what was sent.

MR. NEERCHAL: I thought it had two reviews on this, the number. And I
was wondering, can you give us, you know, a short summary of what the
feedback was?

MS. NORMAN: I think some of the feedback was discussed a little bit in
the recommendations. I think the recommendations from the Office of Oil
and Gas kind of included the feedback that they received from the
Category 1 clearance. I think it was also with the Category 1 clearance
that the point was mentioned about bias not currently being
statistically significant. Prior to that Category 1 review, we had been
focusing in on the bias a little bit more, and not to say that we should
stop focusing on it. But we\'ve been paying a lot of attention to the
concern of the bias in the multinomial method. And it was pointed out to
us that it was not currently \-- although it could become so \--
statistically significant.

MS. KIRKENDALL: The modelers who reviewed the methodology, there was
somebody from the Office of Integrated Analysis and Forecasting, as well
as the Category 1 reviewer was Fred Joutz, who\'s from the GW Department
of Economics. And those two people really thought you ought to be able
to get rid of the bias in some way, usually. They had a summary of a
discussion of the current methodology, the one with the \-- what are we
calling it? The parametric, I guess.

MS. NORMAN: Yes, parametric.

MS. KIRKENDALL: That they didn\'t understand the description of that
methodology. They had questions about the ?? how could you have ?? some
of the same questions that this Committee raised when they first saw the
description of the methodologies. So that was pretty much ??

MS. NORMAN: I don\'t know. If I\'m remembering, I think also something
was mentioned about the ability to use prediction intervals with the
multinomial model. I think that they thought that that was a good thing.

MS. KIRKENDALL: Fred was the one who provided the tables in the back,
where he did a regression of the model on the best data to see whether
the coefficient ?? and what you would like is that the coefficient is
about zero, the constant is about zero, the coefficient is about one.
And he found that that was true for both models, so he actually did that
analysis and provided the tables and we rolled it into the report.

MR. NEERCHAL: And that was the other question I was going to ask you.
You know, for the parametric method there\'s no standard illustrated, so
you cannot really put out prediction intervals. So for the multinomial
you have prediction intervals, so do you have kind of statistics on how
many, or what percentage of the parametric estimates fall in the
prediction interval of the multinomial model?

MS. NORMAN: I don\'t think that we did. It wouldn\'t be difficult to
calculate. It would be difficult for us to go back and look at that, but
I don\'t think that we have that off-hand. I know that we compared it
with the multinomial model, and all of our predictions fell within a 90
percent confidence limit.

MR. NEERCHAL: The reason I\'m asking is that if you look at the Table
S-1, if you look at the numbers \-- right \-- the multinomial model, the
numbers are much better. You know, all the average statistics, you know,
the mean to max distance and so on. But the picture ?? you know, I
expected a lot of ?? you know, looking at all those numbers, I said
okay, I didn\'t look at the graph. Then I said if I\'m looking at the
graph, I would have expected one to be sort of dominant. And they kind
of cross each other quite a bit in this graph. That\'s why I\'m
wondering, and also then I look at the statistic you showed here,
because I didn\'t read it in the paper very carefully.

It seems to me that 2 percent, if you take the 2 percent off, then the
number of estimates falling in the 2 percent interval are roughly the
same.

MS. KIRKENDALL: One is zero and one is one.

MR. NEERCHAL: Right.

MS. KIRKENDALL: So yeah, they\'re roughly the same.

MR. NEERCHAL: So I was wondering, you know, are these predictions
statistically significant, statistically different from the multinomial
model? That\'s what I\'m asking.

MR. WOOD: Oh, I see. You can almost count them here, 90 percent of the
estimates on the parametric model would fall inside of the projection
interval.

MS. NORMAN: Yeah. I think that we agree that the estimates that were
produced by both methods were very comparable. And I think the only
major improvements with the multinomial, are it\'s a statistically
rigorous model. It\'s easy to understand and implement, and comprehend.
You do have the ability to produce the prediction intervals, all of
these things. I think that we\'re all in agreement on that. The
evaluation of methods was something that we created in order to compare
the two models, and we\'ve basically made the selection of the alternate
or multinomial model that was proposed.

Again, I want to try and draw your attention back to the evaluation
itself. If we\'re in the future comparing, say, two completely different
estimation methods, how could we improve the evaluation itself?

MR. BREIDT: So these four criteria, the transparency, timeliness,
accuracy and reproducibility, do those ?? are those lifted directly from
an EIA statement of quality or something?

MS. KIRKENDALL: I believe they came from the OMB quality guidelines.

MS. NORMAN: I believe they came from EIA quality guidelines.

MR. BREIDT: Okay. And is that all of them?

MS. KIRKENDALL: And we\'ve adopted those too. Those are the big ones.

MR. BREIDT: Okay.

MR. WOOD: It\'s too quiet. For example, if possible and feasible, should
we use a longer period of time for a comparison \-- instead of three
years, six years, et cetera?

MS. NORMAN: Or maybe make that more general for more estimation methods,
or just would there be some way of analyzing what time period we should
be using? I mean, should it just be based on the data that we have
collected? Does it make a difference?

MR. BURTON: You have the data extending back how far?

MS. NORMAN: We have it back going to \`94?

MS. KIRKENDALL: \`96.

MS. NORMAN: \`96.

MR. BURTON: Why was the decision made to not use all of the data?

MS. KIRKENDALL: I think mostly because reporting patterns have been
changing over time. We certainly could.

MR. NEERCHAL: So I think maybe you should address one bullet at a time,
so that I have something to report to the Committee later on. I have to
watch my job. So in the first bullet, you\'re specifically looking, or
asking us to look at the different criteria, like average and maximum,
those statistics specifically. You are asking the Committee, are we
happy with those choices. Am I correct?

MS. NORMAN: Yeah. For the Evaluation of Methods itself, the different
criteria that we list out, we\'re asking if there\'s anything glaringly
obvious that we might have missed? Is there anything that could be ??

MS. KIRKENDALL: It\'s more than that. It\'s the transparent,
reproducible, timely and accurate first. Those were the criteria we\'re
after. And then after it, under \"accurate,\" is there something else we
should do?

MR. NEERCHAL: So I\'m addressing it to the Committee. I want you to
address these specific ones. You cannot be quiet. Right? I\'ve been a
chair, so I know how hard it is. So let\'s ?? I think Jay kind of
touched on one aspect, that these four criteria, the overarching
criteria are coming from OMB. So I think that\'s a very official source.
And I think the good one ?? I think we should \-- I say, from Committee,
we should kind of bless it and say hey, that\'s the way to go. And you
take something that is generally accepted as the criteria, and it seems
to me that transparency, I think is ?? I think ?? I understand the
methods much better than I did last year, by the way. And I still don\'t
think I could reproduce it myself. That\'s maybe a shortcoming in me.
But I think it is definitely ?? I think you ?? I see that as a great
thing and this is, of course, more readable, definitely.

MR. HENGARTNER: The only issue with accuracy, of course, is what\'s the
sample size. And that goes back to John\'s question of: Should we use
three years or six years? I think I would agree with Mark that we should
use all the data available. Although, Nancy pointed out there might be
variation in time, and that \-- I don\'t know if it\'s addressed in this
report, that the estimates are varying over time, or not.

Glancing rapidly at the final estimate, I don\'t think that in the three
year period, the reason of the bias is because of changes in time. I
think the reason that there is a negative bias is because the estimate
is of the quantity divided by an estimate. And if you think of taking
the expected value of 1 over X, the answers in equality will tell you
that that\'s always bigger, equal to 1 over the expected value of X. And
so there might be some slick little ways to adjust the estimate by
simple bias corrections.

I would hope you would be able to get back to Crystal with some of these
questions, since she wrote up the initial report. She probably still has
that in the back of her mind. I think she gave a talk about this this
week, as a matter of fact, so it would be helpful for her to get some
input on your findings that you found.

MS. NORMAN: Also, as far as time \-- it\'s been mentioned a couple times
\-- the issue of how long to go over the period that we chose from \`99
to 2001. I know that within even just that one period alone, there\'s a
lot of factors that kind of played into that, as far as how long it
would take for reports to come in. I know prior to \`99, I think almost
all of the revisions were made sometimes within a year. And then over
time, that was increasing. By the time it got to \`99, it was taking
almost ?? \`99 through 2001, it was taking almost two years. And there
are even some now that are taking longer than that.

Again, also within that, I briefly mentioned that we had decided between
using a period of consistency or \-- I don\'t know what we ended up ??
what was it that we called it? M equal to 6 or M equal to 9, the assumed
period of stationarity, or ??

SPEAKER: Stability period.

MS. NORMAN: Yeah, stability period. Thank you. And just within those
three years we had compared both M equal to 6 and M equal to 9, and that
plays into it, as well. And I don\'t know how that would be affected by
extending the data out into the earlier years, just because the reports
aren\'t coming in as late as they were.

MR. HENGARTNER: Well, to some extent I would argue to actually play the
game and go back the six years, knowing that there are changes, because
we don\'t know if tomorrow is going to be the same as today. There\'s
going to be changes again. And if the method is not robust to changes
yesterday, it\'s not going to be robust to changes tomorrow. And,
therefore, whatever estimates we\'re going to bring out and tell them
\"yes, those are the numbers,\" if the method is not able to adapt, then
actually we\'re doing a disservice to ourselves. And so, I would
recommend going \-- looking at the full data. I think just as a
statistician, my gut feeling is, if I have data, I should use it. And
that goes towards the accuracy point that you raised before.

There\'s a notion of stability or robustness that isn\'t addressed
explicitly in those guidelines, and the reason is that they came from
OMB. I think statisticians always hope to say well, if something happens
dramatic, the method is not going to go down the drain.

Suppose that for some reason all of Texas shuts down for a week because
of power outages \-- who knows \-- that would affect the output,
possibly. And the question is, will our method be completely wrong, or
is it able to adjust to things like that, as well?

But robustness is not the main criteria. I think precision or accuracy
is more important. But nevertheless, we should protect ourselves against
the unforeseen. So on that account, I would recommend going back to the
whole point.

Now what\'s the next thing you want \-- method applied correctly?

MS. KIRKENDALL: Evaluation of methods.

MR. WOOD: Actually, is there some kind of procedure that you use to
judge robustness in a formal sense?

MR. HENGARTNER: Yes. There are books about that, John.

MR. WOOD: That\'s a long enough answer.

(Laughter.)

MR. WOOD: In fact, it is interesting, especially if you only looked at
three years of data presented. It started in January of 1999. There was
an extraordinary event which occurred in December of 1998. And if you
look at the data for eight years, you see almost a step function down.
And it was the lowest oil price since the middle of the Great
Depression. The gas associated with oil plummeted, and the whole data
pattern looks different in \`99 than it did in the preceding five years.
It\'s an interesting thought.

MR. HENGARTNER: That\'s why it was over-estimated at that point.

MR. WOOD: It did. You might even look at the ??

MR. HENGARTNER: Yeah. No, I saw it.

MR. WOOD: So it really makes a difference, and you tend to try and tune
it so it can be robust for disturbances like that.

MR. HENGARTNER: So then let\'s go back to the six years and see what
happens.

MS. KIRKENDALL: Okay.

MR. BREIDT: You can also simulate different kinds of disturbances and
see how accurately things are ??

MR. HENGARTNER: I mean, what you do is take a data point and move it up
and down, and see how influential that point becomes, or how much crap
it generates.

MR. NEERCHAL: In fact, I think Jay\'s comment, I think, can actually ??
the second bullet, I think you are doing adequately, except for the
length of the data, in terms of actual database. The other evaluation
methods you can really think of is like simulation, simulated data with
cortex in a long tailed error statement, for example, heavy tailed
errors in your data, and see what happens, whether it really changes
things.

MR. HENGARTNER: I mean, you can even do simpler things. Physicists talk
about sensitivity analysis which exactly, that if your perturbations \--
simulation with heavy tail is interesting, although it might give a
distorted ?? it might be too bad a picture.

MR. NEERCHAL: A related question I did have when I was reading the paper
is that, if you have zeros, does that really throw off your maximum
accurate calculations? Does it matter?

MS. NORMAN: I don\'t believe that there were any zeros.

MR. NEERCHAL: No, suppose, suppose.

MS. NORMAN: You shouldn\'t have any zeroes.

MS. KIRKENDALL: Yeah. The State of Texas, I don\'t think, is going to
report a zero production.

MR. WOOD: So actually, if you look at the deltas there were before the
data was cleaned, and in the mononomial, several of the differences were
negative. I believe Crystal just set them equal to 1 and went forward.
It didn\'t take much to make it go up, so it was assumed that those were
made 1, and then there were periods where the data simply was not
estimated for a couple of months, especially further in the past. And so
that data was ??

MS. KIRKENDALL: So you would have missing values.

MR. WOOD: Yeah.

MS. KIRKENDALL: You have a missing C-1, you thought?

MR. WOOD: In fact, the data file that we handle has all of those changes
footnoted, that this was done because \--

MS. NORMAN: Are you talking about revisions down, or are you talking
about just missing initial values?

MR. WOOD: Oh, that no changes were made for six months, or the estimate
simply was not made.

MS. NORMAN: A particular point.

MS. KIRKENDALL: Well, the first one is no problem.

MR. WOOD: In the ordinary sequence ??

MS. KIRKENDALL: The first one is no problem. That just doesn\'t change
it. But if they did not report anything, then you need to do something
different.

MR. WOOD: Right.

MS. NORMAN: Because the revisions, whether they were made consistently
every month, or whether they were made at a later point in time, it
shouldn\'t affect the model too much. I mean, if you go back even at the
beginning of \`99, there would ?? you hit a point where there were no
revisions, and so you would just have zeroes as far as your change in
production values once you got out far enough, but that doesn\'t affect
it.

MR. HENGARTNER: I think more worse than the data was the fact that
instead of people reporting or not reporting, they would report guesses.
And you wouldn\'t know that it is a guess. Well, no. You probably did,
but they were just guessing. Then that generated downward revisions.
That\'s something I think Crystal\'s model isn\'t able to handle.

Crystal\'s model thinks that the data comes in, and as we go along we
learn. If there\'s more things coming in, we learn more. It doesn\'t say
that someone replaces some unknown quantity by a guess.

MS. NORMAN: There were a couple of points in the data where it was
revised downward. It was usually a fairly nominal amount. I don\'t know
how it would handle a drastic revision downward. I can\'t honestly say.
But as far as some of the smaller ones, there were a few, where they
were revised down, but it was \-- we\'re talking under a decimal place.
It just didn\'t make much of a difference.

MR. HENGARTNER: There is one more thing ??

MS. NORMAN: It hasn\'t been addressed a whole lot. If we were comparing
two different methods, just kind of throw it out there, try and think
towards the future. If we were comparing two completely different
estimation methods, how would this change? I mean, I know that we\'ve
talked a lot about comparing these two particular models. If we were
comparing two other models, is there anything else that you can think of
that might be applicable?

MR. HENGARTNER: There\'s one more thing I\'d like to see, and also
including in this model, since we talked about the time series, I\'d
like to have ?? I mean, a statistician would like to see residuals. And
I\'d like to actually know what this did to correlation of the
residuals, because there might be some ?? the reason is there might be
some temporal effects that we have missed.

MR. BURTON: But we\'re talking about the changing of the reporting.
That\'s exactly what I was thinking, that you could model that. You want
to be able to model that.

MR. HENGARTNER: Well, that\'s what Crystal is doing. Crystal\'s model is
trying to model the changes in the reporting by modeling how the
reporting comes in. Does that make sense?

MS. NORMAN: I think maybe ?? I don\'t know. This is something that I
think that Nancy and I and our group had discussed a little bit, was to
examine the reporting patterns themselves, separate from the models,
either of them, and see. I think it was either Gary or John, or the
combination of the two, that had come up with a possible improvement for
the bias with the exponential smoothing, and I don\'t know if they had
gotten that by looking at the different reporting patterns.

MS. KIRKENDALL: One thing you asked about the ACF that were the
residuals. In the little regression that Fred did, he observed ?? that
was another thing he observed, that there is ?? that Durban-Watson shows
there\'s correlation in the residuals for both models. He\'s regressing
an estimate on the final values.

MR. HENGARTNER: I mean, the reason I\'m bringing this up, it\'s a time
series. It\'s something one should do.

MS. KIRKENDALL: Well, it\'s not a straightforward time series. It\'s a
really funny time series.

MR. HENGARTNER: It is. That\'s why I talked about residuals. I mean,
that\'s what you\'re showing us. You showed us the plot of the
residuals. I mean, how much are the final numbers, the estimated numbers
different from the final numbers. Although, you\'re right. It\'s a funny
thing. But still, the residual will form a time series, and we can then
do all kinds of ?? the fun stuff you know how to do.

MR. NEERCHAL: But the residual series, you can only output after you get
the data. Right? It\'s not clear to me exactly what is the next step
after you get the error correlation.

MR. HENGARTNER: Well, ideally you would hope that the errors you\'re
making are white noise, because that means you cannot improve on it. If
there\'s serial correlation then, you know, there\'s something in there.
There might be some clever way of using the residuals from last month to
improve your current estimates. I mean, now the more you add on, the
less transparent the method will become.

(Laughter.)

MR. HENGARTNER: And I want to warn you against taking my advice at face
value, because there\'s one way to just simply say, I look at the data
and I do things. The other way ?? I mean, what I like about Crystal\'s
model is that I can understand it. I\'m going to be honest. I look at
it. I glance through it, and I can even see oh, yeah, there might be
bias in there, and I even know how to correct for it. I can make
suggestions for that.

MR. BURTON: But by adding an auto-regressive component, you\'re not
making it that much more complicated.

MR. HENGARTNER: Yeah. But making auto-regressive on the residual of the
multinomial ??

MR. BURTON: That\'s the part I don\'t know.

MR. NEERCHAL: You\'re with the auto-regressive on those probabilities
then ??

MR. HENGARTNER: Another thing is to put your ?? but be careful. That\'s
where she actually uses a six-month or nine-month lag. That\'s the
stationary period. So there is somehow already a moving window of data
on which you\'d calculate the estimates, which you\'re going to use to
make the prediction. So there is already a temporal component here.

MR. NEERCHAL: I think, you know, once you buy into the multinomial
thing, I guess \-- okay \-- I think we should probably recommend that
\-- look at the density structure of the residuals and see if there is
something out there that could help you. I think I\'m not that much
worried about the transparency, because, you know, once is okay, if
someone, you know, estimate, I think, you know, you have to trust the
person who\'s doing them, at least to some extent. I think the problem
is it\'s a multinomial, it\'s a multi-variate kind of ??

MR. HENGARTNER: The method is, it simply says: if I have production
today, it\'s going to be reported in the next 12 months, or 24 months. I
don\'t know when it\'s going to be, and so the reporting month is my
random variable, and we modeled this as a multinomial. There\'s a
probability that my production today is going to be coming up 12 months
from now. That\'s the multinomial. I don\'t see that as a big problem. I
mean, it\'s not multi-variate. It\'s more the way we want to model time.

MR. WOOD: But going back to ?? well, there\'s an assumption that there
is a stationary period of some time in the data inherent as then this
works. And in fact, there isn\'t. And the longer time period you look
at, the more obvious that becomes. And that is why the sixth month
stationary period worked better than the nine month which, in fact,
works better than the 12, which works better than the 18, because the
last couple of years, there was a change in that statistic of almost 1
percent a year. And then it got built onto that, and part of the
estimator is based on data that\'s 24 months old. And so you\'re using
one type of relationship that is old as you go through that multinominal
process to build a coefficient.

And those lags accumulate through the system. If there\'s a steady
downward trend, you know, if you need to multiply by a larger weight
every month to get a correct estimate, then that ?? you know, you\'re
going all the way back in the multinomial method for 24 months. So, you
know, during that period, there\'s a very large change actually in the
bias, and the weight needed to transform from the third estimate to the
24^th^. So actually, I suspect the bias is mostly the result of that.
And fairly simple regressions tend to remove it.

MR. HENGARTNER: Let me just see. I\'ll see Crystal on Monday.

(Laughter.)

MR. HENGARTNER: That\'s all I can promise you. And I\'ll bring up these
issues. And I think really you should keep Crystal in the loop, because
as you evolve and try out these models, her thesis is written, but the
paper hasn\'t been published yet. If there\'s a place where you think it
should be published, and now if you adopt this method, her paper should
be published in some form or the other. That now becomes critical, and I
don\'t think she was aware of that. And keep her in the loop with the
difficulties you have. Please, do.

MR. WOOD: Is she working with you ??

MR. HENGARTNER: Well, no. I\'m serious. I mean ??

MR. WOOD: No, I mean, I lost track of her. My e-mail did get returned.

MR. HENGARTNER: Okay. She\'s still Randy\'s student. Randy is visiting
Los Alamos and Crystal is with her. And hopefully, I\'ll be helping
guide her dissertation \-- small world.

MS. KIRKENDALL: Ahh, we have contact with Crystal. Well, that\'s a
wonderful suggestion. We\'re happy to keep in contact.

MR. HENGARTNER: I think she would welcome your input. One last comment.
If stationarity is an issue, one thing we can do with the likelihood,
and that\'s off sort of my hip, is maybe we can weight the likelihood,
use weighted likelihoods and weight the contributions of observations
further down the line less. And think about \-- what you\'re weighting
is now ?? the weight is like only a six-month period or nine-month
period, you could also just imagine the weight. You weight them all one
and then zero, or you can have a smooth transition. It\'s not that
different from what you\'re doing. It might have a different type of
tuning parameter, but it might actually be useful, especially if the
fact that we\'re using six month to predict what\'s happening in 24
months down the line, is an issue, as John says.

MS. NORMAN: You\'re saying heavier weights on the more recent months,
going down over the past six ??

MS. KIRKENDALL: Actually, in that scheme of things, John has already
played around with exponential smoothing.

MR. HENGARTNER: But that\'s different. Making exponential smoothing the
likelihood, not the estimates, which is slightly different, at least ??
and it still falls in the likelihood paradigm, which means we still can
then do the confidence intervals, we still can do all the goodies we get
here. But again, I\'m always ?? I like the simplicity of the method, and
I hate to make it more complicated and more arbitrary. Although, it is
arbitrary too. I mean, you have a six-month period, M equals 6 here.

MS. KIRKENDALL: So we ought to try M equals 1.

MR. HENGARTNER: Well, no. Any ??

MS. KIRKENDALL: There\'s nothing wrong with the theory for doing that.

MS. NORMAN: We might have, or maybe it was just me fiddling around with
it, and I didn\'t actually produce anything with it. But I think I might
have looked at M equals 3 a little bit, so maybe reducing that even
further. But at the same time, I don\'t know ??

MS. KIRKENDALL: Probably increase the variance. I mean, that\'s my
guess.

MR. HENGARTNER: There\'s actually another issue, now that I come to
think of it. If there is a tuning parameter, how are we going to select
it? I mean, here somehow we glanced over you said M equals 6, and then
you made the comparison. But there\'s this tuning parameter, M. Should
we actually look over the whole family, how you do?

MS. NORMAN: We actually ran the model with M equal to 9 and M equal to 6
over that full period, and compared both of those versions of the model
with the parametric, and then made the conclusion that M equal to 6 was
better. I think Crystal did a similar thing with her paper.

MS. KIRKENDALL: She did something similar, but I think her conclusion ??
she had older data. She didn\'t have quite as recent data.

MS. NORMAN: She had I think 6, 9 and 12.

MS. KIRKENDALL: Yeah. And I think she concluded that 9 was best. But I
think that shows how much has changed over the recent past, that now the
6 looks better.

MS. NORMAN: And maybe as part of the yearly or annual revision, that\'s
something that should be examined, because that does affect how the
estimates are produced.

MR. NEERCHAL: Let me get this. I mean, it\'s simple and it\'s objective,
except for the choice of M?

MR. BREIDT: So this gets to the issue of reproducibility, which it
sounds like what you mean by that is automaticity, you know, is this an
automatic procedure that was run with, and that tuning parameter is
something you need to choose in order to make it run.

MS. NORMAN: Yeah. The model is already set up right now. It\'s just a
really simple spreadsheet, Excel file, and didn\'t take very long to set
up. It takes about 5 minutes to execute, once you copy and paste and
drop the data in, and changing that parameter, M equal to 6 versus M
equal to 9, takes you maybe 10 minutes to do it.

MR. HENGARTNER: Why didn\'t you run all Ms, from 1 to 24?

MS. NORMAN: I think the issue was time as far as the comparison. Well,
no, I think ??

MS. KIRKENDALL: Well, no. Then you\'ve got to copy the results of the
spreadsheets.

MR. NEERCHAL: Oh, yeah. That\'s true.

MS. NORMAN: We also had to remember that we had to cut off the data at a
certain time so that we were pretending as if this was the present, so I
had to remove the data on an angle being dropped into each file. It\'s
not that terribly time consuming, a little monotonous, but ??

MR. NEERCHAL: Moving onto your third bullet, I think I will probably
add, you know, something that also came up, is that if you have a method
at least put it in the open literature, when you feel comfortable and
get the M feedback. I think that\'s the ?? you know, if it gets
acceptable for a journal, I think that is the best thing that can
happen. You\'re going to get people referring to that published article.
In addition to those four things, I\'d add another thing \-- publish it.

MS. KIRKENDALL: Publish it.

MR. HENGARTNER: Crystal will. My name is not on it, but Crystal will.
No, I really think if there is a change in methodology, unless it\'s
trivial, it should be documented in publications. And if there\'s a
government publication which this is the right place to put it in, then
let\'s do it that way. But it needs to be refereed. And it would be
better to have the refereed journal publication before you implement it.

MS. KIRKENDALL: We can\'t wait that long.

MR. HENGARTNER: I know. AS I said, this is the Heisenberg Principle.

MR. NEERCHAL: So I think on those four things, I would say transparency,
reproducibility, timeliness and accuracy, and \-- I don\'t know, what is
it, correct one word, you know \-- get it blessed by the open, you know,
scientific literature. Right? Get a publication. I mean, that will work
for any kind of comparison or methodology. I think last year, I remember
when this discussion came up, I think that something like \-- when you
want to change the methodology, maybe we need to follow the same road
test these computer guys \-- develop standards. Right? They have a whole
way of doing it, and everybody comes up with their own models, or their
own method, algorithms, and then they discuss and so on. It takes them
quite a bit of time before J-PEG or M-PEG or something like that becomes
a standard. I think so this is the same kind of processing \-- just put
it out there, let people discuss, and we\'ll make the differences,
compromise the differences, and finally get it okayed in the open
literature. Put it out there for criticism by anybody.

MR. BREIDT: I think that fits into the transparency bullet pretty much.

MS. KIRKENDALL: Yeah, it does.

MR. WOOD: Well, are you even hinting that publishing this in a popular
statistical journal would get it more scrutiny than putting it out in an
EIA publication on a website?

MS. KIRKENDALL: Certainly, scrutiny by a different group.

MR. WOOD: No, it\'s just, you know, stuff goes out there, and the
documentation eventually gets out there, with the data sometimes backing
it up. But there is the possibility that not very many people read it.

MS. KIRKENDALL: That\'s true.

MR. WOOD: And, you know, to put it out as many avenues as possible,
sounds like really a good idea.

MS. NORMAN: I don\'t know if it was already touched on a little bit.
Within those four criteria, the accuracy, was there anything else that
anyone wanted to add that we maybe should include, as far as our ways of
determining accuracy?

MS. KIRKENDALL: We talked about stability or robustness. I guess you can
sort of illustrate that if you look over a longer time period.

MR. HENGARTNER: As I said, I would play with this placing a point
that\'s easy to do in a spreadsheet, or see what ?? simulate a
disruption. And as you say, by looking at the longer period of time, if
you know things are changing, then that\'s somehow a test of what you
realistically would expect to see also in the future.

MR. BREIDT: So under accuracy we discussed the simulated disturbances,
working with the entire data set, and the residual analysis, and in
particular, looking at all the correlation, things like that.

MR. HENGARTNER: If it\'s single in the residuals, let\'s use it.

MR. NEERCHAL: And then basically, the improvement you might see for the
multinominal method that you already have, bringing sometimes this
aspect to it.

MS. NORMAN: Other basic comments on the evaluation itself?

MR. HENGARTNER: Good job.

(Applause.)

MR. NEERCHAL: So when the Full Committee comes, a summary of the ??

MS. KIRKENDALL: No, you give a summary.

MS. NORMAN: You get to talk.

MS. KIRKENDALL: I\'ll let him do it, and then, Kara, if you need to say
something, say so.

MS. NORMAN: Okay.

(Whereupon, the proceedings in the above-entitled matter went off the
record at 2:42 p.m. and went back on the record at 3:07 p.m.)

MR. NEERCHAL:  Okay. If you turn to your agenda, the paper we were
discussing was given by Kara Norman. She said again, if you have any
questions on the paper you should ask her, obviously. And this paper had
to do with the select and implement estimation procedures for comparing
?? a comparison of Texas production methodologies. This was some ??
those of you who were here last year, I think this is the estimation
procedure presented by Crystal Linkletter last year. And the problem is
simple to state, I think.

EIA publishes state level natural gas production estimates on a monthly
basis, and that\'s the easy part. The hard part is that every month, you
only get a partial estimate of the true production number. And you get
the complete picture on a 24-month scale. And that\'s making things a
little bit complicated.

And the current methodology, I think referred to as the parametric
method in the paper that you have in Tab 5, Tab 6 \-- sorry. Tab 6, the
paper is there in Tab 6, and it uses a parametric smoothing procedure.
Basically, based on the last few ?? the data available, a smoothing and
a prediction.

The multinomial method that is being suggested and is going to be
selected is a likelihood based method using the multinomial model. Okay?
And the specific questions Kara had for the Committee are there on the
screen right now. So there are two methods of estimation that are going
to be ?? that are going to be used. And we want to figure out which one
is better, and if we can get some objective criteria. And can I have the
other slide, please.

And they have chosen to take this guide ?? these guidelines which are
from the publication, OMB publication, as Nancy said. Is the method
transparent? Is the method reproducible? Can the method provide timely
and accurate estimates? And so let\'s go back to the previous one. And
Kara\'s question was, are these outline criteria adequate and
appropriate? And I think the Committee\'s unanimous suggestion is yes,
it is adequate and they are appropriate. And we went a little bit into
details into this one.

For example, transparency. What does transparency mean? Obviously, there
has to be adequate documentation, so the paper you have in your notes in
Tab 6, we felt that we used adequate documentation for both the
methodologies. It makes them understandable. And one of the questions
that came up was, you know, to improve upon transparency is to put it on
the ?? put it in the open literature. Make it widely available, and also
that way, you know, you can get feedback from a much wider audience than
just the people who use it, or who are very close to it.

Reproducibility \-- a couple of questions. I think the methodology, both
the methodologies have this aspect to it, and you have to choose a
window. And specifically, multinomial model assumes stationarity on a
certain window. And the current implementation uses a window of six
months. And there was some discussion about how to choose that window,
and treating that a tuning parameter, is it possible to get a best
window, and so on. And this is part of the reproducibility.

If someone else uses the same methodology, would they use 6 or 9, and so
on? How do we deal with that? So Committee suggested that they should
address that and use some guidelines on how to use the window, how to
choose the window, and so on. And also, protect yourself from the
unforeseen. Like for some reason, certain data points you get is zero,
easier method, robust against strange values. Or in a sudden drop, maybe
a real one, not the data artifact but a real one, is your method robust
against such things?

So one of the suggestions was to run some simulations with some
simulated disruptions or perturbations, run them and see how the methods
perform against that.

Timely, I think there was good argument that these methods are fairly
easy to implement and able to produce timely estimates. And in terms of
accuracy, Kara has listed a number of statistics \-- they are on page 6
of your Tab 6 paper \-- a number of estimates, a number of statistical
measures of error are listed there, and I think the Committee felt they
were adequate, and give a good feel for what\'s going on.

However, one of the things that was brought up was the multinomial
method does have an alarming error bias. It is not statistically
significant as one of the reviewers pointed out. However, it is
alarming. It can raise alarm. And I think there were some suggestions
from the Committee, basically somewhat technical, suggestions to see if
the bias can be improved a little bit.

And also, addressing the last bullet, you know, what kind of future
research ideas on the multinomial model. The Committee felt that the
multinomial method doesn\'t really explore the time series structure
very formally \-- let me put it that way \-- very formally, and that is
something to look at. For example, look at the residuals from the
estimates versus the real, and look at correlation, look at ACF block on
it, and see if somehow we can explore the data collection information to
improve your future estimates, for example. These are some suggestions
that were given by the Committee. And I think, perhaps ?? go back to the
?? in addition to the four, transparency, reproducibility \-- one thing
we want to add, the Committee felt that we need to add as part of
transparency, to bring it to higher level of visibilities, this idea of
matters being put in public in open literature, and make it available
widely. People can go and read it, critique it, copy it, do it over, and
so on.

MR. BREIDT:  All right. So we\'ll turn it over to Jae Edmonds for the
summary of the second breakout session.

MR. EDMONDS:  We had a very interesting session looking at a real natty
problem, which is estimating industrial natural gas prices. And just to
review, the situation that we find ourselves in today is as follows. The
EIA has traditionally published the price of natural gas as simply
looking at all of the expenditures on gas and dividing it by the amount
of gas, giving you a very nice average price. But as a consequence of
deregulation, the sources of information have covered decreasing
fraction of the industrial market, such that today only 32 percent of
sales are actually available for computing a price. And they come in two
different pieces \-- the traditional piece, which covers about 20
percent of the market, which is in EIA form called 857. And that\'s
sales to industrial customers. And then a new piece which looks at
non-utility generation of electricity for facilities that are over 50
megawatts. And that\'s an EIA form called 423, and that gets you another
12 percent, so you\'ve got about 32 percent.

So the question is, you\'re missing 68 percent of the information. So
the question that was put forward was well, can you ?? is there any way
to get that additional information? And so they talked about four
different alternative approaches, and so the first was well, you could
use the average of the available, what\'s called an on and off system
pricing values, basically using the 32 percent of information you have.
Using the off system sales prices only, just using a subset of the
prices. Using the most recent manufacturing energy consumption survey
\-- something called MECS, the acronym for that, which was most recently
available for 1998. And somewhat in the rearview mirror, and somewhat
susceptible to the deregulation phenomena that has been such a ?? caused
such difficulty. And then fourth was to develop a model to estimate the
price of independent spot market prices.

The problem is, of course, pretty staggering; that is, you\'re missing
68 percent of the information. The last time you had a full set of data
was in 1998, and so while we spent most of the time talking about
estimation procedures, to go from spot market price to see if, in fact,
you could predict the price that was available to those who were buying
combined heat and power. And there\'s some really interesting and
innovative work that was presented there. I think the fundamental
problem remains that if you want to publish a price for industrial
natural gas, you\'re in real trouble if you\'ve got 68 percent of the
data missing in action.

And so the questions before the Committee were does the Committee think
we should add a column showing the prices paid by industrial combined
heat and power plants; that is, breaking out the CHP information from
the information for direct users. Those two are separate pieces. The
one, the former covering 12 percent, and the latter covering 20 percent.
So you display the information that you have along with the fraction of
the marker that it covers, and my own reaction was that it seemed like a
useful thing to do, to display what you know.

And then the second question was, what does the Committee think of
presenting a separate table to show several estimates. An attempt to
represent the entire industrial sector based on a variety of different
options, and I think there was a pretty unanimous conclusion by the
Committee. If the Committee wants to contradict me, I\'m perfectly happy
to hear that, but I didn\'t hear any contradiction to the conclusion
that it\'s premature. You really don\'t want to go down that road when
you have no way of testing whether or not your model is actually
delivering the goods. So that, in a nutshell, is our breakout session.

MR. BREIDT:  Okay. Thanks, Jae. Well, we\'re kind of desperately late at
this point, and so I\'m going to suggest that as we ?? we\'re going to
have an announcement, but when we do go to break, if you could just grab
your cookie, grab your drink, and then just go straight to the breakout
session and enjoy your cookie there, that would be great.

There is an announcement. I think Bill\'s on the phone right now, but
tomorrow morning we will reconvene in this room prior to the break. But
after the break, we are going to be changing venues, and this is a map
here showing where we\'ll be. So we are here. Tomorrow morning we will
be here. This is the hotel where many of us are staying, so this is
right next door. So if you are staying at the hotel, there\'s no need to
lug your luggage around. You can just leave it there. We\'ll meet here
after the break tomorrow morning.

SPEAKER:  But the good news is we could walk up as a group.

(Laughter.)

MR. BREIDT:  Okay.

SPEAKER:  Why don\'t we just meet at the annex, or what we used to call
it, why don\'t we just meet there for the whole thing so we don\'t have
to walk down and walk back?

MR. BREIDT:  Because I don\'t think we could have gotten the room for
that entire period.

(Whereupon, the proceedings in the above-entitled matter went off the
record at 3:21 p.m. and went back on the record at 3:34 p.m.)

B-R-E-A-K-O-U-T S-E-S-S-I-O-N

MS. KIRKENDALL:  This session is estimating natural gas production, this
time in the Gulf of Mexico. The Dallas field office has recently
acquired a new data stream from the Gulf of Mexico, and they\'re trying
to do creative things to come up with estimates. So I\'ll turn it over
to Gary Long.

MR. LONG:  Thank you, Nancy. I\'m Gary Long with the Office of Oil and
Gas Reserves and Production Division in Dallas, Texas. How are we going
to estimate Gulf of Mexico production? The Gulf of Mexico is about 25
percent of the nation\'s gas production, along with Texas. We\'ve got
half of the gas production in those two areas. You\'ve heard the last
session about Texas, so now we\'re going to talk about the Gulf of
Mexico.

Up until ?? well, in past recent years, we used the Minerals Management
Service well data. Up until recently, that data was pretty good. It
lagged four or five months behind. It was 95 percent complete. What
means, according to the MMS, is 95 percent of the well completions were
included. Due to some IT miscalculations in the MMS outfit, suddenly
we\'re looking at a year and a half lag. And so at our encouragement and
the encouragement of others, they started putting out on their website
what they call suspended data, which is not verified or edited. It\'s
just dumped out there. And we use that data. That gets us back to
current months with somewhere in the neighborhood of 500 to 1,000
missing well completions out of a total 6,500 or so.

They started doing that, putting out the suspended data, as they call
it, this spring, early this spring. When we first started looking at
this data this spring, we were looking at still in the current month 2
to 3,000 missing completions. And since that time, now we\'re down to
500 to 1,000, so that\'s actually getting better and improving. They\'re
getting more and more data into the system.

As far as the data goes, there\'s actually three kinds of data on their
web site, what they call the accepted verified 95 percent complete data.
There\'s also accepted and verified data that\'s less than 95 percent
complete. And then there\'s the suspended data, which is neither
verified or edited, which is newly available since early this spring.

We use SAS to do all of our data manipulations and edits. When we first
starting looking at this data early this spring, the suspended data,
there was about 95,000 records, about 9,000 duplicates that we found.
What we\'re looking at now is about 60,000 records, and 3,700
duplicates. Total file size is about 1.4 million records. A record is
one month\'s completion for ?? one month\'s production for a well
completion. A well completion is, I guess, a single producing interval
in a well. A well can have multiple completions, and each one is
reported separately to the MMS.

Let\'s see. As an example for September, September\'s data, I guess we
would make an estimate in June, and in September we would make an
estimate for the June production. In June we had 5,900 completions, so
we were down somewhere in the neighborhood of 600 missing. It\'s much
better than what we started with back in the spring.

This may change again in the very near future. I just learned yesterday
that the Department of Interior, through the BLM, Bureau of Land
Management, and the BIA, Bureau of Indian Affairs, has started a new
program where they\'re actually going to start collecting data in real
time with a remote data acquisition program.

They\'re going to start in with the BLM in the San Juan Basin, which is
New Mexico. The BIA has got several Indian tribes that have production
on Indian lands that they\'re going to throw into the mix. And beginning
next month, the off-shore Gulf of Mexico MMS group is going to kick-off
their program. And hopefully, some time in the spring, if we choose to
join in, we can actually get the Gulf of Mexico data in real time.

As far as the methodology goes, the concept is pretty simple. The first
thing we did when we looked at this data was use it as one lump, a
single class distribution, if you want to call it a distribution. And
later on, we divided it up into 12 classes. There was some
experimentation involved in coming to 12. We looked at all kinds of
things between 5 and 50, eventually settled in on 12.

IN each case, we have two ways of I guess determining the expected well
completions. The easiest way, the simplest way is just to hold them
flat. The other way is we model the well completions based on a rig
count in the Gulf of Mexico. We divided ?? for the 12 class
distribution, we divided those into the 12 classes by production rates,
and a little later, I\'ll describe how that was done.

MR. HENGARTNER:  Did you look at the total well completion?

MR. LONG:  In the particular well bore, you\'ve got in the Gulf of
Mexico, usually multiple zones that will produce. Okay? Each zone, if
you perforate, you drill a well, you run by and perforate through that
pipe into that zone, and that zone would be a completion. In any
particular well bore, you could have multiple zones, or multiple
completions producing into that well. So basically, it\'s the zone or
producing interval we\'re talking about.

In the single class estimation process, it\'s a pretty simple model.
Basically, it\'s the expected well completions, multiplied by the
reported average production per completion. Multiply those together, you
get the production, you divide production by the well completion to get
the average per completion.

This is some statistics which should make you all happy. This is kind of
what the distribution looks like. It\'s a very skewed distribution. If
you look at the box on the left, look at the 50 percentile here, what
this number right here is, there\'s about 300 mcf a day, or 50 percent
of the well completions are 300 mcf a day or less. There\'s a lot of oil
wells in this mix of 6,500 that actually produce some gas. And so about
half of, a little less than half of those 6,500 completions are actually
labeled as oil well completions, even though they produce gas associated
with the oil. About 20 percent of the total volume of gas comes from oil
wells, but it is a skewed distribution.

In the 12 class estimation process, the underlying assumption here is
that the incomplete reported data we assume is a sample distribution of
the final distribution. So in order to get from the sample to the final,
we need to know what the final looks like, and so we went back to the
last month\'s data where we knew it was absolutely complete, which
happened to be at the current time is September, 2001.

We take that month and the five months before it, get an average, and so
we have an average distribution for those six months. We call that our
standard or the final, or our calibration period. We have lots of names
for it. You can pick one that suits you. And then we divide that into 12
classes according to production rate.

The first class and the 12^th^ class are done a little bit differently
than the rest of the classes. In the first class, it\'s all the
completions that produce less than one million a day. The 12^th^ class
is all completions that produce more than 100 million a day. Most of the
oil wells will probably get in those lower classes, especially the first
class. The 12^th^ class, sometimes you get one or two in there,
sometimes you don\'t.

A lot of times what we think happens in the 12^th^ class is that there
will be a new platform, especially in the deep water, three or four
wells drilled. They\'ll test one well for a month, and they shut it in.
Test another well for a month, test a second and third well for another
month, so you\'ve got three wells that will produce over 100 million a
day. When they produce them all at the same time, the platform may only
produce 250 million a day, so you had three months in a row where you
had at least one well that was over 100 million a day in the 12^th^
class, and then suddenly there aren\'t any in the 12^th^ class, because
they\'ve all dropped to the 11^th^ class, due to the production
limitations on a platform.

The rest of the classes in-between 1 and 12, we have a formula there
which determines the well count in each class. It\'s a power function
that gives you a geometric progression from 2 to 11. And basically, once
we pick the first class of less than a million a day, then the next
class has got 690 wells, and so we go look at the ranking of the wells,
and we see that the next 690 wells get us up to 2.1 or so million a day.
The next 471 wells get us up to 3.6, and so those rates, boundary rates
on the classes were determined in that fashion.

And basically, we ended up with 12 classes. We wanted to have as many as
we could possibly have, and still have a significant number in each
class - ended up with 12. And this is what they look like. You can see
it\'s geometric progression, and this is classes 2 through 11 -
twenty-two well completions in the 11^th^ class. The 11^th^ class is 50
to 100 million a day.

MR. NEERCHAL:  The well completions number, that column is actual data.
Right?

MR. LONG:  Yes.

MR. NEERCHAL:  And so this model treats that data so ?? I mean, the
exponential curve practically used the same time?

MR. LONG:  Well, the exponential curve was determined by that function,
which then determined the well counts.

MR. NEERCHAL:  Okay.

MR. LONG:  We\'re saying we used those well counts in each class to
determine what the limits were, the production limits, top and bottom
for each class. And it\'s those production boundaries for each class is
what we use then to break all the other months\' distributions into 12
classes.

MR. NEERCHAL:  How do we get the formula then?

MR. LONG:  The formula was somewhat arbitrary, chosen by us to give us a
decent number of classes, and still have significant data in each class.

This is just a comparison between the standard distribution and two
other months of data, January, 2002 - January, 2003. You can see in the
first class that has the vast majority of the well completions, and only
22 in class 11 at the tail end. This is the same comparison on
production. See, the first class here has probably the lowest production
as many of the classes, even though it had the vast majority of the
wells.

If you look at classes 8, 9 and 10, or so there at this end, you see a
little different pattern comparing the January \`02, and the January
\`03 data to the standard data. In January, we can kind of explain that
a little bit with demand changes, I guess. January \`02 we had a very
low gas price. January, February in \`02 were down in the low \$2
something, demand was low, so whenever the production changes like that,
there\'s a shift in the distribution. We think that\'s what\'s happening
here.

MR. HENGARTNER:  I would keep the big ones open and close the small
ones.

MR. LONG:  It doesn\'t seem to happen that way.

MR. BURTON:  It may be just guessing, the small ones by ?? because
they\'re smaller, you can adjust your production in smaller, more
precise increments. Does that make sense? Do you see what I\'m saying?

MR. HENGARTNER:  I\'m not an expert. I\'m just thinking ??

MR. LONG:  I suspect it\'s an economic question. The bigger wells,
probably the newer wells, you get some investment there that hasn\'t
been paid out yet, so you want to keep the cash flow coming on those
wells, those fields, those platforms.

MR. HENGARTNER:  That would keep production up exactly for large wells
which are C-11, 10 and 9, and you might close the smaller ones which
don\'t produce much, but cost a lot to maintain.

MR. LONG:  Well, if you look at this one right here in class 10, you can
see that in January \`03, where the demand in January and February of
\`03 was higher, the prices were higher.

MR. WOOD:  Yeah, there\'s something that ?? you know, if you look at
this, it\'s just totally striking. Those wells in class 12, out of 6,500
wells, one of them represents 1 percent of the production in the Gulf of
Mexico, the second largest producing area in the United States. About
two of the class 11 wells represents 1 percent of all production coming
out of the whole Gulf of Mexico. So if you want to adjust production a
little bit, you know, all you have to do is cut back on some of the
major wells, and you\'ve made a couple of percentiles in production.

MR. LONG:  Each one of these classes has roughly 10 percent in
production, so these upper classes, a couple of wells coming and going
can change the way these bars look dramatically.

This is, I guess, a comparison of wells and production on a percent
basis for May, 2001. You can see in the first class there the wells are
between 65 and 70 percent of the total, while production is about 7
percent of the total. Production runs somewhere between 7 and 10 percent
by class. We ran some stability tests, I guess to be sure that the final
distribution or standard distribution was similar to monthly
distributions of real data. We took a few months before and after the
standard time period. This is August, 2000 and this is March of 2001,
just as an example. And you can see they\'re both showing to be similar
to the standard that this last column right here where it says test
percent is the standard distribution that we\'re comparing against.

This is one month that\'s after the distribution. This is April, 2003.
Most of the incomplete months were pretty good. There\'s a couple of
exceptions, September through December in 2002, are all exceptions. This
is basically due to hurricanes or two storms, one in September and one
in October. You see right here, this is October, and this is not a
similar distribution. Any time you have something like a hurricane in
the Gulf of Mexico, you more than likely are going to have to do some
analyst intervention to get a good estimate.

This is January of 2002, and February of 2002. You can see that these
are not similar distributions to our standard. Again, in this time
period we had the low prices and low demand.

This is 2003, January and February, same thing here. These are not
similar distributions, although February is getting closer. If you look
at the sample size on January, it\'s fifty-nine forty-six, when compared
to February, sixty-one seventy-eight. It could be that the January, 2003
here has still got some influence or still recovering from the hurricane
in October.

MR. HENGARTNER:  Can I ask a question? If they mislabel class 1 to 10,
instead of 2 to 11, your model was class 2 to 11.

MR. LONG:  Yes. Right.

MR. HENGARTNER:  But then why does the first class ??

MR. LONG:  No. This is class 11 right here, 4,000. Class 1 - I\'m sorry.

MR. HENGARTNER:  Class 1.

MR. LONG:  That\'s class 1 through 10.

MR. HENGARTNER:  So this is class 1 that you\'re using, although the
model is ?? have looked at class 2 to 11. It was J minus 2.

MR. LONG:  Yes. Yes, we fixed class 1. We just said everything less than
one million a day goes into class 1. And then we use that function to
determine 2 through 11.

MR. HENGARTNER:  Yes. But here, you\'re comparing class 1 through 10.

MR. LONG:  Right.

MR. HENGARTNER:  With that function.

MR. LONG:  Right. That function was used to ultimately get the
boundaries, the production rate boundaries of the classes. And so all
the data ?? the production data for any given month was divided into 12
classes according to those production rate boundaries that were
determined in that standard distribution.

MR. WOOD:  The top of that first class was one million period, so it was
set. And the formula only spread out the wells that were between the
second and 11?

MR. HENGARTNER:  It was set at one million. Then you used that function
to determine your class boundary, hoping to essentially have a similar
number of cases in each group. I mean, more or less, that\'s the ??

MR. LONG:  Basically, this function determines the number of wells in
each class. Okay? Then we had our data sorted rank order. Okay? And then
we just counted out the number of wells. The next 690 wells gave us an
upper limit in for that class of 2.078.

MR. HENGARTNER:  Based on the model.

MR. LONG:  Right. So these classes, these production rate boundaries
determine the classes.

MR. HENGARTNER:  So then what was expected, if you can ?? if you go back
to the previous, or not on this one, on the chi-squared, you had an
expected column. Here, okay. So one is frequency percentage, I suppose
that\'s ?? for example, in the first case, you have 66.55. That\'s 4,297
divided by the total ??

MR. LONG:  Right.

MR. HENGARTNER:  Then you have test percentage. Then I missed what that
meant.

MR. LONG:  This is the standard distribution for that six month average
ending in September of \`01. These first three columns are for the
August, 2000 data in that month.

Okay. This is the same idea that we had in the single class model. We
used the same thing in the 12 class model, except we do it for every
class. It\'s expected well, times the production per well completion.
Add up 12 of them, and you\'ve got the production. But we prefer to
write it this way with the first 10 classes being summed here. J is 1 to
10. The average production per completion times an expected number of
completions in each class, 11 and 12 we do a little bit differently. We
select the ?? or we choose the expected well completions here
differently. And in class 12, we just take whatever is reported.

MR. NEERCHAL:  You just said ?? what is the P dash and P?

MR. LONG:  Right here?

MR. NEERCHAL:  Yes. No, I mean the first formula you have there, MIJ=to
PIJ or WIJ.

MR. LONG:  This is just a blank calculation of what\'s reported in the
data.

MR. NEERCHAL:  Okay. And P prime is the model one.

MR. LONG:  Right.

MR. NEERCHAL:  Okay.

MR. LONG:  Okay. Here\'s an example of how we fill in the missing wells
to get the expected well completion count by class. In this example,
this is the standard here, the fraction expected, the counts expected.
And then this is what\'s reported in April, 2003, and that is, 497 short
of the expected 6502 completions.

You can see for class 9 right here, that the class 9 is already full, so
we do a proportional redistribution amongst the 9 other classes that
aren\'t full, and this is their redistribution fraction here, zero for
class 9. Missing wells are added to the classes like this, none added to
class 9. With our final estimated well count, 6502, same as the standard
distributed like this, and then lo and behold, we take that and this
distribution when compared to our standard, and it\'s a similar
distribution.

MR. HENGARTNER:  Now if there are 497 missing, that comes simply because
you look at the actual counts, and you said in the last six months on
average I had 6,502 wells, and you look at that difference.

MR. LONG:  Right. We had 6,502 in the standard distribution in that six
month average. And then for April, 2003, we had 6,005. The difference is
497.

MR. NEERCHAL:  Is it conceivable that for some months, your expected
count is actually less than the ones that reported too. Right? Once in a
while it\'s going to be lower too.

MR. LONG:  Yes, it could happen.

MR. NEERCHAL:  Right?

MR. LONG:  Usually, as you get closer and closer to the recent time, you
know, you get more and more missing. When you\'re close to our standard
time period, a lot of these classes will be full already. This is what
class 11 reported well completion count looks like. Back in \`96, we\'re
looking at 3, 4 well completions, something like 2 percent of the total
production. And now we\'re in the neighborhood of 20, something like 22
percent of production. Excuse me, 12 percent of production. These
numbers out here are ?? this is reported data, so I\'m not sure. The
closer to recent time you get, the more likely it is that there\'s some
missing.

Okay. And so this brings us ?? this is an example of how we choose how
many wells are missing in class 11. We take all the wells in class 11,
all the wells that have ever produced more than 50 in a two or three
year time period, and just plot their production. And in places where
you see the colored in cells, is where we would add a well, or say that
there\'s one missing here.

For instance, in January of \`03, we\'ve got 4 wells missing here
greater than 50 in the month of August. In the month of September for
January of \`03, we still have 4 missing. And then we had a second
update of the data in September, late September. And you could see that
all those wells were filled in, all these were filled in. We\'re only
left with that one right there missing. This is kind of how the data
fills in over time. And basically, what we do for class 11 is just look
at each one, and decide where there\'s a month that\'s missing. What we
put in that month is the average class 11 well reported production.

MR. HENGARTNER:  Why when you look at the two first columns, there is a
bunch of ?? a square of four blocks that\'s orange, that you say is
missing. And above them there are two white blocks.

MR. LONG:  Right.

MR. HENGARTNER:  Why aren\'t these missing? I see that they\'re filled
out further down the line.

MR. LONG:  Yes. Class 11 is greater than 50 million a day production.
See, this well hadn\'t produced 50 million a day since December of \`02,
and so we didn\'t add any there.

MR. HENGARTNER:  This well that produced .07 in December, and so the
production is very variable.

MR. LONG:  Yes, it\'s real data.

MR. WOOD:  But a well could produce for one day in a month, and then it
could produce - the same well could produce for 30 days.

MR. BURTON:  Why? That doesn\'t make sense to me. I don\'t know anything
about gas, but that doesn\'t make sense to me.

MR. LONG:  Sometimes they shut down a platform for maintenance.

MR. BURTON:  So it\'s ?? production is being controlled that way. It\'s
not ??

MR. LONG:  Yeah. I mean, it\'s a non-production ??

MR. BURTON:  Now the gas was there, now it\'s gone, and it\'s back
again.

MR. LONG:  No, it\'s a mechanical manipulated thing.

MR. BURTON:  Okay.

MR. LONG:  It\'s not necessarily a natural flow.

MS. KIRKENDALL:  Couldn\'t there also be typos in the data?

MR. LONG:  Yeah. And in September when we were looking at June, the data
for June was about half from the suspended data set.

MR. HENGARTNER:  Now let me get this straight. So now wells from month
to month are going to slip from one class to the other, back and forth.

MR. LONG:  As production goes up and down in the Gulf of Mexico, or as
demand may go up and down, they shift.

MR. HENGARTNER:  So demands or whatever.

MR. LONG:  And the distribution can shift.

MR. HENGARTNER:  So somehow what we would like to have is to know in
which class, given this month\'s production is going to be given, I know
last month\'s production, or last five months production. What you\'re
saying is you\'re just looking at the average, or how do you determine
if those two things were black or brown? That\'s what I\'m trying to
see.

MR. LONG:  Up here?

MR. HENGARTNER:  Yes.

MR. LONG:  We look at this history. This well is consistently over 50
million a day, so we think it\'s probably over 50 million a day here.
Same with this well, so we would add ??

MR. HENGARTNER:  What we\'d like to have is what is ?? given that it was
in class 11 last month, what\'s the probability that it\'s going to be
in class 11 next month? Is that what ?? I\'m just trying to figure out
in the data ??

MS. KIRKENDALL:  Well, that\'s not what they\'re doing, but it\'s an
interesting idea.

MR. NEERCHAL:  That\'s not what they\'re doing. I mean, I\'m just trying
to understand the data, and what you\'re trying to convey, such that ??
because I realize when I wrote the paper, I didn\'t understand that part
of it. And a lot of the things prepared for it is irrelevant.

MR. LONG:  Well, you know, the reason why we actually look at these
wells individually, the potential class 11 wells, is they matter. Each
well is a half a percent of total production.

Earlier, I mentioned that there were two ways of ?? we had two ways of
determining what the expected well completion count might be. One was
just to hold it flat, the other one was to use a completion, a well
completion model. This is a well completion model. This is different
than what\'s in the paper that you probably have. Nancy pointed out a
failing in that model so we changed it.

Basically, this is a decline function here. You have wells that deplete,
they get old, they become economic, they\'re plugged. This side of the
equation here is based on the total rig count, and a smooth total rig
count times a constant. You\'re adding new wells all the time to do the
drilling, so you\'ve got old wells disappearing on this side, and new
wells coming in on this side, and we fit that ?? the B and C parameters
here over the January \`96 period on through the standard time frame. We
get this. This is about half a percent decline every month, something
like 30 well completions. And this indicates in the neighborhood of 25
or 30 well completions being added, so it\'s a fairly stable kind of
thing. And I think we have a picture.

The green line here is a 12 month exponentially smoothed well count, and
then held constant. That\'s our flat case. The purple line here is the
model that was on the previous slide. It shows some decline here.

There\'s a analyst override adjustment right here for the hurricanes in
late 2002. You see back here in \`98, we had a major hurricane that
would require these well counts, either the flat case or the model one,
to be adjusted if it was going to be right.

There was an interesting comment about the Texas production, and what
happened in December \`98, also happened in the Gulf of Mexico. It was
like a step function here. December \`98 was when we had the lowest oil
prices since World War II or so. Somehow in the Gulf of Mexico, we lost
about 300 completions.

This is the final production. We have the single class, the 12 class
method, and for each one of those, we have the flat well counts, and the
model well completion counts. And if you look at the tail end of the
plot right here, this dark curve on the bottom here is what\'s reported.
These other curves are four estimates from those four different methods,
and they vary about .2 bcf a day, .1 to .2 bcf a day.

And then the questions. Does our approach seem logical? Are we using the
monthly distributions properly, making correct use of the information
that they have? Is there other information in these distributions that
we could use? Are there any other statistical processes that we might be
able to take advantage of here? Is there anything else in general that
we can do to make better estimates? That\'s it.

MR. EDMONDS:  All right. You may have already said this, so bear with me
if you have. But as you\'ve gone through here and developed these
procedures for filling in the missing pieces, I presume your time has
passed and you\'ve been able to go back and look to see how good that
process has worked, how well it\'s worked in filling in the data, as
those data systematically go through that year and a half lag period.
And could you just ?? if you\'ve done it, could you just review it for
me. And if you haven\'t talked to it ??

MR. LONG:  Well, we\'ve only had this data since about February or March
of this year, so we haven\'t had much of an opportunity to do that. But,
you know, what we have seen, as near as we can tell, looks okay. But
there\'s really ?? since we only got that data that late, we really
can\'t go back in history and come forward and see how well it works.

MR. WOOD:  And there\'s no steady state. The data, some of the data two
years old is missing more than the stuff that\'s five years old. And
what did they do? There have been two court orders and a Oracle Data
self-referential probably imposed on themselves and they couldn\'t get
anything out. But there is no steady system like they used to have in
MMS, or like you saw in Texas, if you looked at that. So you ?? you
know, again, there are strange ?? you know, maybe they get all of
February of 2002, and they put it in all somewhere, and it was on the
bottom of a pile in some warehouse in Illinois where they stored the
data when they\'re under court order. And just so a year from now, and
since they\'ve got almost twice as much data available now in the month
we\'re trying to estimate as they did five months ago, in a year from
now, they may be up to more of a steady state where the data flows in
more smoothly and there\'s a lot more you could probably do with it.

MR. EDMONDS:  Now is what you\'re saying, that eventually you always get
the full data set, but the rate at which that information becomes
available or its quality is improved, is highly variable?

MR. WOOD:  Highly variable.

MR. EDMONDS:  But eventually, though, they turn it over.

MR. WOOD:  They do in most states, we don\'t actually know what they\'re
going to do, because they haven\'t established a pattern.

MR. EDMONDS:  So there actually is a question about the universe of
data; that is, you may or may not actually have the final answer, even
when you go back into the 90s.

MR. LONG:  What happened, there was about a year or so when there was no
data processed at all. And then when they ?? and they just dumped it all
in a file. And when they did start processing that data again, you know,
they would take one from the top, pull one from the bottom, pull one out
of the middle. It wasn\'t done in any chronological order.

MR. BURTON:  Help me understand here why this is hard. They\'re selling
this gas - right? They\'re not ?? that\'s what they\'re doing with it.
Right?

MR. LONG:  Right.

MR. BURTON:  You can\'t sell stuff and not know how much you\'re
selling, or at least you\'re not very smart.

MR. WOOD:  There\'s a four billion dollar question ??

MR. BURTON:  They know. Yes, they know ?? okay.

MR. WOOD:  Four billion dollars that the government thinks maybe they
charged them too much or gave them too little for royalties.

MR. BURTON:  I really wish I hadn\'t heard that.

(Laughter.)

MR. EDMONDS:  So the problem is actually the producers. That\'s where
the system is clogged, where the information is clogged. They just
don\'t want to turn that over.

MR. LONG:  No, not necessarily. This is required to report the data ??

MR. BURTON:  Or is it the state? Where is the data getting ??

MR. WOOD:  Our federal cousins have done more than their share to screw
the data up.

SPEAKER:  It\'s the MMS. And because of various court orders, they have
been prohibited from releasing it, so that\'s a big part of it.

MR. LONG:  That\'s not exactly true. But back in October of 2001, they
were scheduled to switch to a new computer system, so they were going to
discontinue the old one and make the switch over to a new system. This
is the Minerals Revenue Management Group in Denver that processes all
this data for the MMS. On the appropriate date, they discontinued the
old system, shut it down, took it off, threw it away. The new one
wasn\'t ready yet. It wasn\'t ready for another year, and so during that
year there was the Indian lawsuits, and orders from the federal judge to
take all their stuff offline, and all that kind of stuff. But the main
thing for the MMS production data was that it was an IT operation, I
guess.

MR. EDMONDS:  And who\'s producing MMS, is that DOE, DOI?

MR. WOOD:  DOI.

MR. EDMONDS:  So you have an inter-agency problem, as well, layered on
top of all the other problems.

MR. LONG:  We have a pretty good working relationship with MMS. It\'s
just within their own shop they have these problems.

MR. BURTON:  So the producers are producing the gas. They\'re selling
it, they\'re reporting how much they sell within 65 days.

MR. LONG:  Right.

MR. BURTON:  And this is what we got?

MR. LONG:  Right.

MR. BURTON:  You don\'t need a statistician.

MR. WOOD:  We\'ll take that as the Committee\'s recommendation.

(Laughter.)

MR. LONG:  No, there are ?? you know, a lot of this is, in fact, a
problem that arises from the way the data was processed by the Mineral
Management Service of your Department of Interior. But there ?? even as
they straightened that out, there are still always lags and corrections
to the data, which don\'t sometimes get resolved for years. And so it
could be three or four years before ?? or more before all accounting
irregularities are settled. And that process goes on all the time,
reconciliation of what you thought you paid for, and what someone else
thought they delivered.

MR. LONG:  Any other comments?

MR. NEERCHAL:  I think already you\'ve said all of it, but can we know
anything about the location, probably be another location of these well
completions?

MR. LONG:  Yes.

MR. NEERCHAL:  And is there something, you know, some engineering
information about what the well production should be or should not be?

MR. LONG:  All of our information comes from reports, company reports to
the MMS. Now there\'s a whole boat load of forms that they fill out that
go into the MMS, and we have access to a lot of that data. But a lot of
it has the same problem that the production data does.

MR. BURTON:  In essence, you\'re telling me you\'d have the same problem
in Texas if it wasn\'t for the Texas Oil Commission.

MR. WOOD:  Going back to your question, there is one of the fundamental
operations that a petroleum engineer does is to analyze how a well will
decline under various consumptions. And so in that class 11, you saw
that when the wells were producing 90 million a day, and then 80, and
75, that\'s a very ?? in the aggregate a very predictable decline. And
so if there\'s a gap, and you assume that it shifts the data in there,
and so it ?? and there\'s a lot of engineering data, the pressure at the
bottom of the hole, et cetera, et cetera, that tells you, and just
operating procedure, what you would tend to expect. But you can model
all that, but until you actually get the data systematically in to
verify it, you know, then you\'re where we are.

MR. NEERCHAL:  And also, you know, you can ?? there is a part I don\'t
quite understand, because there is ?? even if the well is \"producing\",
they can not release it. Right? So that would be zero production. Right?
If they turn the tap off so the true report that day should be zero,
even though it is producing and building more pressure, obviously.

MR. WOOD:  More typically, they may shut a well that could be producing
100 million a day down for two weeks, let\'s say, if they wanted to cut
the production. So now it\'s producing 50 million. Now that is possibly
a way to, rather subtly, use the distributions to predict a production
change. Gary is a little nervous that we may never be able to do this,
but since he\'s going to do it - not me - I\'m going to suggest - and I
will stop here and say there\'s a sign in our conference room that is,
\"Nothing is impossible for the man who does not have to do it
himself\".

(Laughter.)

MR. WOOD:  But we\'ve already played with the data, you know, just to
make sure that we were really thinking it through correctly. You just go
in and say every single well dropped its production by 20 percent. How
does the distribution inside the classes change? Every single well drops
its production by 10 percent, how does the distribution of wells inside
the class change? And then if the data were consistent enough, you could
look for subtle changes in the distribution and say the production is
going to drop 3 percent this month. We\'re not there but, you know,
it\'s ?? it is in fact, in thought experiments, you know, you can tell
exactly what happened. Wells flow into a class, if you just cut back
some production, the actual mean in every class goes up, because you
drop out some low ones into the following class, and you gain some high
producing wells from the class above you, which was actually kind of
surprising when all of the models predicted that and, in fact, it
happened. I guess I shouldn\'t have been surprised.

MR. NEERCHAL:  And I should ?? you know, it was ?? I mean, I have not
read the papers. Obviously, my comments should be taken in that light.
You know, the way you are dividing things into a class, you know, it\'s
not clear to me. And I\'m kind of just wondering exactly what role does
that function play, because I can think of myself, if there are 6,490 -
I\'m sorry - six four five seven wells, I order them from, you know, the
highest to lowest and keep on chopping 10 percent, 10 percent, 10
percent. And that will give me 10 classes, for example, even without the
declining function. And so it is not clear to me exactly what the role
the function is playing, that\'s number one comment. And the other
comment is that I am curious as to is there a pattern in which certain
wells kind of come on production and off production, on production and
off production? If there is a pattern over time, and something like
that, that is in addition to the distribution. Because when you do the
distribution, you kind of lose the well, or completion level
information, because you\'re grouping everything together.

And also, you know, the big players, it may be worth monitoring them
individually. Those guys producing lots of oil, you know, I\'ll put a
tap on their phone, I think, to see what exactly is going on, you know,
or statistically speaking maybe what modeling ?? you know, keeping ??
pay special attention to those guys, instead of just grouping them. You
know, those are very ?? obviously, a very distant view, I should say,
because I don\'t ??

MR. LONG:  Well, when we get production data by well completion for all
6500 and some of those completions, and so that\'s possible to do that.
It just takes time to sit down and do it, set up a system.

MR. WOOD:  By the way, we did the very thing you said, and you notice
that those classes there, except for the two smallest ones, have roughly
10 percent of the production.

MR. NEERCHAL:  Yes, I see.

MR. WOOD:  And it\'s not quite an accident. But some of the things we
hoped would be okay, the ?? some standard algorithm would put you with 6
wells, or you\'d have 10 wells, and it only changes to 13 wells, and it
only changes to 21 wells. And this way, the number of wells in the class
is more like a 50 percent change as you come down the class. And it made
the changes in the way the wells were producing more obvious, and just
easier to look at. So the absolute class limits are not particularly
important, as long as they each have a reasonable number of wells, and
they each have a method of putting a well in that class, the upper and
lower boundary.

MR. NEERCHAL:  Uh-huh.

MR. HENGARTNER:  I have a question on well completion. I\'m going to
admit that I didn\'t understand what the well completion was, but
something that hasn\'t been done or should probably enter in your
calculations also, the decommission. Because you\'re trying to predict
how many new wells are going to come in, or anticipate the number of
wells. Is that correct? So you\'re trying to fit actually the observed
number with the expected number, and that ?? I think I\'m going to agree
with Nagaraj, that there\'s something a little bit fishy there. In
particular, when you do the ?? even the suggestion, for example, of
zeroing out and say oh, I already have the expected value. Well,
there\'s nothing wrong. I mean, if it\'s random, there might sometimes
be more, and sometimes be less. If you only add in wells when they\'re
less than that, you\'re going to actually have a downward bias, so
there\'s a sense in which maybe, for example, that particular adjustment
might not be needed. But also understanding when wells are
decommissioned, when do you not need to make an imputation. You don\'t
report because you will never report again.

What you\'re looking at is that ?? in the original well and you see its
production month, after month, after month. When they stop producing, or
they decide to close business, I mean, that\'s something that should
come into the consideration. And I doubt you have that data. And I doubt
that it\'s easy to get, but it\'s simply something ?? as I said, I\'m
not going to do the work. Right? Why bring up ?? the other thing that
you should be considering is really going on with this idea that you put
in a completion. It\'s going to produce. And there is somehow a time
series of production, and indeed, the production is going to decrease
over time. And somehow there is information in there. I mean, this idea
of ?? I was saying finding the conditional distribution going from one
class to the other is a poor man\'s way of doing that. A better way
would be if you had the age of the well. If you know that this well was
then ?? or this completion was done last year, that\'s part of the
information that you should be able to say yes, I\'m going to have to
impute something. And furthermore, since it\'s one year old, this is the
kind of production I expect.

So what I\'m seeing here is, in hindsight ?? as I said, I read this very
carefully, and I\'m very thankful for Gary\'s presentation, because he
set me straight, and he did a great job. But there\'s, I think, a need
of trying to model the production of each individual well, if that\'s
really what is needed. It might be overkill at this stage, but it\'s
something maybe to do in the aggregate. And there might be things one
could do. But the age of the well is clearly an area that might matter,
as well as other engineering expertise.

I mean, if I knew the pressure of each well, that would be helpful. I
don\'t think I have that. I don\'t think you have that.

MR. WOOD:  Yes, I do.

MR. HENGARTNER:  You do?

MR. WOOD:  The bottom hole pressure is a standard engineering ?? of
course, sometimes they pencil the number in, but they\'re supposed to
run at least one test a year, Gary?

MR. BURTON:  But, I mean, you still have the issue of them ??

MR. HENGARTNER:  Of them shutting it down.

MR. BURTON:  Yes. Adjusting production based on market conditions. Let
me ask you, how closely is Gulf production tied to Texas production? Are
they correlated?

MR. LONG:  Somewhat.

MR. BURTON:  I would think they\'d have to be somewhat. Is there some
way to use that correlation to try to fill in the gap?

MR. LONG:  We\'ve made estimates and ??

MR. WOOD:  We did that with three other states, and that was our limit.
They are ?? you come at this from an economic viewpoint? There used to
be a very large price differential between the gas that was produced in
the Gulf of Mexico, and was delivered to the shoreline, versus the gas
produced one mile inland. And it was because of all the contractual
relations. As things were deregulated, that went away, as you might
hope. What doesn\'t change is some of the operational procedures in the
very large wells on a month by month basis, where you can change their
production by a factor of 3. They can produce 1 percent or a tenth of a
percent of all the production in the Gulf of Mexico in one well.

We have a series of reports called the Natural Gas Production Capacity
in the Lower 48 States, that has maybe a 15 to 20 year history. And in
that, the aggregate production levels of all the wells drilled year by
year are actually calculated, plotted, and then those curves are brought
forward and the expected production from new wells are added in. And, in
fact, we\'ve put on the website the distribution of new well production
for almost every state. But in the Gulf, there has not been for the last
few years a consistent data set where we know how ?? you know, we can\'t
put that file together yet, even though that\'s what we\'d really like
to do, that you lay in every month the distribution of wells and their
production rates, how fast they\'re going to decline. And that is as
good as the model gets. And we just ?? that would be a final plan. And
so that was good thinking.

MR. HENGARTNER:  As I said, it\'s wishful thinking at this stage. The
other issue that came up, or my thinkingwhen I read these papers, why do
you we care about the classes? In some sense, we could actually go
directly to the production of each individual well. I mean, the idea of
dividing it into 12 classes is a useful device. It simplifies our
calculations. But, in fact, we could say well, okay. I mean, I know
again ?? it goes back ?? I know the history. Let\'s make projections
without looking at the classes, knowing full well that there are
problems. And as you say, if they turn off the knob, they won\'t
produce. But nevertheless, I mean, one could actually even thinking
continuous production, I have yet to be convinced that going into those
12 bins beyond a clear simplification of the model, brings me anything.
So the answer might be it\'s too complicated, let\'s forget it, which is
fine. But it\'s worthwhile, a little bit of thought, say well, could we
actually get a rate by trying to look at the production of each well as
that comes from a continuous distribution. And I don\'t need to bin it,
and that might resolve a little bit of this arbitrariness feeling that
some of us initially had, and maybe still have about how the binning was
done. I don\'t think it\'s a big issue, honestly. I don\'t think that I
would spend much time on it, on the binning issue. However, thinking if
we can make it in a continuous model, it might actually simplify some of
the modeling later on. That\'s a suggestion. Again, it\'s a matter of
how complicated things become.

MR. WOOD:  Again, I think Gary and I would proceed exactly as you
suggest as soon as we\'re close to having enough data. But the original
files were missing half the wells, and so you ?? and they were not
consistently missing. Every other month for 24 months might be missing.
Now that is not a particularly reliable data set to fit. If the data was
continuous, you know, that you had up until a certain point almost all
the data, then you could project forward with some confidence. And
eventually, we\'ll be able to do that. We think we can do it in Texas,
for example.

MR. HENGARTNER:  But if you have a model you can also fill in the holes.
I mean, that\'s the nice thing about the model is that not only can we
make projections, but there are methods for ?? I mean, I don\'t want to
use the word imputation because it has many connotations, but in some
sense that\'s what you can do, is say let\'s just plug in the holes that
we don\'t have, so you don\'t actually need everything.

True, if you don\'t have all the data, your estimates are going to be
less reliable. I knew that. Nevertheless, when ?? I mean, models are
useful. Models are to supplement data. And if the models are good, then
supplementing the data might be useful, or at least enlightening. If the
model is rotten, I mean, you won\'t get anything either. It\'s not a
silver bullet. And I\'m not claiming I have ?? but it\'s ?? I think your
objection because the data is not complete wouldn\'t have stopped me.

MS. KIRKENDALL:  From trying to model.

MR. HENGARTNER:  To model, yes. I think ?? well, I mean, because you can
try to model, and once you have more data you can be happier, but it
should not be a barrier to say I\'m not going to do anything before I
have the full data. I mean, it might not be a good model, and you might
not even realize it until you have the full data set, but it still could
be done. I don\'t think that that should be a stumbling block.

MR. NEERCHAL:  And I think your individual well model has another
implementation. I think that you might be able to take advantage of
neighboring well information, which is the same company owns several of
us, that kind of information. Because it seems to me that this is not ??
we are not just trying to model a natural phenomenon only. It\'s also
combination of a natural phenomenon, plus some production positions.

So that is what I think is throwing a wrench in a lot of straight
modeling approach. Basically, some advantage of doing these classes is
that, you know, then instead of the physical characteristics, you are
actually looking at the production capacity and things like that, that
are more the drivers for the production decision. So this classing,
binning method has some advantages because it takes into account those
kind of same drivers for production and those kind of positions.
Whereas, the individual level well model, you can take into account the
physics much better, or the engineering information much better. So you
need to find some kind of a combination of the two.

MR. HENGARTNER:  But you can actually include ?? I mean, ownership ??
some graphical information, I mean, proximity to other wells also in a
well-by-well model. I mean, you have a spatial model with a few
covariates, age, company, so forth. So I think that has a lot of
potential.

MR. EDMONDS:  Well, what I am hearing you say is to move down the road
toward actually ?? I think rather than wells, you really want to think
in terms of the reservoir, because you\'re actually tapping a reservoir
and the number of wells in some places, an economic reflection of how do
you mine that resource. And that\'s what gives you your ?? I mean,
you\'re going down that route, you\'re ultimately taking not only the
engineering information, the geologic information about, you know, what
does the reservoir look like. But you, in fact, moving down that same
road as the producers, certainly the large producers. I mean, that\'s
how they\'re making their decision. They\'re building a model of that
reservoir, and they\'re looking at the price structure.

On the other hand, it seems to me that what you set out to do is you\'ve
got some missing data in an 18 month period, and you\'re trying to fill
in some cells in a way that\'s not entirely out to lunch. And if you go
down this other route, it sounds like you\'ve got a fairly
straightforward way of filling in these data.

Now it would be really nice to have some retrospective; that is, going
back and say does this thing ?? you know, is this a good practical way
of doing it, before you decide to abandon it and go down, you know, what
I would think would be a fairly large scale investment in modeling and
analytical capability. Now I may not be right, but my sense is that
modeling the Gulf reservoir basins and production basically on a
reservoir by reservoir basis, I would ?? if you\'re going to go that
route you only need ?? the wells become an intermediate variable rather
toward the final variant, which is production. I don\'t know. That seems
like a big task.

MR. NEERCHAL:  I think I got lost in the trees, so I\'m asking a forest
question now. So final objectivity is really to come up with one
production number for the entire Gulf of Mexico, or ?? I mean, how
detailed an estimate are we looking for, at what level? I should have
asked that the first thing, right?

MS. KIRKENDALL:  One number for the Gulf of Mexico.

MR. NEERCHAL:  One number for the Gulf of Mexico. See, that is one of
the things that I feel that, you know, 600 will get 10 percent missing,
you know, out of potential 6,500, you have 600 missing. I don\'t know
how big a deal is it.

MR. HENGARTNER:  Well, if one completion can make 1 percent difference.

MR. NEERCHAL:  No, I understand. How many of them are in the other ??
there cannot be too many because all together, there are maybe 6 or 7
wells there, so there could be one guy who is missing there. You cannot
have 600 missing in c-11. There\'s no way. Right? Because there are only
10 possible candidates for there, so most of the 600 are in the C-0-1
probably.

MS. KIRKENDALL:  Well, we should wrap this up. I think it\'s time to ??
I didn\'t realize the entire other group was back.

MR. LONG:  Let me just say one more thing about the production decline
analysis, which is what you\'re talking about on a well by well basis.
You can do that provided you have sufficient history to forecast a
decline. And it doesn\'t work very well if you have curtailed
production. Like if your production is curtailed on a platform, in other
words, a well can\'t produce all it could produce because of production
facility limitations or whatever. It doesn\'t work very well when you
have curtailed production, and you need some decent period of history.

(Whereupon, the proceedings returned to the regular session.)

MS. KIRKENDALL:  I guess we\'re supposed to have our wrap-up. Should we
go ahead with that? And we have our chair back. Now I guess, Nick, you
get to do the summary.

MR. HENGARTNER:  I get to do the summary. Well, if the whole group was
here ??

(Laughter.)

MR. HENGARTNER:  I will keep it short. The problem is some ?? as far as
I understand the problem now, is to try to impute production of
individual wells or completions in the Gulf of Mexico. The situation is
analogous, although not exactly the same as what happened in Texas. The
big issue is data quality. There is ?? they presented an interesting
methodology, in that they divided the output of wells into categories in
trying to predict how many wells in each category there ought to be, and
trying to proportion the ones that didn\'t submit production into those
12 categories, and then use that to complete the data. It seems like a
very reasonable approach, one that seems to work well, although the plot
so far doesn\'t show much difference, unfortunately with ?? if you look
at the plot, they didn\'t show much difference with the naive, just
replaced the missing data production by the average production. It makes
some small differences, so that\'s still something to be explored.

And then we went on on the modeling band wagon, of which I am
responsible. I apologize. And in part, it\'s ?? I think some of the
feeling of some of us here is that if you really wanted to get the
numbers for the individual well correctly, then new effort might be
needed. I think the counterbalance is we may not need to do that much if
we just want one single number for the Gulf of Mexico. And we may not
need to duplicate the efforts done by the oil companies, which are in
the business of doing similar things.

What else can I tell you? I think ?? I mean, personally I find this
problem very interesting. I hope there\'s going to be a follow-up on it.

MS. KIRKENDALL:  Probably.

MR. HENGARTNER:  If there\'s probably going to be one ??

MS. KIRKENDALL:  Yeah, I don\'t think this problem was solved, so ??

MR. HENGARTNER:  No. I think this was more an introduction to new
problems, which I think this group will have future exposures.

MR. BREIDT:  Polly will summarize the other breakout.

MS. PHIPPS:  We looked at the survey of marketers, basically, for
residential and commercial customers. Basically the survey of marketers
is to fill in the gap in monthly natural gas prices, so there\'s a
survey ongoing in five states now called the 910. It goes out to
marketers to report the volume sold and the revenue. And then it\'s used
with the 857 to report volume weighted prices.

So basically, what Tom and the group did was look at comparing the 910
to the 857. This is for the volume that\'s transported by the long
distance carriers, but sold by the marketers, so they looked at the
comparison of those two, the 910 to the 857 ratios for volume and for
prices. And then they looked at the price estimates and potential bias.
And really, the data looked pretty reasonable, and looking at the
national estimates, except for maybe in one state, there wasn\'t any
real type of bias based on some early numbers that he gave.

So there are questions, and the first question for was new states for
the survey. And first we kind of went around and talked a little bit
about doing a sample survey of marketers, basically looking at a sample
that was kind of staggered in time and in location, but kind of stopped
talking about that because it sounded like logistically it wasn\'t a
possibility.

And then talked about really looking at most states account for a
trivial amount of these off-system sales, so to take like the 10 states
that have the greatest share and survey that group, and you can do
estimates basically to see if there are any kind of changes in how it
affects the national estimates.

The second suggestion there was to look at kind of do some kind of a
different kind of function in selecting states, where you look at the
number of marketers, you look at the percent of off-system sale, and the
total volume in a sense that if you have one marketer in a state, it
doesn\'t cost much more to go in and survey that marketer. So basically,
those were our suggestions on how to select the new states.

The second question was do people want state level prices, or does it
really need to be a smaller area? And I guess we felt like that needed
to go to the users. We really weren\'t able to answer that question.

The third question was should EIA show both the 910 and 857 prices, or
is the average okay? And I think the sense was that probably the 910
couldn\'t be shown due to confidentiality, the five states marketers.
Although somebody might want to expand on that because I wasn\'t ?? they
talked a little bit more about the confidentiality issues.

And then the last question was do the variance calculations that treats
910 as a constant, is that okay? And yes, because there isn\'t any
sampling. Basically, it\'s a census of all the marketers in the state,
so that was a summary. And anyone feel free to add, because I didn\'t
put in the details. We talked about a number of things.

MR. BREIDT:  Okay. Let\'s see. I guess at this point I\'ll turn things
over to Bill, who has a few announcements about tonight and tomorrow
morning.

MR. WEINIG:  For you Committee Members who are joining us for dinner
this evening, the restaurant called Lee Garden is expecting us at 6:00,
or before then if you like. There is a cash bar, and we can order from
either of four menus. Somehow the menus got away from me today, and I
apologize for that, but when we\'re over there we can get menus and take
a look at them. And they\'re very accommodating, and I expect that
we\'ll get individual checks for those of you who are dealing with the
ASA reimbursement policy.

The second thing ??

SPEAKER:  Where is the Lee Garden?

MR. WEINIG:  629 H Street, N.W., and I would recommend from the hotel
that you walk down to the Metro and take one of the two lines that goes
up there. I think it\'s only two stops right to China Town. Get off
toward the front of the train and take the escalators to the street
level.

SPEAKER:  Which station?

MR. WEINIG:  China Town. Gallery Place. And when you get to the top of
the escalator, you\'ll see it across the street. It\'s between the CVS
and mid-block on the other side of the street, very convenient.

Then the second thing is, I\'d like to share with you, tomorrow morning
we\'re going to have a change of location after the break. Here\'s
what\'s going on. Tomorrow morning there will be a fire drill at 11:00
in this building, and for that reason, we thought that would be reason
enough to move to L\'Enfant Plaza. We have offices on the 6^th^ floor at
L\'Enfant Plaza, at 950 L\'Enfant Plaza. It\'s the building that as you
come out of your hotel, it will be the building on the left, and it\'s
between the hotel and the Potomac River.

The meeting will be on the 6^th^ floor in Room 6089. We\'ll have lunch
up there after the conclusion of the second session, so the first
session will be Doug Hale on the adequacy of electric transmission data,
and the second session will be shorter, and we are interested in your
feedback on suggestions for the spring 2004 meeting. As always, we have
found that very productive use of 30 minutes. So we\'ll reconvene, those
of you who are going to dinner, we\'ll see you at Lee Garden, 629 8^th^
Street, N.W., China Town, Gallery Place at about 6:00.

SPEAKER:  Yellow Line or Green Line.

MR. BREIDT:  Yellow Line or Green Line, and one way to get that is just
to go down underneath both lines and come back up, and then take the one
to Gallery Place.

MR. HENGARTNER:  Before we leave I\'d like to announce, some of us are
trying to go to the Blues Alley after dinner. The truth is, I don\'t
know D.C., so which means we\'re just going to go ?? I\'m not sure where
it is. It\'s in Georgetown?

MR. WEINIG:  It\'s in Georgetown, about a block and a half from the
intersection of M and Wisconsin.

MR. HENGARTNER:  So please, if you have time and energy, join us.

MR. WEINIG:  Thank you, Jay. You want to ask for public comment?

MR. BREIDT:  Okay. So one quick announcement. I need to ask, to invite
public comment. Is there any public comment? I didn\'t think so, so the
other is that if you\'re staying at the hotel and want to meet in the
lobby at 5:45 to go to dinner together, that would be great. So we\'re
adjourned. Thank you.

(Whereupon, the proceedings in the above-entitled matter went off the
record at 4:57 p.m.)