AMERICAN STATISTICAL ASSOCIATION

\+ + + + +

COMMITTEE ON ENERGY STATISTICS

\+ + + + +

FALL MEETING

\+ + + + +

Friday, November 3, 2000

\+ + + + +

The Committee met at 8:30 a.m., in Room 8E-089, Forrestal Building, U.S.
Department of Energy, 1000 Independence Avenue, S.W., Washington, D.C.,
Dr. Carol Gotway Crawford, Chairperson, presiding.

PRESENT:

CAROL A. GOTWAY CRAWFORD, Ph.D., Chairperson

F. JAY BREIDT, Ph.D., Vice Chairperson

THOMAS G. COWING, Ph.D.

JAMES K. HAMMITT, Ph.D.

CALVIN A. KENT, Ph.D.

W. DAVID MONTGOMERY, Ph.D.

WILLIAM G. MOSS, Ph.D.

PRESENT (Continued):

POLLY A. PHIPPS, Ph.D.

RANDY R. SITTER, Ph.D.

ROY W. WHITMORE, Ph.D.

GUESTS PRESENT:

JOHNNY BLAIR

NICOLAS HENGARTNER

ALSO PRESENT:

STAN FREEDMAN, Designated Federal Official

MARK MAZUR, Acting Administrator, EIA

LARRY PETTIS, Deputy Administrator, EIA

C-O-N-T-E-N-T-S

PAGE

Introductions 4

Recognition of Dr. Calvin Kent 7

Monte Carlo Analysis of Uncertainty in CO~2~

Emissions, Perry M. Lindstrom 12

Discussant Jay Breidt 31

Discussant Roy Whitmore 43

Optimization and Visualization of the N.A.

Easter Interconnect Power Market:

Douglas R. Hale 63

Thomas J. Leckey 69

Discussant Bill Moss 87

Discussant Calvin Kent 102

Feasibility Study for Surveying Industrial

Natural Gas Consumers, Renee Miller 113

A Briefing: Winter Energy Assessment for

the U.S., Neil Gamson 133

EIA Research Fellow Program, Nancy Kirkendall 149

P-R-O-C-E-E-D-I-N-G-S

(8:33 a.m.)

CHAIRPERSON GOTWAY CRAWFORD: We\'d like to go ahead and get started.

There is some unfinished committee business. First of all, for members
of the committee, there is a sheet that the ASA puts together that has
biographies of all of you on them.

So, Kevin, you\'re exempt from this since you won\'t be here for the
next meeting. If you\'re on here, could you look at your bio and revise
it? Maybe your interests have changed or you\'ve done something else
that you\'d like to put on here.

There\'s on the front, I think, like Tom\'s and Calvin\'s and Jay\'s,
more of the format that they kind of like, you know, your title, where
you came from, your research interests.

So if you have some changes to be made, let me know, and I\'ll
coordinate that with the ASA office.

Any ASA committee member, member of the public, or EIA staff that
wasn\'t here yesterday is asked to introduce themselves. Were you here?

MR. LINDSTROM: I was here, but I didn\'t introduce myself.

CHAIRPERSON GOTWAY CRAWFORD: Well, introduce yourself because you\'re
important today.

MR. LINDSTROM: I wasn\'t important yesterday, but \--

(Laughter.)

MR. LINDSTROM: \-- I\'m Perry Lindstrom, and I\'ll be doing the talk in
the Monte Carlo analysis, and obviously EIA.

MS. KEARNEY: I\'m Diane Kearney, and I work in OIAF working on the coal
model.

MR. DORSEY: Bill Dorsey from the Office of the Administrator.

CHAIRPERSON GOTWAY CRAWFORD: Could you speak into the microphone? Sorry.

MR. DORSEY: Bill Dorsey in the Office of the Administrator of EIA.

CHAIRPERSON GOTWAY CRAWFORD: Okay. Thank you.

And you\'re also supposed to sign in at the desk either now or at break
before you leave.

And lunch for the committee will be held after the meeting is over. I
probably should say lunch for the committee and invited guests will be
held at the conclusion of this meeting in the same room where we had it
yesterday, E-226.

Returning to the position of Vice Chair, are there nominations for Vice
Chair from the floor? Bill.

DR. MOSS: I nominate Jay Breidt.

DR. KENT: Second.

CHAIRPERSON GOTWAY CRAWFORD: All in favor?

(Chorus of ayes.)

CHAIRPERSON GOTWAY CRAWFORD: Any opposed?

PARTICIPANT: Jay\'s opposed.

(Laughter.)

CHAIRPERSON GOTWAY CRAWFORD: Congratulations, Jay. You\'re our newly and
duly appointed Vice Chair.

(Applause.)

CHAIRPERSON GOTWAY CRAWFORD: Now I\'d like to move to Mark, who is going
to recognize Calvin for all of his work on this committee.

MR. MAZUR: Okay. Dr. Kent, first of all, I just want to say I appreciate
the opportunity to get to speak with you in the last day or so. I found
that a very good experience and had heard a lot of good things about
you, pretty much all true. So I appreciate that.

What I want to do today is just express the appreciation of EIA to you
for six years of service to the committee here, and like I said earlier,
we value the contributions of folks on the committee, and we especially
value the contributions of former Administrators since they both know
the subject matter and the way EIA works and provides a good mix for
getting us to do our work better than otherwise we would.

I also want to present you with two things. First is a letter from
Secretary Richardson thanking you for your outstanding service as a
member of the American Statistical Association Committee on Energy
Statistics. He notes that you provide valuable advice and your work has
contributed greatly to the success of EIA. I want to second that.

DR. KENT: Thank you.

MR. MAZUR: This is from Secretary Richardson.

DR. KENT: Okay.

MR. MAZUR: And then second, from us at EIA, I just want to present you
with a certificate for appreciation of your public service to the
committee. I very much appreciate it personally.

So thank you very much.

DR. KENT: Thank you very much.

(Applause.)

CHAIRPERSON GOTWAY CRAWFORD: Calvin, did you want to say a few words or
\--

DR. KENT: I don\'t want to get us behind, but I will say just a couple
of things.

I don\'t know why I was thinking of this, but I do remember it was a few
years ago back when I was in Texas that the champion broncobuster, whose
name at that time was J.R. Cody, got thrown from his horse and was
trampled, and as he was being carried bloody and broken to the ambulance
was heard to comment, \"Boy, wasn\'t that a great ride?\"

And I kind of feel that same way.

(Laughter.)

DR. KENT: Boy, it\'s been a really great ride. I will miss very much
being voted down in the annual graphics contest every year, which you
have consistently done. It was obvious that the ghost of Yvonne Bishop
stalks us in this endeavor.

But it has been for me a very great privilege to get to serve on this
committee because when you finish your term as Administrator, and in my
case went on to become a Dean of a business school, I wondered how I was
going to be able to keep up with what the developments were in energy
markets, and this has been a phenomenal way for me to be forced to
continue with my interest in energy and not completely get bogged down
with the problems of griping faculty and dysfunctional buildings and
limited budgets and those other things that deans spend the vast
majority of their time on.

And so it has been a way for me to keep myself renewed, and I\'ve been
very, very grateful for that, and I hope that I\'ve been able to bring
some of the insight, some of the historical to the committee that the
committee would not have otherwise had and experienced, and I\'ll even
be talking some more about those when I comment on Doug\'s paper.

But the real pleasure of being part of this committee has been the
opportunity to associate with some really wonderful people, many of whom
I hope will continue to be my friends and my colleagues in the years to
come because I have enjoyed you all immensely, and I am going to truly
miss coming to Washington and having the opportunity of being with you.

Carol did ask me to prepare a short history of this committee, and I
have decided to expand the project. This was to put on the ASA Web site,
and I think I am going to do an analysis of the impact of this committee
on the gathering and dissemination of energy information to try to
determine what impact this committee has actually had, and can we say
that this committee has accomplished, as I feel we can, the goals that
this committee was established or that this committee had when it was
established. Have we made a difference?

And I hope that when I\'m through with that analysis that maybe the
committee will invite me back and allow me to share then that analysis
with you. I hope to have it done about a year from now and to be through
with it and to be able to make the point that these committees are not
an academic exercise because I know that there\'s always the feeling
from new members as they come on the committee does EIA pay any
attention to us. Does what we do and what we say here have any impact?

And I think it is true that this committee has had a phenomenal impact
in its 20-plus years of existence, and I just intend to write that up
and to be able to demonstrate to anyone who would question the need of
the efficacy of such an organization that, indeed, this organization
has, this committee has done what was expected of it, and that the
committee\'s work needs to continue because of the tremendous value that
it provides.

It\'s been fun to be a part of it. I\'m going to miss you all. I hope to
see most of you again in the future.

(Applause.)

CHAIRPERSON GOTWAY CRAWFORD: Thanks, Calvin.

I\'ll turn it over now to Harry Lindstrom, who will make his
presentation on Monte Carlo analysis of uncertainty in CO~2~ emission

MR. LINDSTROM: This was easier when I was younger. Let\'s see. Where\'s
the high tech device?

Let\'s see. Was everybody here last spring?

MR. WEINIG: No.

MR. LINDSTROM: So there are some new people. This is really part two.
Last spring I presented kind of the proposal, and so this is the
follow-up to that.

Originally there was just going to be kind of one analysis of energy
CO~2~ emissions, and I was hoping to have a paper by now. The bad news
is I really don\'t have the finished paper. There has been a very early
draft that the commenters have looked at.

The good new is that we\'re going to expand the analysis and do some
more and do some of the other gases, and we\'ll probably be working on
it through mid-summer and then maybe I\'ll have something, you know, a
good paper to present probably a year from now, if I\'m asked back.

Oh, by the way, there is these packets that I gave out. Page 19 through
23 is just a substitution for the packet that you already have. So when
you get to that you can refer to those.

Actually I should point out that it\'s been kind of a busy week. This is
the \'99 report of emissions of greenhouse gases. It went up on the Web
Tuesday morning and went to the printer. It\'ll be back in a couple of
weeks in terms of hard copy, but if you want to download it, it is in
PDF form right now on the Internet, also HTML.

And so that\'s what we\'re doing the analysis on, the data that goes
into this report. In your packet I\'ve got kind of a review of what I
talked about at the beginning of the packet last time, but I\'m not
going to spend much time on that. I really want to get more into the
factors that go into the uncertainty, the results of this kind of first
analysis, some of the lessons that we learned, and you know, where
we\'re going to do from here.

Just a quick review. This is from page 9 of your handout. This is why
we\'re so interested in the energy CO~2~ component. It\'s why it\'s the
first thing that we did, and of course, that\'s the area that we work
in. That\'s why Congress way back in 1992 got us involved in the
greenhouse gas business, because of the large component of the energy
related carbon dioxide.

You see that the other carbon dioxide is another two percent. Methane is
about nine percent; nitrous oxide, six percent; and the other more
exotic, manmade gases are total two percent.

These ratios tend to remain consistent from year to year.

Just a very brief history of why we came to do this. In October of 1999,
there was a meeting of the Intergovernmental Panel on Climate Change in
England, and their Committee on I think they\'re called Good Practices
rather than best practices for some reason recommended that we do a
so-called Tier 2 analysis of uncertainty, not just us, but all of the
other Annex I countries, which are essentially the developed countries,
Tier 1 analysis being a more simplified square root of the sum of the
squares when you combine the uncertainties, and Tier 2 analysis being
the Monte Carlo method, which is considered to be better. At least
that\'s what we\'ve been told through that committee, that it\'s better
for the combining non-normal distributions as some of our probability
density functions are.

Right now, we\'re just going to finish the first part of the energy
CO~2~, but we\'re going to do some more analysis in that area. I think
we need to do some sensitivity analysis, and I\'ll be talking about that
later.

Then we decided that we should go ahead and do the other gases even if
it\'s just a first cut at taking the Tier 1 numbers that we\'ve already
kind of put together and moving them into the Monte Carlo approach.

EPA is going to do a more in depth analysis in which they\'ll be going
out and talking to the experts in, you know, the various manmade
chemicals and doing some very extensive elicitation, which we don\'t
have the resources to do, but at least we\'ll be able to put all of the
gases in the same format and produce an overall number for all of the
gases, and then I think it\'s going to take about two years for the EPA
analysis of the other gases.

And as you see if you look through the packet, and if you were here last
time, we do have a memorandum of understanding with EPA where we both
work on this stuff together, and we provide them with the energy data
that then goes into the official inventory that EPA submits to the U.N.
usually in the spring of the year after we do our report.

So their report just came out for 1998, and their \'99 report will come
out probably next April.

Now, this is really the crux of what drives the analysis, and what we
realized is you really have to spend a lot of time on getting the right
numbers because after this part it\'s basically mechanical, and if you
don\'t get the numbers right here, you know, it\'s the old garbage
in/garbage out principle.

So talk about a couple of the categories, starting with the bias. Of
course, the maximum represents an under bias, a fear that we\'re under
counting; therefore, you know, we\'re moving it up by that amount.

If you look at, for example, the petroleum, these are for the various
sectors, residential, commercial, and transportation. If you look at the
petroleum category for the bias, you see that there\'s more symmetry
than in some of the other categories. That\'s because there\'s more of
an opportunity for over counting because of double counting the
complexity of the petroleum industry. That\'s a higher probability event
than in some of the other fuel categories. So that\'s why you see, for
example, more minimum bias here than you would, for example, in coal.

When you look at, for example, non-fuel use, you see the potential here
of four percent under bias. That\'s probably \-- well, that\'s mainly
because of the issue \-- now, this says non-fuel use, but it\'s
emissions from the non-fuel use of petroleum.

So, for example, oil, motor oil, technically it\'s a non-fuel use,
doesn\'t get burned. However, under EPA regulations it is now supposed
to be disposed of in that way.

Now, we\'ve got a little difference of opinion with EPA over this
particular number because we think, well, maybe people are supposed to
burn all of their motor oil, but probably not all of it is being
disposed of in that way. So it\'s just a judgment as to how much of it
is going to end up in the atmosphere versus how much of it is dumped.

If you look at the coal category, for bias, you see a fairly significant
possibility of an under bias in the electric utility area. I\'m going to
be talking about this later. It\'s a fairly important number because of
the size of the category.

Even though the under bias is potentially higher in the other sectors,
commercial, residential and industrial, it\'s a much smaller number.
Therefore, it doesn\'t represent as much total uncertainty.

In the natural gas category, you see that the great difference between
flared gas and the gas that goes through the economic system, and
obviously we have a situation where gas is flared. There\'s no real
reason to count it. People don\'t keep very good counts of it.

In fact, we get a category that\'s flared and vented, in which case if
it\'s vented, it\'s a methane emission, which is after you account for
the weight of the CO~2~ versus the methane is about a seven times
difference.

So that\'s another uncertainty. It doesn\'t affect directly the CO~2~
number. It really affects the overall number, but we really don\'t have
good numbers on that, and that\'s obviously reflected in these high
percentages here.

And you\'ll see that even though that\'s a low emissions category, that
high uncertainty drives a fairly large overall uncertainty.

Now, this has all been activity data I\'ve been talking about. When you
look over at the emissions factors themselves, you see that there\'s
much more symmetry in the bias, potential bias errors because it\'s
really a function of what you chose when you first came up with these
numbers for the coefficients, and so you\'re going to get a very \-- you
know, it\'s not likely to be under or over biased in any different way.
It\'s just a function of the range you could pick up when you were first
sampling.

So you see a little more consistency there, and some of the \-- the more
homogeneous the fuel, such as natural gas, you see that there\'s very
little or no opportunity there for the bias and just a small random
error. Whereas petroleum that\'s got a lot more heterogeneity to it,
you\'re going to have more opportunities for that sort of uncertainty.

And, again, the flared gas category reflective of the fact that there
may, in fact, be non-methane that\'s being burned. So it\'s often
difficult to even tell what the chemical composition of the flared gas
is.

You\'ve also got this obviously in your packet in case it\'s hard to
read.

These are the simulated values. When I actually sent this out earlier,
we couldn\'t even send out the actual values because they were still
embargoed, but, for example, this number here, the 1999 bottom line
would be fifteen, fifteen in the actual report. So you see there\'s a
little difference in what the computer simulation is versus what is
generated by our estimates, but that kind of makes sense.

One thing to notice \-- let\'s see. Actually maybe I can skip to the
next one.

Essentially when I\'m going through the next part of the analysis, just
remember that this is your denominator when we\'re coming up with the
various percentages for each category. We\'re putting the range of
uncertainty over the total emissions category. So these are the total
emissions in a million metric tons of carbon.

So this represents the ranges around the mean value. So it\'s the plus
or minus. When you total them, you get this range. So we\'re talking
about for the total category of 1999 about, you know, plus or minus 40,
41 and a half million metric tons or a total of about 5.5 percent, which
is \-- and there\'s a little skewedness to the upward side because of
some of the biases we talked about earlier.

Let\'s see here. I\'ll just go into some of these in a little more
detail. These are some of the slides, I think, that you have in that
substitute packet.

You see kind of reflective in here obviously some of the numbers that
went into these. The total category is 21.7, and when you put that over
the earlier number that I put up, that\'s where the 6.8 percent comes
from.

So for the total natural gas emissions category, it\'s 6.8 percent of
the total.

Now, one thing you notice is here\'s a number of 3.3, and we have a
number for 3.5. Well, 3.3 is 7.1 percent of the electric utility natural
gas consumption, but 3.5 is 72.9 percent of the flared gas, and
obviously that\'s reflective of the inputs that I showed you earlier in
terms of both the bias and random error of both terms, the coefficients
and the activity levels.

For coal we see that for the entire category it\'s a little under ten
percent. For electric utilities it, again, about ten percent which makes
sense, and it\'s going to be close to \-- while these are not totally
additive, they\'re going to be in the same neighborhood obviously.

And then for the other sectors, it\'s a little higher uncertainty, but
again, it\'s from a lower number. So that\'s why the total category is
much closer to the electric utility category.

Similar for petroleum. We have about a nine percent uncertainty for the
entire category, and you see the different ranges by different sectors.
Probably the industrial sector is one of the more interesting sectors
because of both the various fuels that are used there. It\'s more
difficult to get a handle on all of the fuels that are used in the
industrial than, for example, here.

Most of this is obviously transportation sector. We pretty much know
what\'s there and what we\'re dealing with. So that\'s a little more
stable than the industrial number.

Again, the non-fuel use is the issue that I talked about earlier. Where
does that carbon actually end up? And there\'s going to be a lot more
analysis done on that by EPA, and I think we\'re going to, you know,
probably work with them on that issue because if, in fact, more of it
does end up in the atmosphere, we need to reflect that in the numbers.

So this is just to sum up in terms of the overall category again. It\'s
about 5.5 percent, and this is kind of the range we thought. It is
towards the high end of what we originally thought. We thought it may be
three to five percent would be the range for energy CO~2~, but as I\'ll
bring up here shortly, I think there may be a reason for this, and it
may be slightly lower than that when we change some of the numbers.

So what are the lessons learned? I think the most important lesson I
learned is I kind of went into this not knowing. I\'m not a
statistician. I\'m an energy and mineral economist, and I always left
statistics to other economists, and what I learned in an assessment of
this nature, it\'s getting those numbers up front. It\'s very
subjective, and it\'s an art rather than a science because you have to
sit down with the experts and, without offending them, try and figure
out what kind of errors are possible in the data.

We\'re not, for the most part, dealing with sampling error. We\'re
dealing with errors that are much more difficult to quantify in terms
of, you know, who got the survey at the oil company. Is it a new kid or,
you know, did they spend an hour on it? Did they spend five minutes on
it?

And then it\'s follow-up. How much follow-up is going on here? Is it a
person with 30 years\' experience in the coal industry that\'s doing the
follow-up? Is it a new hire?

All of those sort of things go into this kind of uncertainty, and it\'s
very difficult to quantify.

I think I also found that you need a good understanding of the
statistical methodology which in this case was provided by Dr.
Benramdhan, who\'s not here today, but she\'s been my contractor on
this, and whereas I\'ve been able to provide some of the particulars of
how we collect the data, and I\'ve learned a lot about how EIA works in
the process of doing this, it\'s been very helpful to me, informative.

And finally, I think I\'ve learned that redundancy is in this case
probably very helpful in terms of multiple surveys where you can compare
across surveys and see, you know, do these results make sense here in
terms of what we know about energy use over here.

For example, the 1998 manufacturing energy consumption survey data is
about to become available, and it will give us an opportunity to check
against what we\'ve been thinking has been going on in the industrial
sector, specifically manufacturing, which is a subset of our industrial
sector, and to look at, you know, has \-- because if you look at our
data the energy consumption has been relatively flat over the last few
years even though our economy has been taking off. Mainly people have
thought, okay, this has to do with the new economy versus the old
economy.

But we really want to look and cross check the data and say is that
really what\'s going on. Are we missing some of the data because of the
changes in both the natural gas industry, the electricity industry,
consolidation of the petroleum industry. All of these things are going
on simultaneously, and of course, they affect the uncertainty and the
level of the data that we can expect.

Also with EIA and EPA working together, I think it\'s also a very \-- we
can kind of check each other, and our report kind of serves as the first
draft, the first public draft, and then their official inventory goes
through much more rigorous review from the outside and comment period
and all the sort of things that EPA is subject to.

But in a sense it works out well because, you know, we get things out
quickly. We\'re an information agency. We work in kind of a different
environment than EPA works, and the two together, I think work well. So,
you know, I think redundancy can be a good thing in this case.

How are we doing on time? Are we okay?

The other thing that kind of jumped out at me as I realized that we
really need to do more sensitivity analysis on these parameters, and to
do that I want to move down in some of these levels in more detail.

For example, we have this category called unfinished oils, which is \--
we don\'t know what it is really. We don\'t know its coefficient. It\'s
a general category that we use when one refiner ships kind of
semi-finished product to another refiner.

And we have that category so we won\'t double count the product, and so
it\'s a subtraction from the emissions, but we really don\'t know
whether we\'re subtracting the right amount or not. So there\'s probably
a fair amount of uncertainty in there, and it\'s a fairly large number,
probably around 50 million metric tons, in that neighborhood. So it\'s
something that we need to pay attention to.

We\'re going to go ahead and do the other CO~2~ sources and adjustments,
which include U.S. territories, and what is the other category? I\'ve
forgotten what the other \-- oh, bunker fuels. There\'s probably a fair
amount of uncertainty associated with the bunker fuels also.

So even though, again, it\'s a small category, two percent, actually
it\'s even smaller because we include in that two percent the flared
gas, which we\'ve already done the analysis on in the first bullet. But
it still has a fair amount of uncertainty associated with those
adjustments that we make.

And then we\'re going to do the other greenhouse gases, as I mentioned
earlier, as a first cut, letting EPA do the more in depth analysis where
they, you know, go out and interview experts in the various gases.

And we want to get at somehow a cost-benefit or value of information
approach so that we can say, you know, given this amount of money that
we have to invest, where should we put the money and where do we get the
most bang for our buck?

Now, I came up with one question. I\'m sure that there are many
questions I have, and it relates to this idea of the under bias in the
electric utility coal.

What happened is we actually found that four percent and put it in the
numbers, and for this study we just didn\'t have time to adjust the
analysis. I think we actually have done another Monte Carlo simulation
with a lower number because my feeling is these biases are kind of like
an oil field. You\'re going to find the bigger ones first. So if you
find four percent under bias in the coal category, you probably want to
reduce that maybe to two percent, you know, knowing that there might be
some more out there, but it\'s probably not going to be a four percent
chunk. At least that\'s, you know, my general feeling on that, and
that\'s something I\'d like to get input from the committee on.

Is that a sensible approach to take on that sort of thing?

And I think I\'ll just stop for now and get your input, and really
appreciate the opportunity to address this group because I find that
your input is very helpful, and I\'ve learned a lot in, you know, the
last six months working on this project. So I really appreciate it.

Thanks a lot.

CHAIRPERSON GOTWAY CRAWFORD: Thank you, Perry.

Our first discussant is Jay Breidt.

MR. LINDSTROM: Do you need this on? Let me see if I can get this off.

DR. BREIDT: Okay. Well, Perry has described an attempt to quantify
uncertainty in carbon dioxide emissions, and the hard part here is you
need to systematically account for the source of uncertainty that go
into these numbers and then go through the dirty work of trying to
quantify, put some magnitudes on those errors. So that\'s the hard part.

At the end there\'s a simulation exercise tacked on to propagate that
uncertainty down to the level of actual emissions, and that\'s where
I\'m going to focus my comments.

And I\'m going to try to offer a peek inside this black box that\'s
creating these numbers we\'ve been looking at in tables and try to set
the stage for some further discussion of this modeling exercise.

And what I\'m working on as the basis is a draft report, and I recognize
it\'s a draft, but I\'m going to offer some criticism of that draft
because this is the time to do it.

So a lot of these simulation exercises get fairly complicated. This one
turns out to be not that bad. There\'s basically just two components.
You\'ve got activity data, and you\'ve got these emission factors. All
right? And you take the product of those two and you get a carbon
dioxide emission.

You\'re going to model the first as uncertain, the second as uncertain.
So the product is uncertain. You do that by sector, by year; add things
up. So that\'s the basic model.

Each one of those uncertain factors is decomposed into two components,
which Perry described. There\'s a bias component, and the model here is
that that\'s uniformly distributed. So you take a min. and a max. and
put a flat PDF on that interval.

The min., if you look at that table that Perry described in some detail,
and you take the minimum bias, take a function of reported consumption,
that\'s your lower value. Function of reported consumption is your upper
value. That\'s the first component of the bias PDF \-- sorry \-- of the
activity data PDF.

And a second is described as the random PDF. It\'s modeled as normal,
zero mean, and with a standard deviation, which is a percentage of the
mean.

Add those two together and you\'ve got your uncertain activity. So
that\'s the first component.

The second component, exactly the same thing for the emissions factor: a
uniform component, a normal component. Add them together and that\'s
what you\'ve got. Take the product, and that\'s your uncertain carbon
dioxide emission.

Okay. So uncertain CO~2~ emission is uncertain activity times uncertain
emissions factor. This is not normal because it\'s uniform plus normal;
non-normal because it\'s uniform plus normal. Take the product. It\'s
non-normal. So a natural thing to do is to use simulation to study the
distribution of that uncertain CO~2~ emission.

But it\'s always worthwhile to see how far you can push analytics here,
and from here on out I have to do a bit of inference because there\'s
not much description of the model in the draft report, but I think that
independence is assumed everywhere.

So what\'s going on here, if you want to take means, take the product of
the means. The mean emission is the mean to activity times mean for the
coefficient, and variances are simple functions of the means and
variances. So you can write these things down analytically, and if you
like, you can work out skewedness, kurtosis, distributions, whatever you
want to do. These are straightforward computations.

So that\'s a good way to check the answers, and it\'s really not that
hard. So I just did them all because I had the draft report in
electronic form so that I can just cut and paste the data, and this is
easy to do.

So these are exact means and standard deviations by sector for the year
1999, and I pulled that out because it\'s the worst case, and I don\'t
think you can read this very well, but if you could believe me, these
are virtually identical in the two columns until you get to coal, where
things look close but not quite on, and I think that\'s a typo in the
table. I think the coal data for \'98 got repeated for \'99, and I\'m
using the wrong data because I\'ve got that typo table.

But everything looks pretty good. So the simulation results certainly
look plausible. it looks like the model is doing what it\'s claiming to
be doing or not quite claiming to be doing.

I want to distinguish here the number in the table that I just shoed are
the standard deviations due to the model uncertainty, not due to the
simulation uncertainty. Okay?

So if you simulate for longer and longer time periods, the simulation
uncertainty will go to zero, but the basic model uncertainty is always
there. So the numbers I was reporting were the actual model
uncertainties.

There are no simulation standard errors reported in the draft report,
and those would certainly be worthwhile. You don\'t need to rely on some
vague idea of mean and standard deviation stabilization. Just give the
simulation standard errors.

There are some other numbers in the draft report. There are percentiles.
Typically the 95th percentiles, and I thought, well, let\'s take a look
at those and see what they look like, and the quickest, dirtiest
approximation would be to say that this distribution is not normal, but
it\'s sort of bell shaped. So let\'s just see what a normal
approximation would give you.

Take the mean plus a standard normal look-up, times a model standard
deviation, and that turned out to be so good that I thought it was done
incorrectly in the draft report because this approximation shouldn\'t be
perfect, but it nearly is. And, again, sorry these numbers are small,
but 270.2, 270.3, 47.12, 47.12. They\'re just right on the money.

So I had to check it again against some simulations, and that seems to
work. So if you need a quick and dirty check for percentiles in this
particular model, you can do that and it works very well.

Now, remember the goal here is to quantify uncertainty, and the draft
report is extensively using expected ranges, and I think Perry called
them something a little different in his slides, but they\'re basically
ranges as a measure of uncertainty. This is the key output, okay, the
range.

And if you think about this for a second, I didn\'t know what an
expected range was. So I poked around, and we\'re taking the maximum
simulated value minus the minimum simulated value.

And this, if you think about it for another second, doesn\'t make any
sense because if you run the process for an hour, you get a min. and a
max. If you run it for two hours, you get a smaller min. and a bigger
max. So this is just going to go to infinity as the simulation size goes
to infinity.

And if you don\'t believe that, here\'s a few different orders of
magnitude simulation sizes. The expected range goes from 7.6 up to 18.8,
and the particular simulation in the draft report was about 10,000
reps., and this number is in agreement with what you see in the draft
report.

So those expected ranges just need to be deleted throughout. They\'re
just not meaningful. So that\'s a few comments on the output of the
simulation.

Now, going back to the original model that generated the simulation,
there are independence assumptions throughout here across years, across
sectors, everywhere. Everywhere you could assume independence you do,
and I worry about that a little bit because, as Perry pointed out, what
we\'re modeling here is the uncertainty, not really the activity data
themselves, but the uncertainty in the activity data, and most of that
is due to non-sampling error, okay, and these kinds of things tend to be
dependent.

Why would there be uncertainty in the activity data? Well, you\'ve got
common survey elements, common processing algorithms, common staff.
You\'ve got ways in which the non-sampling error could be dependent, and
I think that\'s worth looking into.

What about the emissions factor data? Well, if you look at the actual
emissions factor, they\'re exactly the same number for lots and lots of
years and lots and lots of sectors. They\'re derived from the same
studies. So you take some materials and you test them, and you get a
carbon coefficient.

So these are certainly dependent, and now you start combining these
things and you\'re going to get dependence in the combinations. So what
kind of impact might you see if you ignore dependence? What\'s going to
be the problem?

If you look at a particular sector in a particular year, it doesn\'t
really matter. The simulation will give you the right mean. It will give
you the right variance. It\'ll give you the right percentiles, but now
when you start aggregating, you\'re going to have the wrong answer.
You\'ll have uncertainty in totals, which is understated because you\'re
ignoring covariances.

Okay. So just to pull out an example, 1999 natural gas, and I took out
the flared just to do two components, but if you do that without
dependents, you would get a variance of the sum, which is about 10.1,
but if you take into account that they have exactly the same emissions
factor, then there\'s a covariance term which would bump that up to
10.5.

Now, if in addition the activity uncertainties were correlated, you\'re
going to bump it up even more, and I did \-- here\'s the theoretical
computation of what you would add on, and I plugged in some numbers to
that last night, but it was after dinner, and I had had wine. So I
can\'t trust it, but I think the additional component would be somewhere
between zero and 2.2.

Okay. So you\'re bumping it up a little bit each time, and now you
aggregate this over additional sectors, and you\'re accumulating some
additional uncertainty that\'s not reflected currently.

So I\'ve got a number of worries about the statistics here. I\'m a
statistician, and I don\'t leave statistics to economists.

(Laughter.)

DR. BREIDT: And that\'s nothing against economists, but \--

DR. MOSS: And I nominated you.

(Laughter.)

DR. BREIDT: But in the draft report at least there are a number of
improper or incomplete summaries of the simulation data. These expected
ranges, as I pointed out, are just nonsensical, and there\'s no point in
reporting them.

There are no simulation standard errors, which would be good to see. At
the end of the day one of the key numbers is the total emissions
uncertainty. That\'s modeled as a beta, a beta distribution as a bounded
support. None of these variables have bounded support, and this is a
worry to me because I think the message of Perry\'s example of the coal,
they found this big error of 4.4 percent, and they want to know, well,
how much should we reduce this coal uncertainty.

Well, I\'d say increase all of the other uncertainties because there are
potentially large, surprising errors out there. I think that\'s the real
message here.

You don\'t want bounded support. So that\'s why I\'m glad that there\'s
a normal component here so that you don\'t have bounded support, and I
wouldn\'t fit something with bounded support at the end of the day.

Ignoring dependence is a concern. I think there is dependence across
sectors, and that will impact uncertainty of totals, and those are
really the numbers of interest.

And it wasn\'t clear to me if forecasting was really of any interest,
but certainly there could be dependence across time, as well, and in
some miscellaneous things in the draft report, there are just various
misstatements like maximum entropy did not yield the least biased PDF,
and 95th percentile confidence level, they mean 95th percentile, and the
central limit theorem doesn\'t say what they say it says.

So that\'s kind of critical for a draft, but I think that the good start
here, this is a good start to a hard problem because, you know, you\'re
trying to quantify what you know about what you don\'t know, and that\'s
inherently difficult, and so the hard part is getting that table of
percent errors

Then you get around to the simulation methods, and simulation methods
are not 100 percent foolproof. Analytic methods are always useful to
check the output, and the concern I have with simulation methods is that
they\'re very sexy. You get this very rich output, but if you get rich
output from a poor model, you don\'t want to be fooled by that rich
output.

And I wrote here on the bottom this project could benefit from
continuing input of statisticians. Quantifying uncertainty is really
what we do, and things are going to just get more complicated, add more
gases, add dependence, disaggregate some of the things that are
currently aggregated, and I think that we have useful input here.

So I, for one, would certainly like to see another version of this at
another meeting.

So those are my comments.

CHAIRPERSON GOTWAY CRAWFORD: Thank you, Jay.

Our next discussant is Roy Whitmore.

DR. WHITMORE: Okay. I guess we\'re in business.

In reading the paper, it wasn\'t entirely clear to me exactly what the
simulation model was, and I didn\'t think the purpose of it was clearly
explained as well. There was a lot of discussion in terms of
characterizing variability and uncertainty, and I think the real purpose
was characterizing uncertainty.

I think Jay got a little further than I did in terms of understanding
exactly what was going on in the model. I thought it would be useful for
me to start off by presenting what I thought the paper was saying the
model was, and to the extent that my understanding is incorrect, that
might inform what needs to be explained better in the paper.

So I start off with that as just my take on what was there and a
presentation of what seems to me the essential things to model are the
paper needs to explain clearly whether it\'s written for a technical or
non-technical audience.

The first part of it, I think, is very clear that the annual CO~2~
emissions for years \'90 to \'99 are being estimated as the sum over
sectors and fuels of the amount of fuel consumed in Sector S in that
year multiplied by the emission coefficient for CO~2~ for that fuel in
that sector and year.

A Monte Carlo, basically those numbers are fixed values, but unknown,
and the purpose of the Monte Carlo simulation is to characterize the
uncertainty regarding the estimate of the CO~2~ emissions and
characterizing that uncertainty using a probability distribution of the
estimated values.

And the method, you know, you could do simulations of how to model what
the annual consumption is and simulations to model the emission factors,
and then multiply those simulated values together. But I think rather
than that is simply describe the uncertainty regarding the consumption
and emission with probability distributions based on these discussions
with various experts.

But it\'s worth noting, I guess, that those consumption and emission
factor values for each year \'90 to \'99 are, in fact, constants, and
we\'re just characterizing the uncertainty. If we were trying to predict
CO~2~ emissions for the future, that would be a little different
situation.

So what I gleaned from the paper as the model is that the consumption
was being the EIA\'s uncertainty regarding the amount of consumption in
Sector S, and fuel F was being represented at the normal probability
distribution, and the mean of that was taken from some uniform, and the
variance or standard error, standard deviation of that normal
distribution was assumed to be proportional to the mean with the ranges
for the uniform and the proportionality factor for the standard
deviation being based on these expert opinions.

I think maybe I didn\'t get that exactly right based on what Jay did.
His analytic values agreed exactly with the simulation. Clearly he \--

DR. BREIDT: I think that\'s exactly the same thing.

DR. WHITMORE: Okay. The same thing? All right. Good.

DR. BREIDT: Discretion.

DR. WHITMORE: Yeah. So it would help a lot to clarify in the paper
exactly what the model is, whether you\'re trying to make it for a
nontechnical or a technical audience.

And with regard to the way that the bias values were \-- well, let\'s
see. Let me not miss this segue. It wasn\'t clear to me also as to
whether that was used for all the factors because there was in Section
6.1 \-- the paper says, in fact, input data were represented by single
value in most cases and made it sound like there were some of these that
were treated as fixed constants rather than \-- and didn\'t have any
uncertainty associated with them.

I\'m not sure if that was true or not, but that seemed to be what the
paper was saying in one place. With regard to the way that this
minimum/maximum bias and variability were shown based on interviewing
various experts, the bias values were all shown as positive, and so
although there was discussion, the paper said something about under
report for some things, over report for others.

I think it would be much clearer if the under report was shown as a
negative bias and the over report as a positive bias. If you just look
at the table by itself and it says bias and they\'re all positive
numbers, they\'re all over reporting.

And in the section discussing fitting tables, Section 3 of the report
said the principle of maximum entropy will be applied to choose the
least biased probability distribution. As Jay pointed out, I\'m not sure
what least biased there means, but it seemed to be that the paper was
saying if all you know is a mean and a standard deviation, you should
use a normal distribution. If the only thing you know concerning
uncertainty is the range of values, then you should use a uniform
distribution. If you know a range and a mode, you should use the
triangular. If you know range, mean, and standard deviation, you should
use a beta.

And that just seemed too prescriptive to me. The distribution is
supposed to be characterizing your uncertainty regarding that input
factor, and you need to consider what best characterizes that, and
certainly if the only thing you know in terms of characterizing
uncertain, well, is it probably in this range? Then the uniform on that
range is probably a good choice. I\'m not sure about the others.

In fact, the others probably could come into play more often when
you\'re actually trying to fit some data. It wasn\'t clear here that you
had data to fit since you\'re characterizing uncertainty. If you\'re
trying to characterize variability, like say if you\'re trying to model
annual residential consumption, you\'d have data regarding residential
consumption from the RECS, and you would have an empirical distribution
of household consumption. You could use that in a Monte Carlo simulation
directly or you could fit a parametric distribution to it and sample
from the parametric distribution.

And I think it\'s valuable there to kind of keep in mind that we\'re
trying to characterize uncertainty rather than fit a distribution, I
think, in the application that we\'re looking at here.

Section 3 also said that the Monte Carlo results will be benchmarked
with the Latin hypercube to insure reliability and reproducibility of
the outcome. I don\'t really understand what that means because the
Latin hypercube is simply a way of sampling from the assumed
distribution in a way that speeds convergence of the simulation. I\'m
not sure what that benchmarking notion is.

And after the simulations, Section 6.1, it says, \"After simulation
results are generated, a tremendous amount of information is available
and careful assessments are to be made.\" It kind of sounded like before
we run the simulation there\'s a dearth of information. We know very
little, and after we run the simulation now we know lots of information.
That\'s just not the case, and it\'s very tempting to over interpret the
results of the Monte Carlo simulations. They\'re just an output that
depends on ‑- the quality of that output really depends on the quality
input distributions that went into the simulation.

And if the model has changed in some dramatic way, like taking some of
the coefficients that are considered to be constants and actually
incorporating the uncertainty with regard to those, that could greatly
change the simulation results.

Particularly, Section 6.3 says that uncertainties might increase with
more input data yet to be considered. Technically if the model has
incorporated all of the sources of uncertainty, additional information
should only reduce uncertainty. As you know more, unless you find out
something you assumed before was incorrect, you assumed that there was
only so much uncertainty. Now you have discovered, oh, it was really a
lot more uncertainty than I thought. Additional information or
additional data should only decrease uncertainty.

It\'s noted that the study did not incorporate certain aspects of
uncertainty because of time and resource constraints. If you\'re simply
referring to not covering some additional sources of CO~2~, I think
that\'s, you know, within the problem statement. If there are certain
aspects that are not covered, it seems important, you know, in terms of
characterizing the uncertainty to try to capture all of them in some
way.

And with regard to the interpretation, the report attached considerable
value in terms of policy decisions to interpreting various
characteristics of the distribution. In particular, the utility for
making policy decisions was discussed with regard to the mean of the
simulated distribution at standard deviation, the 95th percentile, and
the range, and I noted that, well, first of all, in interpreting these,
it\'s important to keep in mind that they are characterizing the
uncertainty, not characterizing a distribution of CO~2~ emissions.

In particular, as Jay noted, the range is totally dependent on the
number of simulations, and if you\'re interested in something like that,
you need to tie it to percentiles of some sort. Looking at an
interquartile range, if you want something more like the full range of
the distribution, the 95th percentile, from the fifth to the 95th
percentile or something like that.

And I\'m not sure this is within scope of what I was supposed to cover
as this isn\'t directly in the paper, but it\'s kind of an expansion of
this Monte Carlo simulation. It seemed to me to be useful to use a Monte
Carlo simulation procedure to try to predict annual CO~2~ emissions for
future years and to do the modeling in such a way you\'re incorporating
things that you could use that would reflect, you know, things that
might be changed as a result of regulations or whatever.

So they\'re modeled in such a way that the effects of legislative
mandates, changes in sources of fuel, et cetera, could be predicted. One
could model annual consumption as a function of population, weather
conditions, economic conditions, et cetera, using empirical
distributions for things were we have some idea of what the
distributions of these factors are, uncertainty distributions for other
factors, input factors where we don\'t have any data, and similarly, you
can model emissions factors as a function of source of fuel, percentages
of contaminants in fuel, economic conditions, et cetera.

And the 1990 to 1999 data could be used to validate the models or
basically whatever historical data you have could be used to validate
the models, and these could be used to generate a Monte Carlo simulation
for predicted annual CO~2~ emissions under various scenarios regarding
population growth, global warming, state of the economy, regulations
regarding impurities allowed in coal and other fuels, et cetera.

So just another idea of some Monte Carlo modeling you might want to
consider with regard to this kind of a slight variation on the same
problem.

CHAIRPERSON GOTWAY CRAWFORD: Perry, do you have any very quick comments
to make regarding the discussion?

MR. LINDSTROM: On the last point, certainly the Annual Energy Outlook
does project carbon emissions into the future. So some of this may be,
you know, able to be incorporated in there, and we do analysis using
them as policy and that sort of thing. So certainly that would be
appropriate for that.

Obviously, we\'re going to have to kind of go back to square one and
take a look at what we\'re doing here, and that\'s why I\'m really glad
that we presented this to you all before we kind of, you know, went out
there and said this is our end product and that sort of thing.

So what I think we should do over the next couple of months is
communicate with you on what is a good representation, you know.
Obviously whatever the range is not an appropriate output. So what we
have to do is come up with the sort of things you were just mentioning
and come up with a reasonable way to represent this.

And before we go on to do anything else, that\'s what we need to do, and
then once we have developed a more tighter model or a way to look at
things, it makes more sense and we get kind of your blessing or go-ahead
for this approach; then I think we can go ahead. But until that point I
would, you know, hold up on doing any more of the analysis at this
point.

So, you know, your input is invaluable to us in this sort of thing.
It\'s easier for me to say this because I\'m not a statistician anyhow.
So I wish Dr. Benramdhan were here to address some of the more technical
comments, and certainly I\'ll make sure she\'s here the next time, you
know, we meet on this.

CHAIRPERSON GOTWAY CRAWFORD: Very quickly, Roy.

DR. WHITMORE: Yeah. Just with regard to the way that you were using
range, it looked like you were using the range as a characterization of
uncertainty. Look in the range relative to the mean for each factor as
kind of the relative uncertainty of that individual factor.

Certainly there are other ways of characterizing that. Sensitivity,
well, there\'s kind of a sensitivity, but what you really would be more
interested in, I think, is the sensitivity with regard to the total
output and sensitivity of the total \-- you know, how much effect does
each individual factor have on the total. There are lots of other ways
of getting at that as opposed to the sensitivity of each individual
factor.

MR. LINDSTROM: Yeah, I think one of the problems we ran into is under
the Tier 1 method, you know, we kind of had \-- we went through all of
the gases and were able to characterize things in terms of the total.

And then when we did this, we were kind of just talking about the one
segment of it, and so probably it is better to look at things in terms
of the whole picture. That\'s one of the reasons that we\'re going to
continue to study, but as I said, first of all, we have to come up with
the right output variables before we continue

DR. WHITMORE: Because, I mean, the sensitivity analysis, what you\'re
looking for is where do we need to invest resources to reduce the
sensitivity \--

MR. LINDSTROM: Right.

DR. WHITMORE: \-- of the estimated total, even if I\'m clear that gases
there\'s a lot of uncertainty, but if it\'s a very small part of the
total, it may not be too important.

MR. LINDSTROM: Right.

CHAIRPERSON GOTWAY CRAWFORD: David, did you have a question?

DR. MONTGOMERY: Yeah, I was going to relieve you from statisticians\'
questions for a little while.

MR. LINDSTROM: Oh, good.

DR. MONTGOMERY: First of all, kind of going back to your very first
slide about the purposes of doing this, do you have any idea of what a
study like this \-- well, whether anything like this is being done in
countries that have really lousy energy statistics who are reporting
their national inventories, like China or Russia, and if they did, what
it might suggest?

Then I have one follow-up question to that.

MR. LINDSTROM: I have a feeling that the answer at this point is no. I
think we\'re the first ones to even start doing this. We started it even
before the paper was released that asked people to do it because Arthur
Rupinski was on the committee, and he kind of came back here and said,
\"Why don\'t you get started on this?\" which was great when he was
here.

He promptly left, threw it on top of me.

(Laughter.)

MR. LINDSTROM: So I\'ve been trying to deal with it ever since. But as I
said, but he covers the whole range, but so, yeah, I think we\'re ahead
of the curve in terms of, you know, we\'re looking at it and EPA is
looking at it, and I don\'t think anybody else at this point is.

I know England did look at it because that\'s mentioned in the report,
but \--

MR. MAZUR: They\'re supposed to look at this, but just you and not
everybody else.

DR. MONTGOMERY: The other question was about applications. I think, you
know, actually doing what I\'m going to suggest depends on how you
resolve, I think, Jay\'s comments about independence over time, but one
of the things I think might be really useful about a study about this is
asking questions like how valuable are the commitment periods that are
defined under Kyoto. A lot has been made that the five-year commitment
period gives flexibility for dealing with year-to-year variations.

This may also confuse variability with uncertainty, but I was just
wondering as a first step on that if you looked at something like a
five-year moving average and how that would reduce the uncertainty in
the reporting and would be a way of asking how much does the five-year
commitment period insulate you against kind of random errors in looking
at your reports, which might come around and, you know, come back and
bite you because if you consistently over estimate your missions during
the five-year commitment period, you\'re going to find you have to do a
lot more that you don\'t really need to do in order to meet the targets.

MR. LINDSTROM: Of course, one of the issues is how you estimate now as
compared to your target year 1990. You know, is there a random
difference?

For example, in that one coal category we found not only has it come on
since 1990, it has actually increased since 1990. It\'s gone from about
15 million tons of coal to 45 or something like that. So we\'ve got an
actual thing and it\'s been increased, and it does affect how we relate
back to 1990.

In terms of the sort of thing you\'re talking about in the commitment
period, I think that we have to crawl before we can run, and that would
be something to consider. I think in the future maybe with AEO analysis
would be more appropriate for that sort of thing.

And I think we\'ve done some of that stuff with the net system already.

CHAIRPERSON GOTWAY CRAWFORD: Nicholas.

MR. HENGARTNER: Yes, thank you very much.

Just a small comment. I\'d like to defend Perry\'s use of the range, but
under one specific condition. You see, what you have done so far is you
quantify the variability, which is very good, but there\'s a second
component which you very carefully did in the beginning. Then you looked
at the biases.

And so a natural question is: how big is the overall bias in addition to
the variability? And how did you find out what the bias is?

Well, the bias is something systematic, and I don\'t know what the bias
is. You could probably play around and figure out what it is. Another
way simply, you simulate those uniforms, right, and figure out what
would be the maximum bias and the minimum bias possible over a large
enough number of simulations, and in those cases the range will not
drift off to infinity because you have nice, bounded intervals.

DR. BREIDT: But there\'s the normal added to that.

MR. HENGARTNER: No, no, you don\'t add the number. You just look at the
bias.

DR. BREIDT: Well, okay.

MR. HENGARTNER: You just look at the variability introduced by the bias.
What is the systematic error? And then that\'s what you have also in
your slides because then you add it. The random component then \--
don\'t look at the range. That\'s a different problem.

One is looking at the variability. The other one is at just how much
systematic error you\'re having from your numbers.

CHAIRPERSON GOTWAY CRAWFORD: So thank you.

And obviously this is to be continued, again, as you\'ve been at every
meeting for a while.

(Laughter.)

CHAIRPERSON GOTWAY CRAWFORD: So we\'ll maybe revisit this at the next
meeting. So thank you.

MR. LINDSTROM: Well, thank you.

CHAIRPERSON GOTWAY CRAWFORD: I\'d like to move along now and move to our
next talk by Doug Hale and Thomas Leckey.

MR. HALE: Good morning. I\'m glad to see you\'re all still very fresh.

Before we get started, I\'d like to introduce a visitor. His name is
Peter Simoyi. Would you please stand up?

Peter is a Herbert Humphrey Fellow from Zimbabwe. He is with the
Zimbabwe National Power Company. He\'s over here studying at Cornell
with Bob Thomas and associates. His job is a very enviable job of
figuring out how to privatize their electric power system. So he\'s
going out to California we hope to learn everything that he should avoid
doing.

(Laughter.)

MR. HALE: He may be floating his resume in hopes of not having to pursue
this. I don\'t know.

How do you start this thing?

My name is Doug Hale, and with me is thomas Leckey. We\'ll be talking
about a new model of the Eastern Interconnect.

Before I get into the talk, I\'d like to acknowledge that this model,
which by the way is totally in the public domain and anybody that wants
it, you can have it, has been built part time by a lot of people spread
around government offices.

Marilyn Walker at the Department of Justice has done an incredible
amount of work in data collection and analysis.

Robin Allen, also at the Department of Justice, has been a very helpful
economic consultant and has been very reassuring about some aspects of
this project.

Thanh Luong and Maria Hasanzadah at FERC have given us invaluable
engineering support, have directed us to the right electrical models to
use, and of course, Thomas Overbye, my long term colleague at the
University of Illinois, developed the optimal power flow software, the
graphical display software, and again has given us incredible support
over the years.

Yesterday I had the opportunity to talk with Dr. Kent, our former
Administrator, who said hello by way of starting out saying, \"Doug,
I\'ve been dragooned into reading another one of your electricity
papers,\" and thinking about it and thinking about his project, he\'s
right. In 1998, Jim Hewlett, Thanh Luong, Joe Mulholland and I presented
a paper, our first in the series, on a full model of New England. It\'s
a full AC model where we pointed to the problems of voltage constraint
and the great difficulties of getting power into New England in terms of
market clearing prices.

In 1999, we added New England, PJM, and some of ECAR and again gave an
ASA presentation, and this year this presentation.

I can guarantee you two things, Cal. There will only be one more paper
in this series, one more, and, two, even though you\'re off the
committee, I\'m bringing it to you to read. So there you go.

So in honor of Cal, I\'ve decided to rename the paper as follows.

I have, as always, questions for the committee. Why don\'t you take a
look at this model? This is a version zero. I promise you it\'s better
than the first release of a Microsoft product, but you know, the
engineering is right. The mathematics is right. The cost data isn\'t so
hot. The behavioral assumptions, who knows? We\'re going with pure
competitive behavioral assumptions right now.

And I\'m asking for your suggestions about priorities, and this is
sincere. We do have some real leverage here. One thing we could do is to
try to refine all of the cost data now. We have sources in addition to
FERC. We have EPA based data. We have data from the Department of
Justice that\'s not normally available that we could use to refine these
cost estimates, and then thereby give the entire research community a
decent tool for \-- okay. I\'m getting there. I\'m getting there.

My personal favorite, however, is to try to validate this model, and
what I mean by that is, as you will see, we have produced terrific
graphs and lots of numbers and all of this, and the question is: do all
of those numbers relate to anything real?

On a qualitative basis we know it does because we\'ve checked our
conclusions against engineers, but we\'re not so much interested as
economists in getting the engineering \-- well, we have to get it right,
but the engineering isn\'t what\'s driving us. It\'s are we getting the
prices and quantities right, and that we honestly don\'t know at this
stage.

And so I really want to do a very \-- at some stage I am going to do a
very careful comparison of what\'s coming out of the model against
what\'s coming out of actual markets.

The final thing we thought about is perhaps introducing some
noncompetitive behavior. The colleagues at Justice are very hot on this.
I think competition is hard as it is. However, there are obvious things
you could do. You could put in Stackleberg. You could put Cournot
behavior. If you\'re into heavy drugs, you might want to go to some full
game theoretic stuff, but you know, we could do that.

The other thing I wanted your advice about is what sort of model
exercises should we be running. What you\'re going to see is a very
simple first taking out of the model and trying it out, and you\'ll see
that if some areas resist cooperating with other areas, they can do real
mischief in pricing. Maybe we should pursue that some more. Maybe we
should try to look directly at the effect of arbitrary transmission
tariffs.

You know, we\'re looking for suggestions about what\'s going to be
interesting to the outside world.

The immediate research question here \-- and, again, this is our baby
trial \-- is simply to look at the question if we go to these regional
transmission organizations, which FERC is now pushing, can we reduce the
likelihood of more San Diegos in the East.

As you may know, prices and bills in San Diego doubled and tripled this
summer. They were not able to get enough power into San Diego to drive
prices down. They blame a lot of it on capacity shortage. There are
other reasons as well, but the simple question is: okay. Is the East
facing such a situation?

So what we did is we first took the trade levels that are in the NERC
2000 summer case. These are administered trade levels. These are what
they use for planning factors. If it gets hot everywhere, how much is
allowed into Florida? How much is allowed into New England while
protecting the system? And calculated a base case.

Then what we did was to allow free trade over the entire Eastern
Interconnect. It came out of an optimization. So what we\'re doing there
is treating the entire Eastern Interconnect as one huge regional control
organization.

So the idea is if the RTO policy won\'t work, you know, if it\'s
universally adopted, it\'s got no hope, you know, with smaller
adaptations. The question is how much relief is there just in principle.

Okay. Now, with that start, I\'d like Tom to talk a little bit about
model components.

MR. LECKEY: Thanks, Doug.

Our three most important model elements are, number one, the
transmission grid and its representation as it\'s reported on the FERC
Form 715, and then aggregated by NERC into this case which we call the
NERC 2000 case. It\'s based on a reliability estimate of the case which
would obtain in the year 2000 summer peak. So we start with this
representation of the transmission grid.

The next model component which is important is the generators which are
reported, again, on the FERC 715 and arriving at some estimate of short
run marginal operating costs for as many of these generators as we
possibly can so that these generators can then be dispatched.

And then the third most important element is what Doug alluded to, is
that ultimately it rests on an assumption that the model we will
ultimately end up with is a representation of competitive equilibrium
when this model tries to pursue the least cost function for operating
and dispatching the entire grid under those conditions.

Now, what we\'re modeling here is the Eastern Interconnect or what is
called the Eastern Interconnect, and it is about two thirds of the
United States. It\'s basically everything in the continental United
States, leaving out the Western Systems Coordinating Council and ERCOT,
which is a large portion of Texas. So it\'s everything else.

Canada is represented in this system. You can see that NPCC extends well
into Canada, as does MAPP, and so there are inputs which come into the
model from Canada.

You may well recognize that this is an updated NERC region map which
reflects the fact that SERC is now much larger in the last year as
principally energy in Louisiana and Mississippi and Arkansas has now
left SPP and moved into SERC, and a few small utilities also in Missouri
have now joined SERC as well.

Some of the salient statistics, which will give you a sense of the size
of the magnitude of the model. In this model we\'re modeling more than
33,000 buses which, as far as I understand it, is one of the largest
models that you could develop for the Eastern Interconnect. Thirty-four
or 35,000 buses is about as big as I\'ve ever heard for modeling the
Eastern Interconnect.

Contained in that model are 5,312 generators, about five or 600 of which
are located in Canada. More about them later. Of the 33,000 buses,
22,000 of them bare load. We\'re representing about 45,000 lines or line
segments and transformers. There\'s about 107 what are called power
control areas. Oftentimes these correspond roughly to historic utility
control areas. There\'s about 130 or 140 of these in the nation as a
whole. So we\'ve modeling about 107 of them.

Total generation at peak in this model is 549 gigawatt hours, and the
model runs for a single hour. So it represents one hour. It runs for one
hour, 549 gigawatt hours, and demand in the case is 536 gigawatt hours,
again, for one hour.

The first task in running the model is to check the actual submission,
which comes from the FERC 715 as it is run through our software, which
is PowerWorld, and upon doing that, we observed 248 line or transformer
violations and thus began a somewhat arduous task of whittling them down
or thereby explaining why the original case filing would have these.

Many of them are radio ends of lines, but it\'s important to note that
there are some of these anomalies in the original submission.

And here, again, is simply what we call a one line diagram. What you\'re
looking at here again is the same geographic area, which is represented
in the NERC map, but here you\'re looking at about 345 KV lines and
higher.

What\'s actually present in the model is all the way down to, say, about
69 KV or something just above distribution level, but even at this kind
of highly aggregated level of 345 KV, you get some sense of the
complexity and the magnitude of the model that we\'re working on.

You can clearly see ERCOT is not connected directly to the Eastern
Interconnect and of to the west, the Western Systems Coordinating
Council, again, is not \-- it\'s equivalent to the model, but it\'s not
directly a part of what we\'re modeling as the Eastern Interconnect.

And now Doug is going to talk just briefly about the objective cost
function itself and how the model solves.

MR. HALE: This is for the statisticians who weren\'t hear earlier, but
as I said, what we\'re going to do is we\'re going to minimize cost
across this system. We\'re going to do it subject to a lot of
constraints. We\'re going to make sure the line flows are okay. We\'re
going to make sure the right voltage is delivered. We\'re going to make
sure the right amount of real power is delivered.

And when we minimize the cost \-- and we\'re going to honor all
generator constraints. Okay? So we\'re not going to be running
generators at levels they can\'t operate.

When we do that, we get price equal the marginal cost, including a
congestion number, and that\'s the good news. The bad news is that in
the New England optimization, we were able to optimize both real and
reactive power simultaneously. This is a minor point, but let me get it
off my chest.

We have had great software problems doing it. We\'ve talked to people in
the computing areas, and they said, \"Yeah, you\'re going to have great
problems doing that.\"

So what we\'ve done is iterated back and forth between a full AC system
with real and reactive power and a linear approximation of the real part
in the minimization.

What we end up with is a solution that respects all of the real and
reactive power constraints. It\'s engineering okay. It\'s physics okay,
but it\'s probably not, you know, the absolute perfect optimization.
It\'s close, but not exact, we believe.

The question is why in the world do we even do minimum cost. I think the
main answer to that is that, in fact, the minimum cost solution
corresponds to competitive equilibrium, okay, in this sort of
environment. This was shown by Schweppe and his associates back in the
\'80s and is an incredibly important result for this kind of work.

One of the things he did do was show the exact impact of system
parameters and congestion on competitive prices and quantity. So he made
a complete integration of the economics and the engineering and the
physics, which to that time hadn\'t existed.

So when we do this, we get a basic price equilibrium equation where
price is related to system marginal cost, losses and flows on
constrained lines. That is why we have to explicitly represent the
network. This is the only way we can calculate these losses, the flows
on the lines, and the associated shadow prices, you know, to do the
price calculations, and in this environment the transmission cost is
simply the difference in price between two locations. So that\'s the
implicit transmission cost.

I don\'t build complex or large models because I like large models. I
hate large models. It\'s hard to think with the small models, but this
is a minimum complexity you can use to solve this kind of problem. If
you make it simpler, you\'re solving a problem that doesn\'t correspond
to the engineering, physics that\'s out there.

So that is kind of the philosophy and the parameters with which we\'re
working. A major input into this whole exercise are the costs and the
coverage of the costs for the generators. I want Tom to tell you just a
little bit about that before we get to the results.

MR. LECKEY: The total capacity rule, which we\'re modeling in the
Eastern Interconnect, is about 570 gigawatts. There\'s about 781
gigawatts in the United States total. So it\'s nearly three quarters of
the entire capacity in the United States.

What you\'re looking at here is a regional breakout, a NERC regional
breakout of the distribution of both capacity and load which is present
in the model. The higher line, the blue graphs, are the capacity by NERC
region, and the red line is the amount of loads which are also modeled
in each of those regions.

SERC, as you can see, is by far the biggest region, ECAR, of course,
second, and then the rest of them on down the line.

Inset here is a ratio of the capacity to the load, which will give you a
little sense of what you might call the tightness of the power pool,
that is, as peak approaches what percentage of total capacity will be
called on to serve load in the regions, and here, not surprisingly at
all, the tightest power pool is Florida FRCC. Somewhat surprisingly to
me at least is the fact that ECAR is represented as the second tightest
power pool, but then the rest of them follow from that.

What we tried to do in arriving at cost estimates for these generators
was to, in effect, break them up into three categories. Some of these
generators, these 5,300 generators, which were reported on the FERC
form, we could identify and match them to other known generators, say,
in EIA inventory, and from there we could begin to arrive at estimates,
operating estimates for how much it might cost to operate each of these
generators over the short run.

So some of these, about 91 percent of the total capacity, we were able
to actually tie to an operating history, which gave us some confidence
at least that we would know how they would be dispatched at least in the
short run.

What you\'re looking at here again is a regional breakout of that
distribution to the amount of capacity which we could identify its
operating character and the amount of capacity which we could not.

Some percentage of the capacity was unknown, unidentifiable, and we
couldn\'t figure out anything at all about that capacity. Some of the
capacity we were able to identify and say, \"Oh, yes. In all likelihood,
I think this is that generator,\" but perhaps it was a small, internal
combustion unit, not reported at that level on the FERC Form 1, and so
it was not possible for us at least in this preliminary stage to develop
an operating cost for it.

Again, inset here is some of the statistics about the distribution of
these types of capacity by region. NEPOOL, not surprising, is the
hardest to identify, is the hardest both to just figure out what
generators were actually being named there, and beyond that it was also
equally difficult to actually develop costs, and that simply has to do
with the fact that there\'s a lot of small generators in New England.

Florida, oddly enough, was the easiest to identify both in terms of
numbers and in terms of actually developing representative operating
costs for them.

This is a brief description of what we found to be the technological
character of the generating units at issue. Not surprisingly the vast
majority of it is fossil steam, mostly coal, 340 gigs of 570 gigawatts
puts it at a little higher than, say, the national representation of
coal.

You may also notice immediately that that portion of the capacity which
is hydro or pump storage, 39 gigawatts, as a little low, reflects the
fact that what we\'re talking about in the Eastern Interconnect is \--
well, it excludes the large hydro resources out West. Okay? So we end up
with a little bit more fossil steam, a little less hydro, and the rest
of the distribution is about what you might expect.

Over here on the far right is what we didn\'t know anything about. We
couldn\'t identify them, 39 gigs, which is about 6.8 percent of the
total. We have 85 gigawatts of nuclear, which corresponds to, I think,
84 units in the Eastern Interconnect. So it leaves out the 12 operating
nuclear units in the western portion of the country.

Interestingly enough, Zion in Chicago is modeled. Zion, the two units at
Zion are two of the 85 units modeled here. They produce no real output
in this model. They produce about 250 megavars (phonetic) of reactive
power, which represents what the planners in Maine wanted to do with
Zion, want to stop producing real power.

This is a rough supply curve for the Eastern Interconnect as a whole and
goes out to about, you know, 570 gigawatts capacity. The left axis is
what we call the operating cost, and the operating cost was simply the
heat rate, which is an efficiency measure of its ability to convert
thermal into electrical energy, times the composite fuel cost for 1999.

So insofar as a fossil unit is able to convert at a rate better than
10,000 BTUs per KWH, it\'s going to reflect a lower, say, composite fuel
cost, and insofar as it may be a gas turbine, 11, 12, 13,000 heat rate,
it\'s going to be slightly higher than the composite cost of fuel.

But basically the operating cost is a rough estimate of the fuel cost
with a small element for the efficiency of this unit. Here to the left
of about, say, ten mLs, we\'re talking about the base load nuclear
capacity and some pretty efficient coal units.

This portion of the curve right here at ten is the 39 gigawatts of
unknown capacity which we withheld from the optimal power flow solution.
So 39 gigawatts were not applied in the objective cost function to
minimize it. Rather, they\'re simply dispatched with the planners and
reported on the from. So if the planner said we wanted to use this
generator to meet peak load, we accepted that and dispatched it, but it
was not used to minimize cost, and that\'s represented right in this
portion of the curve.

Beyond that you have your coal scheme and combined cycles begin to kick
in around 18 to 20, 25 mLs. Interestingly out here at about 25 to 30
mLs, you have some FO-6 units, some old, residual units which are highly
regional, a lot in New England, a lot in ConEd right around New York
City, and a lot in Florida as well. So that\'s a significant supply
resource for Florida as well, and then the gas turbine is off to the
right of that.

And this slide is simply a subset for three regions at issue of the
previous curve, and so it tells you a little bit about the contours of
the supply resources as we modeled them here.

To the right is SERC, the biggest region. Here in the center is ECAR,
the second biggest region, and here is Florida FRCC to the left. You can
see that ECAR and SERC have long, flat portions, base load, large base
load capacities, and then their peak which, again, does not consume very
much of the capacity, a very small percentage of the capacity, but you
can see that with Florida the curve, the supply curve starts to slope up
very rapidly to the right, indicating again the presence of a lot of
FO-6 units, residual units, where the cost is rising fairly rapidly.

So you can see that about, you know, half of the entire Florida supply
curve is above, say, 20 mLs where, you know, your rough estimate for
base load capacity would be below that, 18 mLs, 16 mLs, and you can see
that the other two regions have much more base load capacity.

Okay. Doug is going to talk a little bit about some of the cost results
which came about from this.

MR. HALE: For the timekeepers, it\'s very short at this stage.

So going back to what we\'re doing, the first thing we did was to take
the trade limits that were imposed by NERC and their planners on each of
the regions, and then within the region, to optimize \-- you know,
minimize costs without \-- within the region, and we came up with this
distribution of prices.

From last time you may remember red means it\'s high, blue means it\'s
low. High goes up here. I think we got to \$700 a megawatt hour, which
you know is kind of San Diegoesque. This is the ocean. Don\'t ignore
that.

Any place you see red is kind of a high priced area. Given the number
of, you know, the complexity of the output, these sorts of contours have
been the only way we\'ve been able to think about this stuff. I mean
it\'s just too hard staring at tables.

So then what we did was open it up for free trade, and it really just
flattened everything out. I mean there was Florida, which was actually
in pretty bad shape, and the prices were up to well over a couple of
hundred bucks here, and TVA, which was a disaster, you know, dropped
like a rock, okay, and that\'s simply because you\'re opening up trade
to these areas.

On average, the prices didn\'t drop very much. You\'re talking about an
average price dropping from like \$35 to 33 or so. The standard
deviation of these prices dropped, however, from over 20 to about six.
So what you\'re seeing is a price equalization across markets.

So what it\'s saying is at least in the East, even in summertime, if
you\'ve got a strong enough grid and enough key generators, so even
under pretty good summer load if everybody will trade together, we can
avoid the San Diego type price spikes.

Okay. That\'s a Never Never Land because they\'re real people and
they\'re real coordination problems, but at least the basic idea behind
what FERC is doing is okay, but I do want to point out that there is a
problem.

We took a generator out of Florida under the free trade case just to see
if anything will happen. It\'s not a very big generator, but all of a
sudden life gets real expensive in Florida, and as we showed before last
time, crazy things pop up in other parts of the country. Okay?

So these guys are paying high prices because somebody in Florida blew a
generator out. So this, again, gets back to the pervasive role of
externalities and interconnection in these electrical systems,
especially as you open them up for more and more free trade. You get
very, very concerned about what someone hundreds of miles away, and
it\'s literally hundreds of miles away, you know, is doing with their
maintenance.

So back to questions for the committee. Again, you know, I\'m confident
that what we have here is reasonable. I wouldn\'t take it to the bank at
this stage, and so I\'m interested in your ideas for development
priorities.

The other thing I\'m very interested in hearing is if you have ideas of
model exercises, either of the model in its current state or as fixed
up, that would tell you something of interest in your own studies and
thinking.

And that\'s it. Thank you very much.

CHAIRPERSON GOTWAY CRAWFORD: Our first discussant is Bill Moss.

DR. MOSS: I don\'t need the overheads.

I like this work a lot, and I think the model is tremendous. I wish I
had it. The model is in the public domain, but the software isn\'t, and
it\'s not cheap, but it\'s \--

MR. HALE: We can make arrangements. Talk to me later.

DR. MOSS: But it is. It\'s a real advance than a lot of the models I\'ve
seen.

I will give you some general comments on what I see they\'ve done and
some possible directions, and then kind of address the questions they
pose and the priorities.

First of all, in terms of a lot of this work and whether the FERC is
going in the right direction, it may or may not. In some sense RTOs
controlling the transmission system will take at least in theory the
transmission system out of the hands of people who might use it to
exploit it for their own purposes.

However, if you read, I guess, the trade press, the popular press and
the popular press, the impression is that the transmission system is
vulcanized because these utilities were designed that way. The original
utilities were designed as stand alone businesses, and they had
connections with their neighbors to a minimal degree to allow some
trading in emergency cases and when they had surplus.

So if the transmission system is truly vulcanized and your results
suggest that maybe it isn\'t, it\'s still possible that the RTOs won\'t
solve the problem of gaming the system, and that the people have market
power and will be able to use their generation and their bidding
strategies to raise price.

I had a couple of questions in terms of the fuel cost, and as you\'ll
see later, I think this is an area you should work on because since this
is a competitive model and your marginal costs are driving the results,
the quality of the fuel costs are important here.

But in the paper you refer to calculating fuel costs and heat rates
based on annual average data at the plant level. Now, here you talk
about generators, and when I think of plants, I think of groups of
generators, and if it\'s true that you are doing it at the plant level,
I would think that that would be an important direction to move away
from to go to generators because generators at plants often have
different characteristics, different cost characteristics and may be
dispatched differently.

And when we get to the strategic issues, I\'ll bring this up again.

Also, and this will come, I guess, into another suggestion I have, when
you work on fuel prices, I think it\'s important to try and get some
seasonality in the fuel prices, at least some of them. This is
especially important in natural gas, probably not so much in coal, and
this gets to another suggestion I have, is this is for one hour and one
hour at the peak, and it\'s not clear to me whether this is kind of a
system peak or whether these are kind of local peaks that are then
viewed as coincident when they\'re not really coincident.

But the issue with the peak, and maybe I will \-- this gets into your
results of free trade. I was a little surprised at the results because
logically they look great, and I believe the direction of change, but
when I look at the results that there seems to be no load pockets,
there\'s no congestion, I\'m not sure I believe it, and I have to defer
to you because you have the models and you know the data better, but
when you look at the issues of, say, open access deregulation at the
state level \-- and one of the things that the state regulators are
worried about are load pockets within their state, but also load pockets
within control areas.

Now, I\'ve done some merger work, and we always do it at the control
area. So we\'re throwing the baby out with the bath water there, but you
aren\'t. You\'re working at buses and things like that. So it may be
important to change the focus of the model or in terms of future
directions, is to start looking at smaller areas to see whether despite
on the broader, kind of NERC area scale it doesn\'t look like you\'ve
got congestion problems that they can trade if the institutions are
there to do it.

But what about within the control areas where you may have load pockets
and generators. incumbents or new entrants can exploit market power. I
mean that\'s really the crux of some of the deregulation issues at the
local level or state and local level.

And this gets to perhaps running your model for multiple hours. I don\'t
know how long it runs, but the way I think of it is you have your load
distribution curve, and you\'re taking the hour way up here, but there
may be other hours you can sample, and I would do this for another
reason, and I\'m not sure just picking hours is enough in terms of
developing scenarios.

Rather than looking at the peak, as I see the peak, you could think of
an example where you have a transmission line between two areas, and
during the peak the generators and the loads on both sides are kind of
going full blast. So they\'re kind of balanced. There\'s no congestion.

But that\'s not where you get the problem. The problem is what if in one
area demand is down; the other area is a low cost area. So there\'s a
tendency for power to want to go in this direction, and that you might
not see so much in the peak as in some of these off peak.

For example, I\'ve talked to utility people, and they say \-- well, one,
in particular, said, \"Well, you know, when there was weather
differences down in ECAR,\" and so there\'s a tendency for the power to
want to go from MAPP \-- I\'m sorry. Down in SERC \-- and there\'s a
tendency for the power to want to go from MAPP and ECAR down into SERC.
Our powers bottle up. We can\'t get anything out.

Well, you don\'t see that here. So where this model could be really
useful is to develop scenarios where you get imbalances in demand and
supply so that the system will want to sell power from one area to the
other, and that is going to be another situation where you\'re going to
see some bottlenecks, and then the question is: are they enough to make
a difference in terms of market prices and the ability to exploit market
power?

I mean, you don\'t have to have these load pockets and these bottlenecks
to last you very long for people to make money. I mean, in California
some of the generators, I think, have paid off their units in the prices
they\'ve received over a relatively small number of hours during the
summer, and of course, that\'s what\'s raised a lot of the hackles out
there, and of course some of the solutions will probably aggravate the
situation.

The visualization, I like this, and I agree with you. You\'ve got
tremendous amounts of data. You have what is it, something like over
30,000 buses? And each one of those buses has a price, and if you\'re
interested in transmission cost, since I take it you\'re not putting
tariffs into the model \--

MR. HALE: No.

DR. MOSS: \-- although you could, the total transmission cost, I think,
would be the tariff plus the difference in the locational prices.

But in any event, this is a nice way to visualize it. Now, just as a
simple matter on how you distribute your results, I love the ability to
download it off the Internet, and I\'ve read one of your earlier papers
doing the same thing. I look at it, I download it, and then I print it,
and I print it to my black and white printer.

So when you do that and you look at the scales on your colors, low
prices and high prices are dark. So unless you have a good description
in the paper or you have tables to go along with it, or unless everybody
is printing on color printers are reading the paper on the screen, you
lose the benefits of seeing the imbalance between the dark and the
light.

MR. HALE: I apologize. We put the wrong \-- we have black and white ones
as well \-- we put the wrong shading in.

DR. MOSS: Oh, okay. Well, I like the color. Next time I\'ll print it on
the color printer.

MR. HALE: Actually it wasn\'t we. I did it. I\'m sorry.

DR. MOSS: I think you\'ve answered some of my questions. Losses are
included in the system, right?

MR. HALE: Right, right.

DR. MOSS: Right, and you don\'t charge transmission \-- in terms of the
priorities, you had a list of five questions for priorities, and then
you asked for analytical questions for which you might exercise the
model.

The questions were in terms of prioritizing, refining the cost
estimates, validating, including price sensitive demand, including
noncompetitive bidding behavior, and redoing Northeast.

Now, the priorities I\'m suggesting aren\'t necessarily from most to
least important. I view them as a combination of what\'s most easily
doable, moving into things that are probably more important, but will
take a lot more work, and unless there\'s something about your model I
don\'t understand, a quantum amount of work.

So I would think redoing the Northeast with your model now might be a
fairly simple task. You\'ve got the model. All of the data is there. Do
it and compare it to your other and see whether expanding the model has
made a big difference.

Because I think what it really amounts to is you\'ve kind of in your
northeastern model, except for the fact that you had a full AC
representation, everything outside is equivalence.

MR. HALE: Right. Exactly.

DR. MOSS: And how do you close the model outside the border? Well, you
kind of have some generators and some demand or load as stopgaps.

I guess the interesting question there is: how big a model do you have
to have to get accurate estimates for an area that you really want to
study? And I think you have the ability here to really test that, and
I\'m not sure we really know.

You know, it might be that your New England model is perfectly fine and
you won\'t get many results that will differ, but that\'s not obvious.
It may be that you have to go a couple of NERC regions out before the
inaccuracies dampen out.

In terms of validating against the market, I think it\'s a good idea to
do PJM because they have locational prices there, I think, at the bus
level; isn\'t that right? And they have a market operating.

When you\'re looking at the rest of the Southeast, the question is: what
are you going to benchmark against? You don\'t have market prices
really.

A couple of suggestions. You might look at some of the like Power Market
Week where they have certain markets. They\'re not a perfect substitute,
but they are based on transactions that are between willing buyers and
sellers.

So, you know, I don\'t know whether you\'d be checking them or checking
your model, but it\'s an alternative source.

Another would be if you know that there are some utilities that are
fairly low cost and that their reported system lambdas really represent
their system cost, if there\'s something you know by talking to utility
people, you might check it against that. But most of these system
lambdas aren\'t very reliable because it turns out that the utilities
report system lambdas from their automated systems, the ones that
automatically dispatch their generators, and the peakers and things that
they put on based more on just discretionary behavior that aren\'t kind
of, you know, dispatched by computer, oftentimes they\'re not included
in the system lambda. So you\'ll go into an area and you\'ll get a
system lambda that says marginal cost is \$15 when, in fact, it\'s in a
particular hour, and it\'s really \$25. So that\'s not a good idea.

Talked about different scenarios, especially trying to have imbalance
across areas.

Now, testing the strategic behavior. I think there\'s a lot you can do
here by pursuing the method you\'ve already done, and that is take out
generators, misstate the marginal costs for generators because, you
know, there\'s a couple of ways a supplier can affect the outcome.

One is they can say, \"Oh, my generator is not available.\"

The other is they\'ll say, \"Well, my costs aren\'t \$15. They\'re
\$30,\" or \$100, and you can play with that a little bit.

And that\'s nice because you don\'t have to change your model at all.
It\'s probably a straightforward change in the data.

And I\'ve already said that I\'d encourage you to improve the cost data
because it\'s the linchpin in really the whole model, and I don\'t have
any great suggestions, except there are some commercial sources, and I
think they\'re good in one sense, and I\'m thinking of RDI.

MR. HALE: Yeah.

DR. MOSS: They try and model the energy cost to individual generators
taking into account transportation and seasonality. So that might be
helpful.

Okay. Two more suggestions, and I think these may be the most important
directions to go, but they\'re the hardest. So I wouldn\'t say, you
know, perfect this first and see what you can get out of it, but that is
price sensitive demand. That\'s important, but the question is: how do
you do it in a model like this?

This is a linear programming model, and you can\'t put demand curves in
there because once you put a demand curve in there, you\'re probably
going to introduce nonlinearities. So one of the ways to do it is at
load centers, you put in dummy generators or \"nega\" (phonetic)
generators that kick in as the price goes up, and that essentially
offsets the demand, and you do get a price sensitivity. So that might
work.

And I\'ve seen it done in others, and you know, whether it\'s successful
here you\'d have to play with it, and actually that\'s not a major
departure. It\'s a matter of figuring out where you want to put these
generators and how to have them kick in at prices to represent the price
elasticities you think are reasonable.

The last one is strategic behavior, and I think this is important from a
policy point of view, but I\'m wondering how you\'re going to do it
because, you know, when you think of how to model strategic behavior,
you\'re saying, well, there are people who are going to set prices
taking into account what other people are doing. So you have to decide
who are bidding strategically and who are going to be price takers, and
it\'s complicated because you have lots of markets, and some of these
suppliers can supply into multiple markets.

Now, if you say everybody\'s strategic, in a Cornell model you get
competitive. With enough bidders you get a competitive solution. So
I\'ve often been puzzled with models who have a lot of markets that are
really equilibrium models. How do you put strategic behavior in there
because you\'ve got a lot of players?

And so when you solve that one, let me know.

MR. HALE: Well, that one should be easy.

(Laughter.)

CHAIRPERSON GOTWAY CRAWFORD: Our next discussant is Calvin Kent.

DR. KENT: Since according to the clock on the wall we are on break and I
recognize the fact that I\'m the only thing that is standing between you
and the bathroom, I will try to make this as quick a presentation as
possible and will not cover all of the things that I wanted to say,
except to begin by noting that I think that this paper has focused on
the most crucial single issue in the whole question of energy or
electricity deregulation.

I was looking back over some of the historical material on this and
remembered that back when we passed EPAC, back in 1992, and EIA was very
heavily involved in the passage of that act, and I personally was very
heavily involved in those provisions that led to deregulation, that we
first started off talking about that we were going to reinvent the
electric utility industry.

We then talked about that we were going to be deregulating it, and I now
know that the term that we\'re using is \"restructuring.\" So we are
going downhill as reality is continuing to pile up upon us, that what we
are not doing is creating really competitive markets in electricity or
anything that begins to approach them.

I want to make one thing clear, and that is from the beginning the real
Achilles heel of this whole deregulation or restructuring effort has
been the transmission, and there are three problems that are associated
with the transmission aspect of this.

The first one is the engineering problem, and that is that, as has
already been pointed out, the whole transmission system was simply not
designed for a competitive free flow of electric power either between
the regions or even between systems, and there has been very little that
has been done in the last eight years to change that situation.

For reasons that I think we can talk about in a little bit more detail,
there have simply not been the investment that is needed in either new
transmission lines, refitting the transmission lines, or even adapting
what is already there to allow for a free flow between systems, much
less between areas.

And as a result of this, I think that the engineering situation is one
that needs to be fully considered, and I do know that they have done a
good job of this, but it is the transmission costs that occur when this
congestion appears, and the congestion appears not just because there\'s
not enough capacity, but because of the fact that the capacity is there
is not appropriately interconnected, and I\'m not sure exactly how you
model that.

I do notice that when the model was run by taking a generator off the
line in Florida that we got all sorts of dramatic results. I would be
concerned with how do we model the problems that develop in the
transmission system.

The second point to make here is simply that how do you price
transmission, and I am one that thinks, because I have been told this by
the people in the electric utility industry, that they do know how to
gain this, and that at least in the very short run with electricity now
being sold in nanoseconds over somewhat competitive markets, that they
can gain in the short run.

And exactly how do you then in this model price the electricity or the
cost of transmission, which is something that I think needs to be
included there?

And I\'m running over this rather quickly. The third problem is
certainly both economic and engineering, and that is the question of
dispatch. Because the assumption always is that the least costly models
or the least costly generators are going to be dispatched first.

I think if you look at the transmission system, you find that that is
not always going to be the case, that it may be very difficult to
dispatch the least costly ones in the appropriate order simply because
the transmission system is not going to allow this.

Now, the paper is an extremely important step forward because it does at
least begin to deal with some of these issues. So far as my specific
focus is or criticisms would be, Bill has already made that, and this is
the one about how do you really handle these issues associated with
costs, and there are some statements in the paper here that worried me
about how they had actually gone about estimating these costs.

And so I would, again, call that to your attention, guys, as you go
forward with this, to really spend some more time on your estimations of
cost.

I also liked that in the model that they have the two cases, the first
one being the administrative trade model, and then they have the free
trade model. I was somewhat surprised at the economic gains were only
3.3 percent from the switching from the administrative model where power
did not flow very well between the various regions to the free trade
model where they did.

So far as their priorities are concerned, I would certainly say that the
first thing to do is to validate against the PJM because I think that
that\'s something that can be done. It can be done quickly, and it will
give you a good sense of whether or not your model is working very well.

And I suspect that you will find that it is, but I think that this is
something that you could do.

I think you know that one of the things that requires as a first
priority is to actually look at whether or not there will be problems or
can be problems with the large transmission owners who are going to be
able to gain the market.

Those of us who have read the proposed FERC 2000 recognize the fact that
FERC is doing a lot of wishful thinking in that particular NOPR that is
out there, and that is they are assuming that by telling people to
behave well, that people are going to actually behave well, and with
transmission now being separated at least in theory, there is still the
possibility as the paper points out, particularly at the end of the
paper, for a lot of the people, such as TVA and so forth who control
huge amounts of the transmission capacity in certain areas to certainly
benefit themselves by that control.

Now, exactly how you work that into a paper I don\'t know, and I\'m not
sure if I\'m disagreeing with Bill or not, but I think from a policy
standpoint, and certainly in terms of being helpful to FERC, that this
is something that needs to be looked at, and at least some of these
things might be placed into the model at some points to show what the
results would be; that if there was some gaming taking place by some of
these large transmission owners, what the results would be.

And then the next priority that I would have would be to simply take a
look at the shifts and the level and location of demand. That\'s already
been mentioned. I think that that would be the next step that I would
take.

And then the last step that I would take, and these are in the orders of
my priority, would be to add the price sensitivity demand schedules and
noncompetitive bidding.

But this paper is an excellent example of how you can have research
which begins to answer some of the most important questions, and my
compliments to you all for having done an absolutely outstanding job.

My apologies for running over.

CHAIRPERSON GOTWAY CRAWFORD: Not your fault.

Do you have anything to add?

MR. HALE: Very quickly.

CHAIRPERSON GOTWAY CRAWFORD: Okay. Very quickly.

MR. HALE: Yeah. A couple of things. The longest run we had was 15
seconds. So it computes very, very quickly.

Second, on congestion, congestion actually went up with free trade. It
went up from 21 congested lines or line segments to over 30. The problem
here is resolution.

I mean, we\'ve got guys in the back room still drawing those one-line
diagrams so that, you know, we can get it on the thing, and that\'s a
problem. Yeah, there are load pockets. No question about it.

The question on power transfer is the one we took up in the last paper.
We\'re trying to move power, keep power from ECAR to New England, and
that\'s when we started to break the system pretty badly.

My guess is you\'re absolutely right. We\'re going to have the same sort
of phenomenon if we try to move a lot of cheap power down to Florida.
It\'s going to break. We\'re going to hit that constraint pretty badly.

As far as the actual demand cases, they are FERC\'s view of what a bad
summer day is like. It\'s not well defined. We do have to change these
demands around a lot. It takes work.

The problem we\'ve got is this is an ad hoc effort. I mean, we\'re
grabbing people from agencies all over the place. The biggest slow down
for us is Tom has been involved with the Annual Energy Outlook. The guys
at FERC just wrote a new report for FERC that they just released in
California, and the Justice Department people won\'t talk right now.
They just say, \"We\'re busy. Go away.\"

But, you know, we\'re getting it done. It\'s just catch as catch can.

Two quick things on the small cost savings. One of the reasons the cost
savings are small is that you\'re at a system that\'s practically at
peak. So there\'s very little you can do by displacing generators.

The other thing is there\'s an anomaly in how we\'re calculating costs.
We\'re calculating out-of-pocket costs without including the shadow
price of the resources. So we\'re basically understanding the cost
savings.

You know, I realized that two nights ago that that\'s what we\'re doing.
So it\'s wrong.

DR. MOSS: So those aren\'t market prices that go down. Those are kind of
the margin costs. Those are cost savings. They\'re not price changes.

MR. HALE: We calculate price changes and cost savings differently, but
what we had there were the cost savings, and that\'s the problem. I
mean, if we had total surplus, it would be a lot more dramatic. Okay?
That\'s where we\'re messing up.

Demand, we\'ve got the computational ability to do demand. No question
about it. We don\'t have a clue as to what to put in there. I mean, I
don\'t have any idea for an elasticity amount or anything else. If
someone does, we\'ll do it.

I really liked your suggestion about redoing the Northeast. I think
that\'s a key question for engineers and for the whole idea of RTOs. We
will fix up the cost.

Tom has done a far, far better job than he had time to talk about in the
paper, and I think he did what, you know, you asked him to do, but
we\'ll do that right after the break, I hope.

Thank you very much. It\'s been very, very helpful. Thanks.

CHAIRPERSON GOTWAY CRAWFORD: Thank you.

Let\'s take a slightly abbreviated break, maybe ten minutes since we\'re
late anyway, and reconvene in ten minutes.

(Whereupon, the foregoing matter went off the record at 10:43 a.m. and
went back on the record at 10:58 a.m.)

CHAIRPERSON GOTWAY CRAWFORD: I\'d like to go ahead and begin with our
next talk, and it\'s a feasibility study for surveying industrial
natural gas consumers, and it\'s led by Renee Miller with two
representatives from the Census Bureau who will need to introduce
themselves.

MS. MILLER: Okay. The first challenge is getting hooked up here. Okay.
Can you all hear me okay?

Good morning. I\'m pleased to be here to give you an update on the
feasibility study for surveying industrial natural gas customers. This
is based on work that Jay Casselberry did. Jay is sitting right up here.
Also Susan Bucci and Sarah Teichner of the Bureau of the Census, and
Susan and Sarah are in the back there. Roy Kass; Roy is somewhere in the
back also. And several other members of the Next Generation Natural Gas
Team also contributed to this.

We previously told the committee that we were considering this study,
and we recently completed it. So we wanted to present the findings to
you and tell you where we are and talk about some issues that arose.

Now, going back to this title update on feasibility study for survey
industrial natural gas customers, that sounds like quite a mouthful, and
some of the newer members might wonder, well, what\'s this about. What
problem are we trying to address?

Well, as Mark pointed out yesterday, we have a problem with coverage for
our industrial gas prices, and let\'s go to the graph.

Some of you may have seen this graph before. Roy Kass presented it at an
earlier meeting, and you can see that something is decreasing, and that
something is the percentage of industrial natural gas deliveries
represented by the on system sales, and those are sales where the local
utility owns the gas.

And this is also the data on which EIA prices are based. So by 1999,
which is the last points on the graph, the price data that we\'re
showing for industrial natural gas was based on 17 percent of all
industrial deliveries.

Now, we label our data as on system sales, but still we think we have a
problem here. Well, how did this happen?

And I know some of you already know the answer, but just to make sure
we\'re all together here, let\'s go to the chart. This is a chart we got
from Natural Gas Division, and actually the title of the chart explains
it. It says that financial transactions are no longer tied to the flow
of the commodity.

So starting from the top of the chart, the gas moves from producers to
pipelines, onto the local distribution companies, or using EIA-speak,
LDCs, or you may call them local utilities, then on to the end users or
customers.

And in the good, old days, the financial transactions followed this
pattern also. So the pipelines and the LDCs owned the gas. They knew the
cost, and they\'re our survey respondents. So when we ask them about
quantities delivered to end users and costs, they were able to report.
And we didn\'t know how good we had it back then.

Okay. Moving along to the middle of the chart, there are now more market
participants. So the end users have a choice. They can buy gas from
sources other than the local utilities, like the marketers in blue, or
they can make arrangements directly with the producers, which that\'s
that yellow \-- a technical problem here.

Okay. I think I\'m back on line here. The end users can also make
arrangements directly with the producers, which is the LO-2, purple
transaction. So the LDCs still deliver the gas. So they know the amounts
of gas they delivered, but they don\'t necessarily own it. So they
don\'t necessarily know the price, and therein lies our coverage
problem.

So we\'re just getting the prices for that top purple block. The orange
system end users were not getting the off system end users.

So what to do? Well, we decided to try surveying the end users. After
all, they should know the price.

We also considered surveying marketers, and in fact, we are going this
route for residential and commercial, but several years ago the
marketers told us that they couldn\'t report by state and end use
sector, and that\'s how we present our data.

There\'s also an additional complication for the industrial end users in
that transportation arrangements are often made separately. So the
marketers might not know the cost of transportation.

And also, if we went to the marketers, we would miss that yellow to
purple transaction where the end users make arrangements directly with
the producers. So we decided to consider surveying the end users, and we
approached the Bureau of the Census to help us with this since they have
it framed for the manufacturers. In fact, they conduct the manufacturing
energy consumption survey \-- many of you know the MECS \-- for us every
four years.

Now, I should point out before going any further that manufacturing is
not the entire industrial sector, but it represents a good bulk of it.

Okay. So we know that the manufacturers can provide data every four
years on the quantities of natural gas purchased and amounts paid, but
we had some questions about whether they could provide data monthly. So
let\'s go to the study questions.

We had concerns about whether the respondents would be able to report
monthly on a timely basis, whether they would be willing to report to
us, whether they would find it difficult and how the data from the
customers would compare with the data from the suppliers. So this formed
the basis of our feasibility study.

And we have some more details on the study in the next overhead. We
conducted the study in three phases. The first phase was a mail survey
of the manufacturing establishments purchasing natural gas from sources
other than the local utility, and at that point we asked establishments
to report quantities of natural gas purchased and cost of natural gas
for a three month period that they chose because we wanted to make it as
easy as possible for them to report.

And we also asked them to provide details on their suppliers and
authorization for information released to Census.

Then the next phase was the mail survey of the natural gas suppliers,
and we asked them to report for the corresponding three-month period,
the period that matched the one that their customers selected.

And then the third phase was telephone interviews of the establishments.
this was a joint project between EIA and Census. EIA designed the survey
forms and provided the technical direction. Jay provided the technical
direction. Census selected the sample, did all data collection
activities, and summarized the results, and that\'s where Susan and
Sarah came in, and they work under Judy Dodds at the Bureau of the
Census.

Okay. Some more information on the study design. We took a small sample
of the MECS respondents. We selected 150. Ninety-nine returned it,
giving us a 66 percent response rate, which we\'re actually quite
pleased with since this was a voluntary feasibility study.

Another feature was that we didn\'t do much editing since this was a
feasibility study and not the actual collection. We converted quantities
from the unit reported to million cubic feet, but we didn\'t do any real
extensive editing, and this is a feature that we\'ll come back to a
little later.

Okay. So I gave you a little background, and now we\'ll go on to some
findings on the next overhead.

Okay. Starting with some good news, on timeliness of reporting in
response to a question about how soon would the respondent be able to
report each month, you can see that most of them would be able to report
within 30 days. So we were encouraged by that.

The next one on willingness to report, most said they would be willing
to report monthly. Reasons for not being willing included too time
consuming. Someone said they didn\'t believe in big governments unless
it would help them.

(Laughter.)

MS. MILLER: There is skepticism about how the government used the
information. I mean, we\'ve heard some of these things before.

Okay. About difficulty of reporting, on the next one, most of them said
it wasn\'t difficult. When we asked why it would be difficult, here
again some of them mentioned the amount of time consumed. They talk
about third party invoices that they needed to get, which created
complications.

Okay. Now, going on to the establishment-supplier comparisons, yes,
there were some differences. We had 157 pairs, and I think pretty
amazingly 37 reported identical quantities of both gas purchased and
amounts paid leading to identical prices.

Now, we computed these prices for comparison purposes, but you should
keep in mind that they don\'t necessarily represent a specific period
because the pairs could have been reporting for different time periods.
So one could be reporting January, February, March, and the other, you
know, May, June, July, something like that.

We had 26 pairs reporting identical quantities, but different amounts
paid, and the average price was somewhat different there. We had 18
pairs that reported identical costs, different quantities purchased, and
here you can see a large difference, and it looked like we have a
slipped decimal place for the quantities purchased.

We had 76 pairs where neither matched, but even here I see something
encouraging because the price for the establishments, the \$2.88, seems
reasonable compared \-- as a reference EIA published price for 1999 was
\$3.10 per thousand cubic feet. So the \$2.88 seems in the ball park,
and it was, after all, the establishments that we were concerned about.

This does show that we need to find out what exactly is being reported
for price. There may be different interpretations of what should be
included, but overall we were encouraged by these results, and so we
asked Census for cost estimates to do this for real, and we gave them
six scenarios which were in the paper that I gave you, and some just
included the manufacturers. Others included mining and construction,
other parts of the industrial sector. Some of the scenarios were for all
states. Some were for selected states, and we got cost estimates from
Census and we\'re in the process of working with them to determine which
scenarios we want to pursue, and then we can refine the estimates.

Meanwhile, in conducting the study some issues arose, which is on the
last overhead. There was interest in comparing the EIA published price,
which is the on system sales, with prices from sources other than the
local utility, which we were collecting from this feasibility study, but
we weren\'t able to do this using the data from this study because the
time periods differed. As I explained, we allowed the respondents to
select the three-month period that they could report for.

We also didn\'t do any significant editing or follow-up because our
intent at this point wasn\'t really to collect data, but to determine
the feasibility of collection.

So what we wanted to talk with you about was your experience with this
problem of balancing the burden on the study participants with the
versatility of the study, and also in terms of collecting versus the
feasibility of collecting, like at what point does one draw the line for
a feasibility study.

So we\'d be interested in what you think of these issues and if you have
any questions or want more details on the feasibility study, Susan and
Sarah would be available for more questions.

Thank you.

CHAIRPERSON GOTWAY CRAWFORD: Thank you, Renee.

Are there any questions or comments from the committee? Polly.

DR. PHIPPS: I was just thinking about the issues that you posed to us. I
think there\'s a tradeoff in any of these studies between, you know, how
much information that you ask in a test situation, but I mean, I think
your priority should be to test the scenario that most closely matches
what you want to implement ideally.

And in this case, you know, if you felt you needed, you know \-- you
basically asked for a quarterly set of data, and people were to report
that in any time frame, and I would probably argue that you should have
focused a specific time frame there because you\'re not going to give
respondents the option in the real world to have a choice of time frame,
and that might have helped out with your other study, too.

You probably want a fairly stringent test here because when you want to
know what your response rate is in the long run as you operationalize
it. You might be able to look, I was thinking \-- I mean, you didn\'t
report anything about the time frames that people selected to report or
to see if they\'re close to the time periods, you know. Are they
reporting for, you know, quite a bit in the past or are they current?
Because I\'m assuming you\'ll want the current information. So you might
be able to look that.

To when you add burden by asking more questions in kind of a feasibility
test, you may want to do something like just split the sample. I know it
wasn\'t huge to start out with, but you could have taken, you know, a
third of it and ask extra questions so you would have a sense what it
would be for the additional burden.

A couple of other points. Also in the collecting and analyzing data for
future data collection, I mean, I would argue that you should basically
\-- well, this is repetitive \-- ask them for the data that basically
you\'re going to need in the future for collection.

On the response rate, the manufacturing sector traditionally has a
higher response rate, a good, high response rate. So, you know, I don\'t
know how that will transfer to the other industries, such as
construction and the other sections, something you may want to think
about in the future, but I think a 66 percent response rate is pretty
good.

And, you know, if they\'re generally providing data that\'s pretty close
to the time frame that you\'d want, you can expect that to be the
difference.

I was interested in if you had any reasons why the response rates would
be more different between the different groups, actually getting a
higher response rate from your suppliers. I wasn\'t certain why that
would be.

The other thing I wondered when you talk about the differences you\'re
finding in prices basically if you could do any call-backs to find out
about people are interpreting your definitional differences, your
definitions differently. So that might give you some information about
why you\'re getting those kind of informations or even do a short,
little follow-up mail survey on that.

The final thing, I was just interested in why quarterly data collection
wasn\'t recommended at all. Basically I think it was one of your
scenarios, and it is what you tested, and maybe you need the speed of
monthly data, but I wasn\'t certain why that wasn\'t a potential
scenario, was recommended against basically.

MS. MILLER: Yeah, actually we didn\'t test quarterly data. We just
allowed respondents, you know, to report for any three-month period, and
the quarterly scenario was rejected based on the experience that the
Census Bureau had with that type of collection. I don\'t know if either
one of you, Susan or Sarah, want to say anything more about the
quarterly scenario.

DR. PHIPPS: So you asked them to provide three months\' data?

MS. MILLER: Yeah, right.

MS. BUCCI: I thought that you had asked for quarterly data, but you
still wanted to go back and publish monthly.

MS. MILLER: Monthly, yes.

MS. BUCCI: And I think that\'s what caused us the problem.

CHAIRPERSON GOTWAY CRAWFORD: Johnny, you have a question?

MR. BLAIR: Yes. The response rates look reasonable for the feasibility
study. One of the things unfortunately that is hard to do in a
feasibility study is to estimate the effect of burden over time, going
back to certainly some of the large customers that are going to be
sampled repeatedly.

So it may be that this response rate is kind of maybe the upper end of
what you might end up expecting in the actual study, but in any case, it
looks like it seems reasonable that you\'re going to have some item or
not item, a unit nonresponse problem, a fair number of nonrespondents,
as with most studies. They are probably not going to be, you know,
randomly distributed, but you know, maybe some of the bigger ones that
you have trouble with.

What I would raise in terms of continuing feasibility is to start from
the very beginning of a process like this to look at what it is you\'re
going to do in terms of the nonresponse and to start looking at
feasibility of things like imputation, for example, samples of
nonrespondents, in particular on the issue of burden, if you can find
that from the nonrespondents.

There is some small amount of information that you\'re able to gain from
them that might prove useful in making adjustments for the potential
nonresponse bias.

So I guess my general point is to start to build in from the very
beginning of this a kind of parallel study to both look at reducing
nonresponse bias, and also to estimate the size and direction of it, I
think, is a component that you need in something like this. Otherwise
you\'re kind of behind the curve.

You get your data collection process out there, and then after you start
to see in the real world you do, in fact, have a nonresponse problem,
then you know at that point is, you know, when people typically start to
address it. I would suggest starting earlier on it.

MS. MILLER: Yeah, I like that suggestion because you\'re absolutely
right. That\'s what usually happens. After you\'re out in the field and
you have a nonresponse problem, that\'s when you start thinking about
imputation.

So I like the idea of looking at the feasibility ahead of time of
imputation.

CHAIRPERSON GOTWAY CRAWFORD: Any more questions or comments, Renee or
Jay or Susan?

MS. MILLER: Is there anything you wanted to add, Jay?

MR. CASSELBERRY: I think what the response \-- this was a feasibility
study. So with our clearances, we had to make it voluntary. With the
quadrennial manufacturing energy consumption survey we do, that is a
mandatory survey, and EIA has the legal authority for major energy
suppliers to do mandatory surveys. So it\'s sort of hard to tell.

We would imagine if we were going to do a monthly survey that we would
try and use a mandatory authority to get the response rate up, but we
would still have the problems you\'re suggesting over time with
decreasing or more difficult \--

MR. BLAIR: Right. You potentially have the problem over time and perhaps
turn your unit nonresponse problem into an item nonresponse problem if
it\'s mandatory, as you can make them respond in some fashion, but
getting complete and good data is not necessarily a given.

DR. PHIPPS: I wondered if you chose manufacturing was like your best
test case because it strikes me from just the experience I\'ve had that
they provide data, you know, fairly \-- you have good response rates
with manufacturers, and when you get to some of the others like
construction and other industrials, you may not do as well.

MR. CASSELBERRY: The reason we chose manufacturing is because from the
just recently done manufacturing energy consumption survey we had
information on who was buying natural gas off system. So we could use
that to target and get a fairly small sample and get these people who
were buying off system and get a feel for how willing they were to
report and some of the reporting problems they had with the total price
of gas since they were buying gas maybe from one group and arranging for
the transportation from another group.

So that\'s why we chose those, for convenience.

CHAIRPERSON GOTWAY CRAWFORD: I think Dwight has something.

MR. FRENCH: Just a comment based on MECS\' experience. Even in
manufacturing you will have pockets \--

PARTICIPANT: Dwight, you need to go to the microphone.

MR. FRENCH: Sorry. Just based on MECS\' experience, you\'re going to
have pockets of nonresponse problems even in manufacturing obviously.
Particular industry groups in particular areas of the country create
more difficulties than others.

So it\'s true about agriculture and construction. You may have some
difficulties, too, but you will have some problems in manufacturing.

CHAIRPERSON GOTWAY CRAWFORD: Well, we have a final presentation, which
is a briefing about something that\'s been in the news a lot lately on
winter energy assessment for the U.S.

MR. GAMSON: If you could give me a few seconds to turn this on, to set
up. It\'s loaded in here.

(Pause in proceedings.)

MR. GAMSON: I\'m just waiting for the Power Point icon.

I\'m going to be giving essentially the same presentation that was given
by Dr. Mark Mazur at the winter fuels conference on October 6th. In
other words, this is really just an encore presentation, the exact same
slide show.

We will have another Short Term Energy Outlook release on November 8th.

Okay. WTI. Prices peaked in September at about \$34 a barrel. In fact,
during some parts of the month they hit over 37, prior to the
announcement by the Clinton administration of the limited swap of oil
from the strategic petroleum reserve, and October has fallen a little
bit. We estimate it to be about \$33.

And the swap, according to our estimates, would make about ten million
barrels of crude available to refineries to process into either product
or replenished stocks, and another 20 million would be backed out of
U.S. oil imports and processed elsewhere in the Atlantic basin, namely,
Europe and North and South America, and would be available for all
Atlantic basin customers.

Some of this would be shipped back to the U.S. as distillate, and some
would be reduced distillate exports from the U.S. If you remember last
year, distillate, prior to the big freeze, we exported a lot of
distillate to Europe.

And a swap would also, we figure, flatten out the difference between the
current and futures price accrued. It would sort of remove some of this
backadation that you see, which would then give refineries incentives to
produce more and hold more storage.

Nevertheless, inventories of crude are typed. In a cold weather scenario
we could see \-- do we have a laser? Oh, there.

And we can see it coming up here, the high end.

On October 27th, and this is a little bit of an update from what Dr.
Mazur presented, we had estimated that crude stocks were at about 284
million barrels, which is 17 million barrels lower than they were last
year and somewhat below normal, well below normal.

But we feel that the SPR swap which started \-- oil actually started
flowing out of there last month \-- will improve the outlook.

Let\'s move on to gasoline. If our crude price path holds, gasoline
prices will be coming down, hitting we\'d say about \$1.46 by the end of
the year, with further declines next year, with a little seasons blip in
the spring.

In diesel, the same path as gasoline, except this caveat. If we have a
repeat of what happened last winter with heating oil, diesel prices will
be affected. They hit \$2 in some states on the East for a period.

Now, OPEC had a series of agreements starting last spring to increase
production. They announced last June that they were going to increase
production in July and then again they made an announcement that they
were going to increase it further in September, in fact, I think it was
Monday that they announced that they were going to increase it by
another half a million barrels, and that was going to be starting the
day after that.

In our October Short Term Energy Outlook, we had pegged that. We had
assumed that they would increase production in the fourth quarter, and
it was Eric Krail (phonetic) who went out on a limb, and I think he got
that.

And we estimate that fourth quarter production for this year will be
32.4 million barrels per day, which is about a 1.2 million barrel
increase from the previous quarter.

We expect that world demand will grow by a little over one million
barrels per day in this year, with most of the increase coming from
Asia, from non-OECD Asia, in other words, not Japan.

Next year we expect demand growth to nearly double with increases coming
from OECD, particularly the U.S., and also non-OECD Asia.

Okay. This chart shows the spot wellhead price for natural gas, and the
spot wellhead price that we\'re \-- that we use on this chart is
actually from the \-- it\'s the composite spot wellhead price compiled
by [Natural Gas Week]{.underline}, which is a little more timely than
our survey.

In October it hit \$5.12 per thousand cubic feet, which I believe is a
record for a monthly number, at least in nominal terms, and since June
the spot price has been above \$4 and over \$5 for much of September and
October.

There have been times in the past when the spot price was much higher
than that for a few days, but it has never really been this high that I
can remember for such a fairly long period of time.

And why? Demand has been strong, production gains weak, injections
sluggish, and storage low.

We also had a lot of hot weather over in the summer in a lot of the
electric utility gas consuming regions, in the Southern Plains, Texas,
and in California. So replenishment of storage wasn\'t as fast.

In addition, these high oil prices have allowed the gas price to rise,
and so throughout the whole summer and into the fall, there was a real
concern about the availability of gas this winter, especially if the
weather in the Midwest is normal, and normal in our case means colder
than it was last year or the last several winters.

So in our forecast we see prices considerably higher than last winter,
about at least at the wellhead over 80 percent, and for residential
users, that would translate into about a 30 percent increase in prices.

And prolonged severe weather would really test our supply system, and
prices would respond accordingly, but one thing I do want to point out.
Since this forecast was first presented about a month ago, spot prices
have dropped considerably due to the mild weather that we had over the
last three weeks, although just yesterday and today before I came here,
I stopped off at the Reuters\' wire. They\'ve shot back up somewhat. So,
I mean, the market is still jittery. They\'re still below \$5, but the
December price is almost \$5.

November is a very critical month. It\'s really the last month of the
injection season, and if the rest of November turns out to be cold, we
could see prices going over \$5 again. If not, since the market is so
high, we could see it coming down.

And as I keep pointing out, it\'s becoming my mantra that our forecast
assumes normal weather. In other words, it\'s like a 30-year average,
and since the last several winters have been warmer than normal, we
project increased demand this winter.

Now, we started out the injection season April 1 right above the
five-year average range for working gas storage, and as I said earlier,
during the spring and the summer, the injection rate was below average,
and by the beginning of the heating season on October 1, working gas
storage was nine percent below the five-year average.

Injections have picked up over the last few weeks. The last AGA numbers
that came out Wednesday were encouraging, but nevertheless, supplies
will be tight.

Some pressure relief may be coming from Canada through the recently
opened Alliance pipeline.

These are our natural gas heating cost estimates for an average Midwest
household, and as you can see, it will be about we estimate on average
it will cost \$240 more this winter to heat their home.

In September, residential heating prices were 44 cents a gallon above
the September 1999 price, and based on our SHOP survey \-- that\'s the
state heating oil propane program survey which comes out every week.
That starts in October \-- the price is almost 50 cents higher than it
was a year ago for October. In our winter forecast we see prices about
20 cents per gallon above last winter.

If our declining crude oil price path holds and the SPR swap limit
works, then we will see that the prices for the rest of the winter will
not increase that much.

Nevertheless, stocks are low. Distillate stocks are real low in the
Northeast, and extreme weather like what we had last late January and
early February could send prices at the high band.

Distillate stocks are low. High sulfur distillate stocks, heating oil,
are low, and East Coast high sulfur distillate stocks are also low.

We estimate that the swap and financial incentives for higher crude runs
and distillate yields should help build stocks by 11 million barrels
from the end of September through November. This is faster than a normal
build, which is about five million barrels, but will not quite bring us
up to the low end of the average range.

And this is our estimate for an average Northeast household heating with
oil, assuming normal weather.

One thing I want to point out is that last winter the weather in the
Northeast was actually for the whole winter warmer than normal if you
consider all six months. It was that one period when it was \-- the last
two weeks of January to the first two weeks of February \-- that it was
extremely cold.

Okay. Well, propane prices, which are tied to oil and gas prices, are
also projected to go up, although stocks of propane are actually quite
strong. There was some concern earlier that the drying season, the crop
drying season in the Midwest, might deplete stocks if it was we there,
but the latest data that I\'ve seen is that prices have fallen some. I
guess that it wasn\'t so wet in the Midwest there then.

And these are propane projections for an average residential household.

Finally, electricity, and the electricity growth was strong this year,
driven by a booming economy, in 2001 a little slower growth, and I want
to point out that for the first time in many years that we expect an
increase in residential prices year over year even though it\'s a slight
one due to the higher fossil fuel costs to generate electric utilities,
and that\'s it.

Any questions?

CHAIRPERSON GOTWAY CRAWFORD: Comments?

MR. GAMSON: Yes.

PARTICIPANT: Since I\'m congressional liaison for this type of material,
particularly with EIA and dealing with oil and gas issues, I often get
the questions about how your data compares to, say, API data, to
American Gas Association data; how you arrive at your assumption on your
cases, et cetera. So it\'s helpful to me to understand that type of
thing because I have to answer the staffers\' questions because they get
inquiries from their constituents, et cetera, and also to put together
talking points for their Congressman when they go out into their
districts, for example, and questions are raised by their constituents
out there.

MR. GAMSON: How our data differs from AGA and API? This is probably
something that I can\'t answer that easily over here. You know, they\'re
different surveys. You want to know whether ours is generally higher or
lower?

I mean, I\'m not quite sure of your question.

PARTICIPANT: Well, if they were going to a Web site that AGA would have,
for example, and there\'s data on there, how does the EIA data usually
compare to that? Is it comparable? How are your resources compared to,
say, AGA\'s? How do you arrive at your assumptions where your data comes
from?

MR. GAMSON: Well, we have surveys, and they have surveys, too. I can
tell you in the case of, let\'s say, natural gas storage, I mean, we
look at that data a lot. Our survey is probably a little more
comprehensive, and so our numbers are a little higher, but we look at,
you know, the movements, the changes.

If they say there\'s 2.7 million TCF in working gas storage, that would
translate to us \-- I mean, I\'m just giving you an example \-- as maybe
about 2.8. I\'m not \-- just because of \-- you\'re probably asking the
wrong person, but that\'s \--

PARTICIPANT: All right. Well, maybe you can answer \--

MR. GAMSON: Or maybe you can come down and, you know, spend a little
more time and talk with us on this, but this \-- I mean, this is \--

PARTICIPANT: Okay. Well, how would that then compare? Say, for example,
do you have authority \-- where does your data come from? Do you have
authority to request from different companies that they report to you on
a periodic basis?

MR. GAMSON: Yes. Oh, yes. Oh, some of that, I believe, is mandated by
law that they \-- yes, that the \-- un-huh. Here\'s John Cook. I mean
some of ‑-

MR. PETTIS: Well, Phil, I think if you would come down to EIA we can do
that. As a matter of fact \--

PARTICIPANT: Okay.

MR. PETTIS: \-- we actually have some charts that plot our data against
AGA data, for example. So, you know, they collect data on a weekly
basis. We collect it on a monthly basis, but we actually have charts
that show those trends together.

So, I mean, we can certainly provide you that information if you \--

PARTICIPANT: All right. Well, I raised it here because I thought it
might be answerable here, but it\'s not a problem to go further into it.

Thank you.

CHAIRPERSON GOTWAY CRAWFORD: I think it\'s a rather complicated and
involved question that they probably need some time to prepare for. So
if we could visit them, they would probably be very able to answer your
questions.

PARTICIPANT: Thank you.

CHAIRPERSON GOTWAY CRAWFORD: Bill?

PARTICIPANT: Well, in terms of the gas storage, they have a report
comparing AGA to your own on the Web. So you could download that, and
they do a rather detailed analysis of the two series.

CHAIRPERSON GOTWAY CRAWFORD: Calvin?

DR. KENT: Yeah. I\'d like to ask just one question. Of the amount that
is being released from the strategic petroleum reserve, that\'s 30
million, and ten million of it only is to be processed here in the U.S.,
and I assume the rest of it is being shipped abroad because of a lack of
refining capacity for the type of oil, or is it the quality of oil?

MR. GAMSON: No. I mean, it\'s not \-- my understanding is that that
other 20 million, that ten million is an additional ten million, and the
20 million would mean that we wouldn\'t have to import, be backed out of
the market. It would still be refined here.

MR. PETTIS: Cal, I mean, what they\'re saying is the SPR release was 30
million barrels.

DR. KENT: Yeah.

MR. PETTIS: It\'s put on the world market. Okay? The U.S. doesn\'t get
the net benefit of that entire 30 million barrels.

DR. KENT: Yeah.

MR. PETTIS: Just as if somebody else put 30 million barrels on the world
oil market we wouldn\'t get the full benefit of it.

DR. KENT: Right.

MR. PETTIS: So it\'s an assumption that, you know, the net increase in
supplies to the U.S. would not be the full 30 million barrels.

DR. KENT: But we\'re looking at the 20 million as being displacement of
otherwise imported and that it all is going to be refined here; is that
correct?

MR. GAMSON: No, not all. Some of it would be refined in other parts of
the Atlantic basin.

DR. KENT: Okay. What I\'m trying to get at is how we then get the price
depressing effects that you talked about because then what is refined
elsewhere will then be reimported back to the United States; is that
correct?

MR. GAMSON: Some of that will be, yes.

DR. KENT: Some of it will be?

MR. GAMSON: Un-huh.

DR. KENT: Okay, but it is also possible, depending on what the weather
is elsewhere, that some of that may go where the higher prices are \--

MR. GAMSON: That\'s true.

DR. KENT: \-- and never come back.

MR. GAMSON: Yes. Western Europe, which is the other big heating oil
consumer \--

DR. KENT: Area.

MR. GAMSON: \-- is cold, yes, that\'s true.

DR. KENT: So your projection then is assuming that what about weather
elsewhere?

MR. GAMSON: Normal.

DR. KENT: Normal. Okay.

CHAIRPERSON GOTWAY CRAWFORD: Thank you very much.

We have one more quick item about the EIA Research Fellow Program, and
Nancy Kirkendall is going to tell us about that, and she has some
handouts for you that you might be interested in.

MS. KIRKENDALL: We finally got a research fellow program arranged with
ASA. This will be the first year that we\'ve tried it, and this
committee had expressed some interest in doing things like this. So I
thought this is not the final brochure. ASA is currently printing it. It
should show up in various mailboxes fairly soon, but I particularly
wanted you guys to know about it.

You know a lot about what we do. You may know people in your
organization who might be interested in applying for this. It\'s a small
program to start with. It will probably support somebody to come here
for about three months.

There are just kind of little lists of the kinds of projects that
somebody might work on, but if there\'s something that you know that
would be of interest to us, then that would be of interest to us to look
at your proposal.

Now, these are just suggestions for things that we could think of and we
could be interested in all sorts of things.

The deadline for submissions is January 15th. It\'s kind of a mixture of
statistical things and economic things because that\'s the kind of stuff
we do.

Any questions about it?

DR. COWING: What are the qualifications, Ph.D. in hand or A.B.D. or \--

MS. KIRKENDALL: This is not a lot of money. So it\'s probably, you know,
A.B.D., a good person who wants to come and spend some time with us
playing with some interesting problems.

ASA is going to do a review of the proposals we get. We\'re only going
to support one this year. We hope that this is going to be a program
that we continue into the future.

So, yeah, I would think some A.B.D.s might benefit from coming here for
a while on a little project. You know, this is a nice, little, well
defined project. Come and work with our data on some problems that are
interesting to us, too. So it\'s a win-win.

DR. HAMMITT: Sort of related to that, is it targeted for junior level
people or is it open to people of all \--

MS. KIRKENDALL: Oh, it\'s open to anybody. The limitation is the money.

PARTICIPANT: How much is the money?

MS. KIRKENDALL: We sent ASA \$50,000, and they take a piece for
administering the program. But, you know, that\'s why we thought it
would be for a summer, a three-month program, or if somebody wanted to
do it once a week if they were local during the year, you know, if they
had a day they could spend.

So we\'re flexible on how somebody wants to actually do the work.

DR. HAMMITT: Fifty thousand would go pretty far, wouldn\'t it?

MS. KIRKENDALL: I hope so. I hope we\'ll get something interesting.

DR. HAMMITT: But one possibility is this could be a good way to get
people going on a thesis topic.

MS. KIRKENDALL: Right.

DR. HAMMITT: They could come here and start doing something.

MS. KIRKENDALL: Yeah.

DR. HAMMITT: Doe sit say all of that in here?

MS. KIRKENDALL: It doesn\'t talk particularly about getting somebody
started on a thesis topic, but I would hope people would look at this as
an opportunity to do that.

CHAIRPERSON GOTWAY CRAWFORD: Thank you.

MS. KIRKENDALL: Okay.

CHAIRPERSON GOTWAY CRAWFORD: At this time I\'d like to invite public
comment on any of the proceedings either today or yesterday.

(No response.)

CHAIRPERSON GOTWAY CRAWFORD: If there are no further comments or
questions, then I\'ll adjourn the meeting, and we\'ll have lunch, which
I\'ve just heard has arrived.

(Whereupon, at 11:52 a.m., the meeting was concluded.)