AMERICAN STATISTICAL ASSOCIATION

\- - -

COMMITTEE ON ENERGY STATISTICS

\- - -

THURSDAY, APRIL 23, 1998

\- - -

The Committee met in the Clark Room in the Holiday Inn Capitol, 550 C
Street, S.W., Washington, D.C., at 9:00 a.m., Daniel A. Relles, Chair,
presiding.

[PRESENT]{.underline}:

DANIEL A. RELLES Chair

CHARLES W. BISCHOFF Member

CAROL A. GOTWAY CRAWFORD Member

PHILIP HANSER Member

CALVIN KENT Member

GRETA M. LJUNG Member

POLLY A. PHIPPS Member

SEYMOUR SUDMAN Member

ROY W. WHITMORE Member

DENNY ELLERMAN Guest

JAMES HAMMITT Guest

**I N D E X**

[Page]{.underline}

Introduction by Committee Chair 4

Opening Remarks by EIA Administrator 14

[Greenhouse Gases and Environmental Analysis]{.underline}:

An Introduction and Overview 25

Presenter: Jay Hakes (EIA)

Greenhouse Gas Emissions 33

Presenters: Arthur Andersen (EIA)

Arthur Rypinsky (EIA)

Analysis of the Carbon Stabilization Cases 74

in EIA\'s Service Report

Presenters: Andy Kydes (EIA)

Ron Earley (EIA)

Analysis of Alternative Carbon, SO2 and NO2 95

and Renewable Cases

Presenter: Alan Beamon (EIA)

Future EIA Analyses Related to the Kyoto 108

Agreement. Presenter: Jay Hakes (EIA)

Discussion: Philip Hanser (ASA Committee) 118

Calvin Kent (ASA Committee)

Questions from the Committee 134

[Modeling Electric Utilities]{.underline}:

Network Modeling at EIA: A Status Report 163

Presenter: Douglas Hale (EIA)

[INDEX]{.underline}: (continued)

[Page]{.underline}

Judging for 5th EIA Graphics Contest

Charles Bischoff (ASA Committee)

Calvin Kent (ASA Committee)

Greta Ljung (ASA Committee)

Questions from the Committee 175

[Web-Site Related Issues]{.underline}:

An Introduction to Web-Site Related Challenges 181

Presenter: Lynda Carlson (EIA)

EIA\'s Web-Site: A Demonstration 184

Presenters: Larry Pettis (EIA)

John Pearson (EIA)

Questions from the Committee 199

Measurement of EIA Customer Satisfaction 209

Presenter: Colleen Blessing (EIA)

Discussion: Bradley Skarpness (ASA Committee)

Questions from the Committee 230

Standardizing Data Definitions and Calculating 239

Common Response Rates

Presenter: Renee Miller (EIA)

Discussion: Carol Crawford (ASA Committee) 258

Questions from the Committee 268

Public Comment

**P R O C E E D I N G S**

Time: 9:09 a.m.

CHAIRMAN RELLES: Welcome to the ASA Committee on Energy Statistics
meeting, and thank you for coming.

I\'m supposed to begin with some official announcements. This meeting is
being held under the provision of the Federal Advisory Committee Act.
This is an ASA, not an EIA, committee which periodically provides advice
to EIA.

The meeting is open to the public. Public comments are welcome. Time
will be set aside for comments at the end of each session. Written
comments are welcome and can be sent to either ASA or EIA.

All attendees, including guests and EIA employees, should sign the
register. We are also asking for your E-mail address this year for the
first time.

In commenting, each participant is asked to speak into a microphone. The
transcriber will appreciate it. Also, committee members and speakers at
the head table need to speak clearly and into a microphone. These
microphones have an approximate range of one to two feet except for the
microphone that just died.

I ask all attendees to \-- Oops. Yes. All attendees should introduce
themselves, giving their name and affiliation. I\'d like to begin with
the head table.

MR. HAKES: Jay Hakes, Energy Information Administration.

MS. CARLSON: Lynda Carlson, Energy Information Administration.

MR. WEINIG: Bill Weinig, Energy Information Administration.

MR. HAMMITT: Jim Hammitt, Harvard University.

MR. KENT: Cal Kent, Marshall University.

MR. SUDMAN: Seymour Sudman, University of Illinois at Champaign-Urbana.

MR. ELLERMAN: Denny Ellerman, MIT.

MR. CRAWFORD: I\'m Carol Gotway Crawford with the Centers for Disease
Control.

MR. BISCHOFF: Chuck Bischoff, State University of New York at
Binghamton.

MS. LJUNG: Greta Ljung, Institute for Business and Home Safety, Boston.

MR. WHITMORE: Roy Whitmore with Research Triangle Institute.

MS. PHIPPS: Polly Phipps, Washington State Institute for Public Policy
Center.

MR. HANSER: Philip Hanser, Putnam, Hayes & Bartlett.

CHAIRMAN RELLES: And I\'m Dan Relles of the Rand Corporation.

Please, can we have the audience do the same.

MR. ANDERSEN: Art Andersen, Energy Information Administration.

MS. MILLER: Renee Miller, EIA.

MR. RYPINSKI: Arthur Rypinski, Energy Information Administration.

MR. STENNER: Bill Stenner, EIA.

MS. ELWELL: Carrie Elwell, EIA.

MR. WILMER: Dan Wilmer, EIA.

MS. SMITH: Sandy Smith, EIA.

CHAIRMAN RELLES: Further announcements: There will be a luncheon for the
committee and invited guests at 12:30 in the Lewis Room next door.

ASA and EIA judging for the fifth annual EIA graphics contest will begin
at 1:30 following lunch in that room, and the committee will resume our
meeting with a presentation by Doug Hale of EIA in this, the Clark Room.

Breakfast for the committee will be held tomorrow morning in the Lewis
Room next door beginning at eight o\'clock.

For your information, Dr. Lynda Carlson here at the head table is the
designated Federal official for the Advisory Committee. She may chair,
but must attend each meeting, and is authorized to adjourn the meeting
if she determines this to be in the public interest. I didn\'t write
that. She must approve all meetings of the Advisory Committee and every
agenda. She may designate a substitute in her absence.

Finally, I\'d like to ask committee members and guests to turn your name
tags vertically to comment or ask questions.

I would like to begin by welcoming guests and our new member, Polly
Phipps, Dr. Polly Phipps, who has introduced herself already, as has Jim
and Dr. Hammitt and Dr. Ellerman. I thank them both for coming, and we
much appreciate Polly\'s presence here, too.

We always ask the EIA Administrator to open our session, but I\'m going
to take the liberty of making some opening remarks myself, since I\'ve
just ascended to this throne, and my expectation is that I\'ll be doing
this for two years. So I wanted to try to set a long term agenda.

I guess I get to this agenda by asking three basic questions. The first
is who is the customer of these proceedings? To me the answer is pretty
clearly EIA. We\'re here to help you. We have other lives and other
careers of our own, but all of us, I think, through some form of
patriotism and respect, make the journey here, and we want to make a
difference. We want to \-- We know EIA has very important and monumental
problems it\'s dealing with. We think we can help, and we\'re here to
help

If a session goes by where that doesn\'t occur, i.e., we feel like
we\'ve just kissed one of our siblings, I think we all feel badly about
it. So that\'s definitely the first priority here.

What are we offering? Now that we\'ve identified you as a customer, I
think here\'s what we\'re offering. If you look at the composition of
the membership, most of us are scientists or all of us are scientists of
various types.

We have statisticians who have expertise in survey methods, measurement,
modeling, forecasting. We have modeling and simulation people. We have
economists. We have political scientists. We have people who understand
Washington institutions and institutional behavior.

So we have some basic skills. We\'re offering our general availability
to EIA people. We\'re offering good contacts with academia so that we
can help you identify and help promote access to new technology, new
ideas. We\'re offering a sounding board. Sometimes it just helps to \--
because you\'ve got to make a presentation to the committee, to reach
closure on an item.

Finally, we\'re offering our undivided attention to you two days twice a
year. For many people on the committee, it is two days twice a year, and
that turns out to be their only connection with EIA, and on an
individual basis one might hope that we would be more available, but
that tends not to occur.

So in the two days twice a year, I guess the next question is how can we
make those two days most effective? It seems to me that what we need to
do is identify sort of a small set of themes, and I\'d like to suggest
two.

One is to help EIA develop its core scientific capabilities. Survey
measurement is an important core scientific capability. Modeling and
simulation is an important core scientific capability. We\'d like that
to be a fairly persistent theme through these sessions, and to tack on
the thing that makes it even harder still. That\'s an area where the
money is disappearing to develop these kinds of capabilities.

The second theme I think I\'d like to pursue is help to enhance EIA\'s
scientific reputation as it grapples with the important public policies
issues of the day. There are two where EIA is right in the forefront,
environmental damage or climate change or global warming, and
electricity deregulation where EIA is being counted on to be measuring
and providing signs of what\'s happening in this new world we\'re all
entering in.

Those of us who live in California are especially aware of that. It\'s
very confusing. Anyway, focusing on issues of that nature will also, I
think, help drive the core issues that we need to focus on and work
harder towards.

So as Chair, what I\'m going to try to do is make sure that talks get
scheduled and that people are well aware of how all the talks and all
the discussions sort of have to fit into these larger pictures. I think
that is pretty easy to do with the talks we\'ve got today.

There\'s \-- You know, some talks are going to be introduction talks to
get us into an area, as Art Andersen\'s and Arthur Rypinski\'s talks
are. Some of them will be sort of steps in a process, as the service
reports are. Some of the talks will be just bringing us up to date, as
Doug Hale\'s talk this afternoon will be. We had a big introductory
session on electricity deregulation last time. Doug will kind of tell us
where we are at this point.

I hope that with these themes, we\'ll be able to view talks more as work
in progress than as finished products. So we won\'t get comments of the
sort, well, you should have divided by the spirit of n-1 instead of
spirit of n, but we\'ll be getting comments about sort of, you know,
where are you going with this and how does it fit into the overall
agenda.

Actually, that will make my life a lot easier, too, because if I ever
get to a point where there\'s no real discussion, I could just stand up
and say where are you going with this, and how does it fit into the
overall agenda. That\'s sort of \-- although if I can\'t conceive of how
it\'s going to fit into the overall agenda, I might say where are you
going with this, and why shouldn\'t I pull your fingernails out or
something of that nature.

So with that, I think I want to set the stage for this meeting. This
morning will launch us into the environmental global warming issues.
We\'ll get an introduction to their problems. We\'ll see what the
modeling and analysis reports are, and we\'ll see what directions EIA
plans to take in this area.

This afternoon we\'ll get an update on electricity, but then we\'ll also
be launching into more of the basic science-y kinds of things, first a
discussion of the Web which is an extremely effective tool for
dissemination of information, which is part of the science process and,
of course, for information gathering; and then survey data operations
pretty much dominates our day tomorrow.

I\'d like to recognize as the signs of the high level of energy that
I\'ve seen on both sides of the fence, Bill Weinig especially, of EIA,
who has worked tirelessly in putting this program together and in
hounding me to make sure I do my part of the job.

The 100 percent response rate I got yesterday in visiting EIA to try to
get a sense of what the important issues were to them \-- 100 percent,
and 100 percent response rate I got from the committee, too, when I
approached them, asking them to be discussants.

So I think, clearly, we\'re all trying to make this work and, when it\'s
not, and you feel like you\'ve kissed a sibling, it will be my fault,
and I\'ll want to talk to you about it. I\'m really serious about that.
It\'s not a gratuitous acceptance of blame.

I think with this much energy, we ought to be meeting together on things
that are of common importance to us, and we will succeed. Thank you.

Jay. I\'d like to introduce the EIA Administrator, who will be telling
us what\'s been happening at EIA, introducing the rest of the morning
session, and coming back later to talk about the future agenda in
climate change.

MR. HAKES: Thank you, Dan. I would like to endorse Dan\'s effort to take
a good analytical look at the work of the committee. The committee has
been extremely valuable to us over the years, but that doesn\'t mean
that it couldn\'t be more valuable, and we had some very good
discussions yesterday, and I think this is a good omen for future
developments.

It\'s usually my role at this point to sort of give you an update on
things that have been happening at EIA, and one of the things that I
traditionally do is give sort of what I call the resource outlook, as
well as some other newsy items.

I would point out in this regard that our fiscal 1999 Congressional
budget request was \$7.5 million, and this request is \$3.7 million
higher than the fiscal year 1998 request of 66.8. This is, however,
still below the 1995 level of 84.5.

Now I think this is a significant development, because the
Administration has in recent years been dropping us in the request a
little bit each year, and this year the Administration is requesting an
increase. That \-- It\'s not a huge increase, but at least it\'s a
reversal of the trend, and I think in that respect is welcome.

Of the increase in funding in fiscal year 1999, the request has \$2.5
million to be directed towards carbon emissions issues. The remaining
\$1.2 million would restore funds eliminated in fiscal year 1997 for
energy efficiency and renewables data collection and analysis.

There\'s some money in there to fund Year 2000 compliance projects that
we\'re working on, as well as some other items.

We don\'t have a clear signal yet from the Congress what they intend to
do with this request for additional money. There has ben some comment
that anything that has climate change attached to it is automatically
disqualified, but I don\'t think that that will be the approach for EIA,
and I think that the committee will certainly look at the funds they
have available and see if they\'re able to fund this increase.

The other issue for us besides money, of course, is people. After many
retirements around the turn of the year, EIA now has about 380 Federal
employees, which is down from 460 in 1995. The Department\'s current
plan calls for EIA to reach a target of about 340.

In order to reach these mandated levels, EIA has been unable to
replacement departing employees for more than two and a half years,
creating shortages in certain skill categories. For this reason, EIA has
strongly protested a continuation of this hiring freeze.

The Department of Energy has indicated that it will make some
adjustments in the targets, but the levels of these adjustments have not
been made known to us at this point, although we are expecting some
additional notification in the next week or two.

If the relief granted is insufficient, the Congress has indicated it may
want to intervene to ensure that EIA has the number of employees it
needs to do its job. I think the critical issue right now for us is to
do some hiring in the not too distant future.

There are two reasons for that. One is we do have a retirement bubble
coming up in a few years, and second, it\'s harder to hire out there
today. The job market has changed in many fundamental respects in the
last year or two. People with the kind of skills we\'re looking for are
in demand, and we \-- I don\'t think we benefit in any way by delaying
activities in this area.

I\'ve said to several of you individually that now is a good time to be
in the energy field. Those of us who have worked in it over the years
know that there are times when it\'s a front page issue, and there\'s
times when it\'s a non-issue. Energy work can be kind of dull in some
years, because people aren\'t asking many questions or paying much
attention. Maybe they should be, but they\'re not.

We are in a period now where energy issues are getting more attention,
and it makes our life more interesting and more challenging.

There are three areas that certainly stand out. One of them is an old
standby, volatility in gasoline prices. Gasoline prices dropped almost a
quarter from their August 1997 peak and, while the media usually does
not pay as much attention to prices going down as they do to prices
going up, that was a big enough shift to attract considerable attention.

So EIA has been called on to explain the causes of this. One of the
things I thought was particularly good was that Dave Costello, who is
the project manager for the short term energy outlook, was on the NBC
Nightly News discussing the drop in gasoline prices.

In that particular segment that night, he was the only energy expert
that was interviewed, and I timed it, because one of my common sayings
is when you really have something important to say to somebody who is
really important, you\'re probably only going to have about 30 seconds
to say it, and Dave had 20, and he did a great job and explained it all
in 20 seconds.

Also in the gasoline price situation, we released our outlook for the
summer gasoline prices back just about two weeks ago, I guess, and
Secretary Pena decided that he would like to release it. We usually
release it ourselves, but it was built into a series of announcements
that he was making that day.

Of all the announcements he made, that was the one that got the most
attention, and he was on NBC News that night with film from that press
conference, was interviewed all over the place, and the story was pretty
widely covered. Consequently, everybody in the country now knows that
their gasoline prices will be exactly \$1.10 throughout the summer.

Over the years, I think EIA has sort of earned its place at the energy
table, to a large extent with our work on gasoline prices. One person
who is strategically placed within the Administration told me that they
love to talk to EIA before there was a meeting in the Administration to
discuss what was happening to gasoline prices, because if people
remember the meeting a week or two later, they would know that that
person was usually right on point and knew what they were talking about.
She attributed that fact that she kept in touch with EIA.

So I think we\'ve really earned a good reputation of understanding that
market, and are looked to for expert advice.

The other two issues, I think, I won\'t comment on as much, because
we\'ve been discussing them in recent meetings. In electric
restructuring we continue to play a very vital role, providing basic
information to people who are interested in the issue.

We have a nice brochure on that subject that is available to large
numbers of people. We have nice updates on our Web site, and we do some
pretty intricate modeling work to try to look at the various impacts,
and we are in the process of updating some of those studies.

So I think we will continue to play a strong role here. I would just say
that when our study came out on regional pricing packs, the first day it
came out there were some people who were critical of it. One of the
statements that was made was that California illustrated that you could
cover the stranded costs and still provide full cost reductions, and the
two were not in conflict.

A lot of things have happened in California in the last few months, one
of which was the withdrawal of Enron from the residential market this
week, partly because they said there was a conflict between covering
stranded costs and providing the kind of reductions that people were
looking for.

So I think our work has stood the test of time well, and I think this is
ultimately what we\'re looking for, is when we do a piece of work that,
when we look back at it some years later, it stood the test of time.

Carbon emissions, of course, is the big gorilla right now, and I\'ll say
more about that later.

A couple of news items: You probably read in the paper that Secretary
Pena is leaving as the Secretary of Energy. He showed up to announce
that with his two daughters and young son, and I think understandably
has a desire to spend more time with his family.

His resignation takes effect at the end of June. So he is still here in
an active capacity. In fact, he\'s scheduled a meeting I\'m supposed to
attend tonight during the dinner hour. So I may be a little late
arriving at dinner tonight.

Secretary Pena, I think, has helped raise the visibility of EIA. He
frequently mentions EIA in his speeches and press conferences, and we
certainly will miss him. We have no inside information on what\'s
happening in terms of his replacement.

Another thing that I think \-- there\'s so many things, and I don\'t
want to sort of go down a long list, but I have one other news item that
I would mention to you.

There is a bill that is circulating around the Congress that would
change the structure of the Federal statistical agencies, and it has
some backing in the Congress. It has, basically, two parts. One part
would set up a structure that would create the possibility, and one
might even say the probability, that the Bureau of the Census, the
Bureau of Labor Statistics, and the Bureau of Economic Analysis would be
unified into one statistical agency.

This is an idea that has been circulating around town for sometime. It
would also create a permanent oversight for that statistical system, and
it would create a mechanism that would make it relatively easy to move
other statistical agencies into that agency.

This is \-- could potentially create something that would be more akin
to Statistics Canada than the decentralized system that we have right
now in the United States.

There was a second part of the bill that, regardless of whether there
was this type of reorganization, would provide for more common
definitions of confidentiality among the statistical agencies and would
permit more sharing among the agencies than currently is possible with
different standards for confidentiality.

The Administration has testified on that bill, and has clearly indicated
that it supports the confidentiality aspects of the bill, but opposes
the reorganization aspects of the bill. Franklin Raines, who is the
Director of the Office of Management and Budget, but who is also leaving
the Federal government \-- I think his resignation takes effect in May
\-- has been a strong supporter of statistical agencies.

He has used statistical data throughout his life, and he feels that
there\'s a lot that the statistical agencies should be doing to work
more closely together and avoid duplication, be more efficient,
modernize systems, but that this can be done without a reorganization
into one centralized agency. So he has personally testified to that
effect.

He also sent a letter to all of the Appropriations Committees asking
them to take particular care that they were funding adequately the
statistical agencies. So if that is helpful, I certainly don\'t believe
that there will be any policy change on this bill with his leaving, but
he certainly has been a good supporter of the statistical system.

I think I would be glad to pause here and if there are any questions,
and there\'s sort of a whole range of things we could talk about; but if
there are sort of general issues that are not covered elsewhere in the
agenda, things that you\'re curious about, I\'d be glad to try to
respond.

Well, if there are no questions, I\'ll take off my general introduction
hat and put on my greenhouse gas hat.

Carbon emissions is a big issue for us and for the country and for the
world. How big is it? It\'s really, really big, and it\'s going to be
with us for sometime. It\'s not something that is going to be here this
year and go away.

We do not in any respect want to be viewed as the experts of the impact
of carbon on climate. There are plenty of climatologists and others who
can join that debate. We read that literature carefully. We try to be
aware of it, but our expertise is in the area of energy systems that
produce most of the emissions resulting from human activities.

There has been some debate about whether this is an issue better handled
by environmental agencies or energy agencies. It certainly seems to me
that this is primarily an energy issue in terms of where previous
knowledge lends the most understanding of the current problem.

One of the roles of EIA that\'s been particularly valuable has been the
setting of widely accepted baselines. In other words, in a normal
situation without policy change, how much are carbon emissions likely to
rise in the United States or how much are they likely to rise in the
world?

It\'s, I think, been so widely accepted that sometimes it\'s been
overlooked \-- the importance of this. It\'s been sort of what I call an
island of consensus in a very controversial sea. People are arguing a
lot about carbon emissions, but no one has really taken any real heavy
shots at the baseline.

There was \-- some of you saw this this morning. Lynda was showing it to
people \-- some question whether we were releasing things at various
points in time to benefit one point in the argument. Senator Hegel, who
is the co-author of one of the Hegel-Byrd Amendment that requires that
the Kyoto treaty have participation by the developing nations, said that
it was \-- well, let me just say for those of you who don\'t know,
yesterday we had released our international energy outlook, which looks
to the year 2020 for energy supply, energy demand, and carbon emissions
growth.

Senator Hegel said it is poetic that the Administration slipped the news
out so quietly on Earth Day. It is the first public acknowledgment I\'ve
seen. You can\'t bottle this stuff forever, he said.

Well, I would point out that we were releasing our annual energy \-- or
international energy outlook in the way that we release it every year.
We had a press conference. There were reporters there. Reporters covered
the issue.

Reuters and AP filed several stories. Those stories went out to all the
news organizations. We got two calls from NBC News. It was mentioned
last night on NBC News. So I\'m not sure that we were bottling this up.

There\'s an interesting response after this. A House GOPA noted that the
agency that released yesterday\'s assessment is an independent
statistical branch of the Administration that does not seem to kowtow to
the White House political agenda. I don\'t know if I can stand too much
more praise like that.

After my last testimony on this issue, I was praised for my candor, but
I think \-- My view on this is that in order to achieve the removal of
the Federal budget deficit, it required some analytic system that the
parties could agree to, and this ended up being the Congressional Budget
Office. So there is a way of measuring the extent the budget is out of
equilibrium or to the extent that it\'s balanced.

If there is going to be at some point in the future a coming together
sufficient to create a policy in this area, there will have to be some
analytic system that people can agree on. I can\'t conceive of a policy
of good, bad, indifferent being made without that foundation.

I think, if we\'re not that foundation, I don\'t see any alternative. So
I think \-- and I think it\'s actually likely that we will be that
foundation, but I think it will be an interesting road by which we get
to that point.

Now baselines are one issue, but a trickier issue is to do the \"what
if\" games. If you change the policy on environment, what is the likely
impact, and I think that is a dicier question, particularly since
greenhouse gases is a relatively new issue, and it requires a lot of
innovative thinking.

I think you\'ll see that we have a lot of people at EIA who are capable
of doing that type of thinking. People will say, well, this is like
sulphur emissions. Well, in some ways it is; in some ways, it isn\'t.

People say, well, this is like the oil embargo of the 1970s. Well, in
some ways it is; in some ways, it isn\'t. This is a new issue that has
its own sort of aspects, and if you like an intellectual challenge, it
certainly is a good topic.

One way of looking at it is back in 1973 when we went through the oil
embargo and energy was a crisis on the cover of Time Magazine four or
five times, that was a new issue; because the country, although there
had been some interruptions in supply, the country had never seen that
kind of \-- the impacts of import dependence and price controls. There
were all sorts of things going on that create chaos that people didn\'t
understand.

One issue that I sometimes ask myself is: If EIA had its current level
of expertise and existed as an organization in 1973 when this
controversial battle was raging, could we have made a difference;
because I think there are some similarities in the current environment,
and I think there is some potential that we can make a difference, and
policies based on good data and good analysis are better than those that
are not.

We will need all the expertise we can get, and we look forward to your
help so that we can answer the questions that policy makers ask. I think
also \-- I think we\'re willing to define a somewhat broader role for
ourselves, and that is to contribute to public understanding of this
issue.

Because of its newness, I think there\'s no reason to expect that the
public should have a detailed understanding of it, and when we work with
a reporter or someone to explain a complicated issue, I think that does
filter out to the broader public, and again is a prerequisite for good
policy making in this area.

Along those lines, we have been active over the last year or so in
developing brochures. I don\'t have them with me today. I usually have
them in my pocket, but we have them on the annual energy outlook. We
have one coming out on the international energy outlook. We have a nice
one on electric restructuring. We have some basic brochures on
greenhouse gases.

These brochures, I think, are things that you can put in front of a busy
policy maker or you can put in front of a school teacher. You can get it
down off the Web. I think, more than the thick publication, it can help
people grasp the issues. So we\'re approaching it across the full range.

I would just say you\'re going to hear this morning from Art Andersen
and Art Rypinski, who do a lot of our greenhouse gas work, ranging from
the voluntary reporting program to the annual inventory to a whole range
of things, and have been \-- are widely consulted throughout the
government and throughout the country and the world for their expertise
on this issue.

We have Andy Kydes and Ron Earley who have been involved in inter-agency
and efforts within the Department to look at this issue, often from an
economic point of view. Then we have Alan Beamon, who has done a lot of
analytic work over the years on the electric industry, has considerable
expertise in this area.

So I\'m going to turn it over to them at this point.

MR. ANDERSEN: Good morning. We will try to keep our discussion to the
time that\'s allotted here. Dan yesterday came by and asked some
questions about what kind of enrichment might come out of this exercise,
and I said I wasn\'t sure, but from our perspective the work that we\'ve
been engaged in as a result of the Energy Policy Act has caused us to
worry about a variety of data issues with respect to the emissions
inventory program that we have now reported on for four or five years,
and we were driven to try to develop a data collection system in an area
where no work has been done before.

We will try to in the next few minutes give you a sense of where we are
with regard to these two dimensions, and see where the discussion goes
from there.

Arthur will be talking about the inventory. Then I\'ll try and say
something about the voluntary program.

MR. RYPINSKI: I\'d like to spend a few minutes talking to you about U.S.
emissions and greenhouse gases and the results of the annual report on
emissions greenhouse gases that the EIA does.

As you doubtless have noticed, greenhouse gas issues have become more
salient and much more in the public eye since the negotiation of the
Kyoto Protocol.

What\'s perhaps not so apparent is that the emissions inventories,
defined as estimates of the admissions of national governments in this
case, are central both to the 1992 framework convention and will be of
increasing importance if the United States elects to implement the Kyoto
Protocol.

People often complain about the framework convention. Well, it doesn\'t
require anybody to do anything, and that\'s not exactly true. It turns
out that the framework convention, which was signed by President Bush in
Rio de Janeiro in 1992, requires every signatory to prepare an emissions
inventory of their emissions of greenhouse gases and to publish that and
provide it to the framework convention secretariat, and the United
States and other signatories have been doing that.

The Energy Policy Act of 1992 requires the EIA to produce an inventory
of U.S. emissions of greenhouse gases with annual updates, and we\'ve
been doing that now since 1993. We\'ve produced five such reports.
You\'ll find stacks of them on the \-- now in its second printing this
year \-- on the back table.

In preparing that inventory, we worked closely with our counterparts at
the Environmental Protection Agency, and we found that over the past
couple of years EIA advice is being increasingly sought on energy
related greenhouse gas issues.

On Monday, my counterpart at the EPA asked me to sit in on a meeting
with the framework committee commission secretariat review of the U.S.
submission. I think it would be fair to say that, when the Commission
secretariat started asking hard questions about carbon dioxide and
energy data, they were quite grateful that I was present to answer them.

So the first thing to know about U.S. emissions is that, when you weight
them by what\'s called global warming potential \-- and I won\'t chase
that concept right now, but I can answer a question later, if you like
\-- most of it in the United States is energy related carbon dioxide, 84
percent. Eighty-five percent of the carbon is carbon dioxide. The other
one percent of carbon dioxide is mostly \-- is industrial process
emissions, mostly cement, and then about ten percent methane, two
percent nitrous oxide, two percent other gases.

To give you a flavor of some of the uncertainties, which is an actively
debated question, I\'ll let my colleagues who collect energy statistics
discuss the uncertainty of the carbon dioxide estimates; but in this
coming year we\'re probably going to double our estimate of nitrous
oxide emissions, due to newly available scientific data. So that will go
from two percent to four percent. So that will give you a certain flavor
for the issue of some of the other gases.

Next slide, please. This is a graphic showing carbon dioxide emissions
by sector. I show \-- In this slide, we show electric utilities both as
a separate grouping and also the electric utility emissions are prorated
into the total emissions of the other sector.

We frequently call this internally the layer cake graph, because there
doesn\'t seem to be much going on in it. It\'s one of the least
interesting statistical graphics you can produce, and it was not an
entry in this year\'s annual graphics contest. However, the impression
you get is, in fact, correct, that there isn\'t a whole lot going on.

Emissions are rising slowly, and you have to do indexes and look at
fairly tiny fluctuations to find interesting things.

I will say that there was a period when the United States was able to
stabilize carbon emissions. It was during the late seventies and 1980s
when, as those of you \-- some of you may recall that oil prices
quadrupled during that period, and the price of some of the other fuels
went up a lot, but one of the unintended side effects was a period of
actually declining carbon emissions in the United States.

That was also the years when the nuclear plants finally came out from
their hiatus and started actually coming on line.

Next slide, please. This shows some \-- a few collection of interesting
indexes of greenhouse gas and carbon dioxide emissions. It shows \-- The
blue line is \-- Well, the bright orange line on top is greenhouse gas
emissions per dollar of GNP, and that\'s something that has shown a long
term secular decline.

The first year in which \-- in recent times in which those numbers have
gone up was, in fact, 1996. Another area where there\'s been a long run
secular decline is carbon dioxide emissions per kilowatt hour of
electricity generated.

Once again, 1996 was a turn in that long term decline. That\'s driven by
the rise of nuclear followed by the rise in nuclear gas \-- nuclear gas?
Right \-- natural gas. Do I know what I\'m doing? Clearly, not. Also, it
was helped along, because the 1990s were much wetter, and we had better
hydro years in general than we did in the 1980s.

1995 was really the turn in that market. CO~2~ emissions per quad of end
use energy: We find that that\'s a flatter trend, and then emissions per
capita are rising, though extremely slowly. Of course, you turn back to
emissions. It turns out that both GNP and population are rising and, in
fact, total emissions are rising in a certain inexorable fashion at
about two percent, two or three percent a year after you strip out
weather related factors.

Next slide, please. We\'ll turn to some of the other gases where the
trends are a little less interesting. This is methane broken out by
source. Energy related is methane emissions from natural gas systems and
from coal mines.

The coal mine methane emissions are extremely unevenly distributed
across the population of coal mines. It turns out that a rather small
number of mines account for a big chunk of the coal mine methane
emissions.

Landfills is a complicated story which I can\'t begin to tell you about,
but it\'s the \-- The trend is a very slow decline.

Agriculture is a complicated story in which, to estimate emissions, we
seek the exact characteristics of the national average American cow.

Next slide, please. Nitrous oxide emissions have a number of interesting
stories. The energy related nitrous oxide emissions and that sharp
growth during the 1980s is essentially a function of the spread of
catalytic converters in automobiles.

It turns out that catalytic converters, when they \-- as we now know,
when they\'re cold start and before they get up to operating temperature
convert NO~x~ into nitrous oxide, and then when they hit operating
temperature, they convert NO~x~ into molecular nitrogen, and the
population of catalytic converter equipped cars, like many other things,
filters into the capital stock slowly.

So in 1980 at the beginning of this graph, most cars on the road didn\'t
have catalytic converters. Catalytic converters were first introduced in
the 1976 model year, and then by 1990 almost all cars had catalytic
converters, and we started to see the side effects of the most recent
catalytic converters produce a little less nitrous oxide than the older
ones.

So for the 1990s the trend flattens out, because the new cars produce
less nitrous oxide. The new catalytic converters produce less nitrous
oxide than the old catalytic converters. So that\'s why the trend
flattens.

Agriculture is a very uncertain number. It\'s driven essentially by
nitrogen fertilizer. The drop you see in the last two years is either a
decline in \-- driven by a decline in fertilizer use because of the
agricultural reforms or it\'s a statistical artifact from not very good
recent fertilizer data, and I hope this year to have a better picture of
how that\'s working.

Industrial nitrous oxide emissions are extremely concentrated. They\'re
the output of a small number of plants that make adipic acid and nitrous
acid.

Next slide, please. Here we have U.S. emissions of the other greenhouse
gases, the so called six gases. For those of you who are Kyoto junkies,
you\'ll recall that at the last moment the negotiators at Kyoto agreed
to go from three gases, carbon dioxide, methane and nitrous oxide, to
six gases, the last three being hydrofluorocarbons, perfluorocarbons,
and sulphur hexofloride.

It turns out that the six gases are really more like 20, because there
are many hydrofluorocarbons. There are many perfluorocarbons, but
there\'s only one SF6.

You\'ll notice the stunning growth right at the end of HFCs, and
particularly HFC-134a. HFC-134a is not exactly a household word, but
it\'s the gas that replaced freon in automobile air conditioners after
CFCs were phased out, and the rapidly rising emissions you see are
because every air conditioned 1993 model car and subsequently has a
couple of pounds of HFC-134a in it, and they leak, not very much, but
there are 15 million cars sold every year, and most of them are air
conditioned now.

PFCs are driven by aluminum smelting. There\'s an excellent chance that
that\'s an easily controllable source. SF6, HFC-23, which is the top
today by-product of the manufacture of HFC-22, which is another commonly
used refrigerant, but the big story here is the inexorable growth of
HFC-134a which will be carrying on for many years to come.

Next slide, please. A few \-- Some issues for the future: There is
increasing interest in how do we measure the uncertainty in our
estimates of greenhouse gases. The other gases, the noncarbon gases, are
getting increasing attention, because (a) they\'re low cost in reduction
opportunities, and (b) the state of our knowledge is not very good; and
(c) the Kyoto accord has some very intricate language on forestry and
land use, which I won\'t attempt to discuss in the time allotted to me.

Suffice it to say that it\'s complicated. The numbers are big, and the
data quality is extremely poor, and it\'s something that the United
States government as a corporate body will have to pay more attention
to.

I guess that\'s all I have for the moment.

MR. ANDERSEN: Creating a reporting program for emission reductions was
one of the requirements of the Energy Policy Act. The statute was the
object of a great deal of debate, and those that wanted to have hard and
fast targets and hard and fast systems of credit lost, if you will, to
those that said, well, let\'s learn more about emissions accounting,
particularly accounting with regards to mitigation activities.

That\'s the way I interpret what we got in the Act relative to the
debate that went before. So on a one hand, there were environmentalists
that were pressuring to move in this direction, because they wanted to
really know where the emissions were, who was doing what, and
potentially set the stage for a more systematic program of regulation.

On the other hand, there were also industry parties that were interested
in having some kind of a program, because they felt that they had been
cheated in the past for doing good deeds in anticipation of regulation,
and they didn\'t get any credit for it.

So there were this coalition of interests that led to the creation of a
program designed to obtain information on voluntary reporting.

Next slide. We then were faced with creating such a system, and there
were a large number of puzzles that we addressed. We don\'t claim to
have come up with the perfect product in this regard. I view the whole
effort as a very big success, but we had to create a system that was to
help report \-- obtain from parties voluntarily.

We had to create a system that addressed a whole range of activities. I
think there were seven listed in the statute, and the eighth was \"and
anything else.\" We were also asked to create a system that permitted a
simple project to be reported and a program that depicted the total
picture of a major enterprise\'s emissions activity and emission
mitigation efforts.

So these all went into trying to design a reporting program, which we
came up with, and it\'s electronic, and it\'s relatively user friendly,
and we\'re pleased that we\'ve got it working.

The question came up all the way through, if it\'s voluntary, how can
you say anything about the quality of the data. What we have tried to do
is to make any participant aware of the fact that what is reported is in
the public domain.

So there is an opportunity for somebody else to be looking over their
shoulder. That is the best we can do in terms of working towards
assuring the quality of the \-- the carefulness of the reporting.

The carefulness of the reporting is a bit different from the issue of
the quality of the data, because what is the data really telling you
when you have voluntary participation and when you have, in fact, a
variety of perspectives that can be applied in reporting the activity.

For example, one of the big issues was can a player play if his
emissions inventory is rising or can only a player play if he can report
reductions in absolute levels of emissions? The decision was made \--
and incidentally, there was an EPA-DOE cooperative exercise involving
hearings over a three-month period that led to the issuance of
guidelines which were then the instructions to us in terms of how to
develop a program.

The decision was participants with rising inventories, if they are doing
things that cut back on emissions, should be able to play, should be
able to report, as well as those that would report absolute reductions.

So the New England Power said design a \-- or the representative there
\-- design a program which permits me \-- I am the only one that really
is doing the job in this area; I have absolute reductions in emissions,
and I want to be able to demonstrate to other people that that\'s what
I\'ve done. So be sure, if you\'re going to let, we\'ll say, Florida
Power and Light in, who has rising emissions \-- be sure that you can
distinguish reports that relate to rising levels of emissions and
reports that relate to absolute emission reductions.

We have a system that does that, and we have a system that any potential
interested party can look directly at the database and see who is
reporting what.

In the context of a voluntary program then, the next question is what
social benefit comes out of this. The perspective that we came to was
that there are potentially two benefits. One is we\'re learning a lot
about accounting, accounting for emissions.

I think, you know, we\'re more than 1,000 years into financial
accounting, and we still have a financial accounting standards board
worrying about things at the margin, and we didn\'t have any accounting
system for emissions. So we\'ve started the process of a uniform
framework for thinking about the emissions in the context of a
particular project or in the context of a total enterprise operation.

The other thing is that reporters that are volunteering are
volunteering, presumably, because they think they are doing something
that is worth having noted, either in anticipation of future claims for
permits or just for being green in the current environment.

So we think that there is a benefit associated with possibly broadening
understanding as to what people think is worth doing with regards to
mitigation initiatives, and the imitation process perhaps is accelerated
in the context of the program we have.

We have now three \-- We have the third reporting cycle ending. We\'ll
have the fourth one out by the end of the summer. So we will have caught
up with 1997 data by somewhere around mid-year \'98.

Of course, in the context of all of this work, we\'ve come to be viewed
as something of an expert with regards to emissions accounting in
general, which is scary, to tell you the truth, in the context of a lot
of the discussions with respect to permits and permit trading.

Next slide, I guess. I\'m going to run through these next ones as
quickly as I can.

The last couple of reporting cycles, approximately 1,000 projects of one
sort or another claiming reductions in emissions have been reported, and
the biggest group of projects relate to electricity.

In part, this relates to the promotion of the Department of Energy of
voluntary programs. Climate Challenge was one of the early voluntary
programs promoted by the Department of Energy, and they chose to use the
1605(b) reporting mechanism as an information base for that program.

So large participation from the electric utility industry, lesser
participation from other reporters, but this distribution is relating to
the most recent data that we have.

If we go to the next slide, you\'ll see in some measure there\'s good
news and bad news with this slide. The good news is that you have a
rising level of participation. We have now about 150 reporters, and we
have improvement in the diversity of reporters.

We now have chemical companies, some chemical companies reporting, some
major automobile manufacturers reporting, and there is increasing
diversity of industrial type companies reporting, but overall the level
of participation is still pretty modest.

Frankly, one of the reasons it is, it\'s voluntary, which is all right.
The other reason is people say it costs to do this kind of thing, and
until I know that permits are worth something, why should I report to
you. Then, of course, the third thing is, if I report to you now, am I
not putting in place a dynamic for real regulation.

So all of these things work to dampen the level of participation that we
have, but nonetheless, there is a rising level of participation in terms
of number of reporters and also in terms of number of projects.

Next slide. As I said, the biggest number of projects reported relates
to electricity, and you see a fair distribution here, and you \-- Of
course, as lots of things like lighting, improved efficiency of
appliances and so on, demand side management type programs are reported.

The generation improvements, in general, relate to efforts to improve
heat rates. Then there have been a host of projects that are designed to
reduce electricity losses, and they have been identified. In some cases,
we get really good information \-- it\'s limited, but it is available in
the database, if you want to go find it \-- that even provides the
payout period for some of these, which because it\'s voluntary, you
would expect the payout period to be short, and that\'s consistently
what we do see.

Next slide. As I said, the overall level of participation is still
fairly limited relative to the potential. So in the order of magnitude
of 200-300 million tons of carbon emission reductions are identified.

Now that should not be thought of as necessarily a net reduction
relative to the overall. In that, there may be activities that are
reported on by more than one entity, and there may be other problems
with regards to how you think about the actual number; but what\'s
interesting is, I think, the distribution, and that sort of gives you
some impression of where the potential action is with regards to at
least voluntary like programs associated with mitigation.

One of the \-- next slide. One of the elements of the Energy Policy Act
was to try to encourage diversification towards non-carbon sources of
generation. So we do have a picture of what\'s going on with regards to
projects reported that either reduce the carbon content of fuel use or,
in fact, involve some effort to move away from carbon based fuels.

I suppose the thing to keep in mind is that, again, that\'s pretty small
if you believe that these are a pretty small number of projects. If you
believe that non-carbon fuels are an important wave of the future with
regards to Kyoto, this may be a cautionary flag.

There\'s another dimension that is of some interest in the
hydroelectric, for instance. There is a project in there which somebody
reports a reduction in emissions because they\'ve reduced the purchase
of coal based generation and now are purchasing hydro generation. What
has this done in the big picture sense from the standpoint of national
emissions? Perhaps not very much, because now somebody else is buying
coal generation.

So as Arthur and I were talking about, it\'s the kind of problem where
somebody goes across the border and starts buying hydroelectric. What is
the overall consequence for emissions for the region as a whole if you
simply have had a displacement of purchases?

All right. The last slide here, I say something about issues. Obviously,
I\'ve already mentioned some things which relate to issues. What is the
information value of the data that we are collecting? There are
increasing pressures to move toward some sort of trading system, and a
lot of discussion about the accounting framework that we have as being
helpful in that regard.

There, of course, is the ongoing issue of those that have played now,
what benefit will they get if we move towards a permit system. Then
there is the question of how much do people care about the things that
you have to come to grips with if you move towards, in fact, a trading
system? You have to come to grips with things like property rights, the
issue of verification, a variety of accounting principle issues.

We\'ve had to think about many of these in the last couple of years, and
we have found that other people, as they get closer and closer to
wanting to recommend permits as an approach in the Kyoto context, begin
to worry about it, the closer they get, because of all the problems of
the property rights and verification.

I\'ll stop here and, if there are any questions or comments.

CHAIRMAN RELLES: The agenda lists a whole series of talks with
discussion at the end, but we have all morning, and I would certainly
like to encourage discussion and questions at this point. Greta?

MS. LJUNG: I was wondering if you have analyzed \-- You have 604
electric utilities reporting. How \--

MR. ANDERSEN: 604 projects, about 120 utilities.

MS. LJUNG: All right. Now how representative do you think that those
utilities are, and have you analyzed them by state and looked at whether
they have had executive demand side management programs, for example,
done more than the average utilities to reduce emissions? Have you done
an analysis of that type?

MR. ANDERSEN: The utilities in the program account for somewhere between
half and two-thirds of U.S. generation. So we haven\'t systematically
looked at how reporters differ from nonreporters, but \-- except that
we\'re confident that it\'s the largest utilities.

I think we have all of the top ten utilities in terms of kilowatt hours
generated in the United States, and very few of the smallest ones. There
are about 2,000, I think, electric utilities in the United States, 2400,
something like that; but there aren\'t that many big ones, and we have
almost \-- We have mostly large ones, all of the largest ones.

MS. LJUNG: You have all the states or the difference between states? I
mean, some states have done quite a bit more, you know, in terms of
demand side management.

MR. ANDERSEN: Yes. We have disaggregated some of the data by states, and
basically there\'s some representation in virtually every state with
regards to either a supply project or a demand side management or, for
that matter, even carbon sequestration projects.

So we have a fair distribution geographically within the U.S. and when
you talk about sequestration worldwide. If you think about the claimed
benefits with regards to reductions, what happens on the supply side has
been the most important relative to \-- That is, the claimed reductions
in CO~2~, heat rate improvements and, in particular, increased
effectiveness of nuclear capacity utilization have been the biggest
sources of benefits as reported in this program.

Demand side management is very numerous with regards to projects, but
the overall level of reduced emissions relative to those projects is
pretty small. On the other hand, to the extent they can be widely
imitated, the potential consequences could be larger than what you see
in the reporting statistics.

CHAIRMAN RELLES: I am going to go in the order in which I saw the cards
go up. So I believe Cal was next.

MR. KENT: I want to ask the first Art. You said that the emissions
inventories are now required in all countries that signed the Rio Pact.
Has any investigation been done as to the determination of how accurate
or reliable the inventories are in other countries, because one of us
has a strong suspicion that they are not very good, if they\'re being
done at all.

That\'s only on the basis of having spent a month in China this last
year and having actually gotten their opinion on these issues. So I
would like to have your response, because this is really a key factor.

It\'s obvious that you\'re going forward, but it\'s \-- in an honest and
somewhat transparent method, but it\'s not at all obvious to me that
anybody else is doing anything more than just paying lip service to
this.

MR. RYPINSKI: Well, where to start. There is \-- The treaty requires
inventories from all signatories.

MR. KENT: Are we actually getting those?

MR. RYPINSKI: We are getting a lot of \-- The framework convention
secretariat is getting inventories from many, though not all, countries,
not always timely and not always as frequently as they might wish.

They have gotten an inventory from, I believe, all of the Annex I
countries, which are basically the developed countries. It turns out
that the term of art is a national communication. So the framework
convention secretariat has received national communications.

When they receive a national communication, they actually post it on
their Web site, and then they send a review team out to review that
national communication. The review team prepares a written review of the
national communication, which is also frequently posted on the Web site.

So the United States government developed something they call the
Country Studies Team, which is a joint venture between EPA and DOE, to
fund inventory work in developing countries. The Country Studies Team
has done inventories in 40 or 50 developing countries. I wouldn\'t swear
to the number, but it\'s around that.

So there is \-- There are a lot of inventories. There is a review
process for those inventories. Now on to the third part, which is what
is the quality of those inventories and the timeliness?

The quality of inventories, basically, hinges on the quality of the
national energy statistics, which as you know, varies from good to
abysmal. If you have abysmal energy statistics, it\'s very hard to have
a good emissions inventory of greenhouse gases.

Also, one of the systematic ways in which developing countries are
different from developed countries is quality of their public
administration, generally. The quality of public \-- One of the ways in
which quality of public administration exhibits itself, or otherwise, is
\-- this will come as no great shock to anyone who has worked in the
Third World \-- is in the quality of the statistics.

Frequently, it\'s \-- I think it\'s been hard to do what we\'ll
euphemistically call the technology transfer as part of the Country
Studies Program. You can send a team in, and you can get an inventory
done, but after the team goes again, will the work continue? I think,
frequently, an open question.

CHAIRMAN RELLES: Denny, please.

MR. ELLERMAN: I want to address a question to Art Andersen on the
voluntary reporting. In doing that, I want to draw a distinction
between, say, the problem of measuring emissions and the problem of
measuring reductions in that the former, of course, is something that
there are technical issues involved with how do you measure emissions,
but a reduction is a difference between two objections, one of which is
observed, namely emissions, and the other of which is not observed,
which is what you think emissions would have been or what in the
language, you know, is called the baseline.

I gather what comes under the voluntary reporting program, what is being
reported is primarily reductions, and I guess, given a concern for the
quality and the reputation of EIA, then the issue I raise is then how
does one \-- even assuming that the reduction part or, let\'s say, the
observed emission side of the reduction is being accurately measured,
which is another issue \-- how do you address the issue of the quality
of the counter-factual, the baseline, the unobserved other half that
defines the reduction?

MR. ANDERSEN: What we try to do in this context is to promote
transparency. Okay? As I said earlier, because it\'s voluntary, a key
thing, we believe, with regards to promoting careful reporting is to
make sure that whoever is reporting recognizes that he could be in a
fishbowl. So that\'s number one.

Number two, we do have a review process, and for classes of projects, if
there are wildly different estimations associated with the claims, if
you will, we will proceed to ask questions and ask them to check the
numbers. But when push comes to shove, if they say that\'s our number
and that\'s it, we do not have an independent view with regards \-- that
we will impose into the database.

I don\'t know whether that gets at all of what you were saying, but you
have in the reporting program \-- If you are a player, you have the
option of reporting all of your emissions for the total enterprise and
the emissions reductions associated with all of the projects that are
thought to achieve reductions, or you have the option of reporting a
project and some emission track in relation to that project.

I mean, that\'s the feature of the database that reflects the program
decision of trying to encourage participation more than anything else.

MR. ELLERMAN: In pursuing transparency is, when a reduction is reported,
are both sides of that \-- Let\'s say you say project X had emissions of
Y amount, and this action that we took reduced them by some amount from
which you can calculate the counterfactual, or does it come in as just
saying we\'ve reduced X number of emissions, and there\'s neither a
reference to what is alleged to be observed currently or \--

MR. ANDERSEN: There\'s a reference in a project context, and there\'s
certainly a reference case in the context of an entity-wide reporting;
but as I say, the reporter can choose the context.

CHAIRMAN RELLES: Jim.

MR. HAMMITT I\'d just like to raise some issues on the emissions
inventory. You, obviously, couldn\'t go into this in any detail, but the
questions involve the sources of these data. I think probably none of
the emissions are actually measured.

So all or almost all of them rely in varying degrees on modeling from
some inputs that are measured like fossil fuel use or fertilizer use. So
there are questions about how reliably we\'re extrapolating emissions
from the things we can measure, how well some of those are measured,
especially compounds that are by-products in particular activities, how
these issues are being addressed in the international fora where,
obviously, there are commonalities across countries, and how we would
view this, and what\'s being \-- what\'s the thinking about whether we
want to report estimates of uncertainty or whether we\'re really going
to define by convention how emissions, sort of nominal emissions, are
defined and report that only as a point estimate.

MR. RYPINSKI: Well, I thank you. I think I\'ve just been given an
opportunity to compete in the Fidel Castro endurance speech making
competition, but I will try to be brief.

First, it turns out that in the case of energy related carbon dioxide,
the chemical composition of fuels doesn\'t vary enough to be a
particularly large source of uncertainty in going from fuel burn to
estimates of emissions. In fact, we have this very interesting program
where the EPA put continuous emission monitors on the stacks of all of
the largest electric utility plants in the United States that are
controlled under the sulphur trading in the Clean Air Act amendment.

They required these guys to monitor sulphur for compliance with the
Clean Air Act, but they also set, oh, by the way, you have to monitor
CO~2~. What they learned basically is that placing a CEM in a stack is
an art, and not all utilities had mastered it.

Basically, the problem is that what you\'re measuring is concentrations
at a point in a tube, and then you\'re extrapolating velocity and
density, and there\'s all sorts of interesting things about turbulent
flow that are really more than you really want to know. Unless you nail
down the flow regime in the stack, you don\'t get a particularly
accurate answer.

So it turns out that, in the case of energy related carbon dioxide, mass
balance based on fuel burn may be more accurate than measurement.

Now when we move on to the other gases, the story sort of flips around
and reverses itself. In that case, there are significant issues with are
we getting the right numbers, and are the indirect methods we\'re using
producing accurate issues.

The trouble is, it isn\'t one story. It\'s like 20 stories based on
different sources, which I won\'t pursue, save to say that, yeah, it\'s
a big issue.

Okay, now on to the third part, which is what is the international
community doing about it, and what are the issues associated with the
United States?

The Intergovernmental Panel on Climate Change has set up \-- the
Secretariat has set up a special working group on methods of estimating
emissions. They\'ve published a four-volume set of manuals on how to do
national emissions inventories that reflect the work of various
international working groups.

It turns out, though, that frequently the problem is not the method, but
what is the quality of the underlying data, and is it, in fact,
available. That is, the method can be perfectly good, and the data may
be unavailable or the method can be good and the data is bad, or the
method can be bad and the data is bad. I mean, there are a lot of
permutations.

I think the inclination in the international community among the
politicians is to choose a number and stick with it. The guys at the
policy making level tend to react very badly to the remark, oh, by the
way, our 1990 emissions are several percent larger than we thought they
were last year.

After they get a few of these shocks, they tend to look for ways to
insulate themselves against future shocks. So that\'s an issue, in fact,
in dealing with Kyoto.

CHAIRMAN RELLES: We have time for one more question. Phil?

MR. HANSER: I guess I\'m a little bit concerned \--

CHAIRMAN RELLES: Use the microphone, please.

MR. HANSER: Is that better? Okay.

I\'m a little concerned about the baseline versus the differences, just
because I was involved in the EIA 861 development a long time ago. At
least from my personal experience, the utility\'s ability to estimate
the impacts of demand side management programs, for example, and
forecast what, in fact, the reductions are varies enormously.

I mean, a couple of orders of magnitude are probably not too large of a
range, frankly. I guess I\'m kind of wondering if it doesn\'t get
magnified when you then try to translate that into greenhouse gases.

That is to say, in a sense, almost you\'re propagating error, but you
have one set of errors associated with the initial demand reductions and
then a second set of errors in the conversion from what those demand
reductions translate into in terms of greenhouse gases.

Let me give you \-- Here\'s another variant on that same problem. If I
pursue an energy efficiency program in which I switch out the air
conditioners from their current state to some new one, I will be
switching air conditioners that use probably some form of CFC to air
conditioners which use HFC-134a or maybe HCFC-222 or a whole host of
them, all of which are among \-- one of which is \-- The substitute gas
in the air conditioner is considered a worse greenhouse gas, in some
sense, than the initial gas in the air conditioner we had.

So measured in energy reduction and claimed greenhouse gas reductions is
the result, but in fact not having taken account of the change in the
coolant that\'s used in the air conditioner I\'ve misestimated, in fact,
the true impact of greenhouse gas reductions.

I just feel like \-- I mean this is one of these things where I get sort
of concerned about, you know, how these things get \-- kind of propagate
errors upon error.

The other issue I guess I sort of worry about is \-- and it may be a
slightly different approach. I\'ll give you a kind of a stupid story.

I bicycle occasionally, and some of my friends have very expensive
bicycles, and they will spend fortunes to reduce the weight of a bicycle
by \-- of their parts by a gram or two grams. Literally, you see parts
listed by price and their weights.

Whereas, in fact, if the bicyclist had gone on a diet \-- right? But the
difference is that, regardless of how their weight fluctuates, if they
had changed the capital stock, that doesn\'t vary.

So the question might be that what you really need to get a handle on is
the capital stock and how it performs relative to its use of energy, and
that kind of issue on the car supply side, how that translates into
greenhouse gases, rather than worrying about in some sense the weight as
the volume measure as it goes through; because in some sense, that
produces a limit on how much gases are going to be consumed and produced
under almost any circumstances.

That\'s kind of a different way to look at, I think, in some sense, the
measurements. On the other hand, though, getting into the capital stock
issue is not a simple one at all, and it requires understanding
vintaging and the technology at the time of the vintage and so on. But
if that\'s going to be in the end a requirement for going out and
measuring how much emissions are out there, it seems to me that going to
mass balance and understanding what technology is out there may turn out
to be more accurate in the long run than trying to run around and trying
to do projects and so on and in the end, you know, may be turn out to be
the way that you really need to think about getting to a measurement of
greenhouse gases.

MR. ANDERSEN: I don\'t have anything to say that would disagree with
that. In fact, in our electricity modeling we do try to understand the
character of the capital stock and have an effort to do vintaging.

The thing I was trying to emphasize with regards to the voluntary
program is that I think there are things to learn from what has been
done, and there may well be additional things to learn as the process of
reporting evolves.

On the other hand, don\'t give it too much credit for getting you to
learn in some areas that people seem to have confidence that we can
deliver on.

MR. RYPINSKI: Well, another way of thinking about it is what\'s the
payload of the voluntary reporting program. That is, what is the
fundamental piece \-- What is fundamentally we\'re doing in the
voluntary reporting program.

There\'s a tendency to think of the reduction as the payload, that
we\'re accumulating reports \-- we\'re accumulating reductions. I think
that\'s actually \-- I think that actually isn\'t really true. I mean,
we are accumulating reports of reduction, but I don\'t think that\'s
really the center of gravity of the program.

I think the process is the payload. I think that its greatest benefits
are maybe pedagogical. It\'s teaching people who work for companies that
emit large volumes of greenhouse gases what causes emissions and how
they might reduce them, how to account for their emissions, and ways of
emitting less.

It\'s, I think, part of the education process that will be \-- that is,
in fact, a poorly understood preconditioned, actually, seriously doing
anything about the greenhouse gases. We\'re also teaching people by
example about a lot of thorny property rights and emissions accounting
issues that are poorly understood or not understood at all in Washington
or outside of Washington, for that matter.

CHAIRMAN RELLES: Thank you all for that discussion. My colleague, Bill
Weinig, is a little concerned that we\'re going too slowly, but I think
there\'s plenty of slack at the end of the morning to soak up the
additional discussion we\'ll have. So I\'m not concerned myself. I\'ll
take the chance.

I\'d like to move us on, however, to the discussions of the service
reports. The first presenter on that, I presume, is Andy Kydes, and Ron
Earley, too.

MR. KYDES: Good morning. It\'s nice to see some old and new faces.

We want to give you a brief overview of the report entitled \"Analysis
of Carbon Stabilization Cases: The Energy Impacts,\" and one of the
first things I want to do is to assure you that I\'ve been told by Dr.
Hicks that my graphics don\'t have a prayer at winning this competition
today.

Now this analysis report actually is an outgrowth of the interagency
task force process that began about two years ago, and its purpose was
to analyze alternative carbon stabilization cases with regard to the
energy and economic impact on the U.S. system. That information was in
turn to be provided to the U.S. negotiators who were then going to
somehow work it into their negotiations in Kyoto.

The Department of Energy was one of the full fledged members of this
interagency task force, and the Office of Policy and International
Affairs was, in fact, one of the active participants with their own
modeling system, namely Marcal Macro with which they did some of the
analysis.

Because, in fact, they were getting bogged down with many rounds on the
same small set of issues that they were looking at, they asked EIA to
supplement their work and look at some additional cases for them as part
of this process, but our work directly responded to the Office of Policy
and International Affairs.

Now in this process, we basically were assigned two groups of cases to
analyze. The first group basically said look at carbon mitigation
targets where the work for mitigating has to be done entirely within
U.S. borders \-- so no international trading.

The second group actually turns out to have only one element, and that
said, suppose that you could get all the permits you wanted
internationally at \$40 a ton.

Largely, the approach we used was, because we do not have, in fact, an
international model, that allows us to do this permit trading. Now the
nine cases we looked at included three budget periods. These are in the
context of a protocol. It means that the average over those periods has
to equal the target, and you can choose to do it in any way you want.

The cases, the targets, we looked at ranged from four percent above 1990
levels of emissions down to 15 percent below. Now we started with the
AEO \'97, the Annual Energy Outlook 1997, and then basically, at the
request of the sponsor, changed some of the critical functions for the
interagency task force study in support of them.

Now let me interject here that we simply didn\'t take the assumptions
and blindly put them into the model. It\'s not quite that simple in the
process. There was, first, a lot of negotiation, making sure as a good
customer that we ask, are you sure you really mean this, because the
implications are thus and so.

So we went through a whole litany of all of their requests and made sure
of the question, that they really wanted those assumptions to be
implemented.

Now one of the first ones that we were asked to implement was this
effect of a policy announcement. Simply announcing to the world that
signing a treaty, that we\'re going to abide by some kind of treaty,
that that in fact would accelerate the availability of advance
technologies, and also reduce their costs without a price signal.

Now this is, obviously, one of the ones that we had some digestion with
and, in fact, eventually we got our sponsor to agree that he has a
problem with it as well. The fundamental problem is that as part of this
IAT process this was one of the fundamental assumptions that we had to
go along with.

Now in addition to that set of assumptions of advanced menus that had to
be available, we included them when we looked at the carbon
stabilization cases, a number of other assumptions that \-- Some of the
key ones are listed here.

It\'s lengthy, and this can take the whole period, which I don\'t intend
to do, just let me highlight some of the important ones.

We were asked to assume that 25 percent of the energy intensive
industries would leave U.S. borders. We were asked to assume that all
nuclear power plants were life extended by 20 years. We were asked to
accelerate the retirements of the industrial technologies by 25 percent
so that you would have greater turnover and more flexibility again.

We were asked to put in and use the ACEEE advanced technologies in
transportation, which meant that the costs were \-- the incremental
costs of putting them in were 50 percent lower than we assumed, and the
performance was about 33 percent higher.

There were a whole slew of these. The very key one, though, is at the
very bottom, which is that we were asked to introduce a non-carbonaceous
generation technology by 2015. It looked an awful lot like nuclear.

Now what are the assumptions that \-- Given these very highly
conditioned set of assumptions, what are the implications on the U.S.
energy system if, in fact, we have to abide by a carbon target or a
series of carbon targets?

Well, the first one basically shows you the impact on consumption level.
What it says is that, if you have to abide by a 1990 level of carbon
stabilization, that the first thing you\'re going to have to do is cut
back by at least 12 percent in your primary energy consumption.

The red line for information is the target at ten percent below 1990
levels, and the only reason I show it here is because, in fact, if we
were to look at it carefully and had more time to discuss it, this is
actually closer to the seven percent below adjusted \-- under the
current AEO \'98 assumptions and so on.

In any case, it says that we have a lot to cut back on in terms of the
level of consumption, but not only do we have a lot to cut back on in
consumption, we have a lot to do in terms of changing the mix of energy,
our appetite for which kinds of fuels we consume.

That\'s sort of indicated by the carbon emissions per person. If you
take a look at today\'s levels, we\'re out around 5.6 tons per person.
However, relative to today, to achieve carbon stabilization we have to
reduce that by 20 percent in 2010. Then as population, of course,
continues to grow, that becomes 25 percent relative to today, decline.

Of course, the situation gets even worse when we talk about stabilizing
to 20 \-- to 10 percent below numbers.

Now this next pair of graphs \-- and this is one of the major reasons
why I\'m not going to win the award \-- on the top right basically
illustrates the relationship between the scenarios with a very highly
conditioned set of assumptions. Okay? The relationship between the
target levels that we set and trying to meet \-- these are one-year
budget periods \-- with the carbon permit values that might be required,
given all of the myriad of assumptions that were incorporated. So this
is a very highly conditioned set of relationships we\'re talking about.

It\'s interesting to note that the top line is your AEO \'97, because
that was the latest we had. The green line is the impact of the
announcement effect. That says that we can reduce carbon emissions, if
you believe it, by 60 million metric tons in 2010, simply by announcing
that something is going to happen. It may happen. I don\'t know, but
that was the assumption.

The important thing is to achieve 1990 levels of carbon stabilization,
that in fact the blue line on the right side indicates that the carbon
permit value has to range between \$130 and \$150 a ton. If you drop
that to 10 percent below, which is the next line below the blue, you
have to go to carbon permit values in excess of \$200 a ton.

The next item, just a graph, shows you the impact of, say, stabilizing
at 1990 levels on delivered fuel prices, the fact that the permit fees
or permit values are translated through the price mechanism to
consumers, which in principle they would be.

What you can see is that, in fact, even stabilizing it at 1990 levels
for carbon, that the price, the percentage change, of coal increases by
300 percent. That is, it triples in price relative to the baseline;
whereas, in fact, natural gas only goes up 75 percent, which is still
hefty; but electricity and motor gasoline are far lower, because they
have other fixed components that keep the average from rising. So coal,
in fact, gets hit very hard in this story.

Looking at then the next slide, how would these carbon reductions occur
by primary sector. Well, it turns out that the most competitive market
is the one that does the most moving, not surprising. Electricity
accommodates about seven percent of the reductions in order to meet the
stabilization at 1990 levels.

In fact, an interesting point, and this is one that our staff complained
about \-- why can\'t the demand side do more \-- is in fact that by 2010
the emissions \-- carbon emissions by the utility industry are 25
percent below 1990 levels. That\'s significant.

Now the next slide indicates something which is not very surprising.
Given that the coal price triples, it\'s not surprising then coal
contributes the most to the reduction in carbon emissions. In fact,
contributes about 80 percent of the carbon emission reductions in 2010
to stabilize, petroleum about 20 percent.

In fact, the bottom portion shows you that natural gas is coming in like
great gangbusters in order to basically generate the electricity through
gas combined cycle to displace the coal. In fact, coal consumption by
2010, by the way, drops to less than 50 percent of 1995 levels. So it\'s
getting hit pretty hard in this case.

The next slide indicates basically how the \-- what the source of
reduction in the electricity sector are. It turns out that about 30
percent of the reduction is achieved through demand reduction, less than
electricity demand, but about 70 percent of it is achieved through
basically fuel switching and efficiency improvements.

I\'ve put into that pot the nuclear life extensions, the renewable past
extension, the gas combined cycle for coal, the fewer coal bills. All of
that is in that pot.

To underscore the change in the mix in the way we use energy, that
important fact, we take a look at the mix change in the generation
market for fuels. You see that coal loses about 16 quads of energy, down
two-thirds from its reference case levels in order to make this
accommodation.

Whereas, natural gas, of course, builds considerably. the black line in
the middle is nuclear life extensions. So we\'re not building new
nuclear in this case, it turns out. It\'s just the life extended portion
that keeps on staying put and, therefore, it contributes to the coal not
having to be built that would have been built, or gas.

Also, at the top there you see about 100 gigawatts of renewable energy
that\'s being built \-- capacity that\'s being built, and an increase,
slight increase, in imports.

So the bottom line is that, despite all the flexibility that we\'ve
built into the system \-- that we tried to build into the system through
our sponsor\'s nudging, obviously, because of his assumptions, that in
fact there are some very significant changes to the energy system that
are going to take place if we, in fact, enforce a 1990 cap or something
of that nature.

So let\'s take a look at the case, particularly when you\'re forced to
accommodate all of the reductions within the U.S. borders. Now when you
assume international trading, of course, the story brightens
considerably.

In that instance, for example, where in fact you have a \$40 a ton
international carbon fee or carbon permit fee, you impose that on the
U.S. markets, and you find that in fact domestically the U.S. markets
are able to reduce their carbon emissions by about 130 million metric
tons, and that still leaves about 240 million metric tons to be
purchased at about \$10 billion roughly.

Coal industry survives, but it doesn\'t prosper as it did in the
reference case, but in fact the story with where the reductions are
occurring is still that the utility industry is the one that is in the
best position to make the changes and make the accommodation.

With that, let me turn it over to Ron Earley.

MR. EARLEY: Okay. Basically, I and Kay Smith and Brian Ungaro do the
macro work. We\'re used to being last and having to do things pretty
quickly.

Let me stress a couple of points. Yes, we did participate in the IAT
process. Yes, we did support the DOE office. In the interagency process
our role in that context was advisory to the DOE policy office, but
there\'s a lot of in-fighting. There\'s a lot of battling that goes on
in the IAT process.

Our role is to kind of keep people honest in many respects. Sometimes
they listen to us. Sometimes, they don\'t. Okay? So we may get into some
comments about the Administration statements about what the role of
Kyoto is a little later on, but for right now I just want to focus on
what happened in the macro side during the IAT process for our service
report. Okay.

What I\'d like you to go away from this meeting today are four
fundamental points. One is: The target matters, and how you get there
matters. Okay? There\'s a lot of talk about carbon sinks now, and Ill
talk about emissions trading in a minute, but how you get there and the
level of the target really does matter.

In this, for most of the IAT work we focused on carbon stabilization. A
lot of the fine points dealt with how you do it averaging over a budget
period, but the fundamental point: If you\'re now shooting for either
seven percent below or three percent below, you\'ve made the whole game
much more difficult from the economy side, from the energy side and the
economy. That\'s point one.

Point number two: How you design the permit system \-- everybody calls
it a permit system. Okay? Everybody is afraid to call it a tax, but one
issue is what do you do with the money. There\'s vast sums of money that
are going to be floating around the country. Okay?

The way we basically model this as the Federal government will receive
the money, and then you got to decide what do you do with it. Well, our
central case for what we basically focused on is we give it back to the
consumer by a personal income tax cut, but you can do it different ways,
which we looked at a lot of those. So that\'s point number two. What you
do with the money makes a big difference.

Point number three: Competitiveness. I\'m going to define the term
leakage, which we throw around a lot. In the carbon setting leakage can
mean what has happened, you know, if industry goes overseas or other
countries develop at the expense of the United States, there\'s leakage
in the carbon system by virtue of all this. When I say leakage today, if
I use that expression, I mean loss of our industry overseas.

It\'s a big issue. It was basically dealt with by assumption in this
particular document, given the cutback in industrial output for the
energy intensive sectors of \-- and I\'m looking back here to Kay to nod
up or down \-- 25 percent by 2010. Okay.

So how you deal with the leakage issue is important. That\'s point
number three.

Point number four: Emissions trading. Emissions trading may be the big
ace in the hole. Okay? I\'ll get back to the Administration statement,
is that 80 to 90 percent of the effort can be done through international
emissions trading program, and you really don\'t need to reduce either
energy or carbon very much at all, and they\'re going to end up with a
\$20 price on carbon. Okay? In terms of a permanent price. That\'s the
Administration position as expressed by Janet Yellin. So emission
trading is important.

I\'m going to go back in slides. We\'re going to go back over those four
points very quickly.

The first slide. Now from your presentation I\'ve thrown out a couple of
the cases, but let me focus on four cases. Okay. If you look at the red
line, basically that\'s our five-year ramp down to 1334. That\'s our
stabilization case.

We basically had about \$100 billion loss in GDP to the economy, about
one percent. We do have a feature such that the economy does recovery
back to baseline. One of the things that happens in the carbon price is
you ratchet up very, very fast to a high level, and then actually the
carbon price begins to fall off over time.

So you introduce a cycle in the economy such that you can actually
depress the economy rather quickly, have a fairly substantial impact,
but then you do begin to recover back towards the baseline.

Now the bottom line, okay, which is 2010: That\'s one of the things I
want to focus on before. The red line is stabilization. If you go to ten
percent below, you have substantially larger impacts on the economy.
Things begin to be nonlinear. Okay?

So if you\'re aiming at the seven percent on Kyoto or if you\'re aiming
at three percent, which is what the State Department fact sheet
basically said, you\'ve got a four percent offset giving sinks, and
you\'re aiming for a three percent down target. The level matters.

Now we did look at a case where the green line basically was an
averaging, which basically was \-- we called it averaging for our
purposes. By the time it got to Kyoto, they called it budget period.
That\'s a lot what happened in Kyoto. There was arguments about things,
and people got hung up on words. So they would change the definition of
it to something else and effectively do something very similar. JI
became clean development mechanism.

The green line basically was saying you meet that over an average
period, over a five-year average. So you don\'t have to meet a specific
target in 2010, and by giving you some averaging over a 2008-2012, you
get some flexibility. The GDP loss is a little bit less, but because
you\'ve got the pressure on a little bit beyond 2010, you don\'t rebound
quite as fast.

Then we did look at one where we had a ten-year ramp period where,
instead of a five-year ramp, a ten-year ramp. Basically, you\'re
smoothing out the adjustment cost on the economy. I believe I\'m
accurate. It\'s probably in here. I think the net present value in the
ten-year and five-year basically comes out to be about the same.

So this is the point on the level of the target matters a great deal,
and these are the dynamics that we basically are showing on the economy,
kind of a heavy hit early on, recovery towards base.

Next slide. Okay. Giving the money back matters. If you give the money
back via a personal income tax cut, essentially you\'re giving it back
to consumers. You have an impact on the economy. You recovery back
towards base.

If the Federal government decides to keep the money and use it to reduce
the Federal deficit \-- well, I understand people are going to say it\'s
gone away already. Projections for 2015 and 2020 don\'t really suggest
that the problem has gone away, but if you use it to reduce the deficit,
what happens is you\'re basically taking money out of the system to pay
down the Federal debt.

You have a larger impact immediately on the economy, but one of the long
run benefits you get by using it to reduce the Federal debt level is you
drop interest rates at a fairly healthy clip. So that over the long run,
you\'re basically subsidizing investment over consumption.

If you give it back to consumers, you\'re subsidizing consumption over
investment. So what you do with the money makes a big difference in
terms of how you\'re going to deal with consumption versus investment,
and that can apply regardless of whether the Federal government collects
the money or you have some other permit system where the money is
grandfathered back to business. How you give the money or how you make
the permits available to people, investment versus consumption, matters
a great deal.

Next slide. Okay, I\'m ditching the discussion on the Federal Reserve
reaction. So we\'ll \-- If anybody has any questions on that, we\'ll
talk in the question period.

Okay. One point I wanted to make here. I\'m not going expressly talk
about the leakage issue now. I\'m sure there will be some questions.

Impact on the economy: I\'ve shown you the one percent loss in GDP.
Okay. Taking your broad sectoral level, obviously, mining which is
essentially coal mining here, really gets \-- you know, gets the wind
knocked out of it. Okay.

So we have a loss in manufacturing. We actually had some industries \--
I choose my words carefully \-- You may actually have some industries
that may benefit. So there is going to be a shift in the structure of
the economy, and that will happen regardless of whether you have any GDP
impact.

Next slide. Manufacturing: Basically, what you are seeing here is the
heavy hits are occurring in your energy intensive industries. Okay? So
even here, you\'re going to have some industries that are going to take
heavy hits, and primarily it\'s because they\'re energy intensive.

Part of what you\'re seeing here is the exogenous assumption of the 25
percent leakage cutback.

Next slide. Now like I mentioned, the Administration position is that
most of the action can be done with permit trading. This is another
thing where the client asked us to evaluate a \$40 carbon price,
basically, as a proxy for what would happen if you had an international
trading system.

Not surprisingly, if the price for carbon is half or cut by two-thirds,
the impact on the economy is going to be much smaller. So one of the big
problem areas is going to be to get a handle on how you\'re going to do
international emissions trading.

There\'s a great deal of sentiment amongst other countries against
allowing the United States to basically do it all overseas. There have
been some talks, and maybe Arthur may know about some restrictions, on
the amount of the action that you can buy overseas, but how you deal
with emissions trading is going to be a big issue.

So you got the targets, what you do with the money, what happens to
competitiveness of industry, and fourth, what happens to emissions
trading. Those are the key \-- those are the magic buttons, from my
perspective on the macro side.

That\'s it.

CHAIRMAN RELLES: Thank you very much, Andy and Ron.

You know what. I\'m sort of losing in the battle against time. So I
think I\'d like to bring in the next speaker, and maybe we\'ll see how
we can deal with questions. I know that there\'s a lot that people want
to discuss about these topics, but let\'s try the next one. At least,
it\'s the same type of talk, and then we can hit them both at the same
time.

MR. BEAMON: I will try to go a little bit quicker. Having seen \-- or
Jay talked this morning about that we\'ve had budget cuts and lost
staff, I\'m the only one that lost my slide turner. So anybody want to
volunteer.

What I\'m going to talk about is some work we\'ve done for Senator
Jeffords from Vermont recently \-- not so recently, actually. Last year
in May, he asked us to look at a bill that he had put together to deal
with environmental issues associated with electricity industry.

He viewed it as an environmental bill more than an electricity
restructuring bill. Other people view it differently and think that
it\'s more of a restructuring bill, but that\'s in the eye of the
beholder.

What I want to talk \-- This bill is S.687. It\'s the Electric System
Public Benefit Act, and its main provisions are that it includes \-- we
didn\'t look at all of them. I\'ll tell you the ones we did look at and
the ones we didn\'t.

Number one is it has a renewable portfolio standard, and I\'ll talk a
little bit about what that is in a minute. It\'s a new three-letter
acronym that everybody will have to learn now, the RPS. There are
several of them floating around in different bills.

It includes emission caps on SO~2~, emission caps on NO~x~, and emission
caps on CO~2~. Now one thing to note differently from what Andy and Ron
just talked about, this is just utilities specifically, power
generation. It\'s not the whole economic caps, not caps on the whole
U.S. So it leads to some different results when you do that.

Talking about the \-- Next slide \-- what the RPS is, what the renewable
portfolio standard is, like I said, there\'s several bills before
Congress right now, different proposals on RPS. I don\'t know how many
of you know, and I\'ve become more aware of it, how many things are in
our electric bills that we pay every month.

There\'s quite a few things in there that we weren\'t aware of, things
that we are paying for. We\'re paying for \-- If you\'re a customer out
in California, they all have externalities that they include in their
planning process. They all have renewable standards that you pay for in
their process.

A lot of these things \-- many people are worried that, if you go to a
competitive environment where all generation sectors appeared in the
marketplace, that these things will go away, that these incentives will
somehow not be procured in the market. So they\'re trying to look for
new ways to do this.

One of the most popular ones has been the renewable portfolio standard.
What that essentially says is that anybody that comes forward with a new
power plant is going to have to provide some percentage of that
electricity from renewables. They usually only include the
non-hydroelectric renewables. So hydro is not included.

Now they\'re going to run it as a credit system. So if you build a new
gas plant, you\'re not actually going to have to build a little wind
plant right next door. You can buy credits from somebody else who builds
\-- who chooses to build a wind or if you need to build biomass or
whatever type that\'s available.

Senator Jeffords\' bill \-- his standard \-- what\'s currently in the
bill now, even though he asked us to look at alternative, is that the
percent it grows from about two and a half percent in 2000 to 20 percent
in 2020. That\'s a percent of generation.

He later asked us to reduce that and have it only grow to ten percent in
2010, and stay there as a ten percent standard throughout. That\'s what
we analyzed. As I said, there are still a lot of the bills, and the
Administration\'s guidelines \-- I don\'t think they want to call it a
bill \-- that they released recently. They have a five and a half
percent RPS standard.

Senator Bumpers has one that reaches 12 percent. Sheaffer\'s bill is
four and a half percent. So there\'s several out there that do that.

So what I\'m going to walk through \-- In a minute I\'m going to walk
through what we saw when we imposed this standard and made the power
generation industry meet it.

The next slide is just to mention what the caps are. In the SO~2~ cap,
if anybody is familiar with the current standard, in 2010 you\'re going
to have to meet a 8.9 million ton standard. That\'s what was in the
Clean Air Act in 1990 Amendments. The standard that he\'s proposing is a
three and a half, 3.6 million ton in 2005.

He\'s proposing a CO~2~ standard. Don\'t get confused by the numbers,
because most of the other bills are talking carbon, not CO~2~. This is
CO~2~. It\'s a 1914 million tons. That\'s about the 1990 level for power
generation.

For NO~x~ he\'s proposing a 1.6 million ton standard at 2005. Today
they\'re a little under seven, maybe six to seven. There\'s a lot of
controversy about that, what the NO~x~ number is. I noticed earlier we
were talking about data quality, and Arthur brought up this issue of
these continuous emissions monitoring systems that belong to some of
these utilities.

In fact, with respect to NO~x~, they\'ve found out the historical
numbers were quite far off, that everybody had used some multipliers
that had been used for many years. When they actually put these systems
on the units, they\'re getting numbers that are significantly lower. So
that there is an issue with some of that.

Now with regard \-- I\'m going to walk through just a couple of cases. I
may even skip a few at the end, given that we\'ve got some time.

We looked at each one individually. We looked at just imposing the RPS.
We looked also at imposing the SO~2~ cap by itself, and then we put them
altogether. Originally, we also did a separate CO~2~ and separate other
ones, but we found out that when you put CO~2~ in, it so swamps
everything else there wasn\'t any reason to do an individual one.

So I\'m going to walk through now what it does to capacity and what it
does to generation for these different cases.

Now this first slide, when \-- I notice when you\'re back there, you
sure can\'t read what these headings are, and when I was sitting back
there, I couldn\'t do it. What this shows, if you look at the upper part
of the slide first, these are capacity additions by different fuel
types, the capacity impact, and what happens when you \-- over the next
\-- between now and 2020.

So the first set of bars are the coal and what we expect to happen to
coal in a reference case. You don\'t see very much. We see a little bit
of coal coming on back in the out-years, 2010, 2015. When gas prices
start to get a little bit higher, coal starts to become more
competitive.

The next set of bars are what\'s going on with gas. This is what we
expect with gas without any RPS. As you can see, gas dominates all
capacity additions over the next 20 years or so.

The next ones are renewables. The geothermal MSW, biomass, solar, and
wind. You get a little bit of wind, a little bit of MSW and biomass, not
much of anything else. This is not because \-- and I try to make this
point to people, that renewable prices have actually come down quite a
bit, but unfortunately, fossil prices have come down quite a bit more.

They\'ve been playing this what I like to call chicken limbo game for a
few years, and fossil keeps lowering the bar just as they get close. I
think that\'s made everybody a little annoyed, at least in the renewable
industry.

Now the lower panel shows the impact of imposing the ten percent RPS
standard. If you notice, gas still dominates the picture. Ten percent is
not going to do that much in terms of gas and coal. It does knock down
gas and coal a little bit.

You see the only ones that get significant gain on the very in, wind has
the biggest capacity growth, and then biomass also has a significant
amount.

Now this next set of slides is actually the \-- is the same structure,
except this, instead of capacity, is generation. The only real important
point to notice here a little bit differently is that, when you look at
the impact on the bottom of wind and biomass, where wind had a bigger
capacity impact, it picked up more of a new capacity.

Biomass actually has a slightly bit bigger generation impact, because it
runs as \-- As you know, wind plants only produce when the wind is
blowing. They don\'t run at very high capacity factors, in the order of
30 percent or so. Biomass plants, on the other hand, run at 75-80
percent capacity factor. So they actually produce more output.

Perhaps, you know, the most interesting thing \-- What impact does this
have on prices? The first thing is the blue bars represent what was
going on with prices in the reference case. In our AEO, if you\'re not
familiar with it, we have electricity prices falling about 20 percent
between now and 2020, and this is all \-- The biggest driver in this is
(1) coal prices are falling quite a big, but the main driver is that
just new power plants cost a lot less than the existing stock.

The existing stock of plants today \-- the average cost of the power
sector is about five cents of the total seven cent price, and new power
plants are coming in in the range of three cents. Over time, it\'s going
to move towards about \-- as these new plants come on line, you know.
Whether \-- you know, how fast it gets there is a big issue with what
states do with stranded costs or whether they recover them or not.

Now the red line is your price impact when you impose the RPS. Prices
still fall. They don\'t fall quite as rapidly. Instead of 20 percent
over the whole time frame, they fall about 17. They\'re about five
percent higher in 2020 than they were without the RPS.

Now I don\'t want to spend too much time on the bottom slide, but it was
just that we tried to look also not at a cost of service based industry,
but look at \-- we traded a credit system for the RPS credits and tried
to establish a price for them, and then include that in a competitively
priced industry.

We found out that it had about the same impact, a little more \-- a
little higher price impact when you saw the margin on cost as a credit
in the market than you did in a cost of service, but it was similar.

Well, this slide is just to show that not only electricity consumers are
going to see some impact, if you were to impose a renewable portfolio
standard. I don\'t know if several of you have been familiar. Several of
the gas associations have come out somewhat against these standards, and
they have some reason to be concerned, given that they\'re going to
dominate new additions over the next 20 years. Anything you impose to
bring on a new competitor is going to hurt them more than anyone else.

As it shows here, their revenues are about \$16 billion less in 2010
than they might otherwise be without the RPS standard.

Then the impact on CO~2~ emissions. It does have an impact. About 260-70
million metric tons of CO~2~. Again, don\'t get confused. This is CO~2~.
You have almost divided by four to get it into carbon. So it\'s on the
order of 60-70 million metric tons of carbon. It\'s not that big a
number, but it does have a significant impact when you displace coal and
gas with the wind and biomass.

Now we are treating biomass as a zero carbon. The cycle is continuous.
So you don\'t get any net carbon growth out of it.

This next slide: This one is just I wanted to show \-- I didn\'t want to
go through a whole litany of all the results when we impose just the
sulphur cap. We get \-- The big impacts are you get a lot of scrubbers.

Right now there\'s not very many scrubbers being added because of the
Clean Air Act amendments of 1990. There were something on the order of
between 15 and 20 gigawatts capacity added scrubbers. Some of them
probably wish they hadn\'t, given how low the SO~2~ allowance price is,
but they did anyway.

There\'s been a huge shift to western coal. If you impose the standard
in Senator Jeffords\' bill, you\'re going to get an even bigger shift to
western coal, and you\'re going to get additional scrubbers, upwards of
over 50 gigawatts of scrubbers out of capacity out there of about 300
gigawatts.

I could go through these next few slides, and I think maybe I won\'t.
What they are is they integrate a case where we put them altogether.
Since Andy and Ron just talked a lot about the CO~2~, the impact of a
CO~2~ cap and what that would do, the results are very similar.

The only key thing to note is that, when you do put the CO~2~ cap, in
addition to getting a large growth in renewables, you get a huge switch
out of coal and into gas, and that\'s the other factor that comes into
play here.

CHAIRMAN RELLES: You\'re not going to believe this, but there are enough
similarities of the next talk to the previous two that it makes sense to
insert that talk also in front of discussion.

So I\'d like to introduce Jay Hakes, who will talk about the work EIA is
doing for the House Science Committee, and then we\'ll have our 45
minutes to talk about all we\'ve seen in the last half-hour.

MR. HAKES: In the House of Representatives, the Speaker of the House has
designated the House Science Committee as the lead committee on climate
change issues, and that committee has been holding hearings on this
subject.

I think the committee would not object to the characterization that the
committee has demonstrated considerable skepticism towards the Kyoto
process.

I was invited to testify before that committee several months ago and
share some of our research, and the approach that we took was rather
than get into precise numbers, although those were available, we tried
to characterize in general terms what we thought the impacts of carbon
reduction would be and some of the dynamic factors that would come into
play, and it was based on the studies that you\'ve just heard. That\'s
why I think there\'s a lot of similarity.

As a process of that testimony, the Chairman asked me to \-- whether EIA
could assist in doing work on the price of carbon and the economic
impacts under a carbon reduction strategy, and we suggested that we
thought we could do that, although for reasons that have been explained,
this is a very tricky matter.

We do not claim that we can model international trading, for instance,
at least at this point in time. Our expertise is on the domestic
situation where we have the strongest tools. So we have tried to work in
such a way that we would be able to provide guidance within our area of
expertise without going out of our area of expertise.

Before I get into how we want to do that, I want to talk just a moment
about sort of the politics of it. I think this report will be more
visible than the reports that you\'ve just heard discussed in terms of
public discussion and attention in the Congress and, therefore, the
stakes are high in doing it right.

I think that we\'re off to a good start. Mary Hutzler, who, of course,
heads the office in which these programs are located, suggested to me
that it might be good to request the committee to send the request
signed by both the Chairman of the committee and the ranking minority
member. That way, you would have both parties sort of signed onto this
request, agreeing on sort of the ground rules for it.

I thought that was an excellent idea, although I at the time remember
being somewhat skeptical that this would actually happen, but in fact it
did happen. It happened very \-- seemed almost automatically. So the
work that we were doing from the committee is clearly authorized and
encouraged by both sides of the aisle.

I think this is an important step towards getting to what I was talking
about earlier, some common base of analysis and data.

Now how do we approach this problem of not knowing the impacts of
trading? Well, the idea would be that we would look at various levels of
reduction of carbon in the United States so that we could cover the
possible cases of what might happen elsewhere.

In other words, we would have a case that would show seven percent cut
from the 1990 levels in carbon emissions in the United States, and that
would show no impacts of carbon sinks, which would be reforestation. It
would show no impact of international trading.

That\'s not a scenario that is likely, but it\'s one that we can include
for analytic purposes.

Then there\'s another case that shows a three percent cut below the 1990
level, and that is based on some analysis that\'s going around about the
impacts of reforestation. In other words, the United States would get
credits from reforestation, which would offset the need to reduce carbon
emissions by that much.

Another case that we could look at would be the 1990 levels, which was
the original position in the United States before Kyoto. We could then
look at a case that would allow a nine percent above the 1990 levels
which is based on a State Department estimate of sinks and offsets plus
EIA\'s estimates of the permits available from Annex I countries, which
would be the countries of the former Soviet Union that are actually
estimated to be a million less carbon in 2010 than they were in 1990
because of the economic collapse in those countries.

We could also look at a case that would go 14 percent above the 1990
levels, which would basically mean that you would stabilize at the 1998
levels, or you could look at a case that was 24 percent above the 1990
levels, which would be consistent with the testimony of the Chairman of
the Council of Economic Advisors which suggested that that would be
compatible with international trading and still reaching the goal of the
seven percent decrease.

Then finally, we would look at a case where the emissions grew 34
percent above the 1990 level, and that\'s basically your baseline case.
That\'s what we expect under a \"business as usual.\"

So you would be able to portray this as sort of a menu of choices, and
then a person could take that and say, well, I think trading and sinks
will do this, and then they could identify where on the curve that fit,
and they could get a price for carbon, and they could get a price \-- or
they could get an estimate of economic impact.

Basically, for each case we would assume that the target would be met in
the 2008 to 2012 time period, and we would also express a number of
uncertainties. The ones we\'ve focused on so far are economic growth,
leakage of energy intensive industries, construction of new nuclear
plants, and technology costs, performance and penetration. These are
sensitivities that we would look at.

One of the things that I think is important and wills be helpful in this
process is if we are able to what I would call show the full elephant. I
mean, you\'ve heard 100 times the story of the blind men who examined
the elephant. One feels the trunk, and another the legs and the tails,
and they all describe this elephant as being something very different,
and yet there\'s one elephant.

Energy statistics is often this way. You can prove anything in energy,
depending on what your base year is, or you can take any analytic study
and extract some part of it and prove your point.

We will probably never figure a way to do that. You know, we could have
something that your fingers burn when you touch the data and try to
extract things selectively, but we will always be defeated. But I think
we can portray things in ways that make it more likely that people look
at the full elephant, and they get the good news, the bad news and not
just selectively.

So we\'ve sort of looked at the possibility of taking each of these
cases and then portraying \-- and I think you could do this on one page
\-- the following impacts: What this cap is in terms of tons. You could
then have what the price of a ton of carbon would be, and it\'s very
important to distinguish between the price of a ton of carbon and the
price of a ton of carbon dioxide.

I\'ve had people tell me, well, we can reduce carbon at such and such a
dollar range. Then I find out two weeks later they were talking about
carbon dioxide. A small point, but big difference.

Then I think you could portray the price of a gallon of gasoline. You
could then portray the price of electricity. You could then portray the
revenues potentially available for recycling, which is what Ron talked
about. That way you portray the good news and the bad news.

The bad news is the price of gasoline is going up. The price of
electricity is going up. The good news is you got some money available
for recycling, presumably.

You could then show the impacts on total GDP, and then you could also
show the rate on GDP growth. If you say, well, the economy is going to
be X billions of dollars smaller, that may sound like a big cut; but if
you look at it on an annual growth rate, maybe it doesn\'t look exactly
the same way. I think it\'s important to show it both ways.

I think another way of being helpful to customers would be to take that
exact same table and show it as a deviation from base. Put every value
at zero or at 100, and then show how it moves based on the different
variables. That way, you get some percentage sense of what\'s going on
in these various scenarios.

I\'m confident that EIA has a lot of tools and resources that are, if
not unique to us, at least I think we have a comparative advantage. An
example of that would be stock change. Some models that are very global
and look over very long periods of time do not look at stock change in a
very detailed manner.

Because 2010 is not that far away, I\'ve testified to the fact that I
think stock change is one of the decisive factors that has to be looked
at, and we have the ability to do that.

There are, however, huge challenges in all of this, which I think we
readily acknowledge. To just pick out one, one of the things that\'s
asked is how do you deal with innovation. In a market situation, history
has shown us that people often come up with solutions that you didn\'t
think of before people started looking for them.

There are innovative behaviors, probably less opportunity for them to
have huge impacts in a time frame is short as the one between now and
2010, but certainly, those possibilities are out there.

So I think, basically, what we\'re saying is the work that you\'ve seen
this morning is sort of moving on to the next stage. It has some of the
same analytic problems and challenges that have been referred to before.

We\'re looking at a completion date sometime in September, if we can
pull that off, and so that\'s sort of the time frame that we\'re at.
We\'re now doing a lot of thinking about this. I think we have
established a good framework in which to do it in terms of this
bipartisan request, but any assistance that you can provide to us on
these sticky intellectual issues will be very much appreciated.

CHAIRMAN RELLES: We have two final discussants for the morning session,
and then I\'ll throw it open to the floor. Cal, would you like to lead
off?

MR. KENT: I am well aware of two things: Number one, that we\'ve gone
all morning without a break. So if any of you all need to take one,
please go ahead. You won\'t miss anything of substance, let me assure
you. I always remember what we used to say when I was down in Texas, and
that is good manners have ruined more kidneys than bad whiskey. So
don\'t hesitate to leave if you feel necessary, and that applies to the
members of the committee as well.

The second thing is that, since we have been here today, the room has
gone through menopause, and I\'m not sure what is wrong with it.

I also recognize my place and my function on the committee, because when
our good chairman asked me to be a discussant, and here I\'m quoting
him, he indicated that we have many competent statisticians on the
committee; so we want you to talk about something you know about. So I
will absolutely take his charge to heart and proceed that way.

I want to focus my comments on two of the service reports, the first one
being the service report on the Electric System Public Benefits
Protection Act, and then the one on the analysis of the carbon
stabilization case.

Let me take these from a political standpoint, beginning with the
acknowledgment that, often when these reports are produced \-- In fact,
maybe I should say usually when these reports are produced \-- the only
thing that gets focused upon are the conclusions that are given in the
executive summaries, and this can be an extremely dangerous process,
because they are often treated by the requester, if he likes the results
or she likes the results, and by the press as being fact.

Therefore, policy reports of this nature, when they are done by EIA, are
extremely dangerous and should, therefore, be done and distributed with
utmost caution.

So let me begin by being somewhat offensive. That is, talking about the
analysis of S.687, the Electric Systems Public Benefits Protection Act.
This analysis was an extremely limited and a very narrow analysis.

In one sense it was good business, because I was told by Stu Leonard
when he was Entrepreneur of the Year, when he was asked what was his
rules for success in business, said there\'s only two: One, the customer
is always right; and number two, when the customer is wrong, reread
Rule 1. This is an example of that.

They did what the requester asked them to do, and by so doing, I think,
painted the wrong picture. The inquiry was extremely limited. It
provided no calculation of inputs by geographic regions, which I think
was a major mistake, even though it was not asked for.

It did not provide ranges on the estimated variables. It put no bounds
on uncertainty, and with the ranges that are there, they may be
sufficiently large to cast doubts on the conclusions that were reached
in the report.

It does not detail the impacts on industries, particularly the fuels
industries. It did indicate which would expand and which would shrink,
so far as usage was concerned, but it went no further than that.

It did not comment about the impacts upon end users. How many of the
energy intensive users will be affected or how they will be affected was
not detailed.

It did not provide a financial analysis of the impact on the electric
generating companies. Could they possibly survive what was being
proposed here \-- a question that should have been asked and answered.

It does not provide an analysis of the impact on reliability of the
electric generation system, which I think is the foremost question that
is being faced and, I think, often is being ignored in government
circles today.

It provides no overall macro economic analysis as to what the impacts
will be on the economy if this particular proposal were adopted, and it
provides no analysis of the technological feasibility of implementing
what has been proposed in this bill.

What I\'m saying then is that the report, I think, because it is
incomplete, and even though it may have been what is requested, is one
of those dangerous reports that can be used to give the public a
misleading and a very incomplete analysis of the situation.

The good news report is the one, the analysis of the carbon
stabilization cases, which does not have the problems that I mentioned
in the other report, but here there is a different set of political
problems, and those political problems stem from the assumptions that
had to be included and the way that the report was drafted.

I know that these came from another agency, and they are mentioned in
detail, fortunately, in the report; but let me just mention to you what
some of them were.

They were going to reduce the price elasticity for all end use services
to approximately zero; that we will reduce the hurdle rate for energy
investment by as much as 45 percent; that commercial shell efficiency
for buildings improves by 20 percent. We will assume that E-85, which is
a gasoline 85 percent ethanol, 15 percent conventional gasoline, will be
competitive. We will assume that new generation vehicles will capture 20
percent of the market.

There will be a 50 percent reduction in marginal technology costs, a 33
percent improvement in marginal fuel efficiency, and that after 2015
nuclear power or, as Ron said, some other carbon free generating
technology will be economically competitive.

We will assume that electric providers will have full prior knowledge of
future carbon permit values, and we will assume no deregulation of the
electric utility industry.

I\'m willing to submit to you that those are absurd, most of them
individually and, certainly, all of them collectively. Now what does
that mean?

It means you did again what the customer asked you to do, and on the
basis of that, you came up with some very good and well documented
conclusions. But the politics of this simply are that, if you make
assumptions like that, it is inevitable that you will get the
conclusions that the people wanted from the beginning.

That is where the problem comes in, in being a client agency, in doing
what people ask you. I know when we were putting the NEMS together, and
I see that Bill Skinner has since left, but I remember those days when
we were trying to convince everyone, Mary, that NEMS was a good idea,
and we said bring us your assumptions, and we will run you the programs
and give you the answers.

I\'m not sure that we should have sold the program as well as,
obviously, we did. So what I\'m saying here is that, if you are going to
use the EIA base case but then you are going to make major changes in
assumptions in that base case, which was done here, that at least what
should happen is the base case should be compared to the assumptions
case first, so you see how money changes or the changes that result just
from the changes in the assumptions.

Then you can run the policy case, and that will give you some indication
of what the policy effect is going to be. The problem, of course, is
that these policy reports are being used. The decision makers will use
the conclusions without giving all of the caveats that are there and,
certainly, without detailing the reasonability of the assumptions.

What happens is that bad policy may be promoted and ultimately inflicted
on the unsuspecting populous.

I think that what I am suggesting here, and I\'m glad to see that it has
already begun, and that is that EIA now needs a gatekeeper on its
service reports. I don\'t know who that gatekeeper ought to be. I think
it\'s a step in the right direction to have the Chair of the committee
and the ranking minority member sign off, but I do think there needs to
be a better gatekeeping function.

That would be my primary commentary on what needs to be done as EIA
continues to do what it should do, and that is to serve the needs of the
Congress in specific and the public in general, but it needs to do so in
a way that the public is truly well informed and Congress is truly well
served.

MR. HANSER: You\'ll have to excuse me. I\'m not nearly so eloquent a
speaker as Cal Kent. I\'m going to try to do something that \--

MR. KENT: But you are a competent statistician.

MR. HANSER: And I\'m going to try and do something which I probably
shouldn\'t. I\'m basically a pocket protector carrying geek, but I\'m
going to try to be a little bit more strategic today. It probably is a
mistake, but we\'ll give it a shot.

The first thing is I think people fundamentally misunderstand
electricity industry restructuring. I guess I saw Janet Yellin saying
wonderful savings that were going to be produced from electric industry
restructuring. We\'re basically going to pay for the carbon emissions
reductions that are going on in the future.

That\'s ludicrous, and I apologize to the Administration, but it\'s a
stupid position, and it has absolutely no basis in fact or modeling or
anything else. It was silly for the Chairman of the Counsel of Economic
Advisors \-- I\'m also a card carrying economist \-- to put forward
something like that without a very detailed analysis. It sort of
doesn\'t speak well for the profession when they start to do things like
that, but maybe that\'s the nature of politics.

The electric industry restructuring is going to bring about all kinds of
things, and it will probably lower the long run cost of generation in
the end; and it may, you know, promote the introduction of more cost
effective generation technologies fairly quickly, but the simple fact is
that the embedded costs of transmission and distribution are simply not
going to go away. They\'re going to stay there practically, no matter
what, and you will continue to get a bill that will have that in there.

What you will see in the long run for some customers is substantial
reduction in the total cost of energy they pay, if they happen to be
very electrically intensive. For the rest of us, we\'ll probably not see
huge differences.

I just received my bill from Boston Edison, and my bill was broken out
by generation, transmission distribution and competitive transition
charge or whatever it was called. The costs for generation on a bill of
about \$90 is about \$12.

That means that, if somebody runs around to me and says how would you
like to get a ten percent reduction in the cost of energy. My bill
reduction on a monthly basis would be about \$1.20. Okay? That\'s the
kind of thing you\'re going to see as a residential customer.

On the other hand, if I\'m an industrial customer and I may be consuming
50,000 kilowatts in a month, it\'s another story, and that\'s where
those benefits are going to happen, and that\'s where the real impact of
electricity restructuring is going to occur.

In terms of reliability, I\'m up in the air in terms of what the impact
will be on reliability. There\'s a potential for some product
differentiation to produce different levels of reliability possibly, not
have a uniform level, and that may in turn reduce the overall cost of
providing electricity, just like almost any form of product
differentiation tends to concentrate the costs for those who want the
higher quality product on the people who are willing to pay for it.

Regional differences are important. Frankly, I think they\'re largely a
function of vested political interests. I listened to at the NEMS
conference about somebody from Washington talking about how they didn\'t
worry about electricity restructuring and weren\'t concerned about it.
By the way, they also happened to receive some of the largest income
transfers that I know of from the tax paying American public that arise
anywhere in the country in the form of receiving very cheap power from
BPA, which is all, of course, funded by the American public and whose
benefits primarily flow to people in Oregon and Washington, but we
don\'t care about restructuring.

So I think, you know, in terms of electric industry restructuring, there
aren\'t that many things that are going to happen that people are going
to see in the long run except for the nature of the way the generation
business is going to be done, and even there we\'re really talking
about, in terms of things like greenhouse gases, not major changes in
terms of what would happen, even in the absence of restructuring.

I guess what I would like to sort of suggest is \-- and maybe it\'s
because I\'m old. I remember all the modeling that was done back in \'78
around the energy crisis, and I remember somebody named Martin
Greenberger, who probably nobody remembers anymore, but who wrote a
wonderful book called \"Models in the Policy Process.\"

Ah, some people are shaking their heads. To me, what\'s important is
sort of the view from, you know, 100,000 feet more than it\'s, you know,
getting into the guts of the models. I need to understand what\'s really
crucial and that drives the models.

Back in the time of the late seventies, nobody took account of, for
example, price elasticities. Everybody sort of had these models in which
\-- They were fixed proportion models, as we like to call them as
economists. You just keep on using energy, no matter what, no matter
what happens to the price.

I guess what I would like to see the EIA do possibly is think about,
when it produces these reports, making clear what are those things that
are fundamentally driving. For example, the AEO \'98 forecast has got a
huge penetration of natural gas over the next 20 years.

Nobody has even computed whether or not it\'s possible to put in all the
pipeline that\'s being suggested by this penetration of natural gas, you
know. Can you even manufacture the amount of pipeline that\'s required
and get it into place? That seems to me a kind of simple thing.

It might help \-- and I don\'t know whether this is a possibility \-- to
bring in outside reviewers on some of these things, you know, folks who
\-- I remember at one point in time David Friedman came in and provided
some of more stringent criticisms I\'ve ever seen of any of these
models.

This area is so important in some ways that I almost feel like it needs
to be subjected to the same kind of scrutiny that somebody publishing
in, say, [The Journal of the American Medical Association]{.underline}
about the impact of some new drug that prevents cancer ought to get;
because in a sense, the implications are just as serious.

In fact, in some ways you could argue that \-- and it has been argued
\-- that the economic implications has lots of health implications, and
so to the extent that you produce a policy which in the end impoverishes
somebody and that in turn has health implications, you might even think
about it from that standpoint, because some of these things are so
important.

I think that in general the EIA is caught in a very \-- in a kind of a
bind, as Calvin has said. It\'s producing models, and it\'s producing
projections. Perhaps what it needs to do is send the models out of
house. That is to say, it ought to be the case that somebody else needs
also to be able to run these models just as well as the EIA does. Only
there can you guaranty, in fact, the degree of model transparency, I
think, that comes from having others looking at the model and evaluating
it.

I think this is important, for example, that, you know, David Montgomery
at CRA run the model or the MIT lab, which is looking at global
stabilization, carbon stabilization, look at those models and run them,
as well as for the EIA to do so.

These are \-- In a sense, they are public models, and the outputs of the
models ought to be made public, and the models themselves need to be
made public as well.

So I guess what I\'m sort of trying to get to is to say I don\'t think
there\'s a way to avoid EIA being involved in this debate, but in order
to make the debate informative, it needs to make clear those things
which are fundamental drivers in terms of the way the models are going;
and it needs to do so not at the level of telling me what\'s going on in
equations, but in terms of the general perspective the things that most
drive the models.

I think it needs to move towards model transparency, perhaps by
exporting the models to others. Then finally, I think it needs to
seriously think about having outside reviewers coming in and either
reviewing the reports of the models themselves as a means of ensuring
quality control.

I don\'t think it\'s sufficient at this point to simply run the models,
produce the results, and then send them out and let them sit in the
American public. I think at this point in time these things are so
important it\'s going to require somebody who has expertise in these
areas to come together and review these things before they\'re
published.

So those are my comments.

CHAIRMAN RELLES: Additional comments from the committee? Jim?

MR. HAMMITT: I think it\'s interesting, we\'ve gotten on the topic here
of how to evaluate these models. My understanding is a lot of \-- is
using NEMS in various forms, and for a lot of these questions there are
so many interactive factors that the only way to address them is with
some kind of relatively complex model often.

I know many of the EIA people I\'ve met in the past through the Energy
Modeling Forum, which is a group run out of Stanford University, are
people with energy models, and to some extent, climate policy models,
get together and try and run their models on common scenarios,
standardize some inputs, see the variety of outputs, the result, and
talk to each other in a fairly detailed level to understand why you get
different results from different models and what the strengths and
weaknesses of those are.

I\'m not sure I have a whole lot to say about this. There are,
obviously, many ways one can go about trying to evaluate how good a
model is, most of which or all of which are hard.

If EIA is \-- I\'m somewhat new to this area, but if NEMS really is to
be the focus of a lot of the agency\'s work and output, you can think of
it, as you have, a long term investment in developing this model,
understanding how it works, how it can be done better. You might have
sort of a portfolio of models. In the climate world, integrated
assessment world, Jay Edmunds at Battelle has one of the well known big
integrated assessment models, and he explicitly has sort of \-- one
thing, a mini-CAM climate assessment model.

So he has a very simplified reduced form thing that\'s accessible to
many people to use, and there\'s a productive interaction between having
very simple models or more simple models where you can more readily
determine what the major factors are, what their implications are, but
you leave out a lot of important stuff, and then a much more complicated
model like NEMS where you can incorporate these additional details, and
you get some productive interchange between them.

I\'m not sure whether you guys are still participating in the Energy
Modeling Forum activities, but that\'s one activity. There was comments
about MARCAL was used for some of this analysis and NEMS for others. I
assume there\'s interchange between the people working in those
different models.

MR. ELLERMAN: This is not the point that you still owe me, but \-- I
want to say I do think this is a real problem. I thought Cal was a
little hard on the reports, but I think still there is a real problem
here.

I always have in mind the sort of analysis done back in the eighties on
the acid rain debate where at some point someone funded ICF to explain
how could they do a report that said this, that it would cost virtually
nothing, and how could they also have another report that it cost a lot.

You know, they were quite able to explain it. I mean, it was actually a
very useful piece of, you know, commissioned work, but I think it comes
back to the same point, that the models \-- that there is a real
problem.

You can think as a consultant, and anyone who has dealt in these sort of
things, knows the problems. The solutions here \-- I\'m not sure that
the issue is model transparency, because I think that tends to be mind
numbing detail, and you can do a lot of that, and I\'m not sure that it
helps the basic function, which is sort of informing the public policy
process.

It seems to me, you\'re stuck with either \-- you know, two aspects. One
is that you have almost a religious fixation on a two-handed approach,
so that you refuse to yield to the political demand of \-- for the
one-armed economist, you know, will come down to some hard conclusion,
and you insist on that; but I also think this idea of peer review has
some merit.

I think it is the report \-- I think that EIA has to issue reports and
go beyond the mere provision of data and forecast to where you can
address a lot of these issues, and I think you do a relatively good job
at that, but you also get these sort of demands, and I think the sort of
client attitude, although there\'s a certain merit in that, it can
easily go too far and, I think, damage the basic mission of the agency.

One of the ways to guard against that may be something analogous that I
just think is like peer review in academic articles. I mean, you can do
work that will support one point of view and will certainly have, you
know, let\'s say, some political implication, but in going through peer
review, if you don\'t acknowledge that there are other arguments, this
is not addressed, this or that \-- I mean, it doesn\'t get published.

I think there\'s some sense that that\'s sort of what \-- some mechanism
that would sort of vet these reports may be something worth thinking
about.

MR. KENT: Let me make one thing clear, that my comments about the second
report were not focused on the model or on the use of the model, but
were focused on the assumptions that took us away from the AEO base
case. That\'s where I think we start getting into really big trouble on
these service reports.

Maybe I didn\'t make that point clear enough at the beginning of my
comments, that that\'s where I think you get into trouble; because if
you change the assumptions, you\'re going to get the answers that you
want at the end; and given the assumptions \-- and I don\'t know what\'s
going on with the request that came from the House committee, but I
think that that should be basically an assumption-less request from
them, that they shouldn\'t be saying change this, change that, change
this, change that, so that either the Republicans or the Democrats get
something that they can use to beat the other one over the head with,
which is what I really think they want.

Let me point out, this is a nonpartisan comment, because, certainly,
most of you remember back when we did the first national energy
strategy, back in the 1991-92, and all of the things that the
Republicans insisted that we had to put in as the assumptions, and some
of these were just very farfetched as well, and yet we based a national
energy strategy upon them.

I think that that\'s where a lot of the focus has to be, is on this
whole area of the assumptions, because the model can work fine, and
there won\'t be any problem with the model, but if the assumptions are
unrealistic \-- such as just assuming you\'re going to come up with
something that\'s going to be like nuclear power but not nuclear power
\-- then, you know, the whole process collapses.

CHAIRMAN RELLES: Would the presenters care to respond to some of these
comments, please?

MR. KYDES: Since, Dr. Kent, you talked about the analysis report and
found the assumptions absurd, largely \-- I think that was your word or
entirely absurd. I don\'t remember which of the two it was. Actually, I
think that \-- I don\'t think you\'re really running or suggesting that
we not necessarily provide the service function to the Department or
anyone else legitimate within the government, but rather that what we do
is add additional sort of \-- I think the word \-- I\'m not sure the
word used, but I\'ll paraphrase it \-- basically to show the results of
their assumptions against the base without the targets, for example,
just to show how much impact each of these assumptions have on the
baseline.

So it\'s simply adding a dimension of information to it to simply show
that, yes, when you loosen the system and you apply it with the rest of
the reference case assumptions, you get either a very small or very
large divergence, and maybe that explains a large part of what\'s going
on and how important these assumptions are.

MR. KENT: That was my suggestion.

MR. KYDES: Yeah. And I think that message is gotten, is a worthwhile
one. In fact, we are doing that, from that perspective, on the new work
for Congress. So \-- as part of the process.

The issue, I guess \-- By the way, there were some assumptions that
initially we thought were absurd also, but in retrospect they were not
entirely absurd. I think nuclear life extension, for example, is an
issue where we wouldn\'t have \-- At that point, we fought it, but we
lost. Now when we do the analysis, and we\'ve been doing it for a month
or so, we find that, in fact, the costs are at \$100 to \$200 a ton
carbon, that life extension is a very plausible thing for about 95
percent of the plants.

So, in fact, that\'s a case where we were opposed to a position, but
found, in fact, the sponsor was right, at least in large part. So we\'ve
done things like that.

Let me just make a couple of comments here from Phil. I do agree that we
need to do a better job at informing in the analysis that we do what
really drives the model results. I think that\'s your main point here,
at least one of them anyway.

That\'s \-- and if it\'s the one that I remember, it goes to one I
accept. But let me just \-- you know, the issue of, you know, whether
we\'ve done \-- whether we do these sanity checks either within the
reference case, whether \-- for example, a case you used, part-time
expansion. You know, is this a feasible activity. Is this historically
\-- I mean, I know it\'s an example.

We do those kinds of things, in fact, particularly since my background
particularly is gas; and having worked in the gas industry, that
particular one is something that we do look at, and we do make sure
that, in fact, it does make sense.

I\'m not sure that we do it in every area, but I can tell you that we do
do it. We do have experts. I know Alan does that, for example, related
to the electricity side; but we still, I think, would find valuable the
outside reviewer concept as well, and it passes the laugh test.

I just have those few comments to make. There are \-- I think, from the
point of view of whether we do these kind of service reports or not or
put a yellow cover, a blue cover or some other flag on it that says this
is not our result or whether we don\'t do these things at all, I think,
is a policy issue that I can\'t address, I think. It has to be addressed
at a different level.

MR. EARLEY: I\'ve got a couple of comments that I would like to make
back. Let me take them in fairly rapid fire order.

When Dr. Kent was here and in charge of EIA, we had the NES. Policy came
down and said we will have 2.9 percent GDP growth in the economy, and
that came straight from the Counsel of Economic Advisors, and that\'s
what they wanted to National Energy Strategy to be based on.

Cal came to us, said, Ron, what am I to do. Let me give you one piece of
advice. Don\'t go above 2.4 for the AEO. Okay? So best of both worlds.

Normally, we do a macro case. We have a central macro case. We have a
high and a low economic growth. You go back to all your AEOs. There\'s
one AEO out there that\'s only got two cases in it. Okay? There\'s the
reference case. He stuck to 2.4, because he believed us. Okay?

Then we gave policy their 2.9 case, and then we reported both of those
in the AEO, and then they ran with the \-- I mean with the NES 2.9
percent growth for all of the policy analysis, and they backed
themselves into a corner, because they had oil imports coming out the
wazoo.

I mean, you know, they were having big problems by the time they got
done. Okay? But as EIA, we gave the client what he needed, what he
absolutely said he wanted \-- okay? \-- and then we stuck to our guns
basically for the baseline AEO. So that\'s point number one.

Point number two: For the past ten or so years, I\'ve been the project
manager for the DOE project manager for the energy modeling. Very much
aware, we are routinely involved in working with them. We are not
directly model participants in each and every study that goes on, but we
have been very much involved in the electricity restructuring study.

We have been involved before in climate change, things of that nature.
We recently had John Ryan, Rich Richalls and Jay Edmunds come down and
have a little chat with us about some issues, as we\'re trying to
struggle through some of this carbon stuff.

So, yeah, we take advantage of the energy modeling forum, and we do take
opportunities to put the NEMS system out there for people to take their
shots at it. So maybe you\'d like people to take more shots. That\'s
fine. And if you want us to have more peer review, good suggestion.

I don\'t know if I \-- Can I suggest the names of the people? Okay. We
have contacted Bill Nordhouse. Help me with the other two names \-- John
Wyatt and Mike Tomei at RFF to help us review some of the work that we
are currently doing on the carbon report. So \--

Okay, third point: Dealing with clients at the front end of the study is
not easy. You got to negotiate what the assumptions are. Sometimes they
will flat come out and tell you what they are.

A lot of times it\'s a negotiated settlement. If they come in and they
give you something blatantly absurd, we say, hey, that\'s pretty screwy,
you know; but sometimes they may come in and say, that\'s what we want,
we want to evaluate that. Okay. Do you have a problem? Yeah.

I didn\'t put this up. I gave you kind of a bottom line answer for a
graphical presentation purposes, which basically showed you the impact
on industry. I showed you one bar.

In your chart, you basically have this. We played some games, some
analytical games. Basically, the real teeny tiny, you know, bars that
don\'t show up very well because they\'re both close to the access and
they\'re not presentation quality graphics, basically are runs based
without any of this leakage thrown into the pot.

We basically negotiated with them up front and say, this is a really
important issue, and if you guys don\'t go forth, you know, with some
information about what the impact of that particular assumption is,
you\'re going to look funny.

So this is one of these things where you negotiate with the client in
advance on things. Okay? This isn\'t as though, you know, a set of
assumptions come in and said, bam, that\'s it. You do negotiate, but on
a large number of those assumptions, you know, you accept what the
client wants \-- Okay? \-- within reason.

Now, you know, it\'s a negotiated settlement. I\'ll let Andy, you know,
or anybody else who wants to talk further about that, but I know kind of
what happened in a macro area, you know, side effects.

Now if you think the assumptions that basically the policy office laid
on us to do our analysis were straining at the leash, you should have
seen some of the inner workings of the IAT process and the battles that
occurred in the rooms. I mean, it pales in comparison to what finally
came out in our service report that was independently done to report out
our work in support of the IAT process.

So I hope that wasn\'t too defensive, but you know, it\'s got to come
back.

MR. HANSER: Can I ask a question, just on the IAT process. Just take one
minute.

I have friends of mine who work in the Federal government also, and they
describe something that\'s nothing less than internecine warfare around
some of these issues, and in particular the Environmental Protection
Agency has been known at times to utilize every potential rule that it
can for holding a meeting to ensure that its viewpoint is somehow put
forward or maintained in the way the process goes.

I have a question just about, you know, the nature of the management of
that process for EIA, which is attempting to be even-handed when, in
fact, there are agencies who are not attempting to be even-handed at
all, and whether that\'s an issue that, you know, is just \-- you have
very clever fellows who work at EIA and who know how to deal with this,
or whether there needs to be something else done to sort of deal with
this; because the way it\'s been described to me is it\'s troubling \--
let\'s put it that way \-- not so much just as a member of the ASA, but
just as a member of the American public. Okay?

So either of you can respond.

MR. EARLEY: Let me take a first shot at it. I\'ve been involved in a
number of interagency efforts and, quite frankly, the way in which a lot
of them work is you get \-- you know, you get behind closed doors, and
you know, you\'re not supposed to go public with everything. Okay?

I mean, part of laying it all out on the line is getting in there, being
open and honest and letting battles, you know, ensue. Okay? So all of
that is fine.

I\'m not going to suggest that it\'s all the EPA or anything like that.
I mean, you know, because EPA has it\'s own, you know, internal
conflicts, as even within the Department there are differences; but we
took to heart very \-- I mean, I did, and I know Andy did.

When we got in the room, we were advisory. We were trying to do the best
thing we could to shed light on the issue. We knew that we were not
there to write the IAT report, but we were brought in on the basis of
our expertise, fundamentally, on the energy markets and on the macro
side, and they took to heart what we basically told them in the internal
process in that.

Now the final report coming out from the IAT \-- well, one, there was
not final report. There was a draft of, you know, a document that was
leaked, officially leaked. The fact is we were advisory, and we were in
there. We were doing our job, you know, advising them on how to go about
certain things.

We were not there to write their report for them in any way, shape or
form. We were supporting the policy office in their dealing with the
rest of the IAT membership, but we tried to keep that acutely in our
minds throughout the entire process, knowing that we were going to be
putting out our own report, EIA report, and that\'s the document that
you see there, which was not sanctioned or reviewed by the IAT
membership.

MR. KYDES: I know you want to answer the questions, but let me just tell
you that, in fact, the results that we came out with, with this report,
did not make the IAT happy. In fact, it made them fairly unhappy.

If you had seen what the position that was being pushed within the IAT,
it was considerably more, let\'s say, flexible in terms of what carbon
emissions and what the economic impacts would be.

So in fact, as it turns out, the position and the analysis that we did
was, in fact, far more consistent with what the Office of Policy and
International Affairs that ran with a totally different model than it
was with the DRI and the EPA and second generation model.

MR. HAMMITT: First off, let me say I sympathize with the position
you\'re in. You know, I\'m sure we all understand. We\'re not trying to
attack you and any of that. I\'m here, because I like doing models. I
think they\'re important and very informative, and there are a lot of
tensions trying to use them in a public policy arena, and you guys are
right in the thick of it.

So let me throw in things that I\'d like to see done there, and I know
there are constraints trying to do some of this.

One point is, you know, in the political process one\'s numbers point
estimates, and everybody knows the aphorism about model for insights,
not for numbers, and this idea that there is a reference case set of
assumptions and these are the right assumptions and some other
assumptions are wrong is, obviously, oversimplified.

We should have kind of a range of reasonable input assumptions, and
implicit \-- you know, ideally, when this can be done, some
probabilistic distribution of outputs resulting from probabilistic
distributions on inputs, and in a complicated large model that\'s
extremely hard to do. But that\'s kind of one ideal, and then getting
the politicians to grapple with that in a reasonable way is another big
stretch.

So you don\'t want to be in the position of people being able to say,
you know, the effect of this policy is exactly there\'s 2.4 percent
change in whatever. You want to show over the whole range of
assumptions, this policy is likely to decrease coal use by, you know,
ten to 50 percent or things of that. Get away from exact numbers to the
extent that can be done.

Scenario analysis is a step that way, a good step. It\'s, you know,
often done and people don\'t really know how to interpret it. You give
like a reference, a low and a high case, and people sort of think
that\'s implicitly bounding the range of the likely, but it isn\'t
necessarily.

Probably many people have seen some work done by a colleague of mine, Al
Schlackter and others. One thing he took was an annual energy outlook
forecast, constructed in \'84, \'87 and some other year, all for the
year 1990. So he was able to get the realization in 1990 and see how
often did the realization fall between the low and the high cases.

Not a whole lot of times.

There are a whole lot of cases where the realization is many sort of
interval widths away from the low and high cases. You all know many,
many reasons that might be true.

The main point is these kinds of scenarios are interpreted as some kind
of an uncertainty range, but it\'s not clear whether they\'re intended
that way. They\'re certainly very much conditioned on a whole lot of
factors that people tend to not think about when they\'re looking at
those scenarios.

Then on the point about peer review type things, that\'s good, but I
have, and many of us, have reviewed kind of complicated model things
submitted for publication, and within a report 20 pages or 100 pages
it\'s not really possible to provide enough information for an outsider
to really critically evaluate that.

So I think you need to be looking toward a much longer term building
comparison with other models and things, in addition to just simple
review documents.

CHAIRMAN RELLES: It is hard to say something that hasn\'t already been
covered, but let me try this myself.

You come to EIA and you see this tremendous amount of pride in NEMS, a
ten-year investment. It does everything. It\'s got a few problems.

For example, it requires EIA staff to run it, and maybe it takes a
couple of hours to do a \"what if,\" which maybe by the time you get the
\"what if\" result, you\'ve forgot what the \"if\" was. Those are
serious problems that, I think, will one day erode NEMS\' position as
sort of the premier modeling tool into kind of an outmoded, hard to use
tool that the private sector has come in and replaced.

If you think about your customers for the moment, you\'ve got two
customers, really. You\'ve got the legislative and executive branches
that want to get answers out, and then you\'ve got the scientific
community. The requirements of the two customers are entirely different.

The first set of customers just wants to answer and has \-- even knows
what that answer will be before he asks you. The scientific community
wants its standards to be adhered to, which include all of the issues
raised until now, the uncertainty, the sensitivities, and so forth.

I think it\'s important to note that NEMS \-- Well, certainly, I don\'t
think NEMS is going to be accepted by the scientific community if it
remains static. It may not even be accepted by the political community
if you start charging for it, because as I understand it right now,
you\'re doing all these analyses for free. So why not? But in a more
realistic world where people pay the cost of what they ask for, I could
even see that the political sides that are now coming to you for NEMS
are going to opt for the private sector modeling fora that gain
scientific acceptance and that they can actually manage themselves.

So I guess my feeling is that, if EIA takes a position that NEMS is
super and what we need is a do everything model and we\'ll wait until
faster machines come along and maybe it will run in a half-hour, I fear
that the popularity and importance of NEMS will continue to \-- will
erode and, of course, the agency\'s reputation with it as it continues
to hold onto to NEMS as its only tool, and all of the competition begins
to spring up.

There\'s been a lot of very good suggestions. I think the one I like the
best is to get some of these easier to use models in-house or some of
these smaller models in-house, both to experiment with them so that you
can see what alternative features you might want to put into NEMS that
these already have, and so at the same time when you issue your analysis
for the House Science Committee, you can sort of buttress your arguments
by some kind of comprehensive discussion of what these other models seem
to be saying.

My guess is you\'ll see some interesting situations where model A, B or
C says something entirely different, and it\'s going to force you to go
back and think about your own assumptions, and either bolster your
arguments that you did it right or perhaps even change your assumptions,
because there was kind of an oh, yeah, realization. The model suggested
something I hadn\'t thought of before.

MR. BISCHOFF: I can remember back in the 1970s there were a lot of big
macro models of the economy, and they oversold themselves. Most of them
went out of existence. So DRI and a couple of others are really the only
ones that still remain.

I\'d be worried, as other people said, whether NEMS by not having a lot
of information on accuracy may be overselling itself, too, and making
the EIA vulnerable for attacks. In particular, I\'ve been on this
committee five years now, and the only real analysis of errors \-- Four
or five years ago we were talking about big error analyses, and then, of
course, the staff was cut, and the EIA couldn\'t afford to do it.

Therefore, it was never done. Only in the issues of mid-term analysis in
forecasting 1997 did I ever see some of this, and I have been arguing
for years and would like to continue to argue until I leave the
committee for something between the 18 volumes of analysis that stand up
this high, the 18 volumes of documentation on the model, and then the
one little 60 page booklet that summarizes the NEMS, something on the
200 page level that tells something, including something about
forecasting accuracy and error analysis and uncertainty analysis that
could be used to back up the annual energy outlooks and the
international energy outlooks and the various base cases and risk cases
and so forth.

CHAIRMAN RELLES: Denny. I would like to propose that this be the last
comment, because we are a little bit over. My apologies to everyone in
the room.

MR. ELLERMAN: I wanted to raise a point, move off from modeling and the
sort of service report aspect and go back to some of the first
presentations and issues about data and all the things that have been
said about inventories.

I think, of course, all the interest now very much is in the modeling,
the impacts of Kyoto, you know, whatever might happen there. But I think
there\'s an aspect that EIA ought to be thinking about as well, as we
think about the data requirements of any type of effective action which
is going to be taken. I think it comes back to measurement, and it comes
back to data.

Whether emissions trading, I think one could argue, is going to be a
feature of whatever type of regime might occur, and that may be a big
if, but let\'s say that\'s going to be one of the major issues, and we
know much of the analysis now and how much difference this trading makes
\-- and there are a host of issues about allocating permits and, you
know, sorts of a property rights aspects or do you have an emission
credit scheme versus allowance type scheme.

All of these rest upon the notion that you can measure the emissions,
and I think in the end it also gets down \-- any of these allocation
schemes come down to where you can \-- how do you verify, and where do
you measure for this?

I mean, we have a model in SO~2~ or, you know, we do have monitors on
the stacks at power plants, and everybody can think of that very easily.
But once you get into CO~2~, it\'s clear you\'re not going to measure at
the end point.

This issue, as many of you know, already came up in the pre-run to
Kyoto, but the question is where \-- If you were going to impose a cap
system or some sort of allowance type system for CO~2~, where would you
measure? I think that is preeminently a technical issue.

I mean, there is a whole political aspect of who owns these rights,
who\'s going to get the rents that get created by such a constraint and
so on, but that in a subtle way also depends an awful lot about where
you can measure and how you would verify and how you would establish
that sort of, let\'s say, right, if you want to put it in those terms.

It comes back to this issue of measurement and verification, which is
going to undergird any regime.

I guess I would make a plea that some small effort at EIA in this be
focused on the issue of how would you measure, where would you verify. I
mean, if things do go down the path, however going more into devising
such a regime, those sort of very technical issues become more and more
important, and it would be well worth having had people who have thought
about it for sometime and who could be viewed as objective and have
thought through all the issues rather than dealing with it at the
immediate moment when it suddenly becomes evident that it\'s going to
have to be done somewhere.

CHAIRMAN RELLES: I must say, I thought this was a super session this
morning. We all gave up our break for it, for which I apologizes. I will
make up for it by having an extra break this afternoon.

Thank you for coming, and we will have our \-- resume in this room in
about an hour.

(Whereupon, the foregoing matter went off the record at 12:32 p.m.)

**A F T E R N O O N S E S S I O N**

Time: 1:47 p.m.

CHAIRMAN RELLES: Welcome to the afternoon session of the ASA Energy
Statistics Committee.

I\'d like to introduce Doug Hale from EIA, who is going to be giving us
an update on the reporting he gave us last time on electricity
restructuring.

MR. HALE: First slide. At last November\'s meeting I discussed two
research projects with you. The first was a project in experimental
economics to characterize the effect of market structure on network
economics. Basically, are there ways you can rig a bidding system, a
plausible way, so that you can exploit the network in order to get
noncompetitor prices?

The second project was to build a model with NEPOOL to quantify some of
the network effects, congestion, line losses, that sort of thing.

The reason for doing all these projects is that our existing model, the
energy market module, was built and constructed basically to capture a
regulated industry, a regulated industry delivering a single product,
delivered electricity.

In that model there is no transmission system. There is no reactive
power, voltage support. There is no real problem with distribution.
Pricing was simply to set price equal to the average total cost.

In competitive versions of the same model, you simply put the price in
as marginal cost. So it was a highly \-- in a sense, highly aggregated
model that abstracted from important elements of the system,
transportation and distribution in particular, but elements that
weren\'t so important in the past because the integrated utility managed
those pieces, and they didn\'t consume much money. They weren\'t very
expensive compared to generation, which is what we focused on.

With the restructuring, the economic literature, to a large extent,
focused on two major issue categories. One category was the emergence of
noncompetitive markets in a restructured environment. The second had to
do with the network effects brought about by the transmission system,
this real concern with congestion and line losses, and another cord
having to do with voltage support, and we\'ll get to that more in a
minute.

So the whole purpose of this analysis was to say \-- to look at one
region of the country and see if these effects that were dominating the
literature were so important that we should adjust our forecasting
model. This was never intended to be a prototype for building a network
model of the eastern seaboard to do forecasting. It wasn\'t that at all.

So those were the two projects I presented to the committee. The next
slide summarizes some of the committee\'s reactions.

In the area of the experimental studies, the general feeling of the
committee \-- the experimental studies to look at market structure, the
effect of market structure on pricing. The general feeling of the
committee was that this is not the sort of work EIA should be doing.

There are a lot of reasons for that that I don\'t want to get into now.
As far as the network work was concerned, I think there\'s a general
agreement that it\'s important to look at a network. It\'s important to
see if networks make a difference for overall prices compared to how
we\'re calculating them now, but that in reality market structure is
probably much more important in terms of price formation and the
outcomes of this restructuring.

There\'s a feeling that we needed metrics for cost production and
reliability. We needed to include start-up and shut-down costs, and
there was a real concern expressed mainly by Dick Tabers that, by going
into the network as we were doing, we were taking a real risk of getting
caught in a morass of detail and that he really implored us to try to go
to simpler higher aggregations in order to capture these effects that
we\'ve left out of our models \-- that are currently not in our models.

Well, we did pay a lot of attention to you, and our current thinking on
experiments is summarized above.

I think we\'ve concluded \-- We have concluded that we simply cannot
sustain an in-house program of experimental economics. The level \-- The
kinds of skills required and a level of effort for the work coming out
simply doesn\'t seem to be worth it to us.

It\'s interesting work. It\'s exciting work. It\'s important work. But
we do agree now with the committee that this be done at the experimental
labs.

This means that some market structure issues, issues that Phil raised,
we\'re not going to be looking at, at least not in a leading sort of
way, because they do require some experimentation. Issues of price
transparency, how does that effect consumer demand, bidding, that sort
of thing, single versus integral bidding, the ISO strength and
governance, what difference does it make for behavior of these markets,
and probably the whole thing of Transco versus the ISOs are things that
do have an experimental element that we\'re not going to be getting
into.

In thinking about the networks, our simple conclusion is Relles got it
right. That\'s the simplest way to put it. Dan made a couple of points.
One, the start-up and shut-down of these units are very important.

When we looked \-- When we went back to the NEPOOL model and started
looking at it, we discovered that the minimum operating levels
associated with the plants were running about 26 percent of capacity,
and as we reduced demand down from the peak demands that we\'re at to
lower and lower levels, we started to get an awful lot of
out-of-merit-order dispatch, which then is a result that is contrary to
what we normally assume in our models. How important it is, we don\'t
know yet.

The second thing we discovered was that reliability limits really do
dominate the line limits that we have in our model. In the NEPOOL model,
we simply have the line loading limits as reported to NERP. We don\'t
have the further reductions from those imposed by reliability
considerations.

We talked with one of the engineers at NEPOOL, and he assured us that
this conclusion is right, that what \-- they\'re managing to their first
contingency type issues and not to the physical capabilities of the big
lines.

The final thing that came out of this \-- and maybe \-- or this
relooking based on the committee\'s remarks, was to find that voltage
support really is critical for security and transmission analyses, even
in a well built structure like NEPOOL.

When we shut down many of the generators, the Millstone generators,
nuclear generators, in particular, what we found was that we could not
actually get the power to New Haven and Hartford, you know, at a voltage
that they could use. In fact, about 40 percent of the big demand
locations simply were not getting usable power.

To get power that you could use to them, you had to run Millstone as a
motor, as produced reactive power. Again, we checked this out with our
NEPOOL engineer, and he said, yeah, that\'s about right. You know, these
are the sorts of things that happen.

So it does turn out that, in looking at the ability \-- our ability to
meet emergency needs or looking at how the system will change over time
and our ability to support demand over time, reactive power is something
that you really have to look at in some circumstances.

I would like to point out, though, that in most of the cases that we ran
we had several generators that were kept running, solely for the purpose
of producing reactive power, even under normal circumstances. That\'s
something we really have to expect.

Well, with all the feedback from the committee and, you know, with our
own experimentation, the question is what are we going to do now.
Obviously, we\'re not going to be doing experimental work.

We are, however, quite concerned with this whole issue of aggregation
bias. After all, that is the thing that got us looking, going down this
road in the first place. Are there things we\'re leaving out of our
models that are important enough, though they be in the details, that
they add up to big effect for our price and quantity forecasts?

So we\'re going to continue to look some at how assignments of the
model, the NEPOOL model, might bring out effects for generators\'
reliability and voltage limits.

The other thing we\'re going to do is a project in interregional
transmission. Here the question is: Do the trade flows calculated in the
EMM accurately reflect the physical limits and economic opportunities
for trade?

I talked about the approach to aggregation bias. The first thing we\'re
going to do is improve the data coverage in NEPOOL, especially the
generator costs and the availability of the generators for dispatch.
We\'re going to try to get some information on the start-up and
shut-down costs and ramping times for some of the minimum operating
level plants.

We\'re going to drop the voltage levels of the lines. We have only been
looking at very high voltage lines so far, 345. We\'re going to bring it
down to 128. That is in response to two pieces of advice we\'ve had from
extremely well known engineers.

One engineer said that we could never find the sort of effects we\'re
looking for at the high level of voltage that we built into our model,
that it would simply be not apparent. The other engineer, equally well
known and well positioned, said that, well, if you look at lower voltage
levels, you won\'t find \-- you know, the congestion problems will have
all gone away, because the system is so overbuilt.

So one engineer was telling us to go down, and one engineer was telling
us to stay where we are. We\'re going to go down and see if there is
anything \-- in terms of disaggregation, and see if there is anything
there.

The other thing we\'re going to do to try to get a better handle on
reactive power is to include some of the capability curves for
generators. This is a tradeoff between reactive and real power, as seen
by the generators, but this will only be for a few generators that in
our simulations continue to have significant voltage support
responsibilities.

Finally, working with the engineers at NEPOOL, we\'re going to de-rate a
few of the lines to see if, by de-rating them to reflect first
contingencies, we get real constraints in the transmission system
showing up.

Having done all of this, we\'ll go back to the same program as before.
We\'ll calculate the optimal or the competitive outcomes with and
without these details added in, and see if that indicates that the EMM
forecasts need to be adjusted.

There are actually a couple of analyses that we should be able to do
with this new model or this refined model. We ought to be able to look
at a reliability region size \-- that is, if the region is bigger or
smaller; does it make much of a difference \-- and we ought to be able
to look a little bit at the potential for market power in voltage
support.

The other project we wish to do has to do with the inter-regional trade.
As many of you know, the trade limits within EMM \-- I guess it\'s best
to put it, they\'re conventional, and there isn\'t a whole economic
analysis of inter-regional trade going on in that model.

We anticipate a lot of trade in the future as electricity moves from
west to east, maybe a growing trade from what we\'ve seen. So what we
want to do is to try to do some initial work in understanding
transmission flows through the high power network.

We have in hand now models of the northeast, midwest physical network
that come from Tom Overby at the University of Illinois. We intend to
clean those up, clean up some of the data, some of the inconsistencies
in it, and then begin a process of perturbing the demand in generation
until the tie-line limits are reached or reduced for security limits or
whatever, and get some handle on how big \-- you know, how much power
can actually be moved around.

We might want to take a look at increasing the size or capability of
some of the lines to see how big of an impact that would have on the
potential for trade.

Finally, if the results of this analysis \-- that is, of the ability to
move the volumes of electricity through the system \-- are quite
different from what EMM is assuming, then we would change the
assumptions in the model to reflect our findings from this work.

So that\'s what we\'re up to now. If there are any questions or
comments, I would welcome them.

MR. HANSER: Just a couple. I think you\'re right about dropping the sort
of, you know, experimental economic stuff. I think it\'s just hard to
support. But I would recommend you give serious thought to seeing if
there\'s some way to fund a graduate student or something like that to
get answers to some of those questions.

It\'s not that the questions aren\'t important. It\'s just a case of
where are your comparative advantages at EIA relative to others. So I
certainly think that\'s useful.

I think what you\'re going to find in the future, frankly, is that the
inter-regional transfers \-- and you may want to think about building
this as a capability \-- are going to be done with DC lines and not AC
in the future to prevent, basically, inadvertent flows across ISOs.

You may want to think about that as part of your modeling, just to sort
of see what would happen if you, in fact, put into place DC connections
between ISOs and eliminate the potential for the flows to escape.

Then lastly, I think Dan was exactly right. The start-up costs and the
minimum load costs and all of those sorts of things are very substantial
in a lot of these places, and partly you need it for voltage support and
partly you need it because it\'s just the nature of the generating
system.

It\'s clear that those costs are \-- need to be somehow thought about
and added in, and you do end up with what we think of as uneconomic
dispatcher time as a result of that sort of thing. But again, to some
degree that\'s a function of how do you expect the competitive market to
operate in terms of what\'s actually bid in? Is it an energy only bid?
Is it an energy bid, but you get to cover all start-up costs if you are
brought in, etcetera?

There are enough variance on those rules that you need to sort of think
about that.

One other thing: I thought you were very responsive, by the way, to the
committee\'s comments. You know, it makes us feel good to be on the
committee if EIA responds that way. So I don\'t want to let that go
unsaid.

CHAIRMAN RELLES: I\'d like to ask: I presume this is part of a process
where you gained an initial understanding of what is supposed to be
happening in the deregulated environment, and then things started
happening, and they start \-- people start coming to you to interpret
what\'s happening, and I\'m just wondering what you think the time frame
is for the various steps that are sort of inevitably going to happen.

For example, I presume you\'re going to be getting into some data
collection exercises soon, both of the power grid and also of the prices
and consumption that end users are paying. Are we talking two or three
years down the pike? What\'s your anticipation of the pace at which this
foundational type work is going to become much more visible and produce
some real understanding of what\'s happening out there?

MR. HALE: Were there anymore comments? I\'ll try to answer that right
now.

CHAIRMAN RELLES: I think that was the only question.

MR. HALE: I and several colleagues got into this work not because we
knew anything about electricity. What we knew was the whole nature of
the industry was changing. It\'s changing a lot, and new that the models
that we had didn\'t capture it.

We also knew that there was a lot of reasonably well developed economic
literature out there that we could appeal to. So a large part of the
work on this project and some previous things have been primarily to get
ourselves up to speed so that we could start asking the right questions,
and especially the right questions about data.

My personal opinion is, if EIA gets the data collection for electricity
restructuring done right or approximately right, we\'ll have done an
incredibly important public service. In this new world, you know, we\'ve
been focusing on understanding the basic economics of it and some of the
subtle economics of it well enough that I can go to Lynda and go to the
people who are collecting the data and say, look, you really do need to
know some things about what the network is, what the impedances are
like.

You need to know something about how these ISOs are working, what kind
of pricing mechanisms are used, start to ask questions that you
couldn\'t even conceive of asking in a forum for an earlier environment.

So that\'s one service of this work, and the other way it comes out is
to write various papers that come up in, you know, internal meetings and
in focus groups and that sort of thing to try to bounce ideas off of
people.

So to some extent, that was a first objective of this work, and it\'s
continuing to develop along those lines. The other thing, however, is to
\-- as I said, to get a better \-- get more assurance that our
forecasting models are still more or less right on target, that we\'re
not systematically distorting anything.

That\'s the reason we\'re doing this work, and that\'s \-- I believe
we\'ll have results from what I\'ve described today within a year.
We\'ll be in pretty good shape.

Does that answer your question.

CHAIRMAN RELLES: Yes. Is New England going to be the test bed for \--

MR. HALE: That\'s the one we have right now. So we\'re going to do it. I
mean, we do \-- I think the New England case is going to probably tell
us more about the details of what data we need to collect \-- it already
has. It\'s going to tell us a little bit more.

I think it will tell us some more about these issues, you know, that
we\'ve been talking about, reliability start-up, voltage support, and
whether we need to collect anything in those areas as well.

We\'re sticking with New England. Right now we\'ve got a decent model.
We\'ve got good relationships, and once we\'ve done it well once, we
believe we could replicate it as much as needed for other areas.

MS. CARLSON: Well, actually, in addition, tomorrow there\'s going to be
a talk by Betsy O\'Brien and Stan Freeman, and you will probably find
out more than you expected to find out about a fairly detailed activity
EIA is involved in in just this electricity, determining the data needs
and how to collect that data. Probably Bob has something to say about
what\'s going to \-- No?

CHAIRMAN RELLES: Okay. Thank you very much, Doug. We\'ll get you next
time, too, I hope.

I think it\'s safe to proceed without a break. I guess \-- just
checking. Why don\'t we get started then with the afternoon session on
Web-site related issues.

We\'ll begin with an introduction to Web-site related challenges, which
\-- I know you\'re not Larry Pettis. So I assume you\'re John Pearson.
Good.

MS. CARLSON: I\'m going to give a brief overview on these three talks
first. Actually, this is John Pearson and Larry Pettis. You get three to
do one.

Probably, the title Web-site related issues is a misnomer for this brief
paper. The first paper really is going to walk you through the status of
the EIA Web-site. As any Web-site, this one is extraordinarily dynamic.
It\'s changing rapidly, almost daily. Updates are happening. You\'re
going to see an interactive data query.

You know, many of these improvements are really a result of feedback
from our customers, and the feedback that\'s received from our Web-site
comments box, from focus groups, from EIA annual customer surveys,
special surveys that we\'ve done on the Web.

Initially, in fact \-- and Colleen, I think it\'s three years now \--
the very first comments we received as EIA started to move away from
paper products and putting things on the Web is our customers said, hey,
go slower. You\'ll see that there have been changes in that over time
somewhat.

Colleen Blessing is going to talk about customer outreach and the survey
activities going on in EIA on our customers. In fact \-- and I don\'t
know if she\'s going to mention it \-- EIA is very much a leader in this
field, and we\'re the benchmark agency for a number of other agencies,
because we have been teaching other agencies on how to do it.

The third paper by Renee Miller and Tom Broene discusses another set of
implications from the Web, and it\'s the transparency problem or
opportunity. What that is that previously our data systems and
publications would stand alone, you know. You looked at the supply side
and, you know, consumption, some supply side publications, and you come
to the demand side and commercial meant this, and there wouldn\'t be a
problem with that.

Now, however, a customer will often obtain data from several different
pages on the Web and, lo and behold, they will find different estimates
or definitions from the same item. We\'re starting to tackle that
problem and to be responsive to our customers for clarity, and to
understand why these differences are, and we\'ve \-- I guess the best
way to define the project is we\'re trying to harmonize or, in most
cases \-- and maybe Renee will discuss what in most cases is \-- as how
we try and work on that.

It\'s also become somewhat clear that with transparency that we may have
difference in response rates and how we calculate response rates. We\'re
beginning to tackle that problem as well, but that, we think, may be a
much more difficult problem with time.

MR. PETTIS: Okay, we\'re trying to improvise here today. The dangers of
having a connection to Internet \-- It\'s really a pleasure for me to
talk about our EIA Web-site, because I think it\'s been one of the real
successes of EIA over the last few years, and just as a little bit
history, it\'s hard to believe, but it\'s been about three years ago now
that we first started thinking about developing a Web-site.

There were several experimental projects going on where we were
participating in a reinvention project [Bulletin Board]{.underline}. The
Department was starting to talk about some Web activities. There were
other things going on around the government, and I think it was at that
point we decided we really needed to sort of get our act together and
figure out that we really needed to be a viable electronic dissemination
organization.

So we created a team that we call the Worldwide Electronic Access Team,
and they really laid out sort of a multi-phase plan about how we would
approach this and, of course, our Web-site was one aspect of that.

I might just comment that there have been some changes in this
presentation since they were put in your book, and we did lay a new copy
of it. So I can probably just talk through the first couple of slides
here, because there\'s not a lot to see, but when we had our Worldwide
Electronic Access Team, I think there were several strong goals that
kind of came out of that.

One is what we talk about around DIA, which is fast and easy access for
our customers and a common look and feel on our Web-site \-- kind of the
first bullet there. Secondly, we saw some opportunities to really take
advantage of cost savings in a means a delivering our products
electronically, and also providing them in a way that they would be more
useful to our users and potentially reduce their cost.

The third thing is, obviously, you\'re all aware there\'s an explosion
in these electronic products, and trying to take advantage of those, not
be on the cutting edge but to take advantage of them as they are
developed commercially out there.

So our strategy, really, was to \-- again, was to select certain
technologies to use in disseminating our information, and we had to
recognize also that our users are rather diverse. They\'re not all
cutting edge out there on the computer side of things, too. So we have
to present things in a way that many different people can use.

This really did involve a paradigm shift. EIA had really been a paper
product organization almost solely, and I would say that this paradigm
shift is really evolutionary, not revolutionary. It\'s going on as we
speak.

We have instances where we\'ve really just taken publications and
presented them electronically, and then we have more innovative products
where people have taken databases and put front ends on them,
applications that make them very user friendly. So we are in the process
of making that shift.

Next slide. Anyway, the Web-site really has all of EIA\'s publications
\-- virtually all of EIA\'s publications. We have also presented a
number of databases out there, as I mentioned earlier, and we\'ve tried
to present these in a number of ways, in a variety of formats that
people like to receive, like HTML or the portable document format or
spreadsheet formats.

So we\'re trying to use customer feedback to say, you know, what kinds
of formats do people want out there.

John, you can probably just go ahead and move. Were you going to try to
move to the first page on the Home page on the Web-site? He\'s going to
try to make a shift and get us \-- But essentially, when we laid out our
Home page on the Web-site, we did not take an organizational approach
where we tried to present stuff from various organizations within EIA.

We tried to sort of present it as an informational approach, so people
could not know about our organization but know what kind of information
they wanted. And we tried to present several different approaches to
getting information.

So this, in fact, is our Home page. You see that we have things laid out
by fuel groups, so people can, in fact, click on a fuel group and look
for information or, if they know what they want, we actually have a
bookshelf. If they know the publication, they can just go to a bookshelf
and get the publication.

So John is going to kind of move us down through a couple of things.
This is one of our second level pages, the energy overview. One of the
things I mentioned before, we wanted to have a common look and feel in
our Home page. That sounds pretty simple. You just lay out a format, and
everybody follows it.

When you have an innovative organization like EIA, people want to be
creative. They, in fact, are doing some really good things about talking
to customers about how they use information. So they want to create new
ways to use it.

So things start diverging over time, and this, in fact, is a standard
second level page that we have recently introduced on our Home page, so
they all look alike. A couple of the features of this, as you see,
there\'s a new navigation bar up at the top here, which allows one to
easily move from fuel. So we\'re on the energy overview. All you would
have to do is click on petroleum if you want to go to petroleum on the
navigation bar.

On the lefthand side we have a standard menu format for all pages. Now
this is where we, in fact, do allow a little creativity in that we have
the standard layout there, but different pages can give different items
for people to click onto, based on how they use the information.

So anyway, this is the new standard that we\'ve recently implemented in
EIA.

One of the most popular mechanisms that we have on our Home page right
now is our list serve where people can sign up for various products, and
they\'re E-mailed to them directly. Simply sign up, have a list of the
products that are available, and they\'re automatically E-mailed to you.

We have about 11,000 people that have signed up for the list serve now
for about 50,000 lists, and there\'s over 800,000 of these mailed out in
the last year. So a lot of usage of that.

Another thing is that this has given us a good opportunity to make
visible our analysis agenda. Over the last couple of years, we\'ve had a
much more rigorous process of trying to work with our customers in
setting up an analysis agenda, but we can now put it out on our Home
page.

If you go down here and click on any one of these analysis products, as
John will probably do here, you\'ll get a little summary of what the
product really is, you know, who the staff people that are working on
it, when it\'s supposed to be produced.

So actually, people who are interested in these types of things or have
ideas or input can contact the people that are working on them. This is
changed and updated as the projects change over the year.

A couple of things about future developments: Again I want to \-- You
know, a little bit longer term project here. We are really trying to get
away from the traditional table display of data where we will put the
data that\'s collected by EIA out on the server and allow our customers
to create the types of information they want to.

We are continuing to add new technologies as they come along to lower
the cost on this and improve the services. John is going to show you a
couple of examples of that here in a minute.

We\'re also trying to use our Internet statistics and to make some
decisions about what we retain, improve, keep on our Internet site, and
Colleen is going to be talking a little bit about some of the feedback
we\'ve gotten in the next presentation.

I wanted to mention a couple of things. One is I think some of you may
know, but in 1996 our Web site was named one of the best in government
by the Federal Web Showcase, and we got a nice commendation from the
Vice President for that.

I say that, because I really think that a lot of people in EIA have
embraced this idea, and they have made it a success. It\'s not just a
few people. It\'s really down to all levels of the organization.

We\'ve also gotten several nice write-ups in trade journals and
magazines about our Web-site. There\'s an interesting side benefit that
I\'ve detected form the Web-site, and that is that it\'s a source of
immediate feedback to our employees about the things they\'re working
on.

It\'s sort of surprising, but when somebody puts a product out there and
they get several E-mails the first day or they find that five or six or
seven hundred people clicked on it, they know it\'s something of
interest, and I think it\'s really been a morale booster for our
employees around the organization.

You might wonder why I\'m up here talking today, but the way we decided
to implement our Web-site was we established this information services
and products committee in EIA, and we have \-- I\'m the champion of that
committee, and the role of the champion is to get out of the way and let
all the people who know what they\'re doing do their job.

Fortunately, we have a lot of those people who know what they\'re doing,
and John is one of those. He\'s going to tell us a little bit more about
some of the new things we\'re working on.

If you\'ll just flip to the next slide, which is kind of a summary of
what\'s happened on our Web-site over the last couple of years, you can
see the rate of growth of usage has just really exploded.

Last year, we had about 715,000 unique users that accessed the Web-site.
Just to put that in context, when we were a paper product organization,
we delivered less than 300,000 paper publications in a year and, of
course, there were multiple copies of that that went to the same
individual. So we\'ve really multiplied the customer base of EIA
rapidly.

As I mentioned earlier, again one of the big usages is the list serve. I
think one of the things we find with our Web-site is that customers\'
expectations are growing rapidly. We get some unique inquiries now
about, you know, where to buy pistachio nuts in bulk and how to trade
hogs in China, things that we\'re not exactly expert in.

I\'d say over \-- well, in the majority of comments that we get on the
Web-site are very positive. Some of our tougher customers, a couple of
them I\'ve noticed, are like eighth graders. I think we\'re generating a
new breed down there that expects to go and find exactly what they want
instantaneously in this new information age.

Anyway, I\'m going to turn this over to John, and he\'s going to show
you a couple of the new features that we\'re working on.

MR. PEARSON: One of the features of the slide show, the handout, is if
you wish to quit now \-- Let\'s see if I can make this thing work.

The first thing \-- The next thing in the handout is our experimental
IQ2 where we provide interactive access to a database, so that customers
could query for the information data streams they want. I know what
you\'re going to say next.

I\'m going to try and remember where we put this. There\'s two ways of
doing it. This is one and \-- Oops, I don\'t see it here. I\'m going to
go back here. Go to bookmark.

Okay, the first thing is \-- This is an experimental site. So not
everything works exactly as one would hope, and not all the data one
would expect to see is in here, but hopefully, if this thing is working,
we will get some action.

Okay. What the customer sees is a series of panes. The first one is
where he selects \-- he or she selects a fuel group. Next is a category
element location and time period. Let\'s go for consumption. This is
almost like, I guess \-- what do I do to select? Double click and see if
that does something. Oh, then I hit next, of course.

You can see that usability is a big issue here. Okay. Industrial
consumption, hit next, and then we get the choices of what\'s in the
particular publication we\'ve got. Let\'s choose electricity consumer,
the industrial sector, and then choose USA next, and then we can choose
a series of time periods.

Unfortunately, there must be an easy way of doing this, but I don\'t see
it. Choose specific time periods. I think what we need is a button here
that says choose them all or choose some interval.

Then we get basically the set of choices, and then the question of what
do you want to see or what format do you want to see. I\'m going to go
for an HTML report, which, as you can see in the diagram \-- as you can
see on the next page of the handout, and then here is a table on the
specific data that was in the database the user requested.

This is an example of how we\'re trying to get the customer to be able
to dynamically request what he or she wants instead of us serving up a
table in a fixed publication. This will, obviously, go through a great
amount of development, but this is where we stand at the moment.

That address there is a live address, if you want to go back and try it.

The next miracle we\'re trying \-- if I can go back to the Home page \--
is a new development in the delivery of documents. If we go to our
publications site, here we\'ve got approximately 200-250 publications,
pretty well everything that\'s current except for the, oh, analysis
reports that a particular customer decided they did not want to see on
the Web-site. So we\'ve got everything else.

We\'ve organized things in overlapping categories. So if you click on
one, you\'re almost likely to get what you want. Again, I\'m going for
consumption here, and assuming we\'re lucky, the one we get here is a
list of all the consumption publications.

What I\'m going to do now is \-- Normally, these files are very big. You
see, this one is half a megabyte, one megabyte and so on, and even on a
miraculous modem, it takes a good time to get at them. I\'m going to
choose one of the biggest, if I can find it. Here it is. I just saw it.
That\'s 5.72 megabytes. That would take most of an hour to get to. I\'m
going to open it up, and with the new technology, what this does in
Netscape is automatically fire up the Adobe reader with a new capability
to request specific pages. I guess I should have killed that.

What it\'s doing now is downloading only the index of the publication
and the first page of the document. It\'s, unfortunately, a rather large
graphic. So we have to wait while this graphic comes down, but you can
see we\'ve downloaded almost nothing. At the bottom you can see what
we\'ve downloaded. We\'ve downloaded almost nothing of this 6 megabyte
file.

Now if I now go down to the very bottom of the file and say, okay, I
don\'t want to see that; I want to see something interesting down here
\-- how about the quality of design of the survey? That\'s right at the
end of the file.

Now the way this is organized now, what it does is go back to the server
and say give me the page at the very end or towards the very end, and
there is the page. The customer did not have to download the entire
file. What he received was precisely what he asked for.

What this does effectively is put all of our publications on the same
status as a publication that was, for example, prepared on HTML where
one can literally page through. Now you can page through these very big
documents just as if they were regular Web pages.

So that\'s called byte serving. It needs everybody to install a new
Acrobat reader, and also we had to install new servers. That\'s pretty
well where I\'m going to stop.

What we\'re also doing, which is this very last page, is looking very
carefully at our log files and using some log analysis programs to see
the kind of response that we\'re getting, and they will tell us
literally what people are looking at, who they are, what format they\'re
looking at these documents, and when they\'re looking at them and where
they\'re coming from. So it\'s tremendously helpful as the analysis tool
which will augment the surveying that Colleen is going to talk about
next.

So I\'m just going back to our Home page, so you can prompt any
questions, and I\'ll turn it back to Larry.

CHAIRMAN RELLES: I have a question. It\'s clear how useful the Web page
is for disseminating information. I guess my question is \-- and
understanding something about the demand for your product \-- My
question is whether you are looking at it also as a \-- or evaluating it
also as a potential cost saving tool in terms of reduced publications,
the ability to reach your customer base more easily, and things of that
nature?

I realize some of that might be covered by Colleen, but I just want to
get the question on the table.

MR. PETTIS: Dan, that actually is one of our specific goals, and our
strategic plan this year is to look at reducing the number of paper
publications that we have and, you know, also looking at things like
whether we can repackage products in a different way that might be more
usable to our customers.

So there\'s a variety of things that we are looking at, and in fact we
have reduced the paper publications over the last several years because
of our Web-site, but it\'s an ongoing process.

CHAIRMAN RELLES: Is there a metric you can give us? Is it like ten
percent reduction or is it too early to say that?

MR. PETTIS: I don\'t have the metric. I don\'t have it in front of me,
but I\'d be glad to give it to you.

MR. PEARSON: If I could interject, one example is the annual energy
outlook where we went from a quarterly publication to a monthly Web
publication that involved \-- publishing nothing at all, but since
people still requested it, we printed the Web copy, and we distribute
that.

We didn\'t save anything, but we multiplied our productivity by six, if
you like. The other example is the amount of work required to put
material on the Web is clearly reduced significantly by this PDF type
capability that I just demonstrated, plus the ability to query for data
tables that we didn\'t think of but the user wants. We\'re sort of
working on that idea.

MR. PETTIS: I think you\'ll probably see some of that in Colleen\'s
presentation, but we still have a fairly high number of our customers
that are demanding paper publications. I mean, I think many of us would
have expected that number would drop. It is slightly, very slightly
declining.

I think many of us would expect it would fall quicker than it has, but
there\'s still a lot of demand for the paper publications.

MR. HANSER: One quick comment. It\'s a great site. It really is. I think
one reason for some of the paper publications is that some of them still
have color, and the PDF format doesn\'t support easily the transfer of
color for a lot of folks or they don\'t have the color printer to
download the information. So I think for some of the publications, that
may be one of the reasons.

One other thing that you might want to just check on \-- This is really
dumb, but one of the first times I tried to sign onto the EIA Web-site,
I went in by \-- tried to get in through DOE, and there was no link
between DOE and EIA at one point. Now I don\'t know if they\'ve fixed
that subsequently, but it was \-- and maybe it\'s been fixed. But you
have to check to find out who links to you, in that any of the possible
ways in which users might want to go easily from, say, DOE into EIA,
that those links are available.

I was really surprised to sign onto DOE and find out that you couldn\'t
get into EIA from DOE. That was really sort of strange.

MR. PETTIS: I think that is one of the areas that there\'s been a big
improvement in. I think if you roll down on our page there, John,
you\'ll see there\'s a Fed Stats button on there where, really, we have
linked all the Federal statistical agencies together. So you can move
quickly back and forth between all of the statistical agencies now.

CHAIRMAN RELLES: I\'d like to ask \-- Actually, it\'s probably
unprecedented for me to ask the committee a question, but I\'d like to
do it anyway.

Is anybody aware of research that attempts to use Web based responses as
kind of a national sampling frame?

MR. SUDMAN: There are dozens and dozens and dozens of Web based surveys
of varying quality. Nobody has \-- You know, it\'s not yet possible to
do a good national survey doing that.

For example, if you want to do a survey with a class, you tell your
students to go to a given Web page and do it. That\'s great. If you
invite people to go to a Web page, you can get them to respond very
nicely, but simply taking people who show up at that page or are trying
to do sort of a cold national sample, it doesn\'t work.

MR. HANSER: I have a colleague that did a simulation of the competitive
electricity market through the Web and had participants coming in and
bidding on \-- to simulate an electricity market.

MR. SUDMAN: I have my students doing surveys on the Web, and they love
it, and it\'s good experience, and you have good data; but they\'re
forced to do it.

MS. CARLSON: Both BLS and Census are experimenting at conducting some of
their surveys on the Web, but they would pre-notify the sample
respondents. Actually, EIA may start looking at some of that for its
electricity data, the new electricity data.

MR. SUDMAN: One of the things that will happen, to follow up on this, is
that it may become one of the choices people have of responding. You
know, you can respond by mail or by Touchtone on the telephone, and Web
is another way to do that.

CHAIRMAN RELLES: When you say that national sampling doesn\'t work on
the Web, is there actually some published research that kind of looks at
the issue and looks at the kinds of biases that you might have to \--

MR. SUDMAN: Well, talking now about \-- I was thinking about households.
No one has tried to do sort of a national study of businesses on the Web
yet or, if they have, I\'m not aware of it. The national sample of
households wouldn\'t work, because a huge fraction of households don\'t
have access to the Web yet. I mean, it\'s something like a third who do,
something of that sort, but for those who do \--

CHAIRMAN RELLES: I\'m just curious whether they\'ve quantified the kinds
of biases that they get.

MR. SUDMAN: I mean, in terms of households, yeah. I mean, it\'s \--
Younger people are more likely to have it, higher education. I mean the
kinds of things that you would expect.

MR. HAMMITT: I have seen it used in a sort of medical context where they
went to Web-sites in order to attract people who have family members
with a particular disease, and it\'s a quite efficient way to reach
those people, but I\'m not sure exactly what comparisons have been done
with this mode versus other ways of getting to them.

MR. SUDMAN: There\'s a tremendous amount of interest in that topic, and
all sorts of people are doing all sorts of things, some quite strange.

MR. WHITMORE: Question with regard to the data query option. Is there
any thought of putting in some sort of a data analysis system so that,
when you ask for a query, you don\'t just get a table, but you get also
standard errors of the estimates?

I know the National Center for Education Statistics does that with some
of their publications where you can \-- Their process is cumbersome, but
they basically allow you to request a table, and you get back not just
the estimates but standard errors of the estimates.

MR. PEARSON: That\'s been an interesting discussion. The examples I\'ve
seen of the Transportation Bureau statistics where they had an
interactive SAS program that would do data analysis looked to be very
computer intensive to get any kind of response. Clearly, we couldn\'t
hang it on either of the two machines we\'ve got.

What we have done is printed the \-- strictly in an electronic format,
the standard errors for the consumption tables. Am I correct on that?

They\'re out there. In fact, if you look on the bookshelf, you can find
them \-- the standard errors for the consumption estimates for
households and so on, but we don\'t put them in the normal paper
publication anymore, as I recall. But that\'s the nearest we can to
providing responses to what is a very computer intensive question.

MR. WHITMORE: The national Center for Education Statistics does. You can
E-mail essentially a request for a table. The table runs at some point
and E-mails back results to you.

MR. PEARSON: I\'ve seen that.

MR. WHITMORE: Not real user friendly. They have kind of a limited
audience of folks that use their products, and they\'ve learned how to
use the system.

MR. PEARSON: It\'s a scary sort. Batch processing sounds like something
we could implement, but the question is what\'s the audience. We have
Web statistics that would verify what the audience is clearly enough.

MR. HAMMITT: On that point, you were talking about what formats data
should be available in. I would think you would want to make sure it was
easy for people who wanted to do some analysis to download the raw data
in a very quick and easy way.

So I would imagine it\'s a very small set of people who will want to run
some analyses, but will want to run it on your machines instead of
getting hold of the data and having a lot more flexibility and control.

MR. WHITMORE: That\'s probably much more of an option here than it was
for the National Center for Education Statistics stuff that I was
talking about. That was very confidential information, although you may
have some CBI types of data as well that would have to be protected in
the same way.

MR. PEARSON: We do make available public datasets, and they\'ve been
sanitized so that you can\'t identify who, for example, households are.
But yes, they\'re available. In fact, they\'re available on the Wet. You
can download the files for several of the consumption surveys where you
could do that kind of analysis.

CHAIRMAN RELLES: Thank you very much. Very interesting demonstration. I
agree, the Web-site is terrific. I logged on there in preparation for
this meeting.

Colleen Blessing is going to be talking now about measuring customer
satisfaction on the Web.

MS. BLESSING: Hello. I\'m Colleen Blessing, and I\'m the Chairman of a
customer survey committee in EIA. I\'m really glad that I\'m following
the demonstration of the Web-site. You had a chance to see it, and now
I\'m going to tell you what our customers said about it.

Over the last couple of years, we\'ve received a lot of feedback from
our customers on our Web-site from various surveys and from the Web-site
button, the feedback button that we have. I just wanted to start out by
reading \-- just giving you a broad sample of the types of things we
hear.

They go all the way from the compliment that Larry talked about: Your
site is absolutely wonderful, I refer people to it all the time; amazing
site; unbelievable amount of technical stuff, thanks a million.

Then we get down to sort of technical comments, and I characterize those
as \"I can\'t find it, I can\'t see it, I can\'t download it, I can\'t
print it.\" They can\'t find the data they\'re looking for or they
can\'t see the distinctions on the page.

Then we have one of those eighth graders that Larry talked about. This
is one of my favorite ones. \"This site was a mess. A two-year-old could
make a better site than this.\" And she signed it, \"An Upset
Schoolgirl.\" So you\'re absolutely right.

The comments run the gamut from \"I couldn\'t live with you\" to \"this
stinks\" or worse.

Now how do we collect all that customer feedback, and how did we give
people the opportunity to tell us what they thought about our Web-site?
We\'ve done lots of different customer surveys, but Bill has given me 15
minutes, and in my 14 1/2 minutes left I\'m going to try to review five
of the ways that we\'ve brought in customer feedback on the Web and
briefly I\'ll just list them so you know what they are: The telephone
survey of customers; a Web-site survey that was fielded on the Web; our
Web-site feedback button, which you may have seen on the righthand side
of the screen. We just recently fielded a survey on the Web of the
country analysis briefs, which is a very popular product in EIA.

Then I\'m going to tell you some of the questions we\'re going to be
asking on an upcoming survey. It\'s a survey of our publication
subscribers. So that will be a mail survey that we\'re asking Web
questions.

Then finally, Dr. Relles and I had an interesting conversation
yesterday, and I was going to bring up or just discuss a few of the pros
and cons of using the Web to collect customer feedback, and you\'ve
already talked some about that. So that was good that we opened that
subject.

The very first way that we got feedback on the Web from customers was
the telephone survey that we\'ve conducted. We\'ve conducted four years
in a row now, and we start out by asking them satisfaction questions
with courtesy and timeliness and accuracy of our publications, but then
we move into a section where we ask them if they\'ve used the Web and
their preferences and their satisfaction with it.

You know, you think, okay, you\'re talking to these telephone people.
How many of them are actually Web customers or how many of them are
electronic customers, and you can throw up the first slide real quick.

Over the last four years\-- and this is not just Web. This is
electronic. So it\'s also FAX and a couple of the other ways that you
can get information electronically, although the Web totally overwhelms
this.

Four years ago \-- I don\'t know \-- Can the people in the back see
that? Not very well. The blue \-- In 1995 we had only 39 percent of our
telephone customers using an electronic product, and by this last
February 73 percent of the telephone people said that they were also
electronic customers.

Go to the next slide. Now you really don\'t have to read this, but the
only thing you need to see is the yellow rocketship over on the lefthand
side. That says 69 percent of the people who we talked to on the
telephone are Web-site customers.

So I think EIA used to think, well, we\'ve got our telephone people and
we\'ve got our Web people, but the customer groups of the segments are
blending, and it\'s not as differentiated as we may have thought before.

We asked these telephone people who said that they have used the
Web-site \-- We asked them our basic two questions: Did it meet your
needs, and was it easy to use? Those are our favorites. The lefthand
column, the red and the green, if you add those together \-- those are
people that said it met their needs. These are either \"yes\" or
\"somewhat.\" So those are fairly high numbers for meet your needs \--
no, I\'m sorry, easy to use.

Most people on our survey said the site is easy to use. Not as many
people say it meets their needs, and the main reason it doesn\'t meet
their needs is they can\'t find the information they want.

A lot of people will say \"fabulous site, I love it, but I couldn\'t
find this information. So, no, it doesn\'t meet my needs.\" It\'s not
that they are totally frustrated with the whole thing. It\'s that they
wanted something specific they couldn\'t find.

Then Larry mentioned that we asked our telephone customers, even though
you know it\'s on the Web-site, even though 69 percent of them have used
it on the Web-site, do you still want the paper. My next slide shows \--
well, it\'s hard to see. I think Bill Xeroxed these flags in the packet
for you.

The last four columns show the users that wanted the paper, and it\'s
basically run in the low sixties, 62, 63 percent. Tab 5 is what she
says. But you can sort of \-- I\'ll just tell you. Two months ago when
we surveyed our customers, even though 69 percent were on the Web, 62
percent said they still wanted the paper.

Okay. You can show the next slide. then the second way that we\'ve
gotten feedback about our Web-site, we actually fielded a survey on the
Web last year. It clicked up. Some of you may have even seen it, and I
talked about it at ASA, I think, a year ago. It clicked up on the Home
page. If you wanted to take it, you could go right into it. If you
didn\'t, there was a click-out button where you could go right to where
you \-- you know, to the data, and wouldn\'t be bothered with it.

There was no specific notification. We didn\'t E-mail out to people
saying it was there, just when they logged on they could see it, and
they could choose whether they wanted to respond or not.

We had it posted up on the Web for one calendar week, and we got 632
responses. Again, we asked our favorite questions: Did the site meet
your needs, and was it easy to use? Again, easy to use \-- 78 percent of
the people said yes. Twenty percent said somewhat. So we got 98 percent
of the people that are on the Web saying it was easy to use, and 86
percent said it met their needs.

Now, remember, because the next slide is going to have a different
answer here, \"meet your needs\" \-- these are the people that just
logged onto the Web. It\'s the first thing they saw, and they\'re
probably thinking about in general how it\'s met their needs.

Okay. Then the third \-- you can go to the next slide. The third way
we\'ve collected Web feedback is the button on the righthand side of
that Home page that John Pearson showed. Under customer service the very
first button says Feedback. If you click the Feedback button, it asks
you two questions. I bet you can tell me what they are. Did it meet your
needs, and was it easy to use?

Then it also had a comment field where it said, you know, anything else
you want to tell us. Now easy to use, again it gets a high score, 75
percent yes, 20 percent somewhat; but meet your needs, only 42 percent
said it met their needs.

I\'m thinking that that \-- in contrast to the people who just clicked
on the screen and took the survey at the beginning, these are people
that \-- I\'m guessing \-- went into the site, couldn\'t find what they
wanted, went back to the Home page and wanted to tell us about it. So
that\'s my interpretation of why that score is lower.

We get tons of comments, and what I\'ve done is \-- Well, actually, let
me give you a number. In 1997 we got 420 specific comments from that
button, and so far in the first quarter 1998 we\'ve gotten about 100. So
it\'s running about 100 a quarter.

I take all the comments, and I\'ve categorized them in kind of three
areas: Data availability \-- I say more than half of the questions or
the comments on that feedback button are \"I couldn\'t find it; I want
this and I looked for it, and it wasn\'t there.\"

Now there\'s several explanations. One is it was there, and they
couldn\'t find navigate to get it or it\'s company specific or we don\'t
collect it. So they kind of want to know answers to those things.

Then the second kind of comments: About a third of them are technical
comments. Again, I couldn\'t see it, I couldn\'t find it, I couldn\'t
download it.

Then 15 percent of them are just straight wonderful compliments: I love
it. It was wonderful. Keep up the good work.

Even though this is, obviously, not a statistical sample, it\'s just
whoever happens to give us the feedback, it\'s been really useful for
us. I think we\'ve made some \-- we\'ve certainly done personalized
responses. The Information Center within EIA attempts to respond to
every one of these feedbacks within 24 hours. If someone says they
can\'t find it or couldn\'t download it, they either E-mail the person
or call the person, if they\'ve left the addresses and phone numbers, to
help them out.

Also, as I was saying at lunch, it doesn\'t take a statistical sample of
people to tell you that a link is wrong. If one person tells you a link
isn\'t working, you need to go in, and the Webmaster will go in and fix
it. So it\'s been helpful that, even if it\'s an anecdotal one or two
people, if they say the same thing, people are going to go in there and
check it and improve it or make it better.

Just one of the things I think has been good is we got a lot of comments
earlier from customers saying I can\'t find historical data, it\'s not
there, I can\'t find it. Well, we knew it was there, but there was,
obviously, a reason why they weren\'t getting to it.

So the Web people added a historical data button on the Web, so they
could click right there, and they didn\'t have to go to the fuel and all
the years and all that. Obviously, people were having trouble with that,
and we changed that because of the comments that we got from our
customers.

Then just recently, a group in EIA fielded a survey on the Web asking
customers that use the country analysis briefs what they thought of that
product. Now the country analysis briefs is a \-- they are summaries by
country of the energy situation. You can click on Iran or click on
Venezuela, and you can get an overview.

It\'s a real popular product, and they wanted to get some idea of who
was using it, why they were using it, what was important and how they
might allocate their resources to better serve their customers.

It was fielded on the Web, but instead of being right on the Home page
like that other one was that I explained, this was embedded in the Web
on the customer analysis brief page. So that only people that got to
that point would be even seeing that survey.

On the other hand, everybody who has signed up on the List-serve, the
electronic E-mail, to get the country analysis briefs was zapped saying,
guys, there\'s a survey out there, go take it. So there was some
invitation in addition to \-- Anybody who was just clicking around could
take it, but they also asked the regular customers to go and find it,
which is different from what we did before.

They asked about the person\'s occupation, the purpose of using CABS,
how is it useful, how they found out about CABS. Unfortunately, they
haven\'t finished analyzing the data, tabulating the data. so I don\'t
have the results of that, but I have a couple of interesting anecdotes.

One of them is they got some initial responses in, and the way Doug
McIntyre in EIA characterized it is they kind of put them somewhere, and
then they lost them. That\'s one of the things you have to be careful of
when you do a Web-site survey, when you do an electronic survey, is to
make sure that you \-- You know, you don\'t have your paper here that
you can enter and add up. It\'s out there. So there were some responses
that were lost.

Doug sees that as a learning experience. That\'s what he said. To date,
they\'ve gotten nearly 400 responses. He said he would be putting a
positive spin on it. Now my favorite thing that he told me from this
survey is, you know, we have limited resources in EIA. So there was a
question on there that said, would you rather see EIA cover more
countries with less frequent updates or would you rather see us do fewer
countries with more frequent updates.

Now a couple of you know the answer to this, because I have already
talked to you. Don\'t answer. What did people say? More countries less,
or fewer countries more? What did they say? Both. They said both. Okay?

So there\'s another little drawback of a Web-site survey. Apparently,
Doug was looking through there, and he said they just want everything.
What they did, since there wasn\'t a telephone person like myself
saying, no, I\'m sorry, you need to pick one answer, they said \"Other.
We want more countries more frequently.\" So again, you don\'t have as
much control over those Web people.

Then when you look at your data and it\'s all \-- everybody checked
\"Other\" and they all have something that\'s out of the realm of
possibility for your resources, I guess the survey wasn\'t particularly
helpful. The results weren\'t particularly helpful. Anyway, he\'s
looking at that now.

Then the fifth \-- you can go to the next slide. The fifth way that we
are planning to collect some more information about our Web-site users
is our upcoming mail survey of publication subscribers.

This is going to be EIA\'s biggest customer survey to date. We\'re going
to mail out to people a questionnaire. It\'s like this. It\'s got a
cover letter, and it has about three pages. It\'s going to be color.
It\'s going to be pretty.

Dr. Dillman was out here, and he helped us with the cognitive one, two,
three, four. He helped us design this so it would look pretty. So the
customers would know \-- would be enticed to fill it out and would know
how to follow the patterns in there.

We\'re going to target six publications. So we\'re really drawing a true
statistical sample for the first time, and we\'re going to target six of
our flagship publications and ask specific questions about satisfaction
with those publications, in addition to going to hundreds of other
subscribers of other publications so we can say something general about
EIA overall.

The cover letter is saying, you know, the classic we want you to help us
out and give us your feedback, but there\'s a kicker in there that, I
think, is going to get their attention. One of the first sentences says,
\"We are considering reducing the number of printed publications in
favor of electronic versions.\"

Now if you\'re a paper subscriber and you\'re afraid of the Web, my
thinking is this is going to get you to take the survey, because you
want to get your two cents in here.

We\'re going to ask them about their capabilities first and then
preferences. We have questions in this survey, are you aware that
there\'s an electronic version of this publication, not just if it were
available. It\'s available. We know it\'s available. You know, are you
aware that it exists in electronic version? Have you ever used it, yes,
no; and if they say yes, we\'re going to ask them which version do you
prefer.

There are choices why they would prefer electronic or why they would
prefer paper, and those choices have mostly come from responses that
we\'ve gotten previously on other surveys of why people \-- you know, I
use it in court. I like to read it in my easy chair. I just like paper,
or it\'s so much faster, so much cheaper to get it electronically.

We\'re going to try to get a real read on this segment of our customers,
which again, remember, is our \-- these are our paper people we\'re
going to, and we\'re going to try to get a feel for their Web
preferences and their capabilities. Some people might say I don\'t even
have access.

We\'re going to ask them our favorite question, you know: Which one do
you prefer? Then a final question, and Larry touched on this, we decided
we would try something new and say to them we\'re considering printing
short summaries of publications and then putting all the detail and the
tables on the Web; would that meet your needs.

So that\'s the first sort of market research into a whole different kind
of product line/product distribution. So we\'re right on the brink of
finishing up our sample selection and we\'re going to print and mail, I
hope, within the next month.

We\'re hoping for a good response rate on this. Dr. Dillman talked to us
a lot about contacting people a number of times. So we\'ve got the
prenotification letter and the thank you card and the second
notification and the second \-- you know, we got a whole bunch of
different ways we\'re going to contact them.

Then probably by the next meeting or at least by the spring, I\'ll have
results of that.

Then an interesting question that we talked about yesterday was is the
Web-site a viable tool for getting serious data collection? When we were
talking yesterday, I just brainstormed some ideas, because we\'ve had
experience with \-- mostly with telephone and some with \-- and a couple
on the Web.

So the Web is a cheap way to go. I think it\'s easy. Once it\'s up,
you\'re finished. There\'s no paper coming in. Now see, the flip side of
that is there\'s no paper coming in, and if you lose the electronic
stuff, then it\'s gone.

There\'s no printing. There\'s no mailing. There\'s no stuffing. You
know, that all goes away. The survey person \-- The respondent can take
it anytime he wants. If he clicks onto the Web-site at two in the
morning, he can take the survey then. It\'s not when I\'m calling him up
and interrupting him.

My last point was I\'m not interrupting people. Sometimes when you call
people to survey, you don\'t know \-- you know, they\'re not necessarily
inclined to talk to you for a long time.

The issues that I saw that are sort of on the other side of the Web
survey are the self-selected sample versus the full population, and
anybody can say, well, those are just the grouchy people that wrote in
or those are just the happy people that wrote in.

So it\'s not clear to me how you take the self-selected sample results
and apply them to the whole population. One of my pet peeves is you
can\'t see how long the survey is. We love this, because they can just
say, hey, look at this, yeah, I can fill this out, it\'s 20-something
questions.

On the Web, you don\'t know if you\'re going to be clicking for 20
minutes, how many screens there are. I had several people on another
survey that we fielded electronically that said I couldn\'t get my \--
You know, they would go to the circle and they would click, and their
machine wouldn\'t click on the circle.

So what they did is they E-mailed me the answers: One, yes; two, no;
three, maybe. Or they called me with the answers. A minority of people,
but some people still couldn\'t make \-- Even though they received it
electronically, they were on our Listserver, whatever, they couldn\'t
send the answers back to me, for whatever reason.

I\'ve done a lot of the telephone surveying in EIA, and I feel like when
you talk to people, you really get the sense. You can feel their
enthusiasm. You can feel their anger. You can feel their confusion. I
think when you get the same kinds of comments on the Web, they\'re a
little bit flat, even if they put 20 exclamation points.

It\'s not the same as actually talking to somebody. It\'s also not as
\-- I mean, it\'s a lot more labor intensive to talk to those people on
the phone, but I think you get a better sense of what people are
feeling.

Also I notice on the telephone people say, do you have any \-- is there
anything else you\'d like to say. Ninety-nine percent of the time people
say not really, but you know, there\'s this thing \-- and then they go
on for 20 minutes. They do have things they want to tell you.

Typing \-- I\'m not sure how many people are going to go on for 20
minutes typing their answers. You know what I mean?

Also on the telephone survey you can \-- This isn\'t a nice word, but
you can force people to pick not the right answer but one answer. We had
a difficult question on the telephone survey this time where not only
did we ask satisfaction with timeliness, accuracy, comprehensiveness and
relevance, but we asked them what\'s the most important to you.

Their answer was, oooh, could you read them again? Can I pick four? I
can\'t pick. Well, so I told my interviewers, try to have them \-- even
though they \-- you know, sympathize with them, and they\'re all
important, try to get them to pick one. You can\'t do that on the Web,
and that\'s what happened with the country analysis briefs survey, is
people said I\'m going to pick my own answer. Then you don\'t
necessarily know what you\'ve got, if enough people do that.

Also in this new survey we\'re going to send what I call party favors,
incentives \-- These are two little neat things from EIA \-- to try to
entice them to respond to our survey. On the Web, you can\'t do that.

Also, I don\'t know how you do nonresponse follow-up on a survey on the
Web. Obviously, since they\'re just coming in in waves, there\'s no way
to get back to whoever you wanted to respond to.

That\'s all I had. Thank you. Did anybody have any questions? I also
have copies of my black and red flags.

MR. SUDMAN: It\'s not a question, it seems to me, of the Web or the more
standard procedures. The example is very much what most major companies
are doing today. That is, most companies have a hotline, and the Web is
indeed the hotline, and they are picking up the kind of very important
useful information.

Hey, if there\'s a problem with a product, you know, somehow it\'s not
working, all you need is one person to tell you that to alert you to
that. So there are enormous benefits form having this hotline.

My wife was on it last week complaining to one of her favorite stores
about a product she used to like very much, which was no longer in
stock. The people there said, you know, we\'re having trouble, they\'re
just not producing enough. So we\'re not going to send it \-- we\'re
sending it to the places where it would sell the most, but we\'ll try to
get some to your store.

You know, it made her feel better. It provided them with information. So
it seems to me, the Web has an important role to play in the sense that
you\'re using it, but \-- I mean, it seems to me, you have a nice
conference paper in comparing the results on easy to use and meet your
needs, because you\'ve done a careful sample on the phone, and then you
have these other two methods which are clearly volunteer methods.

My memory is indeed you get, you know, on the Web especially, you know,
on the comments you\'re getting people who have had a problem. So I
mean, it seems to me you might well want to do that kind of comparison
and present it at ASA. It would be a nice little paper.

So there are some major uses of the Web, but I don\'t think it could
give you the overall sense of what your total customers are feeling. I
think you still need to use the other methods.

MR. HAMMITT: I was going to sort of second that about doing it both
ways, explicitly controlled trials and see what the differences are.

I would think the technology will let you get around some of the
problems like the fact that people wouldn\'t choose only one. Can you
set these things up so they\'re only allowed to mark one box and you
shouldn\'t have another category, if you don\'t want people to give
that. So that kind of stuff you can deal with.

I think \-- I know one guy who does this in psychology, and he provides
people payment for responding to the surveys. Of course, it can\'t be
anonymous completely. So there are ways to do that kind of thing, and
you can, as you have done, E-mail to a defined set of people saying,
please respond.

If you are willing to compromise the anonymity in some way or
potentially compromise it, then you can tell who did, who didn\'t,
follow up with them. I think a lot of those weaknesses are things that
can be mitigated at least.

MS. BLESSING: Yes. Anybody else?

CHAIRMAN RELLES: Cal?

MR. KENT: Let me ask just a more generic question, maybe more of my
colleagues than of you.

Are there any problems that anybody else is encountering in getting
academic publications to accept citations to a Web-site? At least one
journal I read through will not accept them. You\'ve got to have a
library type of citation, because Web-sites can be changed. They\'re not
permanent.

Therefore, you know, if that\'s your source, it doesn\'t really count. I
just wondered if that was a problem anybody else had encountered and, if
you all had had, that may be the reason why at least a lot of academic
types are still wanting to have your paper product there, because they
know they\'ve got something that they can, as you said, take to court or
something like that. I just wondered what the situation was.

MR. SUDMAN: I hadn\'t heard that.

MS. BLESSING: I have never gotten that answer as a reason for paper, but
that\'s an interesting one.

MR. HANSER: The problem is that, if you say I\'ve pulled off gas prices
for the last ten years and you cite, you know, www.eiadoe.gov, it\'s not
acceptable. The series has to come from some explicit publication number
or name or something.

I mean, you can solve that by sort of saying \-- by having a paper copy
of which you do a limited number, and then on the data itself think back
to what the original source is, and then have that included when you
download the data off the site.

MR. HAMMITT: The reason journals insist on that is they want some
archival record.

Another \-- A twist on that will be there are some journals which are
published only electronically now, and I don\'t know what their
procedures are for archiving those, to site to those journals.

MS. BLESSING: I have had a few people say they want the paper for
archives, but they\'ve never said the citation. I just thought they
meant they loved to have a bookshelf full of them.

MR. KENT: I think there\'s more reason than needing insulation on the
wall.

MR. HANSER: In regulatory hearings also, they won\'t accept the download
of the information off the site also. They also like to have \--

MS. BLESSING: Some people \-- I\'ve talked to a couple of people who
said they go to court, and they would like to have \-- I mean, you\'re
not going to wave around a diskette or your printer or something, you
know.

MR. HANSER: You can hold up your disk, because in fact a lot of legal
cases are now basically being stored on CD-ROM, the evidence and all. So
CD-ROMs are perfectly acceptable.

MS. PHIPPS: Maybe I missed this, but in the survey you\'re mailing out
of your published, how are you dealing with like libraries, people who
aren\'t \-- you know, the respondent may not be the one who uses your
data. I mean \--

MS. BLESSING: We\'re taking them out of the sample. We\'ve had a number
of \-- We\'ve had some experience. You call librarians and you ask them
does the CD-ROM meet your needs or does the Listserver, and they say I
don\'t know, I just install it and I never use it again.

So we realize that they are not the end user and can\'t answer the
question. Actually, if I were a librarian and got this, you don\'t just
walk around and give it to the first person you see using that CD. So we
decided we would take them out of the sample. That\'s a good question,
though.

CHAIRMAN RELLES: Are you keeping \-- What kinds of databases are you
keeping of access to the sites? The reason I\'m wondering that is I know
from time to time it\'s contemplated that you either eliminate a survey
or reduce its frequency to some degree, and I would think that you have
some valuable information on the site regarding who is interested, how
many people are interested in what, that it might useful to be able to
tabulate when you\'re trying to deal with some of these issues.

Are data of that nature being kept or is it technologically possible to
keep them?

MR. PEARSON: I think we\'re gradually climbing into the 20th Century. We
now are starting to keep a very detailed daily, by the second, log on
who does what on the site, and we\'re starting to analyze that so we can
see specifically which documents get read and pretty well by whom \--
for example, government, Executive Office, whatever \-- or which
chapters of which documents get read.

What we\'re hoping to get to is a stage where we can produce a monthly
tabulation by document \-- not by reader but by category of reader, by
chapter, so we can get that kind of information.

The same applies to the data systems, the publications, the downloads,
and also the queries, the kind of questions people ask of the dynamic
data systems. What were they looking for? Can we make it easier for them
to find?

So we hope we are learning, but it\'s sort of an escalating process.

CHAIRMAN RELLES: You\'re able to capture the complete E-mail addresses
and \--

MR. PEARSON: No. We can capture their IP address and, unfortunately, a
lot of IP addresses are dynamic. So it doesn\'t help. A lot of people
refuse to give E-mail addresses. I tried to collect them, for example,
on the model downloads, and all we got was rubbish. So we just gave up
on that.

CHAIRMAN RELLES: Do you need the full IP address to identify the
institution? I would think that you could drop off the last about three
digits and still identify the primary server.

MR. PEARSON: Well, for organizations like Exxon or Enron, you can
identify those easily enough, but I think you discover that most of your
readers come from AOL or CompuServe, and that\'s useless.

MS. BLESSING: Anybody else?

CHAIRMAN RELLES: Thank you very much. I actually think we\'re going to
have a break now, unless I hear an objection. We\'ll reconvene in 20
minutes for Renee\'s talk.

I want to remind everyone that the committee and certain invited guests
are going to dinner tonight at 701 Pennsylvania Avenue Restaurant at
seven o\'clock. If \"oh\" means I would like to come, I forgot to tell
you earlier, there\'s a place. Can I see a show of hands of whom from
the committee can go to dinner. Okay. So we have about seven or eight.
Well, we have a room reserved which will accommodate about 12 or so. So
we\'re okay. Great.

(Whereupon, the foregoing matter went off the record at 3:20 p.m. and
went back on the record at 3:45 p.m.)

CHAIRMAN RELLES: We\'ll resume. Pleased to welcome Renee Miller, who is
going to be talking about standardizing data definitions and calculating
common response rates, I guess, a problem that surfaced when people had
access to all this information on the Web and began to notice the
inconsistencies. So it follows onto what we just heard.

MS. MILLER: Good afternoon. Can you all hear me back there? Okay. Well,
it\'s nice to see you all again.

You\'ve seen a demo of our Web-site, and you\'ve heard what our
customers think of it, and now I\'m going to tell you about some
challenges, as Lynda called them, that we are facing. That\'s that users
will be more easily notice our inconsistencies, and the inconsistencies
that I\'m going to be talking about are in data definitions and in
response rates.

Now there\'s something else on that overhead. That\'s end use
feasibility study, and that\'s not an inconsistency. It\'s a possible
way, actually, to resolve inconsistencies in end use sector definitions.

It\'s in Italics, because this project isn\'t quite as firm as the other
two. Currently, we are considering how and also if to design a study to
determine whether survey end users frequently is feasible.

So all of these projects are work in progress, and we are presenting
them to the committee now to set the stage for statistical issues that
may emerge later on. For instance, what to do about breaks in a series
if we do substantially change the definitions.

We also want to start a dialogue with the committee on areas that you
might be interested in getting involved later on, and also there is some
conceptual issues that we have been discussing among ourselves, and we
would really welcome your input.

Okay. All of these projects represent team efforts. The common data
definitions team is a formal team. The others are informal, and as you
can see, we have people from all over the organization.

To start with the problem of common data definitions \-- Okay. Well,
first of all, what is actually the problem here? We have an EIA glossary
of definitions that we published a few years ago, and it has multiple
definitions for some terms, for some 200 terms or so.

I should point out that some of these differences aren\'t drastic
differences but, nevertheless, we do have multiple listings. Now you may
wonder, well, how could this have happened, and I was going to point out
how EIA was formed by combining the data collection functions of
different agencies like Bureau of Mines and Federal Power Commission,
some agencies that no longer exist, and we inherited their surveys.

So these surveys were developed at different times in different places
for different purposes, but I also think that Larry made a good point
when he said, well, maybe we\'re just creative and it led to diversity.

In any event, we\'ve made progress over the years in resolving the
definitions, but we still do have multiple definitions, and after a long
and sometimes contentious practice we decided to list all of them in our
glossary.

So why would we want to revisit this? It was a pretty painful process,
to begin with. Well, we have as a strategic goal to have products
relative to our customers, and we\'re afraid that the users may be
confused by the multiple definitions, particularly since it\'s what
makes us so transparent.

So this time around, to avoid a contentious process, we are working as a
team and, after considering a number of options, the Business
Reengineering Steering Committee decided to charter the common data
definitions team with representatives from each of the offices to ensure
common and accurate definitions in most cases.

\"In most cases\" is the key phrase here, because in the past efforts
haven\'t been successful or sometimes not even started because we were
going for all or nothing.

Okay. Next overhead. Before going on to tell you about the activities of
the common data definitions team, this is an impressionistic version of
a schematic that we presented to the Business Reengineering Steering
Committee, giving them options to consider.

The X\'s represent the options, and for each option we rated the
likelihood of success on a low, medium and high scale, and also
estimated resources, same scale, low, medium and high. So the northwest
quadrant is how Bratcher Frederick describes it when he originally gave
this presentation. That\'s the most desirable place to be, because you
have a high likelihood of success and low resources.

By contrast, the southeast \-- that was the least desirable place: High
resources and low likelihood of success, and this was even before we
knew about the tornadoes there.

Going on to the activities of the common data definitions team, where we
are: We reviewed and discussed our charter so that we all have the same
idea of where we\'re heading, and we took finish motor gasoline and
gasohol as test cases, and we reconciled those definitions. We started
prioritizing terms to resolve, and we realized that we needed to group
the terms, and we are now working on coal terms.

Okay. Where we\'re heading: These are our expected outcomes and
deliverables from our charter. In addition to reconciling the
definitions, we want definitions that a broad range of users would
understand.

Many of our definitions were originally developed with the respondents
in mind, and now we\'re expanding our focus to users as well. For the
motor gasoline definitions, the first two sentences of our definition
that we developed is really a generic definition that we think would
appeal to a broad range of users, and then the remainder is more
technical for the survey respondents.

Then when we moved on to coal \-- this is something that just came up,
late breaking issue; I don\'t even have it on the overhead here \-- we
noticed on the coal forms that they don\'t have definitions for the
respondents.

So one proposal was to have just a generic definition, and right now
we\'re discussing, well, is that really a good idea to have a different
type of definition for coal than for motor gasoline. So we would welcome
your comments in that area as well.

We realized in some cases we may not be able to reconcile the
definitions, and so we will have warning links for our users of the Web
and footnotes for users of our publication, pointing out that there are
multiple definitions.

Then we are going to have a testing phase where we would test the
warning links, and also test the definitions to see if people could
actually understand them. We\'re going to have an implementation phase,
taking into consideration the forms clarity schedule and the publication
schedule, and then last but not least, we want to develop a mechanism to
avoid this problem in the future, particularly when we develop \-- or
when new terms are developed in the electric power restructuring area,
for instance.

We don\'t want to have this problem come up, you know, five to ten years
from now, as much as the team likes to meet.

So how can the committee help? Here is where we start our dialogue on
where you might want to be involved. Would you be interested in giving
opinions, for instance, on whether the new definitions we develop would
be understandable by users? Would you want to participate in pretest of
any warning links we might develop?

Something we need help on right now is helping us decide what is a
definition. How consistent should it be from topic to topic? Related to
this is should it include modifiers?

I\'ll give you an example of what I\'m talking about here on the next
overhead. We have a definition of residential. This is from the
Residential Energy Consumption Survey, and it starts occupied housing
units. Many of our other definitions of residential just starts talking
about housing units.

The last sentence here is also a modifier. Residential does not include
vacant housing units or second homes. What we have been discussing is
the last sentence, for instance, and the use of the word occupied.
Should those be part of the definition or is this really the way to
operationalize the residential definition for REC. So we would like your
opinion on that as well as something that we\'ve been going back and
forth on.

MR. HANSER: What do you do for multiple meter folks?

MS. MILLER: Okay. Residential \-- we just write into the end use
feasibility study, because the end use sector definitions, not only
residential but commercial and industrial as well, have been
particularly difficult to reconcile, and we have had many previous
efforts. The most recent was an interoffice team that attempted to
resolve the inconsistencies, and they made some progress, particularly
with the industrial sector, but discrepancies still remain.

What the end use feasibility study has to do with this is that surveying
end users would get us around this problem somewhat, because EIA, rather
than the energy suppliers would be defining the end use categories.

Now we have heard from some people, well, this is a pretty extreme
measure \-- you know, frequently surveying end users. In fact, even
considering surveying end users, you know, we\'ve been told is pretty
extreme, because there are so many more end users and suppliers in the
industrial sector alone. We\'re talking hundreds of thousands of end
users versus thousands of suppliers.

Cost estimates to survey the industrial end users, when we\'ve come up
with them in the past, they\'ve been deemed too high to pursue,
particularly since we wouldn\'t settle for anything less than state
level estimates.

While the consideration of end use sectors came up in the context of
resolving inconsistencies in end use sector definitions, it\'s not the
only motivation.

Taking the long range view here, it would better prepare us for the
future where prices from surveys of suppliers may only represent a small
proportion of the market. What I\'m talking about here is actually in
another paper. It\'s in the paper by Kass, Harris, Martin and
Casselberry on impacts of deregulation on respondent cooperation, and
Roy Kass is going to be talking about this tomorrow.

In that paper there\'s a chart that shows for industrial natural gas
EIA\'s prices only represent about 20 percent of consumption, and this
is kind of giving coming attractions for Roy tomorrow; but he\'s going
to show you a graph that has shown that the proportion has been going
down, down, down over the years, and we\'re concerned that this is going
to happen in the electric power area as well.

Many of you are familiar with what is happening. End users are making
deals directly with the producers and marketers in natural gas, and the
suppliers that EIA is surveying, which are the pipelines in this
situation, they know how much gas is transported, but they don\'t know
the price since they don\'t own the gas. So they can\'t report the price
to us.

At the same time that this is happening, we\'ve been hearing from the
focus groups that we\'ve been conducting in both the natural gas and
electric power area that there is more interest in equity issues, in who
is paying what price. Phil Hanser touched upon some of these issues this
morning.

So you might ask, well, why don\'t we just consider surveying the
marketers in natural gas and, in fact, we have. We did a pilot study,
and we found that the marketers didn\'t always keep their records
classified by end user sector. In fact, some of our traditional
respondents in the gas utilities are having more problems in reporting
data by customer class.

So we thought by considering surveying end users, we would avoid this
problem of respondents\' records not corresponding with end use sector
definitions. Of course, it opens a whole host of other problems like
whether respondents will be able to actually report the price.

Another reason to consider it is that the technology for conducting
surveys has changed. There\'s more electronic collection and processing.
So while the cost estimates were originally off the chart, possibly by
the time we actually do this, if you think of this as sort of a long
range project, they may be on the chart. It may be towards the top, but
they may be within range.

So where we are here: We have our objectives: Investigate potential
issues and cost to move EIA in the direction of frequently surveying end
users, and we\'ve identified four issues relating to the scope of the
surveys, whether frames are available, whether respondents are willing
and able to report on a monthly basis, for instance; if so, what mode of
collection would be best for them, and whether there are creative
sampling schemes that would allow us to obtain the monthly state level
data without going to sizable samples in each state.

For each of these issues, we\'re in the process of developing options
and pros and cons and estimated resources and likelihood of success to
get back to the Business Reengineering Steering Committee with a menu of
options.

If we decide to go forward, we\'re going to need help from the
committee, particularly in the creative sampling scheme.

We are on to our last topic, the common definition of response rates.
The problem here is that the definition or the definition I have up on
the overhead is, as Bob Gross calls it, the most frequently endorsed
definition, number of completed interviews divided by number of units
estimated to be eligible. That definition isn\'t being consistently
implemented throughout EIA.

We just have two problems. What is complete, and what do we mean by
eligible? Why we want to address this again has to do with the Web where
we want to put information about the survey design up on the Web, and we
thought it would be good if we put response rates up, that they were
consistent from one survey to the next.

Another motivation is performance measures. As EIA is moving to being a
performance based organization we\'re developing and maintaining
performance measures, and we want to include response rates as a
performance measure.

So our objective again is a common definition of response rate in most
cases.

Where we are is that we have polled 20 survey managers, and the \"we\"
in this situation is Tom Broene back here. We found that there were
actually reasonable explanations for different definitions of both the
numerator and the denominator of what a response is and who is eligible.

For instance, for some surveys obtaining data up to a certain point will
provide enough information to impute for the rest. So they consider that
a complete response. In other surveys, the data aren\'t useful unless
they have practically every last item. So they don\'t consider it
complete until they have practically everything.

We also heard from the survey managers that they didn\'t think that the
response rate as a publication was really a good indicator of any
problems with the operation of their survey, and it\'s possible that
response rate may be more useful for people at a level other than survey
manager.

We plan to find out, though, how all survey managers are calculating the
response rate, so we can get a better idea of what\'s involved here. We
basically have gone to about half of them so far.

What you\'re waiting for, how the committee can help here: We\'ve
identified some, shall we say, challenges in coming up with common
response rates, and these are some questions that we have been
discussing and that we would like your opinion on.

The first one has to do with volumetric response rates. What I mean by
that is, when the respondent is multiplied by the volume \-- for
instance, coal production \-- and a sample weight if it\'s a sample
survey. So that would be the numerator, and then the denominator would
be an estimate of total volume.

In a way, these rates are appealing, because they give a good idea of
the impact of nonresponse on the published data. Okay, so what could be
the problem here? Well, in a survey like coal production, there really
isn\'t a problem, because coal production is the major item of interest;
but there are some surveys where we collect information on consumption
and expenditures and square footage.

So the question is do we put everything in our database? Do we pick one?
Another question that has come up is do we include the sampling weights.
Speaking to people from other agencies, sometimes they do, and sometimes
they don\'t, and this is something we\'ve gone back and forth on and,
you know, would like to hear from you on.

Another question has to do with the issue of eligibility. For a weekly
survey sometimes it\'s difficult to verify whether a respondent is
actually eligible. So it may turn out that some nonrespondents are
actually out of business or, for some other reason, ineligible, but we
may not find that out until the end of the year.

So the question is, say that that doesn\'t really have much of an
impact. Would it be acceptable to continue doing what we\'re doing and
just caveat the response rate?

The next question has to do with the numerator about completeness, about
what is actually a response. If we define a response to be a usable
record, then does it matter if what\'s considered usable varies from
survey to survey?

Another topic that has come up, mergers. When I first heard about that,
I thought, well, you know, what difference could this make, but it\'s
going to be more and more common, particularly in the electric power
area.

We\'ve heard the argument that EIA is actually obtaining the same amount
of information we had both before and after the merger, assuming that
the companies involved were reporting both before and after the merger.

The argument goes, well, therefore, the response rate shouldn\'t change.
Well, what do you think? When and how should we reflect the mergers, as
soon as we know of them, at the end of the year so that everybody would
be doing it the same way?

Then what about response rates computed prior to publication, since we
were told that the response rates computed as a publication weren\'t
that helpful in managing the operation of the survey? If we compute
response rates at an earlier point such as when follow-up starts, then
how important is it to standardize those rates, since they\'re basically
being used to manage the operation of an individual survey?

Then our final question on this: Thinking of yourselves as users, how
important would standardized response rates be to you? In other words,
is this really worth the effort? Would explaining what the response rate
for a particular survey represents be an alternative to standardization?
We\'re interested in your response to this question and all the others.

Thank you.

CHAIRMAN RELLES: Carol Crawford is our discussant for this paper.

MS. CRAWFORD: This paper was shorter than some of the other papers you
might have read, but the number of questions is much greater, I think,
than in the other ones. So I decided, if I could do a very good job at
answering about 80 percent of these questions, that this would be a very
effective discussion.

So next slide. The challenge that they had was to develop definitions of
energy and energy related terms, and you could tell from Renee\'s
presentation that this is a very philosophical discussion, and it\'s
actually quite profound; because what she wants is a definition of a
definition, and that\'s kind of hard to do, but you\'ll see, I\'m going
to try.

So the first one: What is a definition, and should the definition
include modifiers that make the concept operational for a particular
system, and how should breaks for the series resulting from a
substantial change in definitions be handled?

Next slide. Thanks. So some suggested answers: I attempted to define a
definition. It\'s as a description of a word or phrase that can be
distinguished from other words or phrases.

One thing I\'m absolutely sure of is that there should only be one
definition for any given term or phrase. When I was thinking about this,
I found it very difficult, and I went back and relied on my training in
technical writing.

Several times during the course of my academic career, I took courses in
technical writing, and I found them to be very helpful. So I went back
to my course notes, and I found this definition from technical writing.
A definition consists of the term to be defined, the general category to
which the term belongs to, and then distinguishing features that serve
to distinguish this term from other members of the general category.

Let me give you an example, actually two of them. Next slide, please.
Let\'s say we want to tell someone what a fork is. Okay, the general
category is that it\'s an eating utensil, and what do you have to do?
You have to distinguish it from other eating utensils, most notably a
knife and a spoon.

So you might say the distinguishing features are it\'s a long \-- it\'s
a metal instrument or it\'s an instrument with a long, thin handle with
tines on one end that\'s used to pierce and scoop food. There could be
different types of forks, salad forks, dessert forks, but they\'re all
still forks.

The rule of thumb, and I saw the tendency for Renee and Tom to want to
do this, is to define the most general term first and to make it as
general as you possibly can. Then it\'s okay to have different types of
that or different categories or different states.

You could also go on and then attach a definition to each one of your
states, like salad fork is a fork that\'s used to, you know, eat salad
or it sits in a particular place on the table; but then you run into, if
every time you use a word and you use more words to define it, you end
up having to give definitions of all those words, and it\'s not clear
whether or not that algorithm converges to something that\'s meaningful.

So my suggestion is to define the most general thing you can, and then
let it be okay for there to be states in that area which you could
define, if you absolutely have to.

So I tried to define residential, and I don\'t want to sort of
trivialize their efforts in any way by thinking I\'ve succeeded in doing
this, because I felt the need to be constructive. On the other hand, I
haven\'t thought about it. I\'m not aware of the nature of all the 200
definitions that they have to reconcile, and this may not be a very easy
thing to do, but I tried.

Let\'s try the term residential. Category would be type of energy, end
use sector. Distinguishing feature would be having to do with places in
which people live. Then for your residential energy consumption survey,
you would have a state of residential dealing with occupied residential
units, and then if you ever had the need, there could also be unoccupied
units or various other states, depending on the nature of the survey.

Next slide, please. I thought it was great that you\'re thinking about
what to do if they do change definitions. So many times in my job, and
it actually came up last week where somebody went and they changed all
the questions in a survey that\'s been collecting data for, oh, 25
years, and now nothing matches and nothing is the same, and the same
thing, is it going to be asked, and they couldn\'t understand why that
bothered me.

So it was great that they\'re thinking ahead about what do they do if
they change things. The way I know what to do comes from what I learned
from Census Bureau statisticians. The idea is over a transition period,
the length of which \-- What actually goes on here are details I\'m not
going to get into, but you want to do things the way you used to do them
and the way you\'re going to do them simultaneously.

This came up more sometimes in the area of measurement than actually
changing definitions where measurement technology gets more refined as
time goes on and things are measured more precisely or there might be an
actual better way to measure something.

So what you would do is measure using the same instrument or the same
procedure both ways for a while, and then gradually phase the old way
out. The idea is then, for a certain length of time, you have both
things going on at the same time, and you can calibrate things, and it
gives you a link so that the historical data in your series isn\'t lost.

I haven\'t been anymore specific than this, because I\'m not sure to
what extent they will actually have to deal with this, if they\'re
successful in getting this structure for their definitions.

I do know that Census Bureau statisticians have set a precedent for
dealing with this issue, and there have been several papers in the
[Journal of the American Statistical Association]{.underline} which I
could look up for you, and some other members of the committee might
have some more information for you here, too.

I\'m skipping comments on the end user survey. They asked a lot of
questions there, but because they\'re not sure they\'re going to do it
yet, I didn\'t address them. I addressed one at the end that you\'ll
see.

So we move on to their second challenge, which is to develop a
definition of a response rate. The current definition that they have is
the number of completed interviews divided by the number of units
selected that are estimated to be eligible.

I think the problem that they\'re having is they need to perhaps define
some key terms in both the numerator and the denominator. Different
surveys interpret the term completed interview a little bit differently.

A suggested solution to this is to define a completed interview as any
interview giving useful information, and then allow that to vary from
survey to survey. So in the paper she talked about a weekly \-- some of
their weekly surveys being able to get more information than other
surveys who actually require that all the questions be answered or that,
in fact, all the answers are usable, and that\'s their definition of a
completed interview.

I think you can just leave it as whatever is useful for that particular
survey.

I didn\'t think that the use of volumetric response rates or weighted
response rates gets around this fundamental issue. It\'s just measuring
a response rate in a different way. I thought that it might be useful to
some users, but I thought it was starting to make things a little bit
more complicated.

Another problem is how to define eligibility, you know, what makes a
unit eligible. Maybe in this broad view of things, is there a way that
we could define eligible that could then be allowed to vary from survey
to survey, so we have one definition for eligible but it can \-- there
can be various states, depending on what survey you\'re in.

Next slide, please. As a user, how important are standardized response
rates? I think it would be very useful to have a standard procedure for
determining response rates. I do think that this definition could be
independent of the time at which the response rate is determined.

In the paper in one example, response rates changed depending on whether
or not they were as of publication or at the beginning of follow-up, and
I think if you make your definitions broader, you could have a
standardized response rate with various states, one state being at
publication and at the beginning of follow-up, and still have the
definition the same.

I also think that this has to be balanced by other issues that EIA also
needs to address, and I think that getting other definitions for key
energy terms, residential, commercial, some of the other terms that
Renee talked about is much more important than finding a standardized
response rate.

So I think that, if standardizing response rates continues to be
problematic and very time consuming, then an acceptable alternative is
to provide an explanation for the details in the calculation of the
response rate associated with the particular survey. I think most people
have a general idea of what a response rate is trying to get at, and
then if you just give the particulars, that that would be satisfactory.

So in conclusion, my first recommendation is to continue to define
concepts in the most general way possible, and don\'t let the specifics
deter you from this path. I saw in the paper that Renee and Tom kept
wanting to do that, but there was always this \"but what about this
case, but what about this case.\"

I\'m sure that\'s tempting with the nature of the task they have at
hand, but if they can hold fast to trying to make things general, I
think they\'ll be better off.

I think it\'s okay to have various states within one definition. So for
the example in the residential, define residential, and then let things
be occupied or unoccupied as the case may be.

I wonder if you had a good technical writer on your team or if a
technical writer would be helpful. These kinds of issues are what
technical writers are great at. They\'re good at manuals. They\'re good
at cross-referencing. They\'re good at making things broad and precise
at the same time, and that\'s \-- Technical writing books are actually
where \-- the first place I went to try and help with some of these
issues.

As for the end user survey, I\'m not ready to agree that this is needed.
Of course, I\'m also hesitant to say that it\'s not needed. I guess what
I \-- I\'m still hopeful that you\'ll be able to come up with a system
of definitions that will work for you without having to go to the
survey.

Of course, there are other benefits from this survey, too, that also
make it attractive. So my recommendation is that at least I need to be
better convinced, I suppose, one way or the other as you get more
information.

Finally, my last recommendation is that developing common data
definitions is much more important than determining standardized
response rates. If EIA has the time and the interest to do both of them,
I think that\'s wonderful, but if, as if often the case, the budgets and
the manpower sort of limit your capabilities, I would concentrate more
on developing common data definitions than I would worrying about
standardizing response rates.

CHAIRMAN RELLES: Thank you very much, Carol. Seymour?

MR. SUDMAN: Very interesting paper and a very interesting response from
Carol.

I\'m going to make somewhat different kinds of comments. In the ideal
world, you would want three different groups to agree to definitions.
You would want the people that supply the data to agree. You would want
the people who use it to agree, and you would, of course, want the
technical staff at Energy to agree.

That\'s a really tough thing to achieve. Nevertheless, my first thought
is, especially in situations where internally you\'re not certain, why
haven\'t you \-- or maybe you have \-- Why haven\'t you talked to both
the suppliers of the data, what do you think we mean when we say
residential, and to the people who are actually using it.

If indeed you can find some commonality between what they think, both
the suppliers and the users think, that seems to me to suggest that
that\'s a definition you could use. If they don\'t agree, then you\'re
going to be have to be the referees and define it, and then tell people
at both ends what your definition is. But I mean, if they do agree, it
seems to me that that\'s the way to go.

I\'m going to secondly talk about the issue of the response rates. I
agree with Carol that, in terms of the numerator, the issue should not
be, you know, were 12 out of 16 questions completed, were 13 out of 16.
It should be the decision of the person who is using the data. Does this
survey provide useful information.

If it does, then, you know \-- Most surveys provide perfect information.
There\'s always going to be, in virtually every survey, some item
missing, some items that need to be imputed. If it provides useful
information, it should be included in the numerator and, it seems to me,
that\'s the way \-- and that\'s what the respondents \-- what the users
should be told.

Now the managers, on the other hand, I think, is a different story. I
think the managers are complaining, hey, you\'re worrying about
providing data to the end user, but the data doesn\'t come to us in time
or something of this kind.

It seems to me they\'re really saying you need some sort of an
information system as the survey is ongoing to tell them, hey, are we
having problems with cooperation. Are we having problems finding people
at home? Do we need to, you know, increase the number of call-backs? Do
we need to change interviewers or something?

It seems to me, they\'re asking \-- and it doesn\'t seem to me that
needs to be standardized. It seems to me what you have to do is have a
system which in general works for each of the managers, but which is not
the same thing as the end user survey.

I guess I\'ll stop.

MR. WHITMORE: I\'d like to make just a few limited comments as well. I
think it might be helpful. Maybe I can provide you some more detailed
comments to some of these questions, since you asked so many, you know,
off the record.

With regard to the differences in definitions, it occurred to me that
some of that may be more than just definitional differences. There may
be some real differences in what would we measure.

In terms of thinking of residential energy consumption, if you go to the
utility company and get total power delivered to residences, that really
is a slightly different entity, and that\'s reflected in the definition,
from what you get in the residential energy consumption survey; because
you are dealing \-- I think there were actually two modifiers on the
residential.

You\'re dealing with residences that are, one, occupied and, two, that
are the primary residence, not a secondary residence. You may really be
interested in total consumption of electrical power in residences,
whether they are occupied or not.

A vacant residence that is being shown for sale may well have the air
conditioning going and other appliances, and it\'s part of residential
energy consumption. A second home, certainly, consumes electricity, and
you probably want to account for that in some category. It probably
needs to be residential.

I think there\'s more going on than just differences in definitions.
It\'s also partly what are you trying to measure, and in that case, if
you\'re thinking of replacing the utilities survey with the end user
survey, you have to think as well of can we really measure what we
really want to get in terms of total residential energy consumption from
the end user survey.

I guess I\'ll comment as well on response rates, everybody\'s favorite
issue. I think at a minimum every survey should provide kind of the
standard definition of response rates, and provide that in terms of
getting useful information for that study.

On the other hand, if you want to provide additional response rates,
weighted response rates in terms of sampling weights or volumetric
weighted response rates, I would think those would be additional things
that may be of use for particular surveys; but you should always have
kind of the standard baseline unweighted response rate to go along with
that, but even that, it seems to me, has some limitation in terms of
being able to use it as a performance measure.

If you want to use the response rates across surveys as a performance
measure comparing different surveys, it would seem like that almost
requires a very standard definition to use as the performance measure,
and you might have to contrive something that\'s used within EIA as kind
of your performance measure type response rate.

Maybe it\'s something like at least 90 percent of the items were
completed. That may not even be something you put on a Web-site, but is
used internally as a performance measure, that you have a very standard
definition that you\'re using for comparing how well different surveys
are doing.

MR. HANSER: Just a couple of comments, one on response rates. I guess I
like volumetric measures for response rates. I\'ll tell you why. If you
tell me that you surveyed a bunch of gas users and you surveyed 20 of
them, but it turns out they represent 90 percent of all the gas
consumption in the market, who cares if there are another 100 sitting
out there in some ways who represent only ten percent?

So I think in certain situations it\'s clear that \-- It seems to me
that it\'s pretty clear that you want volumetric measures of some sort.

MR. WHITMORE: I would agree, in certain situations you would want that.
That wouldn\'t be what you want for your standard for everything.

MR. HANSER: Right. It isn\'t the standard for everything, but there are
clearly \-- For example, we\'re talking about things like fuels and all.
I think it\'s very useful to know that.

I also think that it would be useful from the standpoint of the user of
the data to know what that percentage was. If you tell me that you\'re
only catching ten percent of all the industrial contracts that are out
there on gas again \-- right? \-- that\'s a big deal, because that means
that 90 percent of the contracts are not somehow being covered.

It would also be nice to know what that volume represents of the market
and, if possible, since this is done on a regional basis, how does that
response rate vary across regions; because, for example, for some of us
who work with these numbers, you know, it\'s important to us to know
whether or not the number I\'m looking at represents 90 percent of the
market in the Pacific Northwest but ten percent when I look at the
Northeast.

Right now in the publication that you get from EIA, you have no
understanding about what that number \-- what that looks like. At least,
it\'s not listed in the publication at this point. So it puts you at
risk for the potential variability in that estimate.

Secondly, I don\'t think that utilities, frankly, are going to do
surveys anymore. I think those days are very limited in the future,
because there\'s just no reason to. The minute you break up the
integrated utility, their interest in doing surveys of their customers
except for possible marketing purposes goes away.

The regulatory requirements around integrated resource planning go away,
which is what has motivated in the past most of these surveys of their
customers. Therefore, the utilities aren\'t going to do this anymore.

So I don\'t know that there is going to be any way for the EIA in the
long run, if it wants to continue to acquire this data, than doing it
directly by itself, and it will be put in the same position, I guess,
that the Bureau of Labor Statistics is put in relative to all the other
data that it collects that way.

So, you know, I just don\'t see that continuing very long. You may even
have trouble, for example, even getting the consumption, because there
may not be any central clearing mechanism which, for example, retains
the kilowatt hour data, because the distribution entity may not, in
fact, have the billing capability or may not be doing the billing. It
may be billing by the marketing entity that sells the power and, unless
you survey them, you\'re not going to get that information.

So you may find yourself with some fairly serious disconnects between
the information you\'ve been collecting in the past and the information
that\'s liable to be available to you in the future.

So I think, you know, you\'re going to have to sit fairly seriously and
think about what some of those implications are in terms of your
requirements in the future, for example, to calculate greenhouse gas
emissions that are associated with \-- that come as a result from
differing levels of consumption.

So that\'s my comment.

CHAIRMAN RELLES: Well, you\'ve received a lot of very good comments,
Renee. Would you like to respond to any of them?

MS. MILLER: I\'d like to thank you all for giving us these comments, and
Carol, I think that your categorization for a definition in terms of the
term, the general category and distinguishing features, I think, is
going to help us. We\'ll just have to keep thinking about different
types of forks when we have our discussion.

When we do have situations where we create breaks in the series, I think
we will start thinking about bridge estimates. What\'s happening now
with the definitions that we\'re looking at, some of them look different
on paper, but they, I think, really mean pretty much the same thing, but
we\'re using different terms.

We\'ve used the expression real as opposed to apparent differences. So,
so far, we haven\'t come across any where, if we made a change, we would
have a break in the series.

Let\'s see. I\'m just kind of following my notes here. On the response
rate, I think if we do define complete \-- I think several of you have
said, if we define complete as meaning giving useful information, and
even though I used the expression interviews, actually most of our
surveys are self-administered mail surveys; but I think if we do define
it that way, then that would make things simpler.

When Tom and I have discussed it, you know, we have found volumetric
response rates appealing, you know, for the reasons that Phil has
pointed out, and we just weren\'t sure what to do about keeping them in
the database, but it sounds like what you\'re saying is that we
shouldn\'t make those the standard.

Oh, Phil, when you were saying that utilities aren\'t going to survey
customers, I just wanted to make sure I got that straight. So that
means, if we ask for information about what were your sales to the
residential sector, they wouldn\'t \-- or what were your sales to the
commercial sector, that they wouldn\'t be able to report that anymore?

MR. HANSER: Yeah, that may not exist anymore. That simply may not exist
at the utility level, because the entity that will be selling
electricity to the customers is going to be some marketing entity \-- I
mean, eventually. The first few years of deregulation and all that might
not happen so quickly, but already in California, aren\'t you getting
calls?

CHAIRMAN RELLES: Yes.

MR. HANSER: Right. And so that information \-- that\'s simply not going
to \-- It\'s simply going to be a wire company. It\'s not going to sell
anything to you. So they\'re not going to have that information.

MR. SUDMAN: I have one comment on that. You know, it may well be that
what\'s going to have to happen are sort of messy combinations of
surveys. That is, you may very well still want to use the supplier
surveys to get to sort of a detailed data by state, for example, but
then use a much smaller end user survey to be able to distinguish
between residential, commercial and, you know, any problems with that.

So, I mean, combining those may give you what you want without having to
do like huge end user surveys.

MS. MILLER: That sounds like that will have to go. Tom, was there
anything on response rates that you wanted to add? Thank you for all of
your comments. You\'ve given us a lot to think about.

CHAIRMAN RELLES: Before we conclude, I would like to invite public
comments from the floor. Well, in that case, why don\'t we take another
break and reconvene tomorrow. The coffee, tea and juice service will be
in the other room at 8:30, and we\'re scheduled to start here at nine
o\'clock with the presentations of the graphics contest winner.

(Whereupon, the foregoing matter went off the record at 4:38 p.m.)

\- - -