AMERICAN STATISTICAL ASSOCIATION

\+ + + + +

COMMITTEE ON ENERGY STATISTICS

\+ + + + +

SPRING MEETING

\+ + + + +

THURSDAY

APRIL 13, 2000

\+ + + + +

The Committee met in Conference Room 8E-089 in the Forrestal Building at
10th Street and Independence Avenue, S.W., Washington, D.C., at 8:30
a.m., Carol Gotway Crawford, Chair, presiding.

[PRESENT]{.underline}:

CAROL A. GOTWAY CRAWFORD, PhD Chair

JAY BREIDT, PhD Member

THOMAS G. COWING, PhD Member

CALVIN A. KENT, PhD Member

W. DAVID MONTGOMERY, PhD Member

WILLIAM G. MOSS, PhD Member

POLLY A. PHIPPS, PhD Member

RANDY R. SITTER, PhD Member

ROY WHITMORE, PhD Member

JOHNNY BLAIR, PhD Member

I-N-D-E-X

[Page]{.underline}

Welcome and Introduction 3

Chairman Crawford

Opening Remarks 9

Jay Hakes, Administrator, EIA

Panel Discussion on Confidentiality 29

Legislation: Jay Hakes, Jay Casselberry,

and Robert Latta, EIA

System for Analysis of Global Energy Markets 89

Mary Hutzler

Cognitive Interviews on EIA\'s Web Site: 134

Seeking More Reactions:

Colleen Blessing

Summarizing the ASA Advice 164

Douglas Hale, Mark Schipper, Dwight French

Redesign of Electricity Data Collections 182

Robert M. Schnapp, EIA

P-R-O-C-E-E-D-I-N-G-S

(8:32 a.m.)

CHAIRMAN CRAWFORD: Good morning, everyone. I\'d like to go ahead and
open this meeting.

To begin with, I have a few opening announcements about the procedures
of the meeting. This meeting is being held under the provision of the
Federal Advisory Committee Act. This is an ASA, not an EIA committee,
which periodically provides advice to EIA.

The meeting is open to the public, and public comments are welcome. Time
will be set aside for comments at the end of each session. Written
comments are welcomed and can be sent to either ASA or EIA.

All attendees, including guests and EIA employees, should sign the
register. As before, we are asking for your E-mail address.

In commenting, each participant is asked to speak into the microphone.
The meeting is recorded, and there are formal minutes that are typed up
and presented. The transcriber will certainly appreciate it if you speak
into the microphone.

This is also true for committee members at the head table, and you can
see there\'s microphones almost in front of every person.

I\'d like to introduce, actually, three \-- I\'ll introduce two of our
new members. We have three new members at this meeting, but the third
member couldn\'t come. Our first member is Bill Moss of the Brattle
Group \-- welcome; Randy Sitter of Simon Fraser University. And we also
have an invited guest, Johnny Blair from University of Maryland.

Our third committee member, Mark Bernstein, couldn\'t be here today. He
and his wife just recently had a baby. So he will be joining us in
November.

Now I\'d like to go around the table and then to the audience and ask
each of you to introduce yourselves and to give your affiliation. I\'ll
start.

I am Carol Gotway Crawford, and I\'m with the National Center for
Environmental Health which is part of the Centers for Disease Control
and Prevention.

MR. COWING: I\'m Tom Cowing. I\'m with the Department of Economics at
Binghampton University in New York.

MS. MILLER: I\'m Renee Miller from EIA.

MR. MOSS: I\'m Bill Moss from the Brattle Group in Cambridge,
Massachusetts.

MR. MONTGOMERY: I\'m David Montgomery from Charles River Associates in
Washington, D.C.

MR. KENT: I\'m Cal Kent from Marshall University in Huntington, West
Virginia.

MS. PHIPPS: I\'m Polly Phipps at the Washington State Institute for
Public Policy in Olympia, Washington.

MR. WHITMORE: I\'m Roy Whitmore with the Statistics Research Division of
Research Triangle Institute.

MR. SITTER: Randy Sitter, Department of Mathematics and Statistics,
Simon Fraser University.

MR. BLAIR: Johnny Blair with the Survey Research Center, University of
Maryland in College Park.

MR. BREIDT: Jay Breidt from the Department of Statistics at Iowa State
University.

MR. HAKES: Jay Hakes, EIA.

MR. WEINIG: Bill Weinig, the Energy Information Administration.

MR. LATTA: Robert Latta from EIA.

MR. CASSELBERRY: Jay Casselberry, EIA.

MR. MILLER: Nate Miller, EIA.

MS. SPENCER: Lynda Spencer, Edison Electric.

MR. MOREY: Matt Morey, Edison Electric Institute.

MR. BROAN: Tom Broan, EIA.

MR. FREEDMAN: I\'m Stan Freedman from Energy Information Administration.

CHAIRMAN CRAWFORD: Thank you. Lunch for the committee and invited guests
will be at 12:30 in Room 1E-226 on the first floor of the building.

Since we are having break-out sessions directly after lunch, if the
members of the committee could hook up with the presenters for their
break-out sessions, that would be Doug Hale, Dwight French, and maybe
Nancy Kirkendall.

Our dinner reservations this evening are at six o\'clock at the Tiberna
del Abadero. Bill Weinig \-- No, I have the address. It\'s 1776 I
Street, N.W. So for those of us that are staying at the hotel, we could
meet at 5:30 and then catch a cab there. If you\'re not staying at the
hotel and you wish to meet us there, that\'s fine. Otherwise, the
address is 1776 I Street, and the reservations are for six o\'clock.

Who will be joining us for dinner, so that I can get a head count? Raise
your hands. Thank you.

Breakfast for the committee will be here again tomorrow morning
beginning at 7:45. So, again, for those of you that are staying at the
Holiday Inn Capital we could meet in the lobby at 7:30 and walk over
together.

More details about the meeting: Our long time friend, Ms. Renee Miller,
is the designated Federal official for this advisory committee. This
official may chair, but must attend each meeting, and she is authorized
to adjourn the meeting if she determines it to be in the public
interest.

She must approve all meetings of the advisory committee and every
agenda, and she may designate a substitute in her absence.

Next, some of our scheduling for today and tomorrow is rather tight, and
in contrast to the previous chair, I\'m a stickler for trying to stay as
close to the agenda as possible. So your cooperation on that is greatly
appreciated.

Then this is great. You may have noticed that we are currently without a
vice chair, and with some degree of persuasion I\'ve convinced Dr.
Seymour Sudman to allow me to place his name in nomination for that
position, and ironically he\'s not even here to object as I do this.

So is there a second to this nomination?

MR. MOSS: Second.

CHAIRMAN CRAWFORD: Thank you. So noted. Are there any other nominations
for this position from the floor? None noted. So we\'ll let this item
rest until tomorrow morning, and then we\'ll take a vote on the
nomination for vice chair at that time.

Finally, an important meeting mechanic: Committee members, when you want
to make a comment or ask a question, let me know that by placing your
name tent vertically. That indicates that you have a question, and
don\'t forget to take it down after you\'ve made your question or
comment.

So now it\'s my pleasure to recognize Jay Hakes, Administrator of the
Energy Information Administration.

MR. HAKES: Good morning. Welcome to the new members. Like the committee,
we\'ve had some turnover at EIA. Linda Carlson, whom many of you know,
who has headed up our Statistics and Methods Group, has gone over to
head up the statistics program at the National Science Foundation, and
she is now a member of the Interagency Council on Statistical Policy. So
I get to see her at those meetings, but we are currently in the \--
filling that position.

Renee has taken over this important role of deciding when we adjourn. I
don\'t know if being late for lunch is in the public interest to adjourn
or not.

We currently have a vacancy as head of our IT position, which we are
trying to fill. We have a senior position in climate change that we are
in the process of filling. So there\'s a lot of change. I have to add
myself to the list now, since I\'ve announced this week that I will be
leaving EIA at the end of May.

I can\'t say on the official record where I\'m going yet, because it\'s
just not ready for official announcement, but if you talk to me
privately, I can tell you.

I would just say, I haven\'t prepared any definitive comments, but I
think this is one of the best jobs at the Federal government. I think
that we have really been in the forefront of sharing information with
the public, making sure it\'s high quality information, not getting
sucked into partisan political fights, and having a reputation around
that\'s very, very high, and EIA has a superb collection of very
dedicated professionals. I didn\'t realize when I came here what a great
job this was, and I\'m very reluctant to leave it. But that\'s the way
the system works.

I, as usual, make a few general comments. Often what people are
interested in is our budget, and I try to deflect \-- reduce the time
that\'s spent talking about budgets and more about what we\'ve
accomplished with the money we have. I don\'t know \-- Can we get this
set up?

I have sort of three or four measures that I use to assess whether I
think EIA is being relevant in the energy world. In other words, are
people actually using what we provide?

One of those is the extent to which our materials are used by the
Congress. I would say that, on that measure, we would rank very, very
high right now, partly because of the volatility in oil markets. But
last year I personally testified 11 times before Congressional
committees, and so far this year between John Cook and myself, we\'ve
done five Congressional testimonies so far this year.

So there is, I think, a tendency on the committee when they are dealing
with an energy issue, and not just oil \-- they would like to hear what
EIA has to say about it in the process of the deliberations.

Another group that you look to is the general public. One way to measure
that is by Internet usage. I show this chart almost every time, and we
keep changing the scale. So it always looks kind of the same, but if you
look at the scale, it\'s a vastly different graph every time we use it.

Each dot represents unique daily users. It\'s not hits. It\'s users, and
we recently had a day when we had 35,000 people using our site in a
single day. I think that\'s a rather extraordinary number. I think it\'s
since our last meeting that Government Executive Magazine picked EIA\'s
site as one of the 16 best Websites in the Federal government, and the
other 15, I think, all had substantially larger budgets than we do. So I
think that\'s a good indication of interest.

Then another interest is the press, and to the extent that they are
writing about us. These are quarterly numbers of press citations on the
news wires. You can see that the interest in EIA is going up
considerably.

Then this is just the New York Times, Wall Street Journal, Washington
Post, Los Angeles Times, and USA Today, which are sort of your big
circulation, major U.S. newspapers. It\'s now not uncommon for the same
paper to cite us maybe twice in the same day in two different articles.

They will pretty routinely, when they\'re doing graphs and tables, cite
us as the source, and we like that. It helps us be more visible in the
Congress, to the public, and it sort of shows they regard us, I think,
as sort of the Good Housekeeping Seal of Approval on good statistics.

Now it does take money to do this. So I would also report on our budget.
We have had \-- We currently have a budget of \$72 million a year. This
has fallen from a peak of \$85 million in 1995. So we have been hit very
substantially by the budget cuts that took place in the mid-nineties.

We also during that period reduced our Federal employees by about 100,
but in the last couple of years we have been building back on a gradual
basis, and the administration\'s request for the coming year is \$75
million, which would be roughly a \$3 million increase.

Most of those monies would be dedicated to quality improvement issues.
We\'ll talk about this a little bit later when we talk about data
quality, but as you become more visible, particularly when you\'re more
visible on items that are controversial \-- and I assure you, gasoline
prices are controversial, and some of the other work we do on electric
restructuring and climate change and others are controversial \-- there
are a lot of people that would like to catch you making mistakes.

We all know that statistical systems are not perfect, but when people
are making huge investment decisions or making major policy decisions
and you make a mistake, the price, I think, is high then as compared to
if you\'re obscure and no one is using your stuff.

So I think the bar has been raised in terms of we\'ve got to have
quality systems. We\'ve got to invest in redoing frames, making sure
we\'re getting the right coverage in industry. We\'ve had those kind of
discussions before.

The administration and Congress is sort of buying into that, and at a
somewhat modest level supporting increases. Now there is a move afoot in
the Congress in both the House and the Senate and among different groups
to work for a substantially larger increase than the administration has
requested.

I would \-- My appearance before the House Appropriations Committee was
rather unusual for someone representing Department of Energy before the
Appropriations Committee, because all their questions were about what we
would do if they gave us more money and what would be the reasons why we
would need more money than the administration had requested when it put
in its original budget.

Those were unusual kinds of questions to get from the Appropriations
Committee, and some of the committee chairmen and others in the Senate
are working rather vigorously to try to get a bigger increase.

So I am probably more optimistic this year than you\'ve heard me ever
before about the trend in the budget issue, and I think it\'s up to us
to have very good projects that we can explain to people what the
quality payoff is, why that needs to be done. I think, if we can do
that, people are coming to more and more understand why the data is
important.

I\'ve been waiting for some hearing where someone could ask me why we
weren\'t doing this. One of the people who had voted to cut our budget
back in \'95 would be the person that asked the question, and I would
explain to them why we didn\'t have it. But I haven\'t had quite the
moment to be able to do that, but it\'s a tempting possibility.

We have, I think, since our last meeting finalized the new version of
our strategic plan. If you take a look at it, you will see that there is
a heavy, heavy emphasis on quality, how do we promote quality, what do
we need to build in quality.

The idea is we\'ve sort of achieved visibility. We\'ve achieved
usability. I accompanied Secretary Richardson on some of his trips to
the Persian Gulf and actually did most of the talking at the meeting
with the Kuwaitis and the Saudis, because a lot of the issues were what
kind of world oil demand growth we expected in the second quarter and
what world stocks were.

A lot of it was basically data discussions, and it was very interesting;
because as we were debating with people, it was clear that some of their
material had the EIA logo on it, and they had gotten it off of our
website, and the oil minister from Saudi Arabia when he wa talking with
Secretary Richardson put heavy emphasis on how valuable they found
EIA\'s work.

So there was, I think, a tendency for them to listen very carefully to
anything that was perceived as coming from EIA. So I think we sort of
have achieved a visibility, and now we need to make sure, if people are
using our stuff, that we\'ve given them the most timely and best data
and analysis that\'s humanly possible, and that\'s always difficult.

We are \-- The most controversial thing in our strategic plan is an
attempt to phase our paper documents. Some people, that strikes them as
very, very weird. I think, if you assumed that we were going to
implement this in the year 2000, it would be kind of weird, but I think
that over the next three or four years that this will be more common.

We would have a small set of core publications that we would continue to
publish in hard copy, like the MER and the AEO, documents like that, but
some of the more technical fuel specific publications would only be
available in hard copy, although I think you\'ll have the ability to set
them up on the Internet so it would be very easy for a person to print
off their own copies.

I think both the technology of being able to read on monitors and print
off high quality publications yourself, I think, is going to progress.
My guess is we\'ll be fine when we get to that point. If we\'re not, we
can do a recharting of our course. But you know, if you want to make
some feedback on that, what products are most important to keep in
print, I think we would be glad to hear that.

I think what I would do is open it up to any questions that you have
about things and sort of the direction of EIA or big issues that I might
be somewhat uniquely qualified to comment on.

CHAIRMAN CRAWFORD: Calvin?

MR. KENT: Going back in reverse order from how you\'ve presented it, on
the getting rid of the paper copies and everything, what sort of
feedback have you gotten from constituent groups on that? I remember ten
years ago when we discussed that, of course, we didn\'t have the website
or anything like that, but it was \-- many constituent groups greeted
this with horror that we would no longer be putting out paper
publications to that?

What has been the reaction of the public?

MR. HAKES: There are several ways of measuring that. I don\'t think any
one is definitive. In some of our customer surveys, there\'s been a
sizable number of people \-- Tom may remember the numbers precisely, but
there\'s been a sizable number of people who still want the paper
products. But that dropped off substantially this year.

I think it will continue to drop off. If it doesn\'t drop off, I think,
again, we would probably have some correction there. But just on the
Hill I\'ve seen a tremendous change. The reaction two or three years ago
was a hell, no, and now it\'s well, maybe; because most of them are
using the electronic products.

Also I think the assumption is, if the website was going to be the same
today as in \-- or in two years as it is today, I don\'t think you could
cut down paper products, because I still find in some cases I can find
something quicker on a paper product than I can on the website. But the
website will be different two years from now than it is today.

I will be very disappointed if I still am finding things quicker in
paper products in two years than on the site. So I think we owe it to
our customers to make the site even better than it is today, and then
part of the tradeoff is to recapture some money and time and effort.

You save a little bit of money on paper, but there\'s also sort of just
a time and focus thing of getting something ready for paper publication,
and it seems \-- Assuming that time is a limited commodity, if that time
is spent on developing the website, you get more bang for your buck.

MR. KENT: You anticipated my follow-up question, because I certainly
applaud getting rid of the paper publications and think that that\'s an
excellent move in the right direction, but would hope that this would
not be done basically on grounds of efficiency; because I\'d like to see
the money diverted to making the website even more user friendly,
particularly in terms of the cross-indexing by topics, by areas, and for
the people who are not familiar with how to use databases, a lot of the
people in the general public such as the press and so forth.

You\'ve come a long way, and some of the things that we have that we\'re
going to be talking about today, I think, are further improvements. But
I would certainly urge that, when you get rid of the paper products,
that it be with the idea of enhancing even further the usability of the
website.

MR. HAKES: Yes. I think the EIA employees are pretty excited about this,
because one of the advantages of the web is you have a real sense of how
many people are using your products and a little bit of a profile of who
they are.

A lot of our employees, I think, get a real charge out of that, because
they will put out a new products and they can watch on the Internet and
see how many people are tuning into using it. It\'s even, I think, a
little bit of healthy competition in our office.

So I think that the message we want to send out is time invested in
making the stuff you have on the website more accessible to people is a
high priority.

MR. KENT: Can I go ahead, Madam Chairman?

CHAIRMAN CRAWFORD: Go ahead.

MR. KENT: Tell us what you told Congress when they said, what would you
spend more money on? I\'d be interested to know how you would spend the
money.

MR. HAKES: Well, I guess we\'ve had some money to deal with the changes
in the electric industry and the natural gas industry, but you know,
very high priority is not to get behind the curve in terms of our
coverage of these industries. We have, I think, good plans in place, but
we need to make sure that those work.

Some of their interest relates to the current run-up in prices. So one
of the things we would like to do is get more regionality built into the
short term energy outlook, for instance; because if you look at energy
pricing in this country, you have national averages which are often not
very helpful in dealing with run-ups in the northeast, in California,
which have somewhat unique delivery and supply problems.

So we would want to be more able to perhaps anticipate the kind of
problems they had in New England this January. For those of you who
weren\'t following it closely or didn\'t live in New England, the price
of heating oil and diesel transportation fuel went up 85 cents a gallon
over a period of about three weeks. It\'s since come down 65 cents a
gallon, again over a relatively short period of time.

Those spikes get the public\'s attention. I\'ve gone with the Secretary
to Massachusetts, to Maine where we landed at an airport that was snow
covered and should not have been in use, New Jersey, Rhode Island, to
let people scream at us about various things. I think, you know,
there\'s a lot of public education that\'s taken place.

Another thing that we want to do is to get natural gas into the state
heating oil program, because in the northeast it was the
interrelationship between the natural gas market and the heating oil
market that was one of the major dynamics there and one that really no
one had good data on.

So we want to build that into the SHOP program. I think there\'s just a
lot of quality enhancement items that we have had to sort of put on the
back shelf, because the times have been so tight, that we will get to.

So I think there\'s a whole range of things that we would do. Now the
Senate is also interested in the international modeling area. In the
international area we haven\'t had the modeling capabilities like we
have domestically.

I have personally been trying to slow that train down, because I didn\'t
want to do it without substantial resources to be able to do it right,
and I didn\'t want it to become a pawn in the political debate.

My position has been that, if we waited long enough, both sides in the
debate would ask us to get more involved in international modeling, and
we have reached that point. The proponents of Kyoto, the opponents of
Kyoto, the administration and the Congress all think that EIA ought to
get more involved in international modeling.

So now that we\'ve got that set, I think we\'re ready to go, as long as
the support is, in fact, forthcoming. So that would be an item that
would also require substantial resources.

So a lot of it is little quality items that just have tended to get
pushed to the side, an ability to track volatility in the market a
little bit more. When I think of the fact that we, during the
mid-nineties, certainly considered doing away with weekly petroleum
data, that data was just absolutely essential over the last three
months.

To some extent, it almost becomes the leading indicator of global data,
because no one else has weekly data. Since the U.S. is a big part of the
market, that data \-- You\'ll sometimes see the German market move based
on our data on the U.S. stocks. That\'s how interconnected the world has
become, and it is the leading indicator for global data.

So we have been able to maintain a lot of vital systems, but industries
are changing out there very rapidly. They are becoming more
decentralized. So it puts more pressure on us to have accurate
measurements.

MR. KENT: Let me ask just one follow-up question. I was going to ask
later, but since you brought it up: Is the main driver for the
international modeling the environmental issues or what else is involved
there?

MR. HAKES: Well, we get asked a lot the sort of fossil fuel depletion
questions. So if your model goes out to 2050, you are forced to sort of
\-- Right now I kind of duck those questions, but I think there\'s
reason to believe that EIA ought to have some opinions about that, but
it\'s mainly climate change is what\'s driving it.

There\'s been a real change in the Senate Energy Committee right now in
that that committee has been, I guess, perceived as about as anti-Kyoto
as you can get. But they are very interested in doing something about
climate change and saying climate change is an issue we have to deal
with, here are some alternate approaches.

They have hired someone who has a strong academic background in
modeling, including climate modeling. So that committee, I think, wants
to be engaged in this issue. They have a lot of confidence in EIA\'s
ability to do objective work.

So that\'s where a lot of this is coming from. So it is the climate
change issue that is driving it, but there are other issues like the
balance between demand growth in Asia and the ability of the world\'s
oil supplies to meet that demand. That certainly would also benefit from
this.

CHAIRMAN CRAWFORD: Does anyone else have any questions for Jay? Calvin,
do you have more questions?

MR. KENT: Not now.

MR. HAKES: Let me mention one other thing. One of the things that\'s
come up, too, is a lot of interest in us helping other countries develop
good data systems. We\'re very active right now in Bangladesh. We\'re
quite active in the Philippines. We\'ve been quite active in South
Africa.

I think there\'s almost unlimited funds for us to work in those areas
from other government agencies. The constraint is not funds. It\'s the
time of our people to actually engage in that work. But that will be
another thing.

Since energy is such a global issue, it\'s hard for us to analyze trends
in oil or climate change or other things without having access to other
countries\' data. So we\'re just scratching the surface of what needs to
be done, but I would suspect that EIA\'s role in that area will probably
grow.

CHAIRMAN CRAWFORD: We are going to move on to the panel discussion on
confidentiality legislation. I had asked Jay Casselberry and Robert
Latta to join Jay down here in front. Jay Casselberry will start.

I\'m going to turn over administration of the panel to Jay so that I can
be a member of the ASA committee and observe in that capacity.

MR. HAKES: I think you are aware from previous meetings that there is a
specific piece of confidentiality legislation, and I think what I will
do is turn it over to Jay Casselberry to talk about what it is. Bob
Latta can talk about how it would impact EIA, and then I\'ll talk about
where the legislation is. Then I think, Calvin, you\'re the commentator
on this.

MR. CASSELBERRY: Thank you. EIA conducts about 60 recurring surveys, and
most of them go to businesses, and we have a few others that go to
households, to public utilities and to state governments. But our
primary collection has always been on the supply side of the energy. So
we\'re going to the businesses that supply energy and energy related
things.

With confidentiality, our provisions for the confidentiality of any
survey data we collect are based primarily on the laws, and there are
general laws for the government like the Privacy Act and the Freedom of
Information Act and the regulations that are implementing them.

We also get public input when we\'re making decisions on
confidentiality. We hear from the people who provide the data, and we
hear from the people who want to use the data. So we hear from both
sides.

When we want to do surveys, we go to the Office of Management and Budget
and ask them for approval to do them. That\'s one of the considerations
that they make when they make a decision, is what provisions for
confidentiality EIA has and what they hear from the public regarding our
proposal.

When we send out our survey materials, the forms, instructions, or if we
do it electronically, we tell the people how we propose to collect the
data. Are we going to make it available to \-- Are we going to disclose
it in individually identifiable form or are we going to try and withhold
it and just for our uses, and the information we\'ll put out will be in
statistical tables where there\'s no disclosure?

When we tell the respondents this, they look at this along with the
reporting obligations: Are the surveys mandatory or voluntary? How is
EIA proposing to use the data, and why we need it? Then they make a
decision on whether or not they are going to report. So it is one
consideration.

In some of our surveys we hear a lot more comment from the data
providers on these issues.

The Privacy Act regulates the information we collect from persons, any
personal information we get. Really, that\'s just one survey that we do
of residential energy consumption.

The Freedom of Information Act covers the entire government, and that
describes how \-- Its basic premise is that Federal information and
Federal agencies should be made available and disclosed to the public
when people request it, but there are certain exemptions where agencies
can withhold information.

So that\'s primarily the major law that affects the data that we
collect, our surveys from businesses.

We really have three different ways that we treat our survey data,
depending on what kind of survey it is. We have some data that we treat
as nonconfidential, and that means, if someone from the public requests
it, we would make the survey data available, and it would have the
identifier of who submitted that data.

We have our end user surveys, primarily \-- or exclusively \-- we have
contracts or we have other ways to ensure that the data are treated as
confidential. But EIA never receives individually identifiable data.

Then we have the rest of EIA\'s data, which is pretty much \-- which is
the majority of everything we collect. That\'s data that we treat as
confidential. We collect it or we have contractors collect it for us. We
have it. It\'s individually identifiable, but we do not disclose it,
because there is an exemption in the Freedom of Information Act that
says that if the data will cause substantial competitive harm to
someone, then the agency can choose not to disclose it. We hear that
quite a bit from the businesses that provide us with data.

Just to give you some examples of where we are currently with
nonconfidential data. Some of our electric power data is treated as
nonconfidential. In the past it\'s been a very regulated industry. There
was not much competition.

So the argument of that it will cause substantial competitive harm
really didn\'t exist. The industry is changing, and EIA has gone through
one process already to try and determine what data should be held as
confidential and, as the industry changes, I\'m sure we\'ll go through
that again.

With the end user surveys, there are surveys of households, commercial
buildings and manufacturers. That is data that\'s treated as
confidential, but we have contracts or we use other vehicles to make
sure that there is no release of that data.

Then the confidential data where we actually collect it is pretty much
things like motor gasoline production, sales, stocks, anything about the
business operations.

For our end user survey, Bob Latta is going to talk about that in a
little more detail.

We have the residential, which is covered by the Privacy Act, because
it\'s information about individuals and households. We have the
commercial buildings energy consumption survey, which we use a contract
to ensure that the contractor won\'t provide EIA with the individually
identifiable data. So that keeps it out of our hands, and so that
basically exempts it from the Freedom of Information Act, because we
never actually have possession of that individually identifiable data.

For the manufacturing energy consumption survey, the Census Bureau
collects that data for us, and they have a lot stronger confidentiality
provisions for data that\'s in their possession. That is how we protect
that data.

For the data that we have in our possession, again it\'s the substantial
competitive harm that could be caused by release of that. If we released
information about a business and their competitors saw that, could they
use that to get an advantage over the company that had provided us with
data?

It includes almost all our data on petroleum and natural gas and coal
and in the nuclear area and alternative fuel area. There are some
elements on individual forms that we\'ve made a decision would not cause
substantial competitive harm, but for the most case it is most of the
major energy sources.

Under the laws that set up EIA, we have to provide that data for
official use to other parts of the Department of Energy, to other
Federal agencies, the General Accounting Office and Congress and to the
courts. But it still would not be disclosed to the general public. So
it\'s for official use, and we would make it available to some other
Federal groups and such.

A bill was passed by the House just this recent year, the Statistical
Efficiency Act of 1999. It talks about the efficiency of the statistical
system, quality of it, and it\'s directed just toward statistical data.
So it exempts things that are collected for administrative records or
for regulation of the companies.

It talks about creating a sharing system among some statistical agencies
in the Federal government and safeguards that could be put in place to
make sure that, if data were shared among these statistical agencies,
that it would not somehow end up released.

It would establish eight of these statistical data centers around the
Federal government. In some cases, it\'s entire agencies like the
Census, the Bureau of Economic Analysis, BLS, National Agriculture
Statistics Service, National Centers for Education Statistics, National
Center for Health Statistics.

At EIA it is just one division, and the division Bob Latta is from, and
that\'s the division that collects our end user surveys, and it also has
a division in the National Science Foundation. These would be eight
separate entities that would be covered by this bill.

The idea behind the bill is, again, efficiency so as to reduce the costs
and burden and duplication among Federal agencies, because right now the
data sharing is very limited. So the idea is, if agencies could have
access to other agencies\' data, there wouldn\'t be the need to collect
the same data, that it will cut down costs, and it would reduce the
burden on the public who has to report the same information more than
once to the Federal government.

It also tries to ensure that there is safeguards against the release of
that data. The confidentiality is protected by these Federal statistical
data centers, because some agencies like Census have a lot stronger
confidentiality provisions. EIA would be considered weaker, and some of
the other ones have something in between.

So it\'s to try to set up a general provision for the confidentiality,
if this data is shared. Again, it\'s just data that\'s going to be used
for exclusively statistical purposes, and that if anyone wanted to use
it for a nonstatistical purpose, to use it for an administrative reason
or to try and regulate an industry or try and take action against a
specific company that had reported data, that\'s really not covered by
this. So you would have to go back to that company or the person who
provided data and get an informed consent that you could share it for
those kinds of uses.

The agencies would enter into data sharing agreements where they would
acknowledge what kind of data they wanted, what the restrictions are on
sharing it, and the penalties with the release of the data would travel.

Census has a lot stronger provisions for what happens if someone
discloses their data, and if that data was made available to EIA, then
the penalties for release would travel with the data so that any other
Federal employees in these statistical data centers who had access to it
would be under those same restrictions, and OMB would try and track the
agreements that were made under this Act to monitor how well they were
doing, whether or not they were achieving what was designed.

The Act passed the House in the fall of last year. It was sent to the
Senate, and then it was sent to a Senate committee, and I haven\'t
really heard too much on it since that time. Mr. Hakes may be able to
talk a little bit more about that, if he knows anything else.

I think next Bob Latta is going to talk about, if this Act was passed,
how would it affect the division, the Energy Consumption Division.

MR. LATTA: Robert Latta. I work in the one division that would be
covered by this new arrangement. Basically, it would affect how we do
the residential consumption survey, in this case the sampling units \--
the housing units and the commercial buildings energy consumption
survey, and the sampling for that is commercial buildings.

The manufacturing survey would probably still be done through the Census
Bureau, and they have their own requirements on confidentiality. Now the
current arrangement is that we do not take possession of the names and
addresses of the respondents.

There was some fear that, if we did, that we really wouldn\'t be able to
take them under \-- from FOIA, especially through the commercial
building survey.

The proposed changes is that, if we could protect the names and
addresses, that we may ask the survey contractor when they deliver the
file to us to give us the names, addresses and, probably more important
to me, would be the census tract, ZIP Codes, utility identifiers for the
records.

If we had this, we do not plan to routinely contact the respondents. The
contractor does contact them during the interviews and does recontact
them during some editing and examination of outliers. That would
probably still continue.

The main thing we would want to be able to do is there\'s a lot of data
by ZIP Code, census tract. There\'s EIA data by utility. We would like
to be able to conveniently attach that to the records.

Now the current arrangement is that we can always pay the contractor to
do that. It costs a little bit extra money, takes a little bit of extra
time. Even then in certain cases they are going to look at the data.
They\'re going to decide, well, if we put this online, you\'re going to
be able to nail down which commercial building it is.

In fact, some of the data that\'s collected on the form, we really
don\'t get. An example is that, if you have a 50-story building, we
don\'t know it\'s 50 stories. We only know it\'s over 30. So there\'s
certain things that are already left off, and we would like to be able
to have those things.

It would help us in our analyses. It would save money in that we would
not have to, for some things, just collect the data, look at it, send it
up to the contractor. They would attach it. They would have to analyze
it and see if there is any confidentiality problems, and then send it
back to us. That\'s somewhat of an inefficient way of doing things.

What we do now is they currently attach weather data and some utility
base data. Given in some parts of the country, particularly on the west
coast, you have micro climates from ten miles apart, some of the
anomalies we see in the data might be \-- we might be able to understand
better if we could find out is this house or building right on the coast
or is it 20 miles inland. Currently, from what we get, we cannot see
that.

Another way it would affect is that the Census Bureau is considering a
thing called the American Community Survey. This would replace the long
form for the 2010 census. So they are proposing to go out with the long
form questionnaire on a continuing basis, and this would be something
like a couple hundred thousand households every month.

In order to do this, they have to have a master address file, and all
this really is is a \"Dear Occupant\" list. It\'s a list of all the
housing units in the country. It doesn\'t tell you who lives in them or
anything about the people living in them. It\'s just a list of all the
housing units.

Well, access to that sort of list would reduce the survey design cost,
and this is a considerable chunk of money that would be saved off direct
survey design, and it would also save for each iteration of regs when we
do a new construction update. There would be savings there.

It is hoped that, if the bill passes, that we could be able to get
access to this file, if the Census Bureau does go ahead with this
program.

MR. HAKES: Thanks. Let me talk a little bit about the history of the
bill itself and where it stands and what I think its prospects are.

There was some interest in the Congress in having one unified
statistical agency. This comes out of a long interest in particularly
uniting Bureau of Labor Statistics, Census and BEA into one
organization, but the interest really went beyond that to take all of
the statistical agencies and put them together in one agency.

I testified on that, and my argument was, well, I can make some
theoretical arguments why having the unified agency is a good idea. I
can make some theoretical arguments why having them altogether is a bad
idea, but as a practical matter, to reorganize all these agencies,
change all the committee appointing lines, find a building where we
could all be, change everybody\'s personnel descriptions, and all that
this would take, we would spend the next five years reorganizing and not
having time to think about data quality and all these other issues. I
said, as a practical matter, I think this is a step backwards.

I said, what you want is the parts of the statistical system to interact
and coordinate their activities, whether they do that as one agency or
separate agencies, and I can give you evidence of where we have been
coordinating our activities such as the FedStats site, homepage, other
items that we work on together. And by the way, we could cooperate a lot
more if we could share frames and do other things that would make a lot
of sense, but with our different legislation now we can\'t do.

The House committee, which is chaired by Congressman Horn from
California, a very reasonable person \-- he\'s a former political
science professor. I used to use some of his books in my courses \-- I
think, came around to this point of view.

They were able to pass it in the House and pass it on the floor last
year, and then the Senate had a very small window of opportunity after
the House passed it. It would have been a miracle if it passed, but it
was a brief effort. OMB has been quarterbacking this.

Now one of the things that has happened, unfortunately, in the last few
months is statistical surveys of the public have become very
controversial. We now have public officials telling people they don\'t
have to fill out the long form if they don\'t want to.

So what previously had been considered noncontroversial is now
controversial and, frankly, that probably is not helpful in the Senate,
because now people will be looking for what\'s the political angle here
on this.

I hope that I get invited to testify before the Senate, because my
argument would be, if you\'re against this bill, you\'re for respondent
burden, because what you\'re saying is you want the respondents to have
to answer all these different questionnaires and participate in frames
from all these different government agencies rather than just do it once
and get it over with.

So from our standpoint, it\'s attractive because it saves money, but
from the public\'s standpoint it\'s attractive because it\'s government
acting in a more rational way and not bothering the public more than we
have to.

So I think the case is strong, and I really don\'t see where the
opposition is going to come from. But I think the administration is a
little gun shy right now. I did have \-- I got a report yesterday from
the Director of the Census Bureau at the Interagency Council on
Statistical Policy going over all their ups and downs over the last few
months, and it\'s an interesting story.

One would hope that our statistical agencies would not get caught up in,
basically, political fights, but it does happen from time to time.

So I don\'t know what\'s going to happen in the Senate this year. The
Government Operations Committee that deals with it tends to be less
partisan than some of the other committees. Chairman Thompson has a very
refreshing style.

He was taking testimony on oil prices a couple of weeks ago, and Adam
Zaminsky, whom some of you may know, who is an oil analyst for Deutsche
Bank, got up and listed the things he didn\'t know about the world oil
market, and Senator Thompson said it\'s very refreshing to have a
witness who admits that he doesn\'t know everything. He says, of course,
you are speaking to a body that knows everything about everything. But
he said that with a smile on his face, and he and Senator Lieberman, who
is the ranking minority, tend to run good hearings and be reasonable
people.

So I still think we\'ve got a shot at this in the Senate this year, but
maybe people are a little less optimistic than they were a couple of
months ago.

Cal, are you next?

MR. KENT: I did prepare some comments to make about this, and I must
admit that, as I took a look at the piece of proposed legislation, I was
reminded of the quotation that has been attributed to many politicians,
but I understand it first came from Huey Long. That was, \"If you can\'t
get the whole loaf, settle for half a loaf. If you can\'t get half a
loaf, settle for a slice; and if you can\'t even get the slice, settle
for the crumbs.\"

To be sarcastic, I might say that I think from EIA\'s standpoint, this
bill is somewhat crummy, because it really does not go far enough. I was
extremely disappointed when I found what the actual content of the bill
was.

Let me give you some historical perspective from what happened almost
ten years ago, and I know that many of you get tired of old war stories,
but being an old man that\'s about all I have left to talk about
anymore.

As you recall, back ten years ago right at the time that I took over
from Helmut, there was a great deal of concern about the spike that had
occurred in heating oil and diesel prices, and EIA had put out a special
report.

The Department of Justice had become very interested in whether or not
there was collusion and price fixing and restraints of trade and all of
those other nasty terms that they use. So when I arrived as
Administrator, I actually thought for a while that I had a branch of the
Justice Department as permanent additions to EIA, because so many of
their lawyers were over wanting to gather information.

It became quickly obvious to me that the Department of Justice was
interested in getting our confidential data and using it for
prosecutorial purposes, not just for gathering information, even though
they denied that that was what they intended to do.

They said all they wanted to do was to get this individual data so that
they could look for patterns, so that they could look for indicators, so
that they could find what directions they ought to be going and, heavens
forbid, they would certainly not use any of our identifiable data in a
prosecution.

Of course, we did not believe them, for very good purposes. So we
resisted giving them that information. After my confirmation, of course,
as you know, I was confirmed at 3:45 a.m. on the day after Saddam
Hussein invaded Kuwait, and the two events were not coincidental, I
might point out. As a result of this, a great deal of attention began to
be focused on our individually collected data for petroleum suppliers.

Again, the legions from the Department of Justice appeared demanding
that we give them information, because they were concerned about whether
or not there had been collusion, price fixing and restraints of trade
again.

So a battle ensued, a battle which encompassed the Office of Oil and Gas
to a great degree, and conflict went all the way to the White House. In
fact, it was a very tense meeting at the White House with Boyden Gray,
who was at that time the counselor to the President, myself. Larry
Curtis was there, as was Larry Klure, and I think two-thirds of the
anti-trust division of the Department of Justice was also there rather
vigorously denouncing us for our uncooperative attitude on this
particular issue.

I don\'t know what Jay\'s experience has been in the recent run-up or in
the 1996 fight that took place. I don\'t know if there was the same
enthusiasm in the Department of Justice. Their resources may have all
been diverted to the persecution of Microsoft, but there may very well
have been some indications from the Department of Justice that they were
interested in getting the firm identifiable data again to use it for
purposes other than statistical.

I should point out that the really big loophole that we\'re talking
about here is sharing data with other agencies, as was pointed out on
the slide that you saw previously. That is the Department of Justice
coming in and saying you\'ve got to share the data with us.

Finally, the Office of Legal Counsel of the Department of Justice, which
at that time was responsible for settling interagency disputes of this
nature, indeed made a ruling and gave us an order that we did have to
turn this data over to the Department of Justice.

We continued to drag our feet, obfuscate, to do everything we possible
could not to comply. They kept sending orders over for us to comply.
Fortunately, or maybe unfortunately, from your point of view, Jim Reil
did retire as the Assistant Attorney General in some degree of disgrace,
and you all may remember that issue or maybe you don\'t \-- we\'ll talk
about it off the record \-- and the oil prices went down very, very
quickly, and the enthusiasm for prosecuting the oil companies
diminished. So we were off the hook until my departure.

The reason I tell this story is simply that I\'m not sure what EIA is
really gaining out of this. I can see some real efficiencies so far as
the End Use Division is concerned. I can see that there will be some
ease of exchange of information. But what distresses me is that all of
the other agencies that are involved are getting a blanket exemption.

All of their identifiable information is going to be exempted, but for
EIA it is only one type of information on two surveys that we collect,
and it is not the most vulnerable data that we collect which is being
covered.

In fact, there is almost no coverage for data that is being supplied by
individual suppliers to our various statistical surveys. I\'m sure I do
not have to stress with this group the importance of confidentiality and
reliability.

If you talk about data quality, you have to talk about confidentiality,
because you simply are not going to get good data. You are simply not
going to get a high level of voluntary compliance unless your data can
be held confidential.

EIA has made promises. They have made guarantees. They have done what
has been necessary to build up a high level of trust with industry
suppliers, and without the confidentiality protection, the firms
recognize the fact that there is a very small fig leaf protecting them,
and that fig leaf is not really being enlarged all that much by the
legislation which is being proposed here.

So I am disappointed in the bill. The bill is very similar to one that
was kicked around even back when I was Administrator, which would have
only given us this level of protection.

Perhaps I underestimate the benefit to EIA from this, but I certainly
think that a huge problem is being created as we move into having to
collect data from industries which are becoming unregulated,
particularly the oil, natural gas and electric utilities; because in the
past, and particularly in the area of electric utilities, these have
been regulated industries.

So getting the information, as you all know, has been relatively easy.
You go to the firms, so forth, and you get your information. As some of
the other discussions that we\'re going to be having later on during
this session of this committee meeting indicate, in natural gas, and
certainly in electric utilities, you\'re going to \-- or in electric
generation, you\'re going to have to be getting information from
suppliers that previously came in on Federal regulatory forms, and
you\'re going to have to be going to more and more and more suppliers.

So there\'s a real threat to data quality from deregulation, and that
threat is increased and magnified by the fact that there is no
protection being extended to these new suppliers for confidentiality.

As you all followed the deregulation scenarios which have developed
throughout the country as these various industries have been
deregulated, the result is that some people benefit, and other people
may not benefit at all or don\'t benefit as much, and this leads to all
sorts of inquiries and all sorts of tries for more re-regulation or
restructuring and things like this.

So there\'s very likely again to become \-- With an extremely aggressive
Department of Justice Anti-Trust Division or a Federal Trade Commission
or some other regulatory body, there is very, very likely to be more
demands for this individually identifiable information.

So what I\'m saying is the remaining hole that\'s here \-- in fact, the
hole is larger than what is being covered, actually \-- is going to
create a major threat to EIA and EIA\'s ability to continue to do the
job of providing the reliable statistics that it has in the past,
particularly when you face a more and more deregulated industry with
more and more suppliers who are not compelled to furnish you with the
information that you need.

So having said all this, I have six specific questions concerning the
legislation. Actually, I have seven, because I added one after hearing
the remarks that were made here.

The first one pertains to Section 3(a), paragraphs 1 through 8. That is
the overall question of trying to determine why EIA is limited to only
one office and three surveys whereby the other statistical agencies, the
other seven or at least six of the seven, have general blanket
exemptions for all of the individually collected statistics that they
bring in.

My second question concerns Section 5, paragraph (c), and that is a
technical question: Does the Act assume that all the data collected by
these statistical authorities is statistical data? If it is not, how is
it going to be actually determined what is and what is not statistical
data, because the way the Act is written, statistical data seems to be
data held by an agency, and it was not at all clear to me how you would
reconcile that definition.

Also in Section 6, paragraph (b), must the data that is received from
another one of these statistical agencies be confidential when it is
collected, and does it become confidential after one of these
statistical bodies would then take possession of it? And if it is not
confidential at the time it\'s collected, and if it does not become
confidential when it is transferred to one of these statistical
agencies, does this mean that only confidential data can be transferred?

It may not be a major problem, but it was one that confused me from the
legislation.

The fourth question is out of Section 6 again, paragraph (d). This does
seem to provide a loophole which would allow confidential data to be
used for other than statistical purposes if it was required under other
Federal acts.

Here, I come right back to the point I was making before about the
Anti-Trust Division and other agencies wishing to use this data under
their authorities that they already have, and it does not seem to me
that the Act supersedes any of those existing authorities. In fact, it
does seem to maintain and actually entrain them.

The fifth point that I would make comes out of Section 8, paragraph (c).
I was going to ask the question to the people in EIA, have you even
begun to think and to promulgate the rules under which you would be
collecting the data?

I know it\'s premature, because the Act has not yet been finally
ratified by the Senate or signed by the President, but it would
certainly be of interest to this committee to know what your thinking
has been about what your rules would be on how you would be collecting
and maintaining the data.

My sixth question comes out of Section 9, paragraph (c). I think this
whole question, this whole paragraph, raises the issue about exchange of
data with the states. Does this Act create any Federalism issues about
data sharing; and if so, how are you going to resolve them?

Then the seventh question concerns the exchange of international data.
This is the one that I have not previously thought about until I heard
it mentioned just a while ago. That is, does this Act facilitate or does
it in any way interfere with our ability to collect and to use data
which is internationally collected and about our ability to exchange
confidential data with international agencies?

MR. HAKES: I can deal with a number of these questions. Do you want to
deal with them?

Someone once said, and I\'m sure it wasn\'t Huey Long, that sometimes
the perfect is the enemy of the good.

MR. KENT: Yes.

MR. HAKES: I would argue that the benefits from having the consumption
surveys in the system are rather substantial, because that\'s where I
think the cost savings would be the greatest for EIA. I think you\'re
talking about figures \-- seven-figure numbers here rather than
six-figure numbers, particularly when we adapt to the new 2000 census.
That\'s a costly item for us, and those costs go down very
substantially.

I don\'t know, Dwight, if you have the number, but I know Dwight is
quite concerned about our ability to get that from the census.

The bill does provide a mechanism for expanding beyond, and the thought
was that, if we can demonstrate savings in this one area, you would then
have a stronger case to go back to Congress.

When Congress originally passed our legislation, I think they intended,
people like John Dingell and others, that our data would be used for
prosecutions. I don\'t think it\'s a defect in the law or an oversight.
I think that\'s what people actually believed.

We have, I think, tried to interpret the law in a way to protect
confidentiality, and we continue to do that. We almost always side on
the confidentiality side. We\'ve engaged in some rather lengthy court
battles to do that. But this area is a little bit trickier.

So from a political standpoint, you could end up derailing the whole
bill by getting into some of these other issues, but even if you take
all the politics out of it, I think that it is a tougher nut to crack.
I\'ll give you an example.

Suppose there is an issue in stocks, whether it\'s gas or oil, and those
numbers move the market substantially, and we have one particular
company that makes a revision that\'s enough to move the market. So they
are sitting there, and they know they are about to make this revision.

So they have a sense of how the market is going to move, but no one else
knows that this is going to happen. Now, you know, the pure statistical
side of me would say, well, I don\'t want the FDC coming in and looking
at our data, but the public citizen side of me sort of says maybe I
would like it to look at the data; because we might actually get better
data quality if people knew that major revision errors that could even,
in some cases, appear to be intentional might give an unfair advantage
and ought to be looked at.

There are statutes that say that certain data have to be available in
company identified form for environmental purposes. That\'s a part of
the statutes now. Whether that\'s considered a good idea or a bad idea,
I don\'t know that you need to bring that debate into this particular
bill, because if you do, I\'ll guaranty you, one, the bill won\'t pass
and, two, that is a tough issue.

I think our ultimate goal that\'s even more important than cost is
quality and the willingness of people to respond. But our surveys,
corporate surveys, are not voluntary. They are legally mandatory.

We try not to use that approach. We work very closely with all of the
industrial groups, partly because they are big users of our data. So
they have a strong interest that there be high data quality, and a lot
of the professional associations actually work with us to try to
increase the response rates.

You know, I think we went through this in some previous meetings, but I,
you know, personally went and visited with the comptrollers of the major
oil companies, for instance, because we were concerned last year with
the drop-off in response rates.

The key thing they were interested in \-- I mean, there was some
interest in confidentiality \-- but is this required by law? Yes. Well,
if I\'m the comptroller of a major U.S. corporation and I\'m required to
submit some form and it comes to my attention that my company is not
submitting that form, I think you have almost 100 percent guaranty that
that form will come in.

Now if it\'s lower down in the company, that may not be the case, but
the comptroller is going to make sure that gets in there, because it\'s
legally required. I would say I would personally be willing to list the
names of companies that were chronic nonreporters, and I think they
would become immediate full reporters in the system.

So those numbers came back up to their previous levels, even with all
the mergers and acquisitions which make all of this very difficult,
because it was \-- I think the management of these companies want to
participate.

So I would say that maybe we\'re more in the quarter to half-loaf. I
think we get substantial savings from this. I think that some of these
other issues are very, very difficult. I mean, this revisionary thing is
something that I\'ve been worrying more about in the last year, because
we have gotten some that have been very, very large, and we are also in
a situation where stuff is moving the market more and more.

So I think I certainly agree with Calvin\'s point. I think confidential
is an important value for us. I think it is a commitment that we have
made to industry. To the extent that we are able under the law, we
certainly fight for that.

Like Calvin, I\'ve engaged in a few acts of quasi-civil disobedience to
try to maintain confidential data, because I think that\'s where the
major benefit lies. But I think some of these areas are more difficult,
and I think the practical matter is, if we went in with a pure bill
right now, it would cause enough complications that the whole bill would
go down.

I think that, if the bill works well, then you\'re in a much fresher
situation to come back and deal with it. My impression is, and Jay can
correct me if I\'m wrong because I think he\'s more \-- that this does
not facilitate international sharing, to my knowledge.

I\'m not sure what the situation is with the states. I know EIA, I
think, probably before you got here had some unfortunate problems with
the states. I personally would like to see if there was some way of
resolving that, because I think in our consumption surveys some of the
larger states would like to sort of buy into those surveys, and you
really could improve the quality immensely by doing that.

So that\'s sort of \-- My hope would be that we could resolve that, but
I don\'t know that the bill moves us ahead in that area.

The other statistical purposes issue \-- I don\'t know exactly what that
refers to, but it could be referring to some of the laws on
environmental statistics. We went through an extensive series of
meetings on the electric industry, and we moved some of the electric
industry data into the confidential category, not as much as the
industry had requested, largely because of the countervailing
legislative mandate to make certain data available for environmental
purposes.

So we came up with what we thought there was a reasonable compromise. We
also got the independent power producers and what have been known as the
electric utilities onto a common reporting basis, which I think is
healthy for the law and for the data and for having everybody on a level
playing field, and we certainly appreciated the cooperation of the
independent power producers coming into that system and working with us.

We\'ve been getting very good response rates from them, incidentally,
which is very good for when you\'ve got new reporting burdens.

So that\'s sort of my general \-- I don\'t know that I \-- I certainly
haven\'t given a lot of thought to the rules, but I suspect Jay has.

MR. CASSELBERRY: I think, Cal, that some of your questions \-- I think
every time I read the law, I get something new out of it. So some of
your questions, I think I would probably have to go back and read it
again, just to make sure my understanding is correct.

With regard to, I think, one of your questions, would it apply to all
the data collected by the statistical data centers, as I understand the
law, it\'s anything that they collect for statistical purposes.

So I assume that the agencies could still collect some other data as
long as it wasn\'t classified as for statistical purposes, and I\'m not
really sure it\'s clear how they would do that. But I think that
there\'s probably some things that are done at BLS that might fall into
that category.

I know they do a lot of things with the states, and I\'m not really sure
how they would handle it. I\'m not sure what talks they have had over
there. So I think it\'s all data that\'s collected for statistical
purposes. So as long as that was defined, and I\'m sure even within Bob
Latta\'s division, if they were to take up a different survey within
that division that didn\'t have a purely statistical purpose, then it
would have to be considered.

The loophole for other uses: It does require, if you\'re going to use it
for anything other than a statistical purpose, that it does \-- and it
was collected for that purpose, then it does require informed consent by
the person who provided the data. So I would assume that whatever data
that was made available among the agencies that was collected for
exclusively statistical purposes could not be used in the other ways
without someone going back to the respondents and telling them about the
proposed use.

I think Jay talked about just about everything else. He did talk about
the states, and we have had some states that have come to EIA and
expressed a desire to have access to some of our data, and we have
looked at those issues, but we really haven\'t gone too far; because I
think there also have been the questions about some of the data we
collect and that it\'s under the Freedom of Information Act which is
applicable to the Federal agencies.

How does that transfer if data is given to people that aren\'t Federal
agencies? I don\'t think we\'ve straightened that out. I think there are
some other questions about how it would be used in that, how we would
set up data sharing agreements.

So there have been some discussions on that, but they really have not
gone very \-- too far, up to this point.

CHAIRMAN CRAWFORD: Are there any other comments from anyone on the
panel? Bob, do you have anything to add?

MR. LATTA: No.

CHAIRMAN CRAWFORD: Jay, anything else?

MR. HAKES: We should have had Larry Klure come to this meeting, because
I will assure you that the lawyers of the Federal government have
devoted a substantial amount of attention to this bill. I can\'t really
speak for everything that they\'ve gone into, but this thing probably
went through 20 drafts by the time \-- before the lawyers got to it, it
was probably two or three drafts, and then it went through another 16 or
17.

It is very, very complex because of the different rules that the
different agencies have had, and everybody wanted to make sure that they
didn\'t lose anything and then, hopefully, would gain something from the
bill.

CHAIRMAN CRAWFORD: Calvin?

MR. KENT: Let me make a comment off the top, and that is that I would
certainly not want to do anything that would impede the passage of this
legislation; because it is a step in the right direction. How large a
step it is, I think, is debatable.

Certainly, the opportunity for more wide sharing of statistical
information among agencies is going to be of tremendous benefit, and
that, I think, is one of the greater values that is in the bill, that
agencies are going to have more freedom to share data with each other.

I don\'t know what Jay\'s experience was, but a few years ago we were in
fairly constant conflict with the Census Bureau over our data, because
they were concerned that we did not have the level of confidentiality
protection that they did, and so they were reluctant to share with us
even some of our own data that we were paying for to have collected.

I think this will reduce those problems. So I want to make it clear for
the record that, despite my sarcasm and reservations, that I would
certainly be more than delighted, if anyone ever asked me to, to
indicate that this bill should go forward, even though it is not all
that I would have hoped to have seen in this piece of legislation.

I still have some concerns about this conflict on confidentiality of
data that, I think, transcend the whole bill. That is why is data
collected in the public\'s right to know?

I know that back almost ten years ago we were able to unite both the far
right and the far left of the political spectrum in being opposed to
more confidentiality for EIA. The far right simply didn\'t want it,
because they didn\'t want Big Brother to know anything about what
individuals were doing. The far left was opposed to it, because they
didn\'t want to have big business being given any protection from any
possible persecution.

As a result of this, both the far right and the far left is represented
in Congress, and particularly by some Senators who were very unanimous
in making sure that confidentiality didn\'t go anywhere in the Congress.
So the extremes in this case, I think, overwhelmed the center on this
particular issue.

I would not want to have that happen again, because I do think that this
is certainly a positive step forward, and the rule of politics, of
course, is you take whatever you can get and, if this is what we can get
at this time, then we ought to continue to move forward and be grateful
for it.

At the same time, I think we do have to focus on why you are collecting
data. If you\'re collecting data for statistical purposes, then \-- and
I\'m probably in disagreement with Jay on this point, and it certainly
was the point I maintained all the way through my tenure at EIA, that
hell, no, you don\'t have it available for prosecutorial purposes.

I think it is simply wrong that, if you collect it for one purpose, that
it should be available to be used against someone or potentially to be
used against someone in a court of law or an administrative hearing.
That\'s just my own value judgment, and I think that that\'s a value
that \-- or a problem we continue to debate through this legislation.

I think the public has a right to know, but I don\'t think the public
has a right to know everything, because I think that there is a high
level of deception. Even if you\'ve got the club of saying to the
comptroller, yeah, you got to do it, I still think that that\'s a very
poor public policy to say that, well, we may very well let the DOJ or
some other agency carry around in their briefcase your information so
that they can use it against you if it suits their purposes.

My opinion is let them go out and gather their own data.

CHAIRMAN CRAWFORD: Are there any other questions or comments from the
committee? David?

MR. MONTGOMERY: This may be going a little bit more broadly than the
specific legislation, but \-- and your interchange with Calvin brought
this to mind.

As we see data series disappear from forms that we outsiders can use,
things like, for example, the fuel cost data for individual electric
power plants which you haven\'t collected for a little while, about what
would be happening as more data becomes confidential. What are your
thoughts about kind of creating public use data files that have the data
blurred adequately to protect individuals, and could that deal with any
of the kind of concerns about sharing data with other agencies?

That is, if another agency wanted something nominally for statistically
purposes that you wanted to protect, would it be possible to use blurred
data files of that type to get what you need for doing analysis, which
often for micro-simulation might include something that looked like an
individual record, but if you could protect confidentiality that way?
Are you doing anything on that?

MR. HAKES: That is an important question. Nancy or Bob might have a
better sense than I do.

CHAIRMAN CRAWFORD: Nancy, I\'m sorry to bother you. Could you introduce
yourself? I don\'t think you\'ve been identified yet.

MS. KIRKENDALL: I am Nancy Kirkendall with the Statistics and Message
Group in EIA.

We do have a couple of projects, and I don\'t know how far along they
are. Ramesh Dandekar has come up with a way of simulating \-- taking raw
data and coming up with a simulated version that has, we hope, the same
statistical characteristics.

The idea would be that that kind of process could lead to a micro-data
file. It hasn\'t really been tested much. I think he\'s done something
with CBECS and has been working with Mark Shipper whom you\'ll meet a
little bit later.

He also has been working with somebody in Canada doing the same sort of
thing. The question really is does it really have the same statistical
characteristics, and how multi-variate are those characteristics. You
can certainly get the correlations right between pairs, the means right,
but how much further I don\'t know.

So if that would work, then that would certainly be an alternative.

MR. HAKES: Yes. I think the answer is bright people may be able to
figure out a way to do this, but we\'ve got to be absolutely sure it
works; because if some other bright person on the outside figures a way
to crack the code or something, we would be in big trouble.

MS. KIRKENDALL: Yes, part of the problem when you\'ve got companies is
that big companies are big, no matter what you do and, if there aren\'t
very many of them, people might know their names. So the one on the top
of the list is \-- this is sort of by size \-- most likely the company
with the name you recognize.

So even though it\'s not data from that company and it\'s been massaged
a little bit, do you really protect their confidentiality? So there are
a lot of interesting questions about it. I mean, the technique would
certainly work better if data were not quite that visible, if you don\'t
have quite such a skewed distribution. But it\'s a real interesting idea
anyhow.

CHAIRMAN CRAWFORD: Go ahead. If you want to continue, he might have a
different question.

MR. HAKES: Well, I was going to say that those are kinds of questions
that, hopefully, we can work at with other statistical agencies who have
some of the same issues. Nancy has been heading up a lot of our efforts
with interagency work since she came back from OMB, and it is an issue
that sort of cuts across all the statistical agencies.

CHAIRMAN CRAWFORD: Johnny?

MR. BLAIR: Yes, I had a question. Did I understand correctly from your
comments, Jay, that data that had been previously collected, that the
respondents would be recontacted and would have to give consent for that
data to be shared?

MR. CASSELBERRY: I think, if it was going to be used for a
nonstatistical purpose. We really haven\'t gotten into how the agencies
would handle data that they collected in the past, but basically if it
was collected for statistical purposes, then there can be data sharing
and then the confidentiality and the provisions for sanctions on
improper disclosure travel with it. So \--

MR. BLAIR: I guess I\'m not sure on the difference between statistical
and nonstatistical purposes, but I guess what I see here is that if you
go back to people under some conditions and they have the right to \--
you know, under informed consent to say that I don\'t want my data moved
on, then it seems to me you have a data quality problem in that you now
have datasets that have all sorts of missing data that they didn\'t have
before, and that that\'s an issue in terms of the quality of those
datasets that are passed on, if now you have all of this essentially
item non-response.

If on the other hand you pass it on without the informed consent or
without going back to them, then it seems to me that you have an ethical
problem in that the data were collected under one understanding and is
now being used in a different manner.

I don\'t know. Maybe I\'m misunderstanding some issue here, but those
are questions, I guess, I wonder how those would be addressed.

MR. CASSELBERRY: I think they define statistical purposes in the Act. To
paraphrase it the best I can, it\'s pretty much when the data is going
to be used to present characteristics of the population or
characteristics of the respondents. It\'s not really important to know
any one respondent or to provide any data about a certain individual
survey respondent.

So the intent of the use is to give data about something that\'s
happening or gather data about the business or the group of respondents.
So it\'s not important to know an individual one. So I think when they
talk about nonstatistical purposes as the user is you\'re going to focus
on data that was reported by one respondent, and that has some \-- For
some reason you\'re focusing on that individual respondent\'s data and
you\'re going to have some use for it with regard to that.

CHAIRMAN CRAWFORD: Calvin, you were ahead of me, but if you\'re going to
change the topic, I\'ll go first.

MR. KENT: Well, to respond to Johnny\'s comments, Section 2 provides you
definitions. Usual with Federal nonspeak, nonstatistical purposes means
any purpose that is not a statistical purpose. So I thought that I would
tell you that that issue has been clarified by the Act.

CHAIRMAN CRAWFORD: But he raises a point that I just wanted to see if
you knew the answer to. Once this bill is passed, are your respondents
just automatically responsible for allowing their information to be used
in a manner that\'s consistent with the new law or are they going to be
contacted and told that, well, now that this legislation is passed, this
is what we\'re intending to do, do you agree?

MR. HAKES: I don\'t think so. I think that as this legislation is
envisioned, the original deal with a person is that their data will be
confidential. It wills continue to be confidential within the Federal
statistical system.

So there won\'t be anymore likelihood that their data would be
disclosed. In fact, there would be less likelihood if you\'re part of a
system like EIA where there has not been as firm a protection as there
has been for, say, the census data.

The census data would be continued to be under their strict confidential
rules, but ours would be just as strict as theirs for the consumption
survey. So I don\'t see that there\'s any threat to the public or they
have any concerns about this.

Now the issue would be if you do disclose, and then that\'s where
you\'ve got almost a contractual relationship in terms of what you told
your respondent your policy was. There, I think you do have to
communicate with them very effectively.

CHAIRMAN CRAWFORD: Jay?

MR. BREIDT: I think these disclosure questions are a lot trickier in the
shared environment. Even though one agency might have stricter controls,
you could have different combinations across agencies. Nonconfidential
plus nonconfidential could be something that is in danger of revealing
important information.

I guess, as you start overlaying tables, you start to be able to find
out more and more information. So it seems that it\'s considerably
trickier to ensure the confidentiality of respondents once you start
overlaying information from multiple surveys.

CHAIRMAN CRAWFORD: Calvin?

MR. KENT: Just to put this in a somewhat broader context, and again I
want Jay to respond on whether or not there have been any pressure
placed upon him as Administrator of EIA from any other agency to use
individually identifiable EIA statistics or filings for possible
anti-trust or other restraint of trade purposes in recent weeks.

There was also another historical thing, and I remember that we had a
meeting with the National Science Foundation, which was not only at that
time concerned with the confidentiality of EIA statistics, but there was
also great interest in using information that had been collected by the
Department of Education also for the possibility of prosecution. I
forget exactly what the issue was, but I do remember there was also
interest in using for prosecution purposes information that was
collected by Health and Human Services for the possible identification
of drug traffic and things of this nature that they thought that they
would be able to gain some degree of insight into this.

So it was not just EIA that at the time that at least I was
Administrator where people were worried about the confidentiality of
their information being used for prosecution. So I wanted to indicate
that this was not just an EIA issue back eight, ten years ago.

MR. HAKES: There have been several issues during my tenure. We have one
court case that is still ongoing where an electric utility has sued to
get data from an independent power producer who they felt was
manipulating the PRPA law or something. I think that\'s still under
appeal. So I probably shouldn\'t really say much about it.

We do get approached from time to time by the FTC and other agencies,
and I guess the way to put it is, when they arrive, our approach
generally is if you\'re willing to use our statistical data to help your
investigation and treat it as statistical data and not get individual
data, we will respond to you within a matter of days and try to help
you, to the extent that we can. If you want individually identified
data, we\'ll have to consult with our lawyers and look how long it took
Cal Kent to respond to the Justice Department.

What we end up doing is working with them on a statistical basis, and
that generally satisfies what they need. They usually come in and don\'t
realize how helpful statistical data can be in just understanding the
market. Usually, we can come to some mutual agreement that that\'s a
more fruitful approach.

So I think we do have to realize that we may not always win ultimately
in court some of these battles. So you know, it\'s good to negotiate, if
we can, and I think we have a good record that we can show to our
respondents that we may not have \-- Well, in effect, I think we have
won every time, because we have not been releasing confidential data.

MR. MONTGOMERY: This is a fascinating discussion. So I apologize for
dragging it out even longer. I think your observation about being able
to convince the kind of prosecutorial agencies that statistical data for
understanding the market and things like that are quite useful, I think,
gets most of the way at what I was going to ask about.

It strikes me there\'s a fine line here, and I\'m not quite sure where
it\'s located, in what the Justice Department might want to do with
data, because there certainly is one side of trying to understand
whether there was anything happening in the market that they ought to
worry about.

It would certainly be nice if prosecutors thought about that question
first rather than just looking for people. At the other end of the
scale, my feeling is that what \-- you know, getting all of the
individually identifiable data, say dealing with some oil price spoke,
all it\'s really going to do is save them some time and some resources.
That\'s my question.

Are the data that are currently submitted to EIA subject to discovery?
That is, if the Justice Department wanted to send \-- you know, wanted
to kind of demand one of the file cases, they could demand under
discovery anything a company has already submitted to you.

So the issue, I suspect, is more one of figuring out who they want to go
after than actually having the data for prosecution; because they could
get it for prosecution if they wanted it, I assume.

MR. HAKES: I think the argument \-- That sort of diffuses the argument
on both sides. We could argue to the prosecutors, hey, you have other
ways to get this; don\'t taint our statistical system.

On the other hand, from the companies\' standpoint, they know that the
Justice Department could get it anyway, too. My feeling is \-- and I
should maybe let people speak for themselves, but if you take someone
like, say, the American Petroleum Institute, I think their judgment
today would be that submitting these data is one of the best things
they\'ve ever done, because it\'s often very helpful in explaining how a
market operates.

I\'ve testified before numerous \-- several hostile state investigations
of oil prices, and generally been able to diffuse the argument, just by
being able to show quite detailed data on what\'s going on.

So that\'s why, if a prosecutor had a tremendous case, you know \-- My
argument certainly would be go subpoena the information, but I\'m not
sure I\'d say the world was collapsing if they had to get it from us,
and the law demanded it. I mean, I probably would not have to be dragged
off in chains for that.

I think we\'ve been able to mediate a situation where we\'ve got three
or four different laws, and they are basically in somewhat of a
conflict, the public\'s right to know and confidentiality. You read
those statutes, and you \-- but I think if you have a process that you
go through in some consistency, the courts will tend to give you the
benefit of the doubt.

You know, I don\'t like to get in situations where you as a Federal
agency are told by a court that you did something wrong. But we\'ve done
fairly well so far.

CHAIRMAN CRAWFORD: Anymore questions or comments from the committee? Are
there any questions or comments from EIA staff or other members of the
audience?

Before we break, I\'d like to ask the new arrivals to identify
themselves for the record, please. So if you haven\'t announced your
name and your affiliation, please do so.

MR. FRENCH; I\'m Dwight French. I direct the Energy Consumption Division
which is the division that\'s the focus of all the concern about the
existing legislation.

CHAIRMAN CRAWFORD: If you can\'t speak as loudly as Dwight just did, you
need to use the microphone.

MS. JOHNSON: I\'m Marti Johnson, and I\'m the survey manager for the
commercial building energy consumption survey.

MR. MAKENS: John Makens with Electric Power Division.

MS. LEACH: Nancy Leach, the Energy Consumption Division.

MR. DISBROW: I\'m Jim Disbrow, and I started with EIA in 1974 and
probably have a real good historical perspective on many of these
issues.

MR. LU: I\'m Ruey Pyng Lu from Statistical Methods Group.

MR. MILLER; Herb Miller, Statistics and Methods Group.

MR. BRADSHER-FREDRICK: I\'m Howard Bradsher-Fredrick.

CHAIRMAN CRAWFORD: All right. Thank you. If no one else has anymore
questions or comments, we\'ll break, and we will resume at 10:45.

(Whereupon, the foregoing matter went off the record at 10:15 a.m. and
went back on the record at 10:42 a.m.)

CHAIRMAN CRAWFORD: I\'d like to go ahead and get started. So I\'ll turn
things over to Mary for her presentation.

MS. HUTZLER: This is actually the third AS meeting that we are
participating in right now. The first two AS meetings, we had breakout
sessions on the new international modeling work that we are doing.

The very first one was before we really got started in scoping out our
requirements, and what we were asked in the AS at that time was what\'s
your input, and what do you think we should be doing.

The second requirement \-- or the second meeting, essentially, we had
done our requirements analysis which we prepared and finalized on July
8, and I think all of you have copies of it. Then we had gone on to do a
design and development plan, which we had in draft form in October, and
we discussed with the breakout session in November.

We never did finalize that particular document, and the reason for that
was after we produced it, my staff had to go on and to work on the next
international energy outlook. So most of the folks were working on this.
Only about two people got to really delve into the modeling design that
we had talked about and to decide if what we had set out was really the
way to go.

It turned out that we\'ve decided now that we want to take a step
backwards and say to ourselves, or ask ourselves, the question: Is this
really going to meet the requirements that we set out for ourselves, to
begin with?

We\'re doing that right now. This is probably the first time in my
career where I\'ve done such a thing, taken a step back and asked the
question about whether we are going to change our path and what we had
planned to do.

So what I thought I would tell you today is a little bit about why we
are doing this, talk about our requirements that we set out, and tell
you what our first design was, and then tell you what we\'re thinking
about doing right now.

Now essentially we started dealing with doing an international model
probably back in \-- actually in the NEMS days. We were talking about
enhancing our modeling, going out to the long term, and also perhaps
enhancing our international modeling capability.

Then I think it was in FY \'97 the Office of Fossil Energy asked us to
take a look at a longer time frame and dealing more with technologies,
because they had technologies that wouldn\'t actually penetrate until
the post-2020 time period.

At that time we looked at a number of models, and we did a working paper
that we distributed within DOE. Jay wanted us to get the background and
support of all the people within DOE at the time, and support at that
time meant that they had to give us dollars.

As soon as you ask people for dollars, they hesitate about giving you
support. So we actually got support from two agencies within the
Department, Fossil Energy and Nuclear, but we didn\'t get support from
all of the Department.

After that time point, we did a study of the Kyoto Protocol, and we did
it using the National Energy Modeling System, and that modeling system
is U.S. only. As a result, we had to do a number of parametric scenarios
that essentially let us look at cases that had a different amount of
international trading and corporate permits.

Therefore, the U.S. had varying amounts of domestic reductions that they
needed to take themselves. We got different carbon prices within those
scenarios, but they were essentially the marginal cost of reducing a ton
of carbon within the United States. They weren\'t an international
price.

As a result, that made us again say we needed to somehow get a tool that
would help us in the international arena and be able to calculate the
international price of carbon.

After that occurred, we started asking for money in our budget, and
Congress, in fact, gave us money; and we actually have a multi-year plan
and funding to get us to our final goal of having an international
model.

We started getting funding in FY \'99, and that was only a small piece,
and we are getting other larger pieces as time goes on. The first larger
piece was this fiscal year.

I think what I\'d like to do now is to go over our objectives and our
requirements. Essentially, the objective was to provide a model and data
framework that represents international energy markets, serving as a
tool for the analysis of international energy policies and other related
issues that may affect domestic energy markets in the U.S. economy.

From that basic objective, we defined the requirements. Our requirements
were, first, we needed a tool that would support the international
energy outlook. In our international energy outlook \-- and I gave you
the brochure that sort of summarizes what our forecasts look like \-- we
essentially go out to 2020. We have worldwide projections, and we deal
with consumption of all the major fuels.

Also we wanted the ability to address a number of energy and technology
policies. As you can see, technology was very important to us. We wanted
to be able to represent technological progress, because that was
important in terms of carbon mitigation policies.

Technologies can help us reduce the demand for fossil fuels and,
therefore, carbon emissions and other greenhouse gas emissions.

We wanted the system to be able to interface with the National Energy
Modeling System. We only produce one forecast for the U.S., and that
forecast must be commensurate in whatever we say in terms of our
international global forecast.

We do want to have an extended forecast time horizon, because a number
of the technologies that are important to carbon mitigation aren\'t
going to penetrate in the 2020 time frame. We were thinking about
dealing with perhaps annual forecasts through 2020, and then going out
in five or ten-year increments to maybe 2050 at this point in time.

Jay always cautioned us with going out further in time than 2020, and we
are always cautious when we try to deal with that particular time frame.

Then we also wanted to expand our international energy production
coverage. In the IEO we only deal with oil in terms of the production
side, and we wanted to incorporate more of the supply side in this
particular representation.

So essentially the basic features we\'re talking about include a model
that is compatible in the time horizon with NEMS; that has the potential
to go to 2050 for scenario analysis, not for baseline forecasts; that
the geographic coverage could be compatible with our international
energy outlook, and that\'s because we have customers right now that
look at the IEO and want the same information that we\'re producing
there.

Also, we wanted flexibility in terms of being able to aggregate and
disaggregate country and regional detail. The reason for that is we
frequently get asked to talk at other forums such as this APEC
ministers\' meeting that\'s occurring this May, and they want to know
about the regional detail of their specific region, not just of the
global situation.

So if we can disaggregate into country level detail or subregional
detail, it\'s useful for us.

We also wanted to have more information on the demand sector side. In
the IEO representation we have right now, we did add a transportation
sector, but we also wanted to include the buildings and the industrial
sector, because again we wanted to represent technologies within these
sectors.

We also needed to represent electricity generation, since many new
technologies and renewable and nuclear technologies are important to
this debate.

On refineries, we wanted to be able to deal with different product,
petroleum product types, and deal with markups on their prices, not
necessarily have a representation of refineries but to be able to deal
with the products coming out of them.

I did mention we wanted to have a more elaborate supply representation,
and for the fossil fuels we wanted to represent depletable resources.
Right now many of the energy models that deal with these questions do
not contain that, and we thought that was extremely important,
especially to the expanded time horizon.

Other areas that we wanted to deal with were energy prices. We don\'t do
that in the international energy outlook right now. Energy trade \-- we
deal with coal and oil trade, but not in other commodities, and we
wanted to also enhance that representation.

We wanted to cover all six greenhouse gas emissions. Right now we look
only at carbon. Then lastly, we wanted to have a macro economic
assessment that could provide the baseline economic growth rates as well
as looking at the impact on the macro economy of what carbon mitigation
policies could do.

Now just to summarize again what our current capability is: What we use
to produce the international energy outlook is a model called the world
energy projection system. WEPS, as we call it, is essentially a set of
integrated spreadsheet models.

So it\'s actually very simplistic. On one hand, because it is flexible,
we actually can do a lot with the particular tool in terms of
aggregation and disaggregation, but we can\'t get the specific things
that we\'re looking for here, such as the carbon price.

We do produce forecasts of energy consumption and carbon emissions for
all regions of the world, and I\'ll explain what regions in a moment, to
2020; and we do this in five-year increments.

As I already mentioned, it does have a flexible accounting framework,
and WEPS is a top down modeling approach. What we essentially do is we
take inputs that are economic growth rates for various regions and
countries and energy intensity, and we use that as the driving
parameters in the WEPS format.

Now the regions we have now in the international energy outlook are the
list that you see here. There are essentially 14 different countries and
nine different regions. We do cover each of the countries in North
America. We deal with some specific countries in Europe, and then cover
the other western Europe as a region.

We do look at Japan, Australia being Australia and New Zealand. We\'ve
just recently broken out the transitional economies, the FSU, into Annex
I and non-Annex I, and the same for Eastern Europe. We look at China and
India separately, because they have such a large growth rate there, and
we expect them to have a large share of the carbon emissions in the
future.

We do South Korea, other developing Asia, Turkey, other Middle East,
Africa, Brazil and then other Central and South America.

Essentially, the data sources that we use right now for our
international modeling effort are on this list. We make sure we\'re
compatible with the International Energy Annual that\'s produced here in
EIA. We also use the statistics coming out of the International Energy
Agency as a primary source.

Our forecasts for GDP come from DRI and WEFA, and then we supplement the
rest of the information coming from these various different sources.

Now I also included some specific questions that we wanted to be able to
answer with this tool in terms of greenhouse gas policy issues. They are
enumerated here.

Of course, we have the price of traded carbon permits and the economic
cost to the United States of meeting certain targets, what it\'s going
to mean for domestic and international energy markets, what
technological progress means to reducing the cost of greenhouse gas
emissions, the relationship between technology energy prices and
mitigation policies, then energy security issues and the effect of
alternate energy efficiency standards.

Those were just some of the things that people are asking questions
about that we wanted to be sure that we would answer.

Now we set up an independent expert review committee, and the committee
consists of five people. John Weyant, who runs the carbon modeling work
for the Energy Modeling Forum is on the committee. He\'s with Stanford
University.

Rich Richels from the Electric Power Research Institute is on the
committee. Rich has developed and is a primary user of the merged model
which was one of the models that we had considered.

Jay Edmonds from the Pacific Northwest National Laboratory has the
second generation model. Mary Novak was a member of the committee, or is
a member of the committee, and she usually keeps us on track about what
are some of the major issues happening right now in the international
arena. Her outlook is the next ten years rather than the next 50 or 100
years. So she brings us down to reality many times.

Dave, who is a member of the AS committee, is also a member of our IER
committee, and Dave has done modeling with CRA in this particular area.

We met with the committee members a number of times, first on the
requirements analysis in June and then again in November when we were
going over the design and development plan. We are trying to schedule a
meeting in mid-May.

What we did in June was we asked the members, given our requirements,
what model do you think we should use as a starting point; and we
actually put John Weyant on the hook for it. He slept on it from June 24
to June 25, and came back and told us that he would suggest using the
SGM model as the starting point.

He said that for a number of reasons. This particular area is fairly
complex, and the SGM model, he thought, was small enough that we could
accomplish our task with the amount of resources we had available. It
was also used to do carbon analysis in a number of studies, including
what the Council for Economic Advisors had done for the administration.

The model builders were here in Washington just up the street from us,
the Pacific Northwest National Laboratory folks, and that the model was
in the public domain, was essentially available to us. He felt that it
did deal with technology, and it was probably the CEG model that
probably had the most technology levers available to it at that
particular time.

So essentially, we went down that path, and that\'s what our design and
development plan essentially discussed, which was using the SGM model
essentially as the major component of our modeling effort, and then
supplementing it where we thought it was necessary.

The second meeting, which actually dealt with the design and development
plan, went into a lot of detail, and we talked about the objectives and
various issues, ways of dealing with the models, dealing with model
size, and our resources.

They felt what we had done in the design and development plan was
essentially very optimistic and would be a challenge for us and very
difficult, but they encouraged us to proceed. They also brought up
different issues that we were going to have to deal with in policies,
one of those being banking, which we really hadn\'t covered in the first
design and development plan that we had done.

So essentially, the structure that we were planning on was using SGM as
essentially the integrating model or the major focus of this work, but
we were going to rely on trying to enhance SGM with a lot of other
models that we already use for international energy outlook and which
are already EIA models.

So essentially, what happened after that was we finished our
international energy outlook, and then we got together again, and we
asked ourselves the question, now that we got into the SGM model, was it
sufficient to meet our needs, and is it the path that we should really
go down?

We set up different evaluation criteria to rank the various models that
were out there, and we were at this point in time looking at SGM versus
a model called MARKAL, and I\'ll tell you more about that in a minute.
After looking at the various evaluation criteria, we decided that we
wanted to take a step backward and to take a look at MARKAL in more
detail to make sure that we weren\'t going down the wrong avenue for
what we wanted to accomplish.

It was interesting when we did a ranking score on these evaluation
criteria. It turns out that the SGM model and the MARKAL model came out
very close, which sort of told me how complex this subject was and that
essentially none of these models were really going to do the job for us,
and it\'s going to be quite a bit of work for us to be able to get any
model to the stage where we wanted to end up.

Well, our current status is as follows: In terms of the second
generation model, we have reviewed it. We brought the model in-house.
We\'ve played with it. We\'ve done a lot of scenarios with it. We\'ve
read the code, which is a pretty complex code and fairly difficult to
get through.

We also documented the model. The second generation model is essentially
funded by Energy Research in DOE, and while ER has given the money to
enhance the model, they don\'t provide money for documentation.

So right off the bat, it was difficult for us to learn about the model,
and so one of the things we really needed to do was to get documentation
so that the team of folks could understand it better.

We also spent some time enhancing its reporting, because the report
writers that were there were not graphical and didn\'t give us a way to
quickly evaluate the results we were getting. So we added on a graphical
reporting capability.

Now the things that we learned in going through this was that the
technology detail was really not sufficient for our purposes. We really
wanted something far richer in technology. At the time, the technology
detail was really only for the electricity sector. There were plans to
move it into the other sectors, but that was not done yet.

Also, the type of data that computable general equilibrium models need
is \-- or particularly the SGM model, is input/output data for each of
these countries. That type of data is not data that we are used to
deriving. While some data exists and while the SGM folks do have
agreements with different countries for that data, we felt that it was
going to be a chore for us to maintain that data and to update it on a
periodic basis.

As I sort of already mentioned, substantial development was really
needed to get the model to where we wanted it to be on the technology
side, as well as dealing with other things such as investment, foresight
issues, incorporating banking, and other issues that we felt were
important.

Another thing that we found out was that it was also difficult for us to
manage the model in terms of the structure that we have at EIA and the
way we normally work in the modeling community. The NEMS model, for
instance, has a lot of submodules, and we compartmentalize those models
to different groups, and they have the responsibility for maintaining it
and for updating the data.

The structure of the SGM model didn\'t allow us to do that. Essentially,
it meant one or two people were in charge of the model code, and that
the data requirements were fairly aggregated, and we really couldn\'t
distribute that among staff.

Then finally, we learned that the learning curve was very high for us.
We are not familiar with CGE models, and as a result, the rate of
learning that we had certainly was not very high and certainly not
making me happy.

So at any rate, we took another look, and we contrasted it with MARKAL.
Now MARKAL I\'m going to define essentially as a single country or
region model. TIMES, which is a new development of MARKAL models,
combines various regions, and I\'m going to tell you more about TIMES in
a minute.

Essentially, these models were rich in technological detail. In fact,
the way we represent technology in electric utilities in the NEMS model
is pretty much the way MARKAL represents technology, which we find to be
very rich in detail and able to deal with issues of diffusion and
behavior, those kinds of elements that we thought we needed to deal
with.

It turns out that many countries and regions have the model, are using
the model, and there\'s an international exchange program between these
groups of countries. It started out with the U.S., Canada, and the
European countries all having models, but now the Asian countries are
developing them the models and also the South American countries.

The country/region detail can be solved simultaneously. The management
of staff assignments is a whole lot easier, because we can give country
and regional detail to various folks to work on. It\'s also a dynamic
linear programming model. So it\'s a structure that we\'re very familiar
with, and we do have the staff and the folks to be able to work with
more easily.

Now the MARKAL methodology is essentially a bottom up methodology, and
that was one of the major things we were looking for to represent
technology. It\'s driven by sectorial energy service demand. So it does
already have a representation of the various end use sectors. However,
we do need to provide these particular drivers to the model, and I\'ll
talk more about that in my last slide.

It\'s a least cost optimization. Various growth rates can be specified
by user constraints in terms of the rate of growth of various
industries. It\'s also flexible in how you can group regions and in
terms of calculating solutions, and it can already constrain carbon, for
instance, and also add on a carbon tax, which is the two things that we
already allow in the NEMS model.

Now what TIMES is going to do is actually be a regional model,
essentially a linear programming model of various regions. At this point
in time, TIMES only exists as a prototype, and they are planning to have
the regionalization done by the end of this calendar year.

They are also planning to do other features in the TIMES model, such as
having feedback on prices with the energy service demand, which doesn\'t
exist at this point in time. As I already mentioned, they will have a
multi-region capability.

They are going to improve their technological representation. Now MARKAL
\-- A similar region MARKAL model already has vintaging of equipment,
but they are going to do this in the multi-region version of the TIMES
model, and also be able to add on constraints on capacity additions.

Essentially, they are trying to make it more user friendly by having a
generic framework for energy materials, emissions and service demand.

Now the things that we think at this point in time that we need to do to
get the MARKAL TIMES to where we want to be is to bring the TIMES model
in and to evaluate it ourselves. We currently have the MARKAL model for
the U.S. in-house, and we\'ve been running various scenarios off of that
model, but we haven\'t brought in the TIMES model, and we\'re trying to
get it right now so that we can evaluate the methodology that\'s used to
link the various regions and to make sure that we\'re happy with the
formulation.

We also want to expand upon the supply side of the model. While it does
have a supply side, it\'s not rich in terms of detail, and we want to
make sure that we dealt with depletion for fossil fuels.

We also need to evaluate the data that\'s being used. The data in each
of the regions are supplied by the country or the region itself, and
once you join the MARKAL group you get to share the various regional
detail, but we do need to evaluate it and compare the different country
detail against other countries to make sure it makes sense relative to
one another.

We\'re going to need to construct models ourselves to get into the
regional detail that we want for the international energy outlook. The
models that exist today have various extra countries and do not have a
complete listing of the entire world. So we\'re going to have to
construct the models for that.

One of the issues we are going to have to contend with is where we\'re
going to get the forecasts for service demands. We can add on a macro
model to do that or we can look at trying to enhance our world energy
projection system to do that, and we\'re debating that issue right now.

The next issue deals with how to get the macro economic capability. The
TIMES system does not have a macro capability, and it turns out, once
one is added, it makes the problem nonlinear and the run time really
escalates. So it\'s going to be an issue of what to choose to do that
and how much detail we can actually have there.

Lastly, there\'s more work that we feel we\'re going to need to do on
enhancing the user interface in the report writer.

That\'s sort of a summary of where we are right now and what I wanted to
cover. We\'re hoping that this doesn\'t put us back too far, because the
learning curve is going to be so much quicker for us that, hopefully, we
can sometime next year be able to have our first version of the MARKAL
model, if this is the final path we\'re going to take.

I do need to apologize to Tom, because Tom was given our requirements
analysis and our design development plan to look at and to comment on,
without having the benefit of knowing the details of where we were
going. So that\'s what he\'s prepared to talk about, and I\'m not sure
if he\'s going to continue to talk about that or if he\'s going to take
a different direction.

Obviously, what we want to know is how do you feel about changing
course, and what kinds of questions do you have about it?

CHAIRMAN CRAWFORD: Thank you, Mary. Our discussant is Tom Cowing.

MR. COWING: Well, as Mary mentioned, unfortunately for a variety of
reasons that we don\'t really need to go into, my comments were prepared
assuming that the session called SAGE was really about SAGE, and it
turns out that it really isn\'t about SAGE, at least not from what I can
understand.

So what I\'m going to do, I think, is very quickly \-- I think it would
be much more beneficial to EIA if I gave back about half my time,
hopefully, and you got some comments from the committee. I know there
are people on this committee who are far more knowledgeable about energy
modeling in general, and in particular about some of these models. So I
think it might be much more useful to use my time to get feedback from
them, but I will offer a few comments.

I\'m not going to go over all of this, but let me say before I start
that I think that \-- My impression of this thing, whether it\'s SAGE or
MARKAL or whatever eventually gets selected, this is an incredibly
ambitious project, and if you go down a few roads getting started which
turn out not to be the right road, I think it\'s far better to take the
time and get it right than to keep going down that road.

So you know from the documentation, SAGE looked pretty good. If you\'ve
now decided that it\'s not going to fit what you need, then I would
applaud that decision and let\'s find the right road and start moving
down the right road.

So first a few generic comments that I really don\'t want to say a whole
lot about. I really would, I think, rather focus on the bottom points
here very quickly. In modeling in general, it\'s important to be robust
in the sense that you never can always anticipate what the needs, the
model needs, in the future will be. Actually, in your, I think, July
report you acknowledge that, and I think you need to keep that in mind.

Once you get a model going, I think it\'s important \-- and this is,
unfortunately, very rarely done, but it\'s very useful to have some
validation analysis on that model. Validation analysis is simply
sometimes called back-forecasting or [ex]{.underline} [post]{.underline}
forecasting where essentially you run the model over an historical
period, so you know what the answer is.

The idea is to see to what extent the model can replicate the actual \--
the known results, and that gives you some sense of what\'s working and
what isn\'t working.

Another possibility is, again when you finally have a model in hand, a
sensitivity analysis. That\'s where you change some of the key parameter
values and see how sensitive the resulting forecast is to those
parameters, and the results may surprise you a little bit. On the other
hand, they may \-- you may decide that that\'s perfectly reasonable, but
until you do that kind of exercise, you may not know what\'s driving the
results.

Then finally, Mary mentioned this before, having been on the downside of
this issue, the documentation. So I would encourage you, having suffered
from the lack of documentation from previous users, that you be sure
that future users don\'t suffer from the same problem with your model. I
think it\'s important that you have some complete and detailed
documentation.

I had a few comments about SAGE which, I think, are now not worth
talking about, because I really would again like to talk more, or allow
the committee to talk more about your, I guess, tentative decision to go
with MARKAL.

As I said before, this is an incredibly ambitious modeling project. So
it\'s perhaps not surprising that some of these problems that have
occurred in the last six to ten months indeed occurred, in particular, a
worldwide energy/economic model with considerable by region, by sector,
by fuel type, by end use, and by emissions.

The focus of the model \-- If this were any other application, I would
be a little concerned, because it\'s not clear to me. At times the
documentation seems to suggest that what we want here is a model of
international energy markets, and on the other hand, a lot of the
applications that EIA seems to have in mind are international greenhouse
gas issues.

Now in this case, I think I would assert that it\'s pretty difficult to
deal with energy related emissions problems without having a
satisfactory model, a good model of energy markets, to start with. So I
think there\'s less conflict here between these two goals, but I think
you\'ve got to be perhaps very careful about what is the primary purpose
of this model, and not let the secondary purpose drive the development
of the model.

I would think that, while emissions issues are going to be with us for a
long while, as we\'ve seen in the last month, energy issues and pricing
issues, energy pricing issues, have a way of rearing their, I think I
should say, ugly head unannounced at anytime. So again, maybe a little
robustness in model design would be a good idea so that you can
accommodate a number of things in the future.

Several more comments, not critical as much as perhaps some suggestions
for model development, which were inspired by the SAGE documentation but
actually have something to have some application to other modeling as
well. Forget about the top one. MARKAL doesn\'t do it, maybe TIMES. SAGE
certainly did, and so I\'ve lost that.

Again, very ambitious. I suggest that you had a list towards the end
that maybe we could refer back to where there are a number of things to
be done, and you\'re not going to be able to do them all. You\'re
certainly not going to be able to do them all reasonably quickly.

So I think some prioritization is going to have to be done here. You\'re
going to have to decide what\'s most important to be done first, and
then prioritize that list, again because it\'s an extremely ambitious
modeling effort.

Finally, I don\'t know how \-- what kind of reception we\'ll get here,
perhaps not \-- you know, either from EIA or from the committee. But I
would strongly suggest that there is the need for some cooperation and
collaboration amongst Federal agencies here.

I will assume that EIA are the acknowledged experts, certainly on energy
data, and I would assume energy market modeling. But there\'s another
\-- If we\'re really focusing on greenhouse gases use, there\'s another
expertise here that lies in the emissions and in the business, both
energy related and non-energy related, and I don\'t know that EIA has
the acknowledged expertise in that area. It may be EPA. It may be other
agencies, but I guess I would be a little concerned to have one agency
going down a road all by itself in an area where I think there is
broader areas of expertise that may be necessary.

So with that, I\'m going to stop, and I hope that there will be a lot of
discussion from the committee with respect to some of these modeling
issues.

CHAIRMAN CRAWFORD: David?

MR. MONTGOMERY: Does Mary want to respond to any of Tom\'s \--

MS. HUTZLER: There is one that I wanted to respond to, the cooperation
and collaboration with other Federal agencies. I think that\'s very
important, and in fact, we do try to do that.

So far, all the documents we\'ve produced have been distributed to other
agencies, the academic community, and industry to just comment on and
provide their suggestions, and we\'ll continue to do that.

We also try to work very closely with EPA, because they are also
developing their own model to try to look at these particular questions.
At first they were going to go down the same approach we were going
down, and now they are thinking about looking at MARKAL also, and they
are funding some work on the MARKAL model, too. So we are doing that.

MR. MONTGOMERY: Thanks. First of all, for the record I just want to
comment that I was as surprised as everyone by our announcing you\'re
kind of changing directions this morning, Mary. I didn\'t know you were
planning that. But although I was surprised, I wasn\'t shocked by it,
and actually I would maybe go a little further than Tom did in
congratulating you on making a decision of this kind.

I actually think you\'ve shown a lot of courage in doing this, and I
want to congratulate you on that.

As Tom said, in the early \-- I mean, the right way to do R&D is to take
advantage of the fact that you learn lots of things at the early stages
when you look into new things, and that means that it\'s a good idea to
look at several different approaches.

The problem with that, especially in the government, is that when you
adopt that approach, you sometimes have to decide that some of them
don\'t work, and it takes an awful lot of courage to make that decision
at the right time, based on uncertainties in every direction. I think
that doing that definitely continues EIA\'s tradition of telling the
truth, whether it\'s comfortable or not. So I\'m glad you did that.

Another thing is in the beginning of your presentation, I think you had
a really good list of issues that you\'re trying to deal with. I like
the first four better than the rest, if you kind of remember back to
that. I thought they really did get at the heart of things that are both
important and within your capability of answering within a reasonable
period of time.

I am a little bit troubled about the emphasis you are putting a little
later on looking at the response of new technologies to policies, the
idea of endogenous technical change, and I\'m not sure MARKAL is going
to make that any easier for you, because it\'s an area where I think
implementing something in a model is maybe a little beyond our current
capabilities in terms of understanding the fundamental process that\'s
going on there in terms of endogenous technical change. So that might be
something I think you should think about more before trying to leap
into.

The other comment I have is \-- and I think you are moving in the right
direction on this with MARKAL \-- I\'ve always felt that the highest
priority ought to be to get the energy supply and demand right and to do
a good job on a global energy forecast for all the important end use
sectors and all the important types of energy production and all the
important regions. I think this gives you an opportunity to do that.

One of the things that concerns me about MARKAL is there has been a
tendency in the past when MARKAL has been utilized to overoptimize
tremendously, because it is an LP using a kind of a simplified list of
technologies that can satisfy energy service demands that tends to find
lots of ways of satisfying those energy service demands that we don\'t
currently utilize that cost a lot less than what we\'re doing now.

Now the good part about that is the way it\'s frequently been run in the
past, like when Sam Morris was doing MARKAL macro was he set up the
baseline in which he said the baseline is going to satisfy the standard
property of a competitive economy, and I think actually Doug Hill
mentions this in his presentation about electricity markets.

A competitive market will minimize the cost of delivering energy
services. Therefore, we should adopt what MARKAL says is going to happen
as our baseline, and then we work away from that.

That\'s going to be a little difficult to reconcile with the other
baselines you\'ve tended to look at, if MARKAL tends to continue to
overoptimize, and the way people have \-- and there\'s a danger then
which is to try to get around the overoptimization by putting in a lot
of [ad]{.underline} [hoc]{.underline} constraints which you can\'t
explain either, and that then interferes with everything, especially the
evaluation of the cost of policies when you bring them in. So I think
there are things that I kind of encourage you to worry about there.

I was glad you mentioned supply in your slide, because I think that is
where at least all the versions of MARKAL I\'ve ever seen are weak and
where all of us could really use your contributions in getting a good
picture of the world supply picture together.

Just two thoughts following on what Tom said. One is how manageable do
you think sensitivity analysis will be in MARKAL? Can you actually get
at the things that you would like to change, you know, coefficients in
the linear program \-- the matrix of coefficients in order to see how it
responds to things?

The other question is how are you going to handle other greenhouse
gases, if you are going to do climate change policy, because - I\'m not
sure whether it\'s true, but we keep hearing more and more from EPA in
the negotiations about how important controls on other greenhouse gases
are and how much they are going to affect the costs for the energy
sector.

Those are kind of the thoughts that I came up with since hearing about
your change in direction.

CHAIRMAN CRAWFORD: Tom?

MR. COWING: Maybe Mary wants to respond to that first.

MS. HUTZLER: I think Dave\'s questions are good ones, and his comments
to us are good ones. I really don\'t have answers to all those right
now, since we are just \-- On the sensitivity analysis, we\'re playing
around with the model right now, and hopefully, we\'ll figure out a way
to be able to manage that, to be able to get the results out. That will
be one of the things we\'ll have to cover in our new plan, once we
accomplish that.

Also this other greenhouse gases, of course, that\'s a difficult
problem, and I\'m not sure that\'s going to be something we\'re going to
accomplish in the near term. It\'s probably going to be in the longer
term as more information becomes available, that we\'ll incorporate that
into the system. But for the most part, everything you said are things
that we\'re going to have to deal with.

MR. COWING: My point about cooperation and collaboration with other
Federal agencies was really more \-- I would really like to stress
collaboration more than cooperation. I mean, I think it\'s very
commendable that people share results of work they are doing, but it
strikes me that this is a great area for collaboration, joint model
building efforts.

Now whether that is feasible within a fairly competitive environment
between agencies in Washington, I\'m not able to judge, but there\'s
certainly a lot to be said for not having two agencies going down the
same road at the same time, doing exactly the same thing or roughly the
same thing.

MS. HUTZLER: I agree with you and, if you could tell me how to do it,
I\'d be happy to try to do it. I think we are succeeding in DOE doing
that much more than we have in the past, and more and more of the DOE
folks are coming to us.

In fact, Jay had a roundtable with government officials in DOE, and they
were encouraging us in this particular task and indicating wanting to
use the model. But that\'s taken us a number of years to get to that
point.

Unfortunately, EPA has a lot of money and can buy lots of different
models and get lots of people to do work for them. As a result, they
have opportunities to go to other sources. So it\'s been difficult to
try to do more collaboration with them, and I would like to do more, but
I\'m not sure at this point in time exactly how to accomplish that. So I
would welcome any ideas that anybody here has for it.

MR. COWING: I understand what you\'re saying.

MR. MONTGOMERY: Actually, with respect, Tom, I think I disagree
completely. My recommendation would be don\'t. I think it\'s been really
important, looking back over the last \-- what is it now? \-- 25 years,
even going to EIA\'s predecessor, that EIA be an independent agency with
its own capability for doing modeling and analysis, and you have proved
that numerous times in the climate change debate when EIA\'s analysis
has shown results that are directly the opposite of what both the
administration and EPA have wanted to say about policy issues.

When we\'ve looked at them, what we have found is that EIA\'s results
have shown integrity and careful research, and the other parts of the
administration have had to come around to that.

I would strongly urge you not to collaborate in Tom\'s sense with EPA on
this, because their track record, I feel, has not been good in producing
the kind of objective analysis that EIA wants to produce. I think we
need you to stand up on your own.

It\'s uncomfortable, but you do a really good job of it. And the amazing
thing is with how much less money you do that job.

CHAIRMAN CRAWFORD: Someone has a question. Would you please step out to
\--

MR. COHEN: Well, not with a question. It\'s a comment on that.

CHAIRMAN CRAWFORD: Please step up to the microphone.

MR. COHEN: I think it\'s important to have as a background information
on that the MARKAL model is the intellectual property of a group called
the Energy Technology System Analysis Program, which is an organization
created by the International Energy Agency.

The MARKAL model itself is available freely to virtually anyone, and
maybe 80-90 groups around the world currently using it. Where EIA comes
in, where each group comes in, is in setting up the database, the access
database, in fact, that the model uses. But the same groups all over the
world use essentially the same model.

TIMES will be the same way, but each group has to develop its own set of
databases. When you\'re using the model, you set up your constraints and
you, as Dave was saying, overoptimize them into temporal constraints.

The model results are going to be 90 percent modeler and ten percent
model, and that\'s the way it probably should be. Ultimately, TIMES will
be available to you and everyone else.

MR. COWING: Could you provide a little more information about TIMES? It
came at the end of your presentation, and maybe I missed something or
maybe it wasn\'t there.

Who is responsible \-- Is this an EIA project or is this a project being
run by some other group or groups, and how far along are we, and how are
you going to make use of that?

MS. HUTZLER: The MARKAL model was originally developed by Brookhaven
National Laboratory, and now there are other people worldwide that are
using it. In the U.S. the policy office here has the U.S. model of
MARKAL, which means that they are in charge of the database in the U.S.
model.

There\'s a group in Canada that have regional province type models of
MARKAL for the Canadian side. Various European countries have MARKAL,
and I believe they also have one for the European Union. Australia has
MARKAL, and in fact, ABER is now working with a lot of the Asian
countries to set up MARKAL models within each of those countries.

There is an effort ongoing to have the South American countries also
have their own MARKAL models. Now those are individual regional models
that are MARKAL. The next step, of course, is to put them together and
to have a worldwide model.

In fact, one of the reasons why we didn\'t go down this path last year
when we were looking into it was we weren\'t sure whether they could
combine these regional models and get a solution. In fact, they have
accomplished that in the first phase, which was available this spring.

There\'s a group of people \-- it\'s called ETSAP \-- that sort of
communicate on these issues, and different people that are working on it
sort of take the lead for different responsibilities in improving the
model.

One of the things that they have to do for TIMES is to take all the
databases that they have for the MARKAL version and be able to put them
within a TIMES version. The TIMES will be a multi-regional model solved
as one linear programming model, and they plan to have that done in the
end of this calendar year where they will have the rest of the world,
region, and then certain specific regions.

It may, and probably won\'t, follow what we would like to do for our own
representation. So that\'s part of the job that we have to do to set up
the different regional details.

Does that answer your question?

MR. COWING: Well, except the word \"they.\" Who is they, with respect to
TIMES, not MARKAL?

MS. HUTZLER: Okay. Each of the various countries that are participating,
essentially.

MR. COWING: So there is no overall group that\'s going to take the
responsibility for integrating all of these regions into a single model?

MS. HUTZLER: No. It\'s basically the users who want to participate in
this particular program, and that\'s one of the reasons why they are
interested in our participation, too, because it would add resources to
the entire effort of getting all of this together.

What countries do is trade databases with each other.

CHAIRMAN CRAWFORD: Okay. I\'d like to move along. Thank you very much.
It\'s a very interesting topic. I think there is a lot of interest
within the committee on the issues that you are grappling with. So if
you think you have some further development by next time, we would be
interested in hearing from you again. So thank you.

Our next session is on cognitive interviews on EIA\'s web site, seeking
more reactions, and Colleen Blessing will be leading this discussion.

MS. BLESSING: Good morning. I have the great honor to be the speaker
before lunch. So I\'m going to \-- I talk fast, and I talk loud. So,
hopefully, I\'ll keep you awake, and I know that we\'re scheduled to
break at 12:30.

First, I want to say that I am representing and presenting work that\'s
been done over the last year by the Cognitive and Usability Evaluation
Committee, and there are two other members of the committee here today,
Renee Miller and Howard Bradsher-Fredrick, and that it\'s also
Antoinette Martin and Bob Rutchik. There\'s Antoinette in the back, and
Herb Miller.

At the beginning of December I was up at an EPA customer service \-- It
was really a convention. There were hundreds of people there, up in
Philadelphia, and there were just speaker after speaker after speaker.
Every single one that got up, within the third or fourth sentence, they
said, well, our award winning website, and about the eighth sentence it
would be, well, you know, as people look at our award winning site.

Well, by the middle of about the second day, that got to be a joke.
Luckily, I spoke in the morning on the second day. By the time John
Cavalunas from Census got up and was talking about the award winning
site, people were either laughing or booing. I figured either there\'s
too many awards being given out or only people that get awards get to be
chosen to be speakers.

I always like to start out my talks on our website with the fact that we
are an award winning site. In October of 1999 the Government Executive
Magazine picked the top 16 Federal sites, and EIA was selected as one of
those winners.

So I try to position myself so that we say we\'re moving from a position
of strength. We have a good site. We have lots of visitors. We have lots
of good information. So then when I get to the middle of my talk where
I\'m sort of seeing people are getting lost and I\'m hacking up what we
do, then I have to return to sort of the award winning site.

We do have a good site, but we didn\'t let the award let us become
complacent, and then we just say, well, okay, we can kick back because
everybody loves us. We realized that there were still areas where we
could improve.

Now I think I see a couple of new faces around the table, and I want to
take about two minutes to put my talk into context. If you all remember,
the ones that were here in April, we dragooned you all away from this
table and took you to a room \-- most of you, except for Polly and
Seymour \-- took you to a room where we gave you a test on our website.

That was our very first hint, our real testing hint, that we had areas
for improvement. I remember Cal Kent gnashing his head against the
machine saying, I can find this better in a book; and I remember someone
else saying, I would have called in the IC by now.

So we got the message loud and clear from the committee that we needed
to go ahead with this testing with real people. In November, the ones of
you that were here in November may remember that Howard
Bradsher-Fredrick went over the testing that we did in the summer where
we did cognitive individual tests with 17 subjects. I sometimes call
them victims, where we sat them down with a PC, and we gave them the
similar type questions that we gave you, nice little protocol.

They had an hour to complete four questions, ten minutes per question,
and we watched how they navigated, and we asked them what they were
thinking and how they were feeling as they were going through it.

As Howard probably summarized in November, the results that we found
were that there was too much jargon on our site. Sometimes when I hear
people talk about websites, they say their site was designed for
professionals. I don\'t think it\'s that our site was designed for
professionals. I think that our publications and our reports, everything
we do, has a professional air to it, and there were lots of terms that
the people that we tested didn\'t understand.

There were too many buttons on our page, and I don\'t think anybody in
EIA would disagree with that. All the people that we tested said that we
looked at every single button, and we know from our testing that they
didn\'t see probably half of them. They focused on certain areas and not
others.

The search engine didn\'t work \-- The search engine that we had at the
time didn\'t work very well, and the ones of you that were in the
testing that tried the search engine found that out.

Also a fourth finding that I\'ve kind of added is that I think that EIA
is still writing for the web \-- I mean writing for paper rather than
writing for the web. We sort of write for paper, and then we kind of
fold it up and shove it through a sieve, and then it gets out on the web
there, but it\'s not \-- Sometimes people would click on a button
expecting a table or expecting some data or a chart, and there you would
see giving this long executive summary with paragraphs and paragraphs,
and they didn\'t know where they were, and they would kind of bail out.

We had always talked about \-- I know Mark Rodekohr is here; he\'s one
of our Webmasters. We talk about the three click rule. We wanted to get
people the data in three clicks. A lot of our answers for the AS and for
our tests were available in three clicks, but which three clicks? That
was the test.

People, a lot of times on the ones that I tested, could get two clicks
down, and you\'re sitting there saying click there, click there, try it
there; but they couldn\'t understand the table name or the jargon, and
they couldn\'t see it in the paragraph. They didn\'t know what the third
click was.

So what we figured out is basically that we needed to make our homepage
and our paths to the data more intuitive so that people \-- and also we
sort of adopted a theory that we call \"all roads lead to Rome\" where,
if you click petroleum prices, you end up at the same place as if you
clicked prices, petroleum. So you could enter in different ways, but you
would end up getting to the same tables ultimately.

Okay. So we did all this testing, and now this is where I pick up from
where Howard left off. The next step was our committee said, well, now
what do we do. So we decided that we would \-- We talked with a website
usability expert, and we decided that we would set up a joint
application/development session, which basically is just a big meeting.

We had an all day session with between 35 and 40 of the staff that were
intimately involved with the website. We had such good participation.
You know, all day meetings at EIA are not particularly popular. We had
such good participation that I think we had more people in the afternoon
as people left for lunch and said \-- than we did in the morning. So we
had great participation.

What we did in the afternoon is something called parallel design work.
The usability expert said it\'s much better to have little groups of
people come up with their own ideas and then put them together, rather
than having you all dictate to me the ideas and putting them up at once.
It doesn\'t allow for as much creativity.

So we came up with lots of different designs, and voted and voted and
voted, and came up with a new sort of schema that we did send you a
couple of versions of.

Then what we decided that we would do is test these versions. You have
them in your package. I only have two of them, because the third one was
the electronic version, and you may or may not \-- Did anybody have
success opening the electronic version, the third one that got sent?

CHAIRMAN CRAWFORD: What was it supposed to look like?

MS. BLESSING: A purplish and bluish, and it had \--

CHAIRMAN CRAWFORD: Because one thing I opened up just had a search box,
and that was all. That wasn\'t it.

MS. BLESSING: We knew we had problems in sending that out.

Well, basically, the two that you got in hard copy were the ones that
were the basic ideas of what we tested. One thing that we tested was \--
Well, first we just decided that we would say how do you want to access
the information, by fuel, by sector, by geography, by process or by
prices, environment, forecast or analysis?

We tested one version \-- I\'ll tell you about the test in a minute \--
tested one version that just had the big buttons and no hints
underneath. Then we tested another \-- In paper, we tested this. I
handed this to the person right here in paper, tested another paper
version which, reading the usability literature, paper testing is
popular and very accepted.

We tested another version that had the same big buttons with the hints.
I have to stress that we were trying to test for content only. We made
this just as pretty as we could without making it too pretty. We didn\'t
want people saying, well, I really don\'t like the butter yellow,
because that\'s not what we were testing. We wanted to test on content.

We were advised to test in black and white, but I don\'t know. I didn\'t
have the heart for that, because I thought people would say, yuk, it\'s
black and white, I don\'t like it, without even looking at it.

Then the third version that we tested was one similar to this with no
hints, but on a laptop with a mouse, and there were drop-down boxes.
There were mouse-overs on all of these. So the little hints came up as
you moused over the big boxes.

I personally wasn\'t sure that was going to work, and it turns out
people sit down, and they think with the mouse in their hand, and they
started right away starting the mouse-over. Even though the design that
we tested was kind of ugly, because it hadn\'t really \-- We were
putting it up at the very last minute \-- people still didn\'t have a
problem seeing the mouse-overs and understanding what they were.

What we did is we went to the NEMS conference, the National Energy
Modeling System conference that we hold every year over in Crystal City.
We set up a table with all of us sitting around the table, and we were
right outside the door. So as people would come out for coffee or to
wander around, we were just almost grabbing them by the sleeve.

MR. HAKES: What do you mean, almost?

MS. BLESSING: Did we grab you? We were grabbing them by the sleeve. We
had at least one person sort of working the room, asking people to come
over and do the test.

Sometimes the test was 90 seconds. It was probably an average of about
three minutes where we sat them down and we handed them a piece of \--
We handed them one of the three of these, and asked them three or four
questions: Where would you click to find? Where would you click to find?
So all we were doing is looking at very first level navigation.

They still wanted to tell us some things, and we were writing down some
things, but in sort of in the usability field I shouldn\'t really care
which one you like. This one is pretty, and this one. I don\'t care
about that. I care about whether you can find \-- If you can find the
information better on this one, then it doesn\'t matter if you think
this one is pretty or this one is the one that I should use.

So we were trying to go \-- not pay too much attention, because some
people did want to talk about the yellow and the blue, and we tried not
to listen to that.

We tested 69 people in one day. That\'s how many sleeves we grabbed. The
only demographic question we asked them was have you used our current
site, and 70 percent of them had used the current site.

We also wanted to make sure with our questions that we covered as much
territory in EIA that was on the website as possible. Our previous
questions left out \-- You know, there\'s so much on the website, you
can\'t ask a question about every single page.

We sent a matrix to the committee showing you. If you ask a question,
what\'s the price of gasoline in Massachusetts, that\'s a price
question. It\'s a petroleum question, and it\'s a state level data
question. So we have a matrix showing the breadth of coverage on the
site for all the different buttons that we had, and we tried to do a
pretty good job of that.

I guess the main finding is that people seemed to like this new version,
and they also seemed to \-- They didn\'t seem to have a problem with the
mouse-overs.

So the next step: We decided that we wanted to \-- Well, we came back,
and we actually talked in the committee. These talks are ongoing almost
even as we speak, and changes are ongoing. But we had a long still about
whether we were going to use mouse-overs or not, and we have decided
that we are going to go with that idea.

Now what we\'re trying to do is finalize the look and the placement,
make it actually prettier. So what we have now \-- and I\'ll have Nate
throw it up there in a minute \-- is much different looking than this,
but it\'s the same kind of idea.

We are having a usability contractor. She\'s just working off-line for
free, because she likes us, and she\'s real interested in usability and
design. She has been giving us some suggestions on the site, on the
homepage design.

I think the spot where we\'re at now is \-- and this is a meeting we had
yesterday \-- is we\'re really talking about where we should go with the
level 2, level 3 pages and how much work we should do before we go ahead
and put up something, some of the work that we\'ve already done.

Before I show the final version, because I think we\'ll get waylaid on
that a little bit, I just wanted to discuss two of the tradeoffs that
seem to be sort of coming to the surface as we\'re talking in our
meetings.

One of the tradeoffs is getting a better first homepage and maybe a
couple of second level pages, getting those out as quickly as possible;
and it\'s still a slow and laborious process, but getting them out as
quickly as possible and not worrying about \-- not having all of the
navigational changes, not having everything mapped, versus waiting until
you have a complete mapping, until you have \-- I could picture taking a
wall about this big, and I think I could map the whole site if I had
lots of markers and lots of days. You could do that. You could actually
just conceptually map it. I don\'t know how long it would take to
actually do all the work.

So it\'s a question of getting up something that\'s better at the front
end, and people might still be a little bit \-- They might be lost in
some areas when they get down farther \-- or should we wait?

I think we\'ve taken the tactic \-- The committee has decided that
we\'re pretty much going to go with the put up the good things that
we\'ve already developed quicker rather than waiting for everything to
be done.

The other issue that we\'re looking at is sort of the three click rule.
If you have a three click or we even maybe got to four clicks, by the
time you get to the third click, we\'re sort of expecting that people
are going to come to data. But if you still have decision trees \-- do
you want spot prices or wholesale prices, retail prices, and do you want
them monthly or do you want them weekly, or do you want them this? \--
If our goal is really to get them to everything that we have, it\'s
going to take more than three clicks.

That is a philosophical difference that\'s being discussed in EIA. What
the three click rule might entail is EIA experts deciding what table it
is I probably think you want. If you\'re asking for electricity prices,
we might have ten tables with electricity prices, but I could probably
give you one that would be close to what you would want.

With the three click rule, we would have to make some decisions about
what we put up in front of their face first. The non-three click rule
\-- and I can argue this side, too, that sometimes people don\'t mind
doing more than three clicks if they are sure they are on their way to
the right data. I think we could do a good job of that, too.

If you want to just \-- I just want to show you where we are. This
changed since last night. This morning I got the URL, and it had changed
again. So I don\'t know. It might not look like what it looked like
yesterday.

That\'s not a real good representation of what it looks like. Can you do
a mouse-over? There you go.

So as they mouse over all these different things, choices will come up
that are clickable. So you can click right to industrial. These are not
just hints. They are actually live clicks. Then we spent a lot of time
\-- Actually, this is an older version. It\'s still got the MER on it.

Anyway, ignore that man behind the curtain. This is the idea. It\'s sort
of bluish and purplish and reddish. We\'ve gotten rid of the red on blue
that people can\'t read, and we\'ve got some stars. We\'re going to have
a place for an interview spotlight. This is going to be connected here,
and this is our top little report right now. That will be a spot for
whatever our most \-- the interesting thing of the week or the month.

I guess, if anybody wants to see it in more detail, we can probably show
it on a laptop. This is just an idea of what it looks like, where we are
right now.

We did a lot of work yesterday even on this, and some of it is
represented here. There\'s going to be like a \"need help\" at the
bottom right here, so you can contact the info center or call. We\'re
hoping that that is going to be above the fold. So that will be right
there on the page, \"need help?\" right at the bottom. They will be able
to see where to go.

Thanks. You can just turn it off.

Then the last thing before I turn it over to Carol is we asked you to do
some homework, and there were two things that we asked you to do. I hope
that, for the people who tried the homework, it just got you to take a
look at the new design and get an idea of whether you like the way we
redesigned the entry, the gateway.

The search engine that we tested \-- well, when we tested it, we knew
that it didn\'t work very well, but customers confirmed that. Two or
three months ago, we leased a new search engine called Google. We\'ve
got it for about three more months, and we\'re testing it out now.

So we were hoping that some of you had a chance to do the test. I
actually did the test yesterday. We sent you seven questions and said
what would you search on. I did the test yesterday. I don\'t want to
spoil anybody\'s thunder, but, man, I was amazed.

Six of the seven questions, that search engine came up with the answer.
Now it might be that I\'m familiar with the data, and I kind of knew
which one to click, but almost always the answer was in one of the first
ten \-- Six out seven, the answer was in the first ten sites that came
up.

I tried \-- I actually did some \-- I don\'t know if you would call it
sensitivity analysis, but I typed in \"Russia, oil exports\" or \"oil
exports, Russia.\" I was trying to sort of trick it and see if I could
make it not work, and I couldn\'t make it not work. It kept giving me
the right answers.

So I\'d be curious to see, for the people who tried the search engine,
what you thought of it. Also, if anybody had any comments on where they
would click to find the answers or what they expected to see on the
second level pages, we really would appreciate that, too.

I\'ll turn it over to you, Carol.

CHAIRMAN CRAWFORD: Well, it was interesting, because originally this
session wasn\'t going to have a discussant, and then Colleen changed her
mind saying, well, we should make sure that somebody actually does it.
So I did actually do it.

So I can tell you what I clicked on and how easy it was for me to find
things.

MS. BLESSING: Did everybody get the \-- at least get the homework, the
seven questions?

CHAIRMAN CRAWFORD: Okay. There were seven questions. I\'m not going to
stand up here and go through what I did for seven. I\'m going to go
through two of them, and then two that I decided to make up on my own,
because I may be a little bit different than some of your normal users.
So we\'ll see if I could find the information I was looking for.

So the first question was: How much energy was used in the United States
in 1998? I thought, well, why not try 1998. I was thinking maybe I would
get some kind of a list of key publications that summarized 1998 data,
and I did.

This was a summary of the links I got, starting with the natural gas.
There were actually more than this. I just listed these. So in general,
I thought, okay, maybe clicking here might get me to what I want, but I
didn\'t think so.

So I went back, and I refined the search to total energy in 1998. That
gave me what I was really looking for, I think, total primary energy
data. From there, I was able to get a table for energy consumption by
country by year, and I came up with that answer.

I was a guinea pig the last time that we did this when they dragged us
down to the computer room, and I have two different perspectives, how I
managed to do it and what Calvin managed to do; because I was sitting
next to him.

Calvin was very familiar with all the publications and the information
and even practically almost the page number certain information should
be on. So when he went to answer the questions, he would just go get a
certain publication, and try and find it that way, whereas I was using
their search engine.

I couldn\'t come up with anything. I got like zero out of ten answers.
Even when I started to think I was getting close to the right answer, I
could never be quite sure that the information that I was getting was
really what I wanted.

In this particular case, I didn\'t have that problem. I would say the
three click rule worked great for me. In three clicks I had what I
needed, and I was also sure that what I was getting was the right
information.

Notice that I searched on \-- I refined it a little bit more, just to
see what happened. I basically typed in most of the question. Total
energy U.S. 1998 was too specific. They gave me these links that I
didn\'t think was pointing me in the right direction.

So that was part of my homework assignment, was to write down what I
searched on and what I found and how it differed from what I expected.

The next question I looked at I picked for a couple of reasons. Has
Georgia passed deregulation legislation for its natural gas industry?
The answer is yes, and the reason I know is because I just got an \$800
bill for my gas that\'s in dispute.

Okay. This has been a chronic problem with a lot of consumers in
Georgia. We\'re not sure what our bills mean, and they\'re not correct.
Then you call the customer service and you say I have a problem with my
bill, and I got this thing, well, we\'re guessing that this is what went
wrong. Why are you guessing, you know? This is a bill.

Anyway, the answer to this question is yes, they have passed it, and
they\'ve actually implemented it. So I searched under George
deregulation. The third link was this. The answer was yes, and the
legislation was passed in April 1997. So I was thrilled. I don\'t feel
so stupid anymore. Of course, I wasn\'t sitting next to Calvin who had
probably had everything a lot quicker, but this was working very well
for me.

One of the questions I wanted to know which, incidentally, I had asked
before, was what was the current agenda for the AS meeting. I searched
under AS, and the first link resulted in the agenda. So this one was a
one-click rule.

I have to say last time when I went to search for this information, it
wasn\'t there at all. So I think maybe there\'s more information that\'s
being put on your website now or at least I\'m certainly able to find it
better.

Some general comments: I thought it was easier to locate answers to
questions than it was in the past. I thought it was easier to find the
relevant information, and I was more sure of the meaning of the
information that I was getting.

I did have a question about how some of the tab links were chosen, like
for instance when I just searched for 1998 publications, why did it
decide that uranium was going to be more interesting than electric
power? I didn\'t understand that. I guess I wouldn\'t think that to be
the case.

I also tested it was robust to typos and misspellings. I was chronic
that way anyway, but you know, a lot of people are typing quickly, and
they make a mistake. It didn\'t seem to bother the search engine at all.
I mean, I spelled deregulation in like six or seven different ways, and
it always gave me the same link.

The search engine is smart. I thought I would really get it \-- I mean,
part of it is do you get the information that you want. As a subset of
that, does it screen information you don\'t want. A good test is I
searched under energy. I thought, well, let\'s be really stupid and
search the Energy Information Administration for the key word energy.

I liked what I got. I mean, I was surprised. This is what you get if you
search under energy, and you get about the U.S. Energy Information
Administration, and you get some of their top summary documents. So I
thought, wow, this is really smart.

So even when I search on something that I thought was stupid, I get a
smart response. So I agree with you. I think it\'s a great search
engine, and I think it\'s working really well.

About your homepage, I was at a disadvantage, because I couldn\'t do the
mouse-overs, and I couldn\'t see things. All I got were those black and
white copies.

This is what their current web page looks like, just so you can see it
compared to some of the other ones that she was showing. It doesn\'t
show up as well, because it\'s a black and white copy. I work for the
government. We don\'t have any color copiers.

My preference was I preferred the current homepage to any of the
alternatives, and again I\'m going to put the disclaimer on it. I
didn\'t get to see these in action. In particular, I was very biased
from this point of view, because, see, I couldn\'t see mouse-overs. So I
looked at this thinking, gee, this is very plain, and it doesn\'t have a
lot of information on it that I needed to know.

So then you can just disregard that comment, because once you do the
mouse-overs, then everything is as it should be. But I did say, the
current pages \-- It\'s complete, but it\'s concise.

One of the things that I noticed about it, you see, is the hot topic
today is gas prices, and you can find that. Things that are of interest
to you are right up front there in terms of, you know, pre-summer
transportation, fuels outlook conference, testimony on rising heating
oil and diesel. I mean, those kinds of things are right up front.

So I liked it from that point of view, and I didn\'t see that so much in
the forefront. I mean, there is a click on the new page for what\'s new,
but some of the current hot buzzword topics are not just right up there.
My feeling was a lot of people might be coming specifically for that
information.

So that\'s what I mean by hot topics and new information is easy to
find. I didn\'t think it was too crowded or too confusing, but maybe
I\'m interested in \-- Maybe I\'m used to working with it.

This one, I thought, to me, had too many choices. But again, I think
with the mouse-overs they\'ve solved a lot of these problems, because
they have made it a little bit more plain, but sort of like merged these
two together in a dynamic kind of interface.

Then the other thing is your site is an award winning site. So I\'m
concerned that a total revamping of the site isn\'t necessarily what you
want to do. Maybe just tweaking it in a certain way, but then in
retrospect, I lost a lot because I didn\'t have that. So this one might
be a really good one.

So I\'ll take questions from the committee, if anyone else did the
exercise and searched on things or used it for finding the materials for
this meeting.

MS. BLESSING: I had one rebuttal. We did decide \-- and it was hard to
see on the screen when Nate put it up there \-- that the righthand bar
with the special features was something that we needed to have. So that
is now in the new version. There\'s stars beside all those special
features.

So I can\'t even hold it up. It\'s hard to see, but we did add \-- In
the test version, it didn\'t have the hot topics or special features
along the righthand side, but we do now.

CHAIRMAN CRAWFORD: Comment, Randy?

MR. SITTER: Actually, I did the exercise. It was successful most of the
time. I mean, I rarely got the top hit, but in the top ten it usually
had the one I needed. But one thing that was both entertaining and a bit
frustrating was that the search engine is quite nice. It gives a little
quote where it found the information sometimes. It gives a link, and it
actually gives a quote from the document.

So then you click, and you go to the document. It takes forever to find
the quote. Now if they have already found the quote, it sure would be
nice if it actually came up where the quote was. They must know where it
is.

I think in the last two that happened to me. Some of the documents that
you went to, some of them were \-- the information was right in your
face, but some of them were quite long documents. You know, you can find
it easily enough by going down the headings and so forth, but it always
bothers me when a search program finds exactly what I want and then
still leaves me something to do to actually go and find exactly what I
want.

MS. BLESSING: Puts you at the top of the document, and then you have to
go to \--

MR. SITTER: Yes. Then you got to go and try and find the paragraph.

MS. BLESSING: I noticed that, too. Sometimes it came up with it looked
like four or five hits from the same document, and I asked Mark about
that. He said, because it\'s finding different parts in the document.
But you\'re right. Then when you click on it, it\'s putting you at the
top, and you have to find those different spots.

MR. SITTER: Exactly.

MR. RODEKOHR: We can certainly ask the Google people if that can be
done.

MR. SITTER: Because one was particular. I believe I got a quote from
U.S. Coal. The quote at the bottom says 1998 coal production record
breaking. I clicked there, but then I still had to read to try and find
where it said that.

CHAIRMAN CRAWFORD: Any other questions or comments from the committee?
Anything from \-- I think someone from EIA had a question or comment?
No?

Well, if there\'s nothing new, I guess we\'ll break early for lunch. I
have one announcement regarding lunch. Is there anyone that will be
attending lunch that\'s a vegetarian? If you are or if you decide to be
right about now, tell the server, and the server will bring you your
plate. I guess there\'s only a few of these. So request it only if you
really need it.

(Whereupon, the foregoing matter went off the record at 12:11 p.m.)

A-F-T-E-R-N-O-O-N S-E-S-S-I-O-N

(3:22 p.m.)

CHAIRMAN CRAWFORD: I\'d like to get started and have our group leaders
summarize some of the high points of the discussion.

I would ask that they take about five, seven minutes per person so we
can make sure that everybody has enough time.

MR. HALE: Hi. I\'m Doug Hale. In my session we went over the paper that
I presented which is actually called \"Modeling Electricity Markets:
What Next?\"

What I was interested in doing was bringing the committee up to date on
the work we\'ve been doing in modeling electric power in light of all
the deregulation that\'s going on or re-regulation, and getting their
ideas about how we should \-- what we should focus on in the near
future.

There are three projects that I had brought to the table. The first was
one to expand our model from basically New England, New York,
Pennsylvania, Indiana, Ohio for the entire East, basically from the
Rockies east.

That project is essentially underway at a low level. It\'s something
that makes sense. It\'s cheap and relatively easy to do.

The other two projects I wanted them to think a little bit about was,
first, to kind of look at the sort of modeling we are really doing.
We\'re using a kind of modeling that\'s been used in the electric power
industry for sometime, but hasn\'t been used very much in economic
analysis.

What we\'re doing is something called an optimal power flow where you
minimize the cost of delivering electricity geographically, and you take
account of all the constraints on moving power, all the interactions
from where power is put in and taken out, and you take account of the
losses.

There are certain advantages to doing \-- to modeling the electric
system in such detail. It allows you, in particular, to calculate purely
competitive prices for electricity. The other thing it does, even if
you\'re in a noncompetitive model, is to allow you to see how the
various strategies and the various market participants may play would
actually influence their competitors and their customers.

The problem with those models from our point of view is that they\'ve
never really been verified against actual data. Where they have been
used is in the allocation of electric power. The economic version of it,
however, has really not verified.

So the second project that I asked them to consider was that one. That
is, to go ahead and take one of our models, apply it to an area, let\'s
say, like Pennsylvania, and compare a calculator with actual results.

The third project was to go beyond the purely competitive types of
analysis we\'ve been doing and to look at some forms of noncompetitive
behavior. My committee, I think, did an excellent job of responding to
my interests. I think one thing that they pointed out that I\'ve been
aware of but really hadn\'t focused on very much was a different area.
That is the question of how sensitive these models really are to slight
changes in assumptions.

One of the things we\'ve noticed in the models is that you can change,
say, the properties of a particular line in Vermont and have it affect
prices in southern New York. That\'s fairly unsettling, but in fact, to
some extent, it is a real world.

The question was to try to get a better handle on how common these are,
how realistic they are. Are they simply an outcome of the solution
algorithm or is something else going on?

Another version of the look at the algorithm look at the modeling sort
of suggestion was to apply a different calculating algorithm to New
England than the one that we had actually used. It\'s a little
convoluted to try to explain at this session, but it\'s actually a
really good idea. I should have done that.

As far as a verification project is concerned, I think people were quite
supportive of the notion of trying to verify these models against actual
data. But I think there\'s a lot of concern that what we\'re going to
find is that one of the big problems with the model is the quality of
the input data we\'ve got.

So there\'s a real insistence that we pay a lot of attention to the cost
data and the networking data and the price data that we have available
to us for doing these comparisons.

There are also a host of observations about the difference between the
physical world which we model and the actual contractual world that
traders are using. So there\'s a real wedge between the commercial
reality and the physical reality in the real world that isn\'t being
captured in our model, and that may again be a source of some real
problems.

I am very high on doing the verification project, and most of the
committee seemed fairly supportive of it. However, there is one
persistent advocate of doing something about noncompetitive behavior in
the market, and that was Cal Kent.

So given that, we\'re going to do something on it. I\'m not sure exactly
what, but we\'re going to be getting on that. In short, I found it to be
a very productive session. I have a number of very good suggestions, and
we\'ll be reporting back to the committee fairly quickly on them.

So thank you very much. I really appreciate your help.

MR. SCHIPPER: The area that we discussed was disclosure auditing
software. To provide a little background and scope on the problem, to
protect confidentiality agency suppressed table cells that might reveal
sensitive data.

We have software that exists that does the suppression. However, it
doesn\'t provide any way to evaluate those suppressions. By evaluation,
I mean the upper and lower bound on a suppressed value cell.

Now EIA is leading an interagency agreement project that will use LP
programming to obtain upper and lower bounds for all suppressed datasets
within our reports. We have seven agencies on board, ranging from the
Bureau of Labor Statistics, Bureau of Economic Analysis, National
Science Foundation, Bureau of the Census, a host of people that are
looking for a solution to this common problem.

What we basically stepped through were the features of software. The
first part was basically the verifying of an auditable table. In other
words, do our table\'s sum \-- its individual components sum to a
published total?

Now that may seem like a very naive thing to do, because that\'s what we
publish tables to do. But because of independent rounding, totals do not
sum in EIA reports. A message from Jay on the committee said, well, you
can use a different logarithm to basically tabulate your data called
accumulate and round system.

Now I don\'t know the exact origins of the logarithm, but it is an area
that we will be looking at. Now whether we can get seven other agencies
to agree to retabulate their tables based off of the system where totals
will sum to their individual components, that is a little bit in doubt.
But it was a good suggestion, a way that we will avoid the need to
pre-verify a table for auditing.

Once we had the pre-verification done, we went into the LP system,
basically the constraints required to solve the upper and lower bounds
for all suppressed data items.

EIA has with these agencies gone through several different
methodologies, and we had specific questions whether we were using an
approach that was the most efficient and that solved the problem as a
data intruder would look at the problem; because we\'re coming at this
problem as what is an outside individual looking at EIA statistical
reports going to be able to tell and break down our confidentiality in
our tables that would perhaps have a breach in disclosure.

So with that, we\'ve developed a methodology of allowing of modeling
table data, and we felt it was \-- it could need some improvement. The
committee came back with a very thoughtful suggestion that we could
simplify the LP problem by eliminating constraints that do not impact
the LP model.

It\'s something that we hadn\'t looked into. It will greatly aid the LP
processing. However, it will bring about a need to have a further front
end process to the software.

With that, there was some final suggestions that we had brought up a
very complicated problem that we\'re solving using linear programming,
and we\'re using examples to demonstrate the needs and how we go about
solving this problem.

In terms of efficient processing, it was brought to our attention that
we perhaps should run these systems underneath all the different
attributes, underneath a simplified model, underneath a cumulative
rounding approach to generating our tables, and to see what effect that
has in terms of efficient processing, in terms of speed; because we\'re
looking at programs across these seven agencies that perhaps will use
this on a weekly basis and cannot dedicate a huge amount of time to
confidentiality, LP processing, that rely on their suppression package
to handle that type of system.

So the comments we received will greatly aid slimming down the software
to something more efficient.

The last part that we addressed was we expect the software, and we\'re
in the midst of it \-- we\'re coding it currently. We\'re in the
methodology stage \-- on how we\'re going to interface with users. We
expect to release this free of charge at EIA and also through the OMB,
the Federal Committee on Methodology to make this available to all
agencies and all users.

With that, we had questions on how we should interface with these users,
whether it would be a batch processing, whether we would use a web based
application or graphical point and click user interface.

We received comments back that it really depends on cost and budget
issues. In terms of discarding those budgetary issues, which are hard to
do, especially when funding is coming in from so many varied sources and
satisfying so many different needs, that a graphical user interface
would provide the most benefit to users in terms of their ease of use as
well as their ability to learn and to basically handle the software in
this processing, rather than a batch model, and avoiding the more
expensive web based approach.

We got great comments back. I am more than thankful to Jay and Carol for
their information they provided, and hopefully I\'ll get back to them
soon on the changes that we\'ll make based on their comments.

MR. FRENCH: The session that I was asked to chair was a series of
methodological issues associated with EIA\'s energy consumption surveys.
Our reviewers were Polly Phipps, Johnny Blair, Randy Sitter and Roy
Whitmore, and we certain appreciate the efforts they put in during the
session.

The first thing we talked about was the issue of phone versus personal
interviewing in our commercial building survey. The current cycle of our
commercial building survey is being done by phone instead of personal
interviewing, really is cost saving measure more than anything else.

What we have discovered in early returns is that we\'re working hard to
keep the response rate up on phone interviewing. How well that works
right now is an open book, and we\'ll have to see.

The field work period is going to take longer than the comparable field
work period for the in-person interviews, and there was more training
required and so forth.

On the other hand, surprisingly enough, our early findings seem to
suggest that we\'re getting better data from the telephone interviews
than from personal interviews, and we broached these subjects with the
reviewers, and they suggested none of this really surprised them.

Having controls over a localized interviewing process rather than a
nationally dispersed interviewing process, they could understand that
there might be some controls that you would get better information.
However, there was a concern expressed that, since we have at this point
editing built into our interviewing system via the Blaze System that we
are using and we did not have that in our previous survey in the 1995
survey, it\'s possible that if we had that type of system built in, we
could see an improvement in a personal interviewing setting, too, which
might help equalize the situation between personal interviewing and
telephone interviewing that way.

We talked about cost savings, and a couple of the members said, well, it
sounds like you\'re not going to save as much money as you might expect,
and it\'s possible that if you look at costs for an equivalent size
survey versus what our final costs will be that we might not save as
much as we might have hoped at first.

The comment was made that phone does have advantages, but in the
experience of a couple of our reviewers, cost is not necessarily one of
those advantages, at least not significant cost savings. So perhaps we
can\'t really expect that much of a difference in cost.

Another concern that was brought up in this area was where you may have
to treat property managers differently if we run into situations where
we have property managers that are responsible for a lot of buildings in
our sample. Certainly, we are going to have to take that into account as
we go through the field work, which by the way, we are right now in the
middle of the field work period for this survey.

Going to the next issue \-- this was the one that probably generated the
most controversy \-- we are considering as part of the comprehensive
redesign that we\'re going to be doing post-2000 census for the
commercial building survey to go to an establishment based listing that
we use to get to buildings as a subsequent stage rather than doing a
direct area sample based listing of buildings to produce the frame for
future CBECS surveys.

I would call the way that the committee members received that healthy
skepticism. Certainly, there may be some advantages to a list design. I
think it was Roy mentioned that, if you\'ve got addresses from a list
design, you might be able to send advance packets of information and
advance packets of things that you want people to get information about
easier than if you did an area list of buildings; because ordinarily,
when we do an area list of buildings, we note the existence of the
building, but we don\'t have any information about who or what is
inside.

So if you would try to send a packet to somebody, you would be sending
it blind; whereas, with an establishment list indeed you may have a
contact and you may be able to send something to somebody.

The main issue that they hashed over and were concerned about was the
whole business of linking of establishments to buildings and
establishing appropriate measures of size that allow you to do efficient
weighting for your national estimation.

Certainly, this is something that we\'ve thought about quite a bit and
was a subject of a research study that was done as to how one does link
buildings and establishments, if you\'re going to proceed from an
establishment listing.

There certainly are going to be problems with things such as enclosed or
strip malls, multi-building facilities which may share a common address
on an establishment list, and we are going to have to take all those
things into account.

One very useful suggestion is that, if we were to do an establishment
list in our redesigned post-2000, maybe what we ought to do is not think
about telephone interviewing the first time but go out into the field
and do personal interviewing, at least the first time, and try to get
\-- clarify our linkages between establishments and buildings as best we
can, and at the same time, of course, through a personal interview you
would get building information. You would get contact information as
well, and then be able to go out on the telephone from there on and
maybe realize some savings and benefits from the telephone approach in
later cycles.

One other thing that was mentioned: Because we are thinking, we know
that establishment lists will not cover the buildings population
completely because of various logistical issues, and we are going to
have to do some sort of research to determine adjustment factors to
ratio adjust our estimates to estimates for the total building
population.

One thing that Randy mentioned was, if you are going to do personal
interviewing, CAPI based on establishment lists, because if you\'re
going to do personal interviewing you\'re going to have to do some sort
of deep clustering because you don\'t want to be wandering all over
cities to do these interviews, there would probably be benefits to doing
the research that you have to do to develop your adjustment factors on
the same deeply clustered sampling units that you would be doing your
interviewing on.

So that certainly is something that we should take into account, if we
are going to think about the establishment approach.

Okay. The other two issues are very short. So it\'s not a big deal.

The third issue was an issue of separating out non-metropolitan counties
from metropolitan areas in doing the PSU design for our sample
post-2000. The reviewers were all unanimous about this.

If you want to do that, if you want to take away the two West Virginia
counties out of the Washington, D.C. primary sampling unit and put them
somewhere else or put them in a different stratum, go right ahead as
long as you know that they have no real relationship to energy use
relative to the other counties in the Washington area, etcetera. There
was unanimous agreement on that.

Finally, we asked about in the manufacturing survey the fact that we
went out with a linear question and answer questionnaire this last time
as opposed to a matrix form for our questionnaire. It looked a lot
longer, but it actually didn\'t have anymore information, and we had the
instructions along with the form rather than in a separate instruction
booklet.

The reviewers said don\'t worry about that. Don\'t worry about perceived
burden because of weight being an issue. Being easy to do is more
important than bulk, and it was suggested that studies that suggest that
maybe bulk is a concern and that perceived burden is a concern is sort
of muddy, and maybe this type of research that\'s gone on and suggested
this may have been on poorly designed long questionnaires. So you may
have confounding factors as to what is the real problem.

One final suggestion: In the MECS, the survey is passed from person to
person. It may be that a linear form is a good form of questionnaire if
you can separate out pieces of the form more easily, because one person
has to fill out one part of the form and another person has to fill out
another part of the form.

It was suggested, if you can tell a person you do pages 10-12 and
that\'s it, rather than having to separate out some sort of a complex
matrix for people to do, it would be easier for them to deal with.

So those were the suggestions. I think they are very good ones, and
certainly especially with regard to the suggestions for the upcoming
design. We\'re going to be concerned with this over the next couple of
years as we do the redesign. Thanks again, folks.

CHAIRMAN CRAWFORD: Okay. Now I\'d like to turn it over to Robert
Schnapp, and he will be talking about the redesign of electricity data
collections.

MR. SCHNAPP: Thank you. I appreciate you inviting me here to update you
again on our project to update our electric power forms.

I have given this presentation around town. You\'ll see at the end how
many different places we\'ve done it. This is typically an hour
presentation, and at Phil\'s request I cut it down to a half an hour.
We\'ll have to cut it down to 15 minutes now. So there will be some
slides that we\'ll kind of fly right through here.

MR. KENT: Don\'t cut it down too much.

MR. SCHNAPP: You\'ll have to listen fast.

I\'m going to start with FERC Order issuing orders 888 and 889 where
they essentially unbundle the transmission system from the rest of the
utility system, so that there is open access to the transmission lines
by any power producers.

Following that we have now determined that there are actually 25 \--
just found out yesterday the 25th state that has passed legislation or
regulatory orders to allow for retail competition.

In addition to that, Congress is considering legislation for electric
restructuring. In fact, they are holding hearings right now on the
Senate side. With all these changes going on, we felt we needed to take
a look at our data collection forms and figure out what new elements we
should be collecting and from whom, as well as what, if any, we should
be dropping.

What we\'ve seen over the short term because of all the changes that are
going on in the states and at the Federal level, we\'ve seen sales of
utility assets from the utility to the non-utility. I will tell you that
we have now \-- we are now collecting monthly the same information we
collect from the utilities.

So that would be electric generation, fuel consumption and stocks. That
began in 1999, and so automatically if any plant is sold by a utility to
a non-utility, they are automatically picked up on the non-utility form.
Following on that, all of our annual forms are hooked up the same way.
So nobody is falling through the cracks.

We\'ve also seen a lot more, obviously, more open access to the
transmission lines. We\'ve seen emergence of a lot more wholesale and
retail trade and less construction, though, particularly by the
utilities, because they are being required in some cases to give up, to
sell off their generation assets. In other cases, they are doing it on
their own, but we\'re still seeing them getting out of the generation
side. The non-utilities are picking up the slack there. In the long
term, we certainly expect there to be more competition at the retail,
wholesale and generation levels.

EIA historically has collected and published and modeled the utilities
and non-utilities as separate entities, and we\'re hoping to be able to
continue that information during the transition as the industry evolves.

What we think we are going to be moving to is to doing away with those
terms utility and non-utility, so that we would probably be sending
forms to generators and distributors and transmitters or energy source
providers. So that we will be doing away with those terms. In talking
with the many constituents out there, everybody pretty much agrees on us
doing that.

Some of the areas that we\'re looking at more closely to figure out what
to do are in the transmission, reliability, and markets area. We\'ll
talk a little bit about where we think we\'re going at the end of this.

So we\'ve put together a phased approach for addressing all the issues,
and it accommodates the fact that the states are moving toward retail
competition at different rates. Even if a state says they are going to
be retail competition, it may be phased in over time, and all the
different states are going at different speeds and even right now we
only have half of them that say they are going to g to retail
competition. So there\'s a lot of things that we have to put into the
mix.

During that time we are also going to be evaluating what data we should
hold confidential, data that\'s sensitive, commercially sensitive.

So we\'ve put together an integrated plan and schedule that lays out all
of our analytical activities, meetings, trips, with all the industry
groups and Federal/state/local governments and all the respondents; and
we\'ve put together all the costs and technical and schedule milestones.

We\'ve done that in Microsoft Project, which we don\'t really need to
spend much time on here.

What we have done is we \-- This is the phased approach that we have.
The first phase, Phase 1, is to develop our data requirements. What we
did first was we held 11 focus groups with about 165 people. That
represented a whole variety of stakeholders.

So it\'s just not utilities and non-utilities, but municipals,
cooperatives, investment bankers, the media, Congress. We went to as
many different people as we could, particularly those that wouldn\'t
typically respond to a [Federal]{.underline} [Register]{.underline}
notice, if we sent that out.

From that information we developed eight categories of data that we felt
we needed to look at. So these are the eight categories. Certainly,
generation and transmission \-- this is all in the power supply area.
Then peripheral to that would be the environment, reliability and
finance. In the markets area, wholesale and retail; and on the
consumption side, the consumer services.

So what we\'ve done internally is we assigned two or three people to
each of these areas, and we have a group of six people. So everybody is
on at least two or three of these groups, bringing different specialties
to the areas.

We have three analysis people and three data people. So we\'re really
mixing up all of the different specialties that we have.

So groups have gone off and done analysis from information that we
received along with the information from the focus groups, and we\'ve
come up with a conceptual design which is helping us work through our
data requirements. We\'ll get to that conceptual design in a minute.

What we\'re doing now is we are coordinating with all of our
stakeholders and going around doing this presentation to make them aware
of where we\'re going with this. We then plan to come back to them after
this phase is done in June and share it with them again in an informal
manner and get comments from them before we go out with a formal
[Federal]{.underline} [Register]{.underline} notice to everybody.

In our conceptual design we are not putting together just what we\'re
calling primary collections \-- in other words, information we\'re
collecting off of forms that we send out, but there are also secondary
collections, those that we might access through Internet sites, if we go
to ISOs or companies, RTOs, whatever might be the case.

So our goal for this Phase 1 is to come up with what we call a data
dictionary. That essentially is a list of data elements we want to
collect, the definition, who we would collect it from, and how often.

Now in this data dictionary there would be a lot more information for us
to use internally. It would include things like the publication that
would go into the form that it\'s going to be on, the table it\'s going
to be published in, where on the Internet it might be, as well as the
edits that would be done to check it.

All of that together being the data dictionary really is key to
everything else we\'re doing. It will certainly feed into our survey
instruments. It also feeds into the common collection and processing
system which is a system that EIA is building to collect all of its
energy data.

This data dictionary really is the requirements document that is just
handed over to the programmers. That\'s everything they need in one
book, but also would feed into all of our Internet products as well as
our hard copy publications.

So we\'re expecting to have that done in June of this year.

Moving on to Phase 2, which is the forms design, we would design our new
survey forms. As I said, we would probably be going just to generators
or transmission operators.

We think that we may \-- Instead of having the approximately 12 forms we
have now, we may redesign them so that we only have two or three forms,
but depending on how big you are or what area you\'re in, you might fill
out schedule A, B and D or pieces of A, B and D. It kind of depends on
where we\'re going with that. We haven\'t made any firm decisions on
that.

As far as the frame is concerned, we\'re fairly certain we know who is
out there, who we have to collect from. It\'s not too difficult for us
to identify the new players, the marketers, out there; because they have
to register with the states.

So particularly in California and Pennsylvania, who are the two prime
states that have retail competition now, and just going to their
Internet sites, they list all of the new competitors that have to
register. So that helped us a lot during the year in collecting our
survey data.

Once we have designed the forms, we\'ve already gotten commitments from
some of the trade associations to help pilot test the forms to make sure
that they are going to work. We do want to make sure about that before
we start sending them out.

Then in this time frame, we would also come up with our new policy on
confidentiality for the data, and our target is to go to the Office of
Management and Budget for approval in August of 2001.

Phase 3 is collection and processing, which is coming up with a typical
hard copy form. We are, as I said, redesigning the forms. We expect them
to look very similar. So that right now we have about 12 forms. It looks
like they come from 12 different companies. They have different ways of
describing the same thing, and we\'re trying to get away from all that.

We are also making a special effort at electronic data collection. We
certainly get information now by FAX. We are going to be looking into
E-mail. We do get one form returned to us on diskettes.

We are looking at Internet collection right now in two different forms
that \-- two different methods that we are looking at, and over the next
few months for our monthly forms we are going to be trying those out.

As I said before, we have our CCAPs. That is the data processing system
in our schedule here. We have a checkpoint for August of this year. I
understand there are some issues that have come up in the coal and some
of the other areas on the system. We need to have a definite answer on
whether our electric forms are going to be going into that system by
August. Otherwise, there\'s a lot of other planning and work that we
have to do on our side.

So the goal for Phase 3 is to be complete by January of 2002, which is
when we would begin to collect the new data. So that means annual data
for 2001 and monthly data for 2002.

The fourth phase \-- I might add, though, that these phases are not
waiting for each other. We are doing work in all of these phases at the
same time. It just kind of depends on how far along they are.

Phase 4, dissemination, and we don\'t call that hard copy publications
anymore, because we have such an emphasis on the Internet as well as a
lot of our hard copy publications will probably be going away.

So we are \-- Back in Phase 1 with our data elements being defined, we
are asking our staff as they develop an element to think about what the
table is going to look like that that element will appear in. So that
we\'re thinking kind along the whole spectrum here as we\'re moving
along.

Then we need to, as I said, think a lot about how we\'re going to
displaying our information on the Internet, just to make it more useful.
EIA certainly has a very large presence on the Internet now, and they
are redesigning the homepage, but we\'re always trying to improve things
as we move along.

So our goal on this is for September 2002 to have this all done. That\'s
when we expect to have all of our annual data ready for publication.

I\'m just going to fly through these. So our conceptual design has been
put together, so that we are first describing what the present
configuration has been, and then overlay on that what we think the
future direction will be. From this, we are able to define what our
future data requirements are going to be.

So if you take out distributed generation resources for the moment, this
really is the vertically integrated monopoly. We have the company
producing electricity, putting it on the transmission lines, then over
the distribution lines and then to the end user, with imports and
exports moving in and out of the region.

We do see in the future, though, distributed generation resources coming
on more, and there\'s a question as to how and if they can put their
extra electricity onto the transmission system. There certainly is a
question on that.

What I\'m going to describe here now are not new responsibilities that
have to be taken over inside of the electric power industry. It\'s just
they are being broken apart from where they were before.

So what we have being established are our system operators. In some
places they\'re calling them independent system operators. The Federal
Energy Regulatory Commission wants to start talking them regional
transmission operators, and there are all kinds of other names that are
being floated around. So we don\'t know what they\'re going to look like
or be called, but we\'re calling them system operators at the moment.

They would be the ones in charge of the transmission system, since it
wouldn\'t be operated by the utilities anymore. So they need to
communicate and coordinate with the transmission system, with the
generators. If there is distributed generation resources that come onto
this system, they will have to coordinate with them, as well as with the
distribution system.

The other side would be the energy service providers. Those are the
folks that would be calling you up about six o\'clock at night telling
you they have the best electricity that\'s ever been made, and they have
the best price that you can find; and they will be, obviously, talking
with you, the end user.

They will have to be communicating with the generators as well as the
system operators and the distribution system, and probably with those
that are going to be importing or exporting electricity into or out of
that system, and then possibly with the distributed generation resource
as well.

So there\'s a lot more communication with other entities that didn\'t
exist before that are coming into the marketplace.

Behind actually \-- So this is our design for the supply system. We\'ll
get to the market system in just a second. But what we have done is
behind each one of these boxes, we drill down and there\'s another set.
There\'s another conceptual design of just that area.

So that we drill down further and further until we decide exactly what
the elements are that we need to collect from each one of those
entities.

So on the regional electric markets we will have the \-- We certainly
have the wholesale markets and the retail markets that are becoming more
and more prevalent on their own, and we think that there is going to be
more information that will need to be collected from the wholesale
customers.

We have the suppliers that we were talking about before, and there\'s
going to have to be, certainly, a lot of communication between them and
the wholesale marketers and the retail marketers. Again, we drill down
in behind each one of these elements as well, and then information on
the end user.

So let me kind of quickly then jump on to the areas that we are thinking
right now about changing. This is not all of them. No decisions have
been made, but they are areas that we are thinking about, we thought we
would share with you at the moment.

First is on costs, quality and quantity of fuels. For those of you who
are familiar with it, the Federal Energy Regulatory Commission has a
FERC 423 which collects this information from the utilities. if you
compare, say, January 2000 data with January 1999 data, it will look
like the electric utility industry is going away.

Well, it\'s not that it\'s going away. It\'s that the assets are being
sold off to the non-utilities. So they are dropping out of the frame. So
that data collection system is becoming kind of obsolete at this point.

So what we are thinking about doing is we\'re considering instituting a
423 for everybody else; in other words, for all the non-utilities, but
we would do it at a more aggregated level.

For the next item for net generation, consumption and stock data, I said
before that we are now collecting a monthly non-utility form that looks
the same as the utility form. That\'s the 900 non- utility and the 759
utility form.

We kept them separate, because we didn\'t know how the non-utilities
were going to respond to it. They didn\'t want to provide the same level
of information several years ago, and so we kind of built them back up
to what we wanted, and they have cooperated really fully with us on
this.

So we were expecting just to combine the two forms into one and, as I
was saying, not make that separation anymore, that distinction anymore.

Environmental data we collect on the EIA 767, and we are funded
partially by the Environmental Protection Agency on that. So we are
working with them on their needs. We\'ve been working very closely with
the modelers here in the Office of Integrated Analysis and Forecasting,
as well as the Policy Office on their modeling needs in the
environmental area.

EPA would like to expand it, going to non-utilities, getting information
from gas turbines and even nuclear plants.

Transmission side: We are thinking about collecting physical asset
information for the most part. That would be sizes, locations of
facilities, lines, substations. We do think we may have to collect
information on the cost of new construction of these assets, though.

Markets area: Because of the unbundling of generation and transmission
and retail competition, retail sales, it\'s making it more difficult for
us to collect the cost to the end user for their bill. We have been
going to the utility who \-- It\'s easy for them in the past to say,
well, you know, we\'ve been charging them this amount, and so that ends
that. But with that being broken apart now, we have decided to try and
break it into three schedules.

So if you\'re an integrated utility, you fill out the same form
essentially you did before. If you\'re an energy service provider, you
fill out a separate schedule. It looks the same way but only provide
your piece of the information, and the same thing if you are a wires
company, only your piece of the information.

So we think if you then add that altogether for each state, we\'ll have
the answer. We have worked with the Edison Electric Institute on this.
We met with them several times. We\'ve gone out and talked to a couple
of utilities, and they think that it will work. We still have our
fingers crossed on this, but we do need to test it out very thoroughly.

In the reliability area, we have two forms. There\'s the 411 which the
North American Electric Reliability Council collects for us, and that
provides us information on the adequacy. Then there is the 417R which
collects information on reliability. It tells us when there are outages.

So we are working with the DOE\'s Office of Emergency Operations to
determine with them what exactly is needed. Right now the 417R, we
think, is fairly useless. It requires a utility to provide a form within
24 hours and give us an analysis of what happened. That\'s just not
possible, and believe me, it\'s not the first thing on their list that
they have to do.

So we\'re trying to figure out what it is that is required, and we\'re
working with Emergency Operations now.

On the 411 on adequacy, we\'ll be meeting more with North American
Electric Liability Council to figure out what it is that needs to be
collected there. We haven\'t met with them yet on this, but we will be
shortly.

Then the last area, which is data confidentiality, we have been directed
in our last OMB submission on the last forms clearance, OMB requires us,
the next time we come through to coordinate with FERC and with the Rural
Utility Service, the three of us each collect a different piece of
financial information from the electric power industry.

RUS collects it from the cooperatives. The FERC collects it from the
investor owned, and we collect it from everybody else. So they want to
make sure we have a, quote, \"consistent policy.\"

We\'ve already had preliminary meetings with FERC and with RUS. We have
agreed that we will work at that, and we also have agreed that a
consistent policy may not mean the same policy, because there may be
differences for different reasons, but we are going to work together to
come up with a consistent policy. That will have to wait, though, until
we define our data elements. So that will be sometime after June.

Just to let you know, the data confidentiality policy was changed about
15 months ago, and the areas that right now we hold confidential are
consistent between utilities and non-utilities. So that is all future
changes to capacity are held confidential. So that\'s retirement and new
additions.

Heat rates are confidential. Fuel inventory stocks are confidential, and
then the names of the sales to end users \-- the end user sales name is
held confidential. That\'s just for non-utilities. They are the only
ones that have to provide that to us. That would kind of give out who
their customers are, and that really isn\'t fair. So that one just
applies to them.

So this really is very different than what we had before where most, if
not all, of the non-utility data was held confidential, and all of the
utility data was in the public domain.

So we\'re trying to, more or less, level the playing field and treat
everybody equally, but we\'re going to have to work our way through this
and look at how competition is affecting the industry, where there is
competition.

It may require then some areas will give out more information than in
others, but we\'re just going to have to work our way through it.

So this is a list of all of the stakeholders that we\'ve met with so
far. So you can see how many times I\'ve given this presentation. You
can also add to that the Office of Management and Budget who we met with
last week to bring them up to speed on where we are, so they know
what\'s going on. We don\'t like to surprise them with our forms.

So I think that that completes the presentation. I\'ll be glad to answer
any questions. No questions? Okay. Was that slow enough?

MR. KENT: I will begin, obviously, by noting two things on changes that
took place from the slides that you sent out to us and the slides that
you included.

The first one was on the slide \-- these aren\'t numbered \-- where you
were talking about your Phase 2 forms design. You put in confidentiality
evaluation rather than evaluation of sensitive information, and then on
the next to the last slide, your last bullet there, you change from
sensitive data to data confidentiality.

I don\'t think you were around for this morning\'s session, but this was
obviously a hobby horse that I love to ride. So I would like for you to
discuss more about what you\'re doing in this confidentiality area and
if you anticipate problems.

I\'ll tell you where I\'m coming from in advance, and that is I suspect
that there is going to be some degree of difficulty in your getting the
data that you want in unregulated markets, if you don\'t have a strong
protection of confidentiality. Yet you are not covered by the new
proposed legislation, as is the End Use Office.

So I\'d just like to have you elaborate some more on what
confidentiality problems you anticipate facing and how you are going to
be dealing with them.

MR. SCHNAPP: Well, we\'ve changed the term to sensitive data, because we
found out from DOE\'s security consciousness that we don\'t have
confidential data. We have sensitive data. We have commercially
sensitive data.

We\'ve just been calling it confidentiality, because we\'re trying to
keep it confidential from the public. So that kind of clarifies that
point. But we have already met with \-- We have people out on both sides
of the issue that are very strong, people that feel that we should have
everything in the public domain to make sure that we guard the
environment and the consumer, and then we have people on the other side
who say, well, if I\'m in competition, why are you treating me
differently than, say, a petroleum company where most of that
information is held confidential.

So it\'s a very difficult line that we\'re going to be walking at this
point. We have been told by folks like the Energy Power Supply
Association, which represents the independent power producers, that they
know that they have to provide us with more information.

They\'re quite aware of that, and they will cooperate with us. However,
they say that that information is confidential. It is sensitive
commercially. I believe that the Edison Electric Institute \-- I won\'t
speak for Linda here, but I believe that they feel that most of that
information should be held confidential as well where they are in
competition.

It\'s our job to figure out where those areas are, because not
everything in the industry is going to be in competition, and not every
state is going to have retail competition.

So I can\'t tell you the answer right now where we\'re going to come
out, but those are some of the issues that we\'re going to have to
address over the next about 12 months.

MR. KENT: But as you move into these markets where you\'ve got a lot
more deregulation and a lot more players, the question is your ability
to actually get the reliable data that you have, if there is some
concern about not just loss of competitive situation or competitive
status but also that perhaps the use of this for regulatory or
prosecutory purposes.

MR. SCHNAPP: Well, we\'re just going to have to see where things come
out. I mean, I can\'t tell you where we\'re going to be or where the
industry is going to be 12-15 months from now. I\'m not really sure how
to answer the question anymore than that. We\'re going to have to see.

If we deem there to be competition, we can figure out in our minds,
along with everybody else, if there\'s competition in this area. Then I
would say that it strongly tells us that we probably shouldn\'t make
that information available to the public.

Now as far as sharing with other \-- you know, with regulatory agencies,
as far as I can tell, we\'re required to, you know, if it\'s asked for.
Our forms right now say we have to share it with them.

I know that we do work with them to try and avoid giving it to them and,
if they want some analysis done, we might do it for them. That\'s been
offered in the past, do some other kinds of aggregations to make sure
that things are withheld. I believe that we have shared with other
agencies with them saying that they are only going to be using it for
statistical purposes, and I wouldn\'t think that EIA would have a
problem with doing that, if the highest levels of that agency signed
those documents.

MR. FENNEL: We are also working with the trade associations. They were
very upbeat about it and said that they would be happy to see us collect
the data, and their concern was in how we were going to sum the data.

We are working with them. In fact, we have a meeting scheduled for
probably \-- I think it\'s the first part of next month, to discuss the
ongoing data confidentiality and sensitivity of the data.

I just returned from a trip last month where we met with a non-utility,
and in the beginning of the meeting, which did not go very well, they
were very concerned about what we were doing, the data and how we were
going to handle it. By the time we got through it, they were more
relaxed about it, and they were actually asking questions about the
forms and how they could submit the data to us on a routine basis.

So I think we are making a lot of progress there. As Bob said, it\'s
hard to tell where it\'s all going to fall out, but we are making
progress, and we have begun the dialogue.

MR. KENT: Let me run a follow-up on that.

In the past you had a lot of statutory authority. Do you have sufficient
statutory authority on these collection items?

MR. SCHNAPP: Well, the FEA Act has always said that the Administrator
can collect information on anything related to energy, and they\'re
required to submit it.

By the way, this is Dean Fennel. Dean is the Project Manager on this
particular project. I forgot to introduce him before.

CHAIRMAN CRAWFORD: David?

MR. MONTGOMERY: A little more both about the kind of data you\'re going
after and the confidentiality.

Could you say a little bit more about what you\'re planning on
collecting on the cost and quality of fossil fuels? Will that continue
to be providing basically prices of fossil fuels?

MR. SCHNAPP: Well, what we\'re thinking about is, you know, the FERC 423
asks you how much coal and the price and the quality of it that you got
from that mine in Montana, and we\'re thinking more along the lines of
how much coal and what was the price and what was the quality of coal
that you got from Montana, as opposed to from that mine.

So a more aggregated level is what we\'re thinking about right now.

MR. MONTGOMERY: Would you be tying these to deliveries to particular
power plants, the way the 714 did?

MR. SCHNAPP: It would go to all generators, to all generators that
don\'t provide FERC 423.

MR. MONTGOMERY: I\'m sorry. When you ask a particular generator, would
the respondent be a company which might have power plants in a number of
the states, or would the respondent be describing the cost of fuels for
a particular unit?

MR. SCHNAPP: It would be on a generator basis.

MR. MONTGOMERY: That, I assume, you would be keeping confidential, too.

MR. SCHNAPP: Again, we have to evaluate it, but I know that price data
is a touchy subject. And as we get into more competition, saying that
price \-- that the cost things \-- that is much more touchy, and we\'re
probably leaning toward holding that confidential.

You know, the price you sell something at is something different than
what you paid for it.

MR. MONTGOMERY: Where I\'m getting with this is that there\'s kind of
procedural requirement that I wonder how it\'s going to be satisfied,
which is FERC\'s merger policy, Appendix A. The way it\'s been
interpreted up until now, which not everybody agrees is the right way to
do things, but it\'s very clear it\'s how FERC does do things, is to say
you have to calculate Kirkendall indices for all of the markets that
might be affected by any merger that comes before them.

It\'s got to have precedence over the merger. The way everybody has been
doing that so far is basically taking the \-- and they want to look at
measures of economic capacity. So, basically, the way everybody does
this is they take the FERC Form 1 data, the EIA 423 data, and the last
714 data you made available in order to calculate the variable cost of
operating each power plant, which is basically the heat rate times the
price of fuels.

It\'s no problem for the merging parties, because they, of course, can
provide the information; but this is supposed to be a market-wide
calculation, and right now FERC expects the applicants to do it.

Have you given any thought to how this process is going to handled in
the future?

MR. SCHNAPP: No, but I expect that FERC will.

MR. MONTGOMERY: Not necessarily.

MR. SCHNAPP: Well, no, I think they are. We\'ve been invited to come
over there and talk to them about our project, because they are going to
begin the same project, looking at all their forms and seeing where they
should be going with it.

We\'ve talked with Dan Larcamp. We talked with him about six months ago,
and he has said this was going to happen, and now they have contacted
us. So I think that they are moving forward to look at all their forms.

He\'s not sure, should they be collecting all this information, if
competition is coming in.

MR. MONTGOMERY: Right.

MR. SCHNAPP: He\'s not making a commitment any way, but he says we need
to look at it. So I can\'t answer your question as to what they are
going to require. I really don\'t know.

CHAIRMAN CRAWFORD: Are there any other questions or comments from the
committee? From the audience, any questions or comments?

Any questions from the public? I don\'t know, are there any public
people here?

Well, if there are no further questions or comments, I\'ll go ahead and
adjourn the meeting for today.

Remember that dinner reservations are at six o\'clock.

(Whereupon, the foregoing matter went off the record at 4:21 p.m.)