AMERICAN STATISTICAL ASSOCIATION

COMMITTEE ON ENERGY STATISTICS

\+ + + + +

MEETING

\+ + + + +

THURSDAY

APRIL 19, 2001

The committee met in the Department of Energy Training Facility, 8th
Floor, 950 L\'Enfant Plaza, S.W., Washington, D.C., at 8:30 a.m., Carol
A. Gotway Crawford, Ph.D., Chair, presiding.

PRESENT:

CAROL A. GOTWAY CRAWFORD, Ph.D., Chair

F. JAY BREIDT, Ph.D., Vice Chair

MARK BERNSTEIN, Ph.D.

JOHNNY BLAIR

MARK BURTON, Ph.D.

THOMAS G. COWING, Ph.D.

JAMES K. HAMMITT, Ph.D.

NICOLAS HENGARTNER

W. DAVID MONTGOMERY, Ph.D.

WILLIAM G. MOSS, Ph.D.

POLLY A. PHIPPS, Ph.D.

RANDY R. SITTER, Ph.D.

ROY WHITMORE, Ph.D.

[C-O-N-T-E-N-T-S]{.underline}

Welcome 3

Introductions 4

Opening remarks 7

Progress on EIA/ASA Fellowship 21

Briefing: Progress on MARKAL 28

Questions from Committees 41

Briefing: Analysis of Strategies for Reducing

Multiple Emissions from Power Plants 45

Summaries of Breakout Sessions 71

Update and Results of Cognitive Testing of

EIA Graphics 90

Summaries of Breakout Sessions 124

Adjournment 137

[P-R-O-C-E-E-D-I-N-G-S]{.underline}

(8:35 a.m.)

CHAIRPERSON CRAWFORD: Welcome to the ASA Committee Meeting on Energy
Statistics. This meeting is being held under the provision of the
Federal Advisory Committee Act. This is an ASA, not an EIA Committee,
which periodically provides advice to EIA.

The meeting is open to the public and public comments are welcome. Time
will be set aside for comments at the end of each morning and afternoon
session. Written comments are welcome and may be sent to ASA or EIA.

All attendees, including guests and EIA employees should sign the
register in the main hall. We are also asking for your e-mail address.

Restrooms and water fountains are behind the elevator shafts in the
lobby on this floor, and local calls can be made from within the
conference center.

We\'ve labeled the rooms for the breakout sessions Red, White, and Blue
beginning with the room we are in now as Red, White\'s the one in the
middle, and Blue is the one at the end of the hall.

Linda Minor from ASA is here today. Hopefully you\'ve had a chance to
meet her. She has blind forms for your expense reports if you did not
get these via email.

The conference center here may be reached at, and I have the number if
you want to jot this down, 202-287-1622, should you need to be reached
by telephone.

In commenting, each participant is asked to speak into a microphone, and
you can see the microphones are those little round things on the table.
The transcriber will appreciate it and he may often ask you to speak up
if he can\'t hear you.

Also, Committee members and speakers at the head table need to speak
clearly into a microphone. Speakers are asked to use a lapel microphone
or to hold it like I\'m doing if you don\'t really have the lapel, and
Bill can help you if you need it.

I\'d like to introduce four new members and their affiliations. First is
Mark Bernstein from the RAND Corporation. He\'s been a member for six
months or so, but this is his first meeting and Mark is sitting right
here.

Johnny Blair, who was a previous guest, but is now actually a formal
Committee member, I don\'t think he\'s here yet, but he\'ll be here
shortly.

Mark Burton from Marshall University, this is his first meeting, and he
was recently appointed at the beginning of the year.

And Nicolas Hengartner, who was a guest last time from Yale University,
who is now a new ASA Committee member.

Now I\'d like to have each of us introduce ourselves. I\'ll start with
the Committee and then go to EIA people and any others. I\'ll start.

I\'m Carol Gotway Crawford. I\'m with the National Center for
Environment Health. That\'s part of the Centers for Disease Control in
Atlanta, Georgia.

DR. COWING: Tom Cowing, Binghamton University.

DR. HAMMITT: Jim Hammitt, Harvard.

DR. BERSTEIN: Mark Berstein, RAND.

DR. BURTON: Mark Burton, Marshall University.

DR. MOSS: Bill Moss, the Brattle Group.

DR. WHITMORE: Roy Whitmore, Research Triangle Institute.

DR. HENGARTNER: Nicolas Hengartner, Yale University.

DR. BREIDT: Jay Breidt, Colorado State University.

DR. SITTER: Randy Sitter, Simon Fraser University.

DR. PHIPPS: Polly Phipps, Washington State Institute for Public Policy.

MR. PETTIS: Larry Pettis. I\'m the Acting Administrator of the EIA.

MS. KIRKENDALL: Nancy Kirkendall. I\'m Acting Director of the Statistics
and Methods Group.

MR. WEINIG: Bill Weinig, Statistics and Methods Group.

MS. HUTZLER: Mary Hutzler, Director of the Office of \--

CHAIRPERSON CRAWFORD: Could you speak into your microphone?

MS. HUTZLER: Mary Hutzler, Director of the Office of Integrated Analysis
and Forecasting, Energy Information Administration.

MR. FAUZI: \-- Fauzi, Office of International Integrated Office, DOE.

MR. RUTCHIK: Bob Rutchik, Statistics and Methods Group.

MR. SITZER: Scott Sitzer, Office Integrated Analysis and Forecasting,
EIA.

MR. COHN: Larry Cappello Cohn, Office of Integrated Analysis and
Forecasting, EIA.

MR. BROWARD: Tom Broward, Statistics and Methods Group, EIA.

MR. McDOWNEY: Preston McDowney, Statistics and Methods Group, EIA.

MR. FREIDMAN: Stan Freidman, Statistics and Methods Group, EIA.

MS. MILLER: Renee Miller, EIA.

MR. WATSON: Bill Watson, EIA.

CHAIRPERSON CRAWFORD: Lunch for the Committee and invited guests will be
at 12:30 in Room 6097, which is on the 6th floor. We\'ll take the
elevators to the 6th floor and there\'ll be signs, and we\'ll resume
after lunch in this room.

Dinner reservations this evening are at La Brasserie on Capitol Hill and
the address is 239 Massachusetts Avenue, Northeast. We will meet there
at 5:30 and eat about 6:15.

For those of us who are staying at the Holiday Inn Capitol, we\'ll have
to see how our time is. We may want to leave from here, or we may choose
to go back there and then leave together. Bill suggests taking a cab
there.

At the dinner tonight we will talk a little bit about the invited
session for the joint statistical meetings and EIA has asked us to
select four winners for their annual graphics contest, and that\'s
always a lot of fun. So I hope you can join us for dinner.

Breakfast for the Committee will be here again tomorrow beginning at
7:45, and our meeting will resume at 8:30 here in this room.

For your information, Nancy Kirkendall is the Designated Federal Office
for the Advisory Committee. In this capacity, Dr. Kirkendall may chair
but must attend each meeting, and she\'s authorized to adjourn the
meeting if she determines this to be in the public\'s best interest.

She must approve all meetings of the Advisory Committee and every
agenda, and she may designate a substitute in her absence.

I think that\'s all. Does anyone have any questions? Otherwise, I\'d
like to recognize and turn the meeting over to Larry Pettis who\'s the
Acting Administrator for EIA.

MR. PETTIS: Good morning. I\'d like to join Carol in welcoming the new
members and we appreciate you taking your time to participate in this
Committee and assist the EIA in some of the work we\'re doing.

There\'s really a lot that has been going on in the EIA since we met
with the Committee last time and I thought today I\'d give you just a
little bit of an overview of some of the things we\'re doing. Then some
of the sessions today are going to go into those in a little bit more
detail.

I think some of you might be interested with the departure of the
Clinton Administration, Mark Mazure left the EIA and he is now the
Director of Research, Analysis and Statistics of Income at IRS, and Mark
continues to drop by and see us periodically. So if any of you want to
look him up, that\'s where he is.

There\'s two primary factors that are placing some heavy demands on EIA
at the present time. First, energy is certainly back in the news. I
don\'t think we\'ve had a period of time where you see the president and
the vice president and the secretary of Energy all being quoted in the
news on a weekly basis about what\'s going on in energy.

A couple of things that are getting a lot of our attention are the
tightness in both the petroleum and natural gas markets and the high
prices that we\'re seeing there, and also the electricity situation in
California.

The second factor is really the transition to a new administration and a
new Congress, and both of these groups are interested in addressing the
energy issue, so they\'re placing some heavy demands on us at the
current time.

One of the things that we\'ve been asked to do is a number of briefings
on the Hill for new members and staff that are coming into the Congress
to bring them up speed on some of the energy issues. And just to show
you a little bit about what the interest level was, we held more
briefing on the California electricity situation, and typically we\'d
expect to go over there and get maybe 20 staff members to show up. Well,
there were about 100 staff members, standing room only. So there\'s a
tremendous amount of interest in what\'s going on.

We are also doing a number of briefings for the new administration. We
call this the Mary Hutzler Annual Energy Outlook Traveling Brothers
Show. We\'ve had a briefing of the Secretary of Energy. This might be a
first for you. I thing Mary briefed the Vice President directly on the
Annual Energy Outlook, a number of other key policy advisors to the
President and the Vice President, and she also had a briefing for the
Vice President\'s Task Force on Energy which includes a number of
cabinet secretaries in government.

I think most of you probably know that the Vice President does have a
Task Force on Energy, and they are working on developing a new national
energy policy plan. The EIA has been involved in this to a certain
degree. We did some of the foundation materials that really address the
current energy situation and what our outlook in the future is, and it
sort of formed the baseline information that\'s used in this report.

I think the current schedule is that they expect to complete this report
sometime within the next month and we expect that we will probably be
asked to analyze a number of policy proposals that are included in that
report.

Congress is also holding numerous hearings on energy. Again, this may be
a record for us. Over the last two months the EIA has testified in eight
hearings on the bill. Most of these hearings have been before the House
and Senate Energy Committees.

The House committee, for example, is holding a hearing to buy fuel on
the energy side. That\'s a pretty heavy demand to continue doing these
hearings. But also the other committees that ordinarily don\'t have
jurisdiction over their energy seem to be finding reasons to hold
hearings on energy also, if they can find some small reason to have one.
So there have been many of those. We have a couple more coming up here
in the next week or so.

Just out of curiosity, the appointment process in this new
administration in the area is moving along rather slowly. Secretary
Spencer Abraham is the only confirmed political appointee that we have
in the DOE at the moment.

There are six other nominees that have been announced by the White
House, but none of those have gone though the confirmation hearing yet,
and I\'m not sure that we know exactly what the schedule is.

Everyone is most interested in who the next EIA administrator is going
to be. I don\'t know. So if you do, let me know.

We\'ve either completed or are working on a number of activities from
the Congress and analytical reports. Some of you probably have seen \--
and this is going to be one of the topics in the meeting of this week
\-- is a report that we did on strategies for reducing multi-emissions
in electric power plants.

The first report addressed SOx, NOx, and CO2, and there\'s going to be a
follow-on report that addresses mercury and renewable portfolio
standards. We are also working on a study of the new low-sulfur diesel
regulations which we expect to complete by the end of May.

We also have implemented a survey on interruptible natural gas tracks
which was requested by Congress, and I think there may be some
discussion of that also during the meeting this week.

I think again, some of the things that sort of indicate the high level
of interest in energy, earlier this month we had our National Energy
Modeling System conference. We got over 600 registrants for that
conference and about 500 people that actually showed up. And we had a
lot of good topics on current issues, and so I think it was a good
opportunity for us to have good discussion with people outside about
what\'s going on.

Our EIA web site continues to be a great success, over 5 million unique
users on the web site last year. And it continues to grow at a rather
rapid pace. We can sort of see the power of the web every day when you
put a little report or speech that somebody\'s done on one of the
current energy issues and it\'s picked up by all the wire services
almost immediately. So it\'s really proved to be a valuable tool for us.

We\'re also getting really heavy inquiries from the press about energy.
You can\'t hardly pick up a paper without it being on the front page, so
there are a lot of calls to our organization from the press.

I thought I would mention briefly about our budget. The EIA\'s budget
request for FY 2002 is \$75.5 million, which is the same as our request
in FY 2001. The net effect of that to us is really about a 4 percent
reduction in funds for programs because we have to absorb these
nondiscretionary costs like pay raises and overhead increases.

The major focus of our budget this year is on data quality. We really
feel like it\'s important to continue a number of these efforts that we
started in recent years. One to address the changes and surveys related
to the electricity and natural gas due to the restructuring of those
industries.

I know this is one of the topics of the session today, but our new
electricity survey forms and confidentiality policy is currently out for
public comment and our plans are to implement these forms starting in
2002.

There\'s also been a new monthly natural gas marketers survey that is in
the clearance process that will be used in states to collect residential
and commercial information in states where there\'s retail choice and
our current surveys aren\'t able to pick that up.

The other thing that we\'re continuing is our redesign of our commercial
building energy consumption surveys and residential energy consumption
surveys based on the 2000 census, and that\'s an ongoing activity.

We\'re also trying to address the petroleum data quality issues related
to mergers and consolidations and then we also had a number of revisions
related to the heat fuel standards that will be coming in the course of
the next few years.

To keep our focus on these high-priority activities, EIA has proposed to
make a number of cuts in other areas. Unfortunately, we have this nice
presentation scheduled on the International Modeling Activity in this
meeting today and that is probably the largest that we proposed taking
in our budget.

We have also made some proposals to eliminate some publications like
Extra Energy Data System because we really don\'t have the resources to
maintain those on a current basis and also keep some of our more
widely-used publications like our Monthly Energy and Annual Energy going
so we really feel like we need to focus our resources in those areas and
keep those top-quality products.

I guess the last thing I was going to say is this is going to be my last
meeting at the ASA. I\'m going on retirement at the end of May and I
guess I have some mixed feelings about that. I hate to leave while
everybody\'s having so much fun, but I also feel very comfortable that
we\'ve got some very talented and capable people here in the EIA and
things are going to move along quite smoothly.

I sort of had a chance to reflect over some of my time in the EIA and
working here over the last few years, and the thing that really jumps
out is what a great opportunity I\'ve had to work with some great
people. And certainly I would include this committee within that group
and I would like to take this chance to thank you all for all the work
you\'ve done in supporting the EIA and I know that will continue in the
future.

So that kind of is my summary. I\'d be glad to answer any questions if
anybody has any. Yes, your gasoline prices are going to be higher this
summer.

CHAIRPERSON CRAWFORD: How high will they go?

MR. PETTIS: Good question. I think there is a lot of concern about that
right now. Our current forecast is that average national gasoline prices
over the summer are going to be about what they were last year, near
\$1.50, but if you look at our most recent survey, they are \$1.56
nationwide right now.

We\'re kind of in a very tenuous situation right now. There\'s been this
slowness coming out of the refinery maintenance period. There\'s some
things in the market right now. The spreads out there are so large that
there\'s a lot of incentive to produce gasoline. So we really think that
over the next few weeks you\'re going to see increases in production and
imports and then prices will settle down a little.

Any other questions?

DR. COWING: One, Larry. Your comment about the reduction in EIA\'s
effort on world energy modeling, is that a reduction in effort or total
elimination of the project?

MR. PETTIS: Well, I think maybe when Mary and her people talk about
that, they can get into that a little bit. I think that we will be
continuing some effort this year where we will \-- if memory serves me
right, Mary will end up with some regional models out of this. But I
think the integration activity that will probably take place after that,
we\'re really not going to have the funds to do.

DR. HAMMITT: You were talking about the political appointments and said
there were six additional nominees so far. Are those all assistant
secretaries? And how many assistant secretaries are there in the DOE?

MR. PETTIS: You asked a hard question. I really don\'t \-- I think that
there are something like twelve. There is a nominee for deputy secretary
and under secretary, and a couple of assistant secretaries have been
announced.

If you\'re interested, we could probably put together the list of who
those people are.

DR. HAMMITT: Do you think the delay is in making public the list?

MR. PETTIS: I just think that process is just really a long story. It\'s
just typical that once they\'ve selected individuals, it\'s getting the
completed nomination package over the Hill that\'s kind of where the
slowness in the process is.

Okay. Thank you. I look forward to a good meeting.

CHAIRPERSON CRAWFORD: I\'d like to go ahead and move on and turn it over
to Nancy Kirkendall who is going to tell us about her progress on the
EIA-ASA fellowship. And there in your notebook are some blue sheets that
talk about that.

MS. KIRKENDALL: Actually, this program came about, at least in part,
because this committee was interested and thought that we should pursue
a research fellow program. And so last year we actually did establish a
fellowship program with the American Statistical Association. I think
the proposals were due in January. We received three this year, which
isn\'t too bad considering that it\'s the first year. Hopefully, in the
future we\'ll get a few more.

This is the announcement. It describes a number of different projects
that EIA is interested in. I think one of the things that\'s interesting
about this is the variety of projects that we are interested in here in
EIA.

As I said, we had three proposals and we could only select one. The
winner was Steven Gabriel and his proposal was to work on NEMS
convergence issues, so Mary is getting some help this summer on her NEMS
convergence.

One of the other advantages though is that one of the proposals turned
out to be a former committee member that we kind of lost contact with
who used to be working for an oil company and now he is at a university
and he has developed some expertise in alternative fuel vehicles. So we
will probably \-- although we didn\'t pick his proposal, we will make
use of his services for independent expert reviews and things like that
in the alternative fuel vehicles area.

So it\'s a good program for finding people who are interested in working
with us, even though we can only fund one right now. Any questions
anybody has on this?

DR. COWING: Where is Gabriel from?

MS. KIRKENDALL: I think he\'s at the University of Maryland. Local.

DR. COWING: And this is for two months?

MS. KIRKENDALL: Yeah, it\'s two or three months in the summer, however
they choose to set it up. Somebody local could spread it out a longer
period of time because they don\'t have to relocate. But nominally it\'s
set up so that somebody could come here for the summer and work here on
site. He has more flexibility because he\'s located nearby.

So you guys are all encouraged to circulate these announcements. Anybody
that you know who might be interested in doing something, a little work,
coming here working at EIA with the staff on some important projects
encourage them to send in an application.

DR. HAMMITT: What is the funding?

MS. KIRKENDALL: We give a grant of about \$50,000 to ASA. I think in
most of the proposals the person ends up getting are somewhere around
\$25-30,000. It depends on their salary and expenses for coming here.
It\'s basically to cover expenses and pay a little salary for the time
they\'re here. And then they work with us; we make space for them. You
don\'t get rich on it, but you might get a trip to do some interesting
stuff and work with EIA.

DR. HAMMITT: Nominally, it will cover a couple months\' salary is the
idea?

MS. KIRKENDALL: Yes, that\'s the idea.

CHAIRPERSON CRAWFORD: Any other questions? Well, we are a little bit
ahead of schedule and the schedule looks kind of funny because I\'m
supposed to come in here at 9:13, and this is what happens when you ask
Bill to squeeze you in. So you get squeezed in for what amounted to two
minutes, but it looks like I\'ll have a little bit more time.

I found out rather last minute that this committee, and all ASA
committees are invited to submit proposals for Invited Paper Sessions at
the Joint Statistical Meetings that are held in August. And I found this
out when the ASA president contacted me last July and asked me for such
a proposal and basically gave me about a week to put a proposal together
and submit it.

So I did it and the good news was it was accepted, and the bad news was
I had two weeks to confirm all the speakers that I had just sort of made
up without contacting anyone. So I frantically contacted EIA and we
scrambled as hard as we could, but we couldn\'t confirm the four slots,
and we didn\'t actually end up taking advantage of that session.

So this time I know it\'s going to happen again in July the president of
ASA is going to contact me and say, we\'re encouraging all ASA
committees to submit proposals for Invited Paper Sessions, and this time
I want to be ready. So I figure if we start thinking about it in April,
by the time he contacts me in July, I should have a pretty good idea and
I can actually tell him that most of my speakers are confirmed.

So the way it works is I will submit a proposal and the proposal can be
anything we want it to be at this stage, just involving energy
statistics. It could be four or five speakers from EIA. It could be two
speakers from EIA and an ASA Committee discussant. It could be two EIA
people and two people in the energy field. It can be just about
anything.

Originally, when I put the proposal together last year, I thought of two
people from EIA talking about some of the state of the art developments
that EIA had made. I had actually volunteered CKAPS because I thought it
was interesting and dynamic, but then I found out, I guess, it was no
longer with us.

So I encourage you to think about, particularly if you\'re from EIA, if
you have something that you think would be appropriate and interesting
for a statistical meeting, and it doesn\'t have to be hard‑core
methodology. I think a lot of statisticians don\'t really know about
energy, so one session I had envisioned was something like \"Global
Warming: Does It Really Exist and Is It a Problem?\" And people are
interested in that kind of thing.

Or even like some of the statistical issues about how you measure global
warming. Or maybe some of the things that you\'re doing on your web
site. You know, web site design and interactive data analysis are very
interesting to statisticians.

And I also encourage the Committee members to participate in this
session. If you have something that you want to speak on or if you have
some ideas for somebody that\'s even outside of this Committee, but is
involved in energy, this year I\'d really like to take advantage of that
slot.

And you will get an opportunity to publish your papers. There\'s the
General Proceedings that comes out of the various sections of the joint
statistical meetings, and so you will get a paper published in a
proceedings and some time to speak in an invited session.

So I guess I would just encourage you to e-ail me your suggestions for
this or talk to me. We\'ll be talking about it at dinner. Talk to Nancy.
Stan Freidman had some ideas about some ways that EIA people could
contribute to this. And just let me know so that I can be ready to
submit something on behalf of all of us in July.

PARTICIPANT: Where and when is the meeting exactly?

CHAIRPERSON CRAWFORD: Let\'s see. The meeting that I think I\'ll be
soliciting for is in August and I don\'t remember the dates. It\'s early
August I think, somewhere between the 1st and 15th, and it\'ll be in New
York City, so this will be 2002. So it\'s a great place to go, better
than Indianapolis was or probably even better than Atlanta was this
year.

Any other questions? We\'re still running ahead of schedule, but are you
ready, Mary? We\'ll just go on. Okay. I\'d like to turn the meeting over
then to Mary Hutzler, and she\'s going to talk about the progress on the
development of the MARKAL Model.

MS. HUTZLER: Bill asked me to give you another status report on our new
international MARKAL model development, so that\'s what I\'m going to do
today.

I think we have a few new members, so I\'m going to repeat a few things
to get those folks up to speed.

Essentially, I told you last time we were using the MARKAL model system
as Carol indicated, as the basis of this new modeling system. Last time
I talked about the five different versions of MARKAL that exist and told
you that we were planning to use the version called MARKAL-ED. The ED
stands for elastic demand and essentially elasticities at the ED use
demand level are included so that you can get price feedback effects and
the reductions in demand from it. So that was our starting point.

And last time we also told you that there were a number of benefits that
we found in terms of using the MARKAL approach. This slide isn\'t in
your handout, but I wanted to put it up again just to remind you why we
were taking this particular approach.

We wanted a model that was rich in technology detail, and just as a
reminder, when we did the study of the Kyoto protocol, we were not able
to determine an international price with trading and partner permits
because we only had a U.S. model.

So that\'s what we want to use this model for, one of the reasons. There
are other reasons as well. And, obviously, technology is a very
important issue in trying to deal with reductions in CO2 emissions.

Another reason why we thought the MARKAL approach was good for us is
that it\'s capable of handling a different level of detail depending on
the type of country you\'re dealing well. So the developed countries
that are very rich in data, we can support that level of detail in the
model. For the countries that are developing, where the databases
aren\'t that rich we also get an adequate level for this model.

Also, in our development of models in the EIA, we\'ve used a modular
approach where we have analysts working on either different models or,
in this case, different countries. And the MARKAL system allows us to be
able to do the same thing here, assign different countries to different
staff members and to use a team approach.

And then, lastly, there are a lot of countries that already have MARKAL
models and databases for them and also there is a team of experts that
meets twice a year, generally in European countries to discuss the
MARKAL system and talk about enhancements to it. So it\'s a system
that\'s supported by others and also has data that exists already.

Now, what have we achieved since I last spoke to you about five months
ago? Essentially, what we\'ve decided to do was to do a front end to the
modeling system that is depicted here to make it easier to assemble the
data in a consistent format, and to provide it to the heart of the
MARKAL system, which is in this box right here.

So we spent a lot of time essentially developing Excel spreadsheets that
are consistent and will be available for every single region so that we
can take all the data, use these spreadsheets with the core templates
and then set it up so that it could be uploaded into the matrix
generator system and solve it for the MARKAL model.

Now the data that we\'re dealing with are essentially three different
types. One is data that comes from the IEA, International Energy Agency.
And that\'s our basic building block and we have that data set up for
each of the regions that we want to use, and actually for each of the
countries. Because part of the system \-- what we wanted to accomplish
was to be able to take a region but also to decompose it. So we
essentially have that data set up on a country basis.

There are other data that we\'re spending a lot of time gathering right
now that we need. Certain of that data we\'ve used in our development of
the International Energy Outlook, data consisting of population from the
World Bank, the United Nations, the basic drivers, the GDP numbers that
we get from Data Resources Incorporated or else from LEFA.

But we have also dealt with a lot of other countries, trying to get
their energy statistics to support the data that we get from the IEA.
And I have had people who are trying to collect data there. We have
effort right now going with Australia to try to have a relationship
where we can get their data and the data they\'re using from MARKAL.

I have a person in Kyoto, Japan and talk to MARKAL modelers there and
also get a number of their reports. Lawrence Berkeley Laboratory has
quite a bit of data on both Mexico and China so we\'ve got relations
with them to try to obtain all of that data, too.

So a lot of our effort right now is in this data-gathering stage, as
well.

The third major area of data is this new technology data repository.
This particular part isn\'t built because we want that to be able to go
directly into the matrix generator and solver of the MARKAL system. So
that particular data repository we have ABRE, the Australian Bureau of
Resource Economics in Australia working on because they developed the
ANSWER system and we want that being kept in there.

What we intend to do is to populate this data repository with the data
for new technologies that we have in the National Energy Modeling
System, based on the United States. Then we will modify that data by
using adjusted factors to deal with different issues for other
countries, such as labor, the developing countries have cheaper labor so
we need to adjust for that. Also for financial parameters because the
cost of capital will be different in different countries.

So that will actually come through another template over here, but those
are still parts that need to be built.

What we\'ve accomplished so far is we have these five templates listed
here, which deal with supply. That\'s all aspects of supply. Oil, gas,
coal, and also refineries are included in this upstream supply template.
We have a separate one for electric utilities. Grouped together are
residential, commercial, and agriculture; then a separate one for
industry and transportation.

So each region will have these five templates, plus one for the
adjustment factors for the new technology database. And then these
templates will feed into this program that will essentially upload the
information, as I said, into the matrix generator. And then ANSWER will
be used to solve that.

And another area we\'ve been working hard on is this scenario analysis
software. We have a group called HALOA, they\'re from Canada, and they
have, for the MARKAL system, what they have called ANALYST and it\'s
essentially a report writer capability as well as an archival
capability.

And we\'ve asked them to modify that software so it could be directly
attached to the ANSWER system as well as have the ability to get all of
the variables that we use in the NEMS system and this particular system
too. And that work has been completed.

So just in summary, we set up this set of preprocessor systems with the
templates and we spent a lot of time on data gathering and this report
writer and analyst capability at the end.

Now just another reminder, I didn\'t show you this at the last meeting.
We\'re doing 15 country models, or 15 region models I should say, as you
can see listed here. We\'re breaking up North America into the three
countries. We have OEC in Europe, the former Soviet Union, Eastern
Europe, Japan, South Korea, China, India, the rest of developing Asia,
Latin America, Australia and New Zealand together, Africa, and the
Middle East.

Now I have analysts assigned to each of these different regions. There
were 11 analysts working on this project. Unfortunately they\'re not all
full-time. I already mentioned the low-sulfur diesel study. I have one
and half people working on that right now, so they haven\'t been able to
be full time on this, as well as our International Energy Outlook. Most
of these people are also working on that. So when it\'s the time of the
year when we do that particular project they can\'t be working on this
development.

And we just released the International Energy Outlook on March 28 on the
web and next Monday the printed report will be out.

Now just the see the regional structure of each model, it essentially
deals with every part of the energy system. We start with resources
extraction. We have the transportation links to move it into the
conversion areas such as refineries or to electric utilities. Then we
have the technologies in the end-use area, and then we have service
demands.

And essentially we\'ve been spending a good deal of our time working
this way because service demands aren\'t data that\'s available from the
IEA, so we have to be able to calculate those.

Now different countries or regions will have different levels of
complexity. For instance, end-use technologies. We have four of the
developed countries, but not the developing countries. So this system
will be different, as I mentioned before, for each of the different
regions that we\'re looking at.

And I can discuss that better in this particular chart, which is pretty
complex. And the point of it is not for you to look at every single box
and understand it. The point of it is that it\'s a major project to be
able to do this for each region and area. This is just the
transportation sector. And, as you can see, you start with crude oil and
have to put it through the refinery, then you have links to get your
processed crude out to whatever end-use that needs it, such as here\'s
jet planes, trains, vehicles, and your trucks and buses down in this end
of the chart.

Now if you take a look at the vehicles dealing with new technologies, we
have electric vehicles. So the electricity system is part of the
complexity of trying to deal with the just the transportation system,
and you see the different box there.

Now just on this chart, there is the nuclear, renewable and fossil, but
in the MARKAL system we will be having each of these separate
technologies, separate fossil technologies for coal and gas, separate
renewable technologies, as well as nuclear technology.

And then down here we can see coal because we have liquefaction and
gasification. Actually, there should be a link up to the refining
process, but there\'s not, and also natural gas to fuel trucks, buses
and vehicles, as well.

This area here, this column, deals with your demands for the services,
your vehicle miles traveled for each of these.

Just to summarize again where we stand on our software development, we
have two processes going on. One with the ABRE group dealing with this
common new technology, data repository, and the multi-region operation
of the model.

Those were to be completed actually two months earlier than what you see
listed here. That didn\'t happen because we had problems getting the
contract to ABRE so they had to start later, but we still think that
we\'ll be able to get the software in time for meeting the database
pieces that we are putting together.

Then the other part here is the enhancement of the outputs in the
archival capability, and we have Lowe working on that. That work is
completed so that part we have accomplished and that was done on
schedule.

They changed the name of ANALYST to VEDA. In Indian that means truth,
the highest level of truth. So it\'s just a new name that you\'ll be
seeing from us.

I wanted to mention again the peer review. In August of this year we\'re
planning to have the ETSAP community. This is the community of MARKAL
modelers peer review all of the single region model and I\'ve updated
this list of our working relationships. We did have China, Italy, and
Canada and the European Union, but we\'ve recently connected with Brian
Fischer who heads ABRE and who\'s willing to work with us on this
particular project.

And also, we have sent people to Japan, as I mentioned, to talk to the
Atomic Energy Research Institute where the MARKAL modelers are.

Just to summarize the schedule, this is pretty much the same schedule I
showed you last time. All of these steps have been completed. The delay
in the ABRE contract, here, is causing our delay.

And then you can see the rest of our schedule, which essentially says
that by that end of July we want to have all of single region models
done. August is our peer review and then in the fall we are we\'re going
to try to accomplish the multi-region model.

Now, of course, I had mentioned the budget cuts. Software will be
developed for the multi-region model. So if we\'re successful and we
don\'t need to do a lot of additional work to it, we might have it a
multi-region model, but if it turns out that there\'s a lot more that
needs to be accomplished to do it, then we may have difficulty doing the
budget cuts.

The other thing I wanted to mention related to the budget is that the
last time I showed you this list of future work and this area, of
course, will not be able to continue given that we won\'t have the
budget to do it, if in fact, the cut is taken.

What is \-- macroeconomic model as part of the system, so it\'s probably
\-- we can develop that. Also, through the peer review we expected \--
to be suggested and we probably won\'t be able to get any of that done.

Also, we did feel we needed to do a lot more work on the resource models
and try to deal more with the aspect of completion and that obviously
will have to be cut. And then this last issue is that ETSAP is
developing a TIMES model and that particular model has some enhancements
that we wanted to watch and also to capture in our development if, in
fact, we thought that they were worthwhile.

So that\'s about all I have to say today. I\'ll be happy to take any
questions you have.

DR. HAMMITT: Could you just remind me of the structure? What MARKAL does
is find least-cost ways of supplying specified final demands, is that
right?

MS. HUTZLER: It is a regular programming model and it does look at the
least cost solution of the system that you set up for it to solve it. We
have links to the different regions and there\'s a training component
also.

PARTICIPANT: I think you mentioned earlier that we might apply it some
scenarios involving carbon dioxide trading. Is it set up to do a dynamic
solution so that you have a really good way of understanding trading and
carry over of an allowance, that kind of thing?

MS. HUTZLER: There is a dynamic solution capability in MARKAL. Actually
we\'re trying to look at two different versions of MARKAL. There\'s one
currently that deals with perfect foresight, so we have that capability.
We\'re also looking at a timed-step solution procedure where there
won\'t be perfect foresight, but there\'ll be some myopic foresight
nature to the timed steps, and that\'s under development right now and
we\'ll have both capabilities to work from.

PARTICIPANT: Is there anyway to know whether it works well? Have people
applied it retrospectively to an historic period and have some idea
about whether it\'s doing a good job?

MS. HUTZLER: That\'s actually a good question. MARKAL has been around
for 20 or 30 years now. Annie Kidas (phonetic), who works in my shop, is
one of the original developers of the model. I really don\'t know if
they\'ve done a retrospective review of it, but it has certainly been
used for a lot of country analysis.

PARTICIPANT: Do you know if there\'s been any retrospective review?

PARTICIPANT: It\'s been used a lot for comparing one policy to another
rather than developing baseline forecasts. In fact, that is a primary
challenge we face and why we\'re focusing so much attention on these
templates in developing estimates of service demand linked to levels of
economic development. So we\'re trying, we might have elasticity related
to the billions of vehicle kilometers demanded that is very high \-- for
quite time and then trail off.

But no, it hasn\'t really been used very much as a primary forecasting
tool.

DR. MOSS: Did you say you have price elasticities in there? There\'s
price responsiveness?

MS. HUTZLER: Yes. Other questions?

PARTICIPANT: There must be some sort of market clearing going on here
also. I mean it can\'t just be linear programming. Could you speak to
that?

PARTICIPANT: Allowing trade, yes, there is markets in there, so if you
had trade and carbon permits among the four regions, there would be a
single carbon price and there is markets.

PARTICIPANTS: What I had in mind is \-- was motivated by the question
about elasticity. If you have a demand at a given price, to start the
model off and then the model solves the least cost supplies and am
implied constrained or a shadow value. You have to have agreement
between those two things.

PARTICIPANT: You bet. Demand will be modified \-- yes.

MS. HUTZLER: Any other questions? Okay. Well thank you very much.

CHAIRPERSON CRAWFORD: We\'re still slightly ahead of schedule. Scott,
are you ready to go?

MR. SITZER: I\'m ready.

Okay, I\'d like to talk this morning about our study of the strategies
for reducing multiple emissions from power plants. This is the study
that was mentioned in the letter President Bush wrote four U.S. Senators
and told them that he would not be regulating carbon dioxide at the
power plants.

I don\'t think this is the only reason, but it was mentioned that
letter. We completed this study at the end of December of last year and
we\'re now working on the next part, which I will describe a little bit
when I\'m finished here.

The modeling tool that we used, basically was the national energy
modeling system which I think most of you are familiar with, I hope
you\'re familiar with at this point. It\'s a comprehensive model of the
energy, economy, environmental system within the U.S. It\'s certainly a
domestic model. And the reference case that we used for this study is
based on the Annual Energy Outlook 2001. In fact we\'re doing the study
pretty much to currently with the AEO.

The key assumptions that we\'ve made for this study. And I should
probably back up a little bit and give you a little more background.
This is a service report, which we did at the request of Representative
David MacIntosh who at the time was Chairman of the Economic Growth and
Natural Resource Subcommittee of the House Government Reform Committee.
He later resigned from the House and ran for Governor of Indiana. Right
now he\'s unemployed.

He asked us to look at the costs to consumers in the industry of
multi‑pollutant strategies, which is to say reducing emissions of sulfur
dioxide, carbon dioxide, nitrogen oxide, and mercury.

For part one of the study, which is what I\'m talking about now, we had
the models structured to do three of those pollutants, but did not have
the structure to do mercury. We\'re working on that now and that will be
part two of the study which we\'ll be releasing over the next several
weeks.

In doing the study, using NEMS, we did have to make some assumptions
about how we would treat emission caps for these three sources of
emissions. One is, we assumed that there would be a cap and trade
system, which is similar to what we have for sulfur dioxide now in clean
air act amendments of 1990.

Basically a cap is set, allowances have been issued to power plants and
users and they\'re allowed to trade those, which means that what happens
is, you get a market price for the permit. In order to emit an amount of
sulfur dioxide, you basically to buy the right to do so.

We assumed that that would be the system that would be used for nitrogen
oxide and carbon dioxide as well. There are other possibilities, but
that was the assumption to be made.

Second, we assumed there would be competitive electricity pricing in the
wholesale market by which we match the cost of allowance would be
included in the price of electricity, even though some parts of the
country have not yet deregulated and that whole activity has slowed
down, for obvious reasons.

In the wholesale market there is still working competition and we made
the assumption that allowance would be passed through to the electricity
crisis.

And finally, that implies that the marginal generating unit would set
the price of electricity.

The assumptions that we were asked to look at included the nitrogen
oxide and SO2 emissions would be 75 percent below their 1997 levels by a
target date of either 2005 or 2008. We did different cases that includes
these various target dates.

Carbon dioxide emissions would be at 1990 levels by either 2005 or 2008
and that carbon dioxide emissions would be at Kyoto levels, which meant
seven percent below 1990 levels, by the period of 2008 to 2012. I have
Kyoto in parentheses here because this study is only for the electricity
sector. The Kyoto Treaty covers the whole economy, but we were looking
only at the electricity sector for this study and all the targets that
we have assumed are just in the electricity sector, which is about a
third of CO2.

The key findings were that when you run stand alone cases for just
nitrogen oxide or sulfur dioxide, it tends to have little impact on
price because so much of the mitigation strategy is in adding additional
capital costs, which do not increase the marginal costs of electricity,
but it does increase the industry costs.

When we run the CO2 and the integrated cases which have all three
pollutants in them, we find that we get the greatest impact on
electricity price and also that it reduces the need for NOx and SO2
retrofits and a large part of the reason for that is that you use a lot
less coal when you add carbon dioxide to the mixture, which reduces NOx
in the SO2 emissions considerably.

And finally we found achieving the reductions by the earlier target date
of 2005 could pose a significant challenge to be met by the electricity
and the natural gas industries in terms of bringing that natural gas on
line in terms of new equipment, new kinds of technologies, and so on.

The remainder of what I have to say is mainly results and we could get
into methodologies and modeling in more detail in the questions. This
graph shows the reduction requirements that would be needed, 1997 NOx,
SO2 and carbon dioxide are shown. In the base case, our 2010 and 2020,
we have significant reductions in NOx and SO2 without any further
strategies because NOx is responding to the EPA\'s Summer NOx Production
Program and SO2 responds to Clean Air Act amendments of 1990, both
already on the books.

But then if you look at the target, you\'re 75 percent below 1997 levels
for NOx and SO2 and that\'s a significant decline the industry would
have to make.

CO2, there\'s nothing on the books at the moment. The Kyoto Treaty has
not been ratified. So we see significant growth in carbon dioxide in the
base case between 1997 and 2020. The target cap of seven percent below
1990 levels would show a very significant decrease for CO2 in the power
sector.

This is the reference case. Electricity generation by fuel historically
and projections, and as you can see, the biggest growth area for
generation in electricity over the next 20 years, we see as being
natural gas. Natural gas just about triples in its contribution to
generation between 1999 and 2020. Coal stays steady. We don\'t lose it,
but on the other hand its share declined significantly I think from
about 53 percent today down to about 44 percent by 2020.

Oil continues its long-term decline in the power generation. Some
decrease in nuclear as we have retirements due to license renewal and
shifting from nuclear to natural gas, there is a slight growth in
renewables. But this is basically the reference case, the no change in
the laws and regulations scenario.

Looking at the SO2 cases that we did, stand-alone SO2s, these are the
projected allowance prices in 2000, 2010, and 2010. And in the base case
we see allowance prices as being around \$220 to \$230 a ton in the year
2020. If you move to a case where you need to reduce SO2 emissions by 75
percent, allowance prices are in the \$1100 range and this reflects that
very significant increase that would be required in order to meet those
assumptions.

We also ran a SO2 sensitivity case which was roughly half the target of
the requested case. As you can, it\'s not linear. If SO2 emissions need
to be reduced by about 40 percent rather than 75 percent from the 1997
levels, we get SO2 costs of about \$400 per ton of SO2.

This graph shows carbon fees over the forecast period and basically they
cluster from the CO2 case where we have 1990 levels of CO2 going on
through 2020, to the cases where we bring the seven percent below 1990.
In the former cluster we tend to run about \$100 a ton going up to about
\$120 by 2020. But in the cases where we require a seven percent below
1990 level, they range between \$120 to the highest at about \$140 a ton
in 2020.

Again, this is somewhat lower than the report that we did a couple of
years ago where we looked at the Kyoto Treaty as a whole because this is
just dealing with the power sector.

In the integrated case \-- this is basically the most stringent case.
The integrated 1990 case with carbon at seven percent below 1990 meeting
it by 2005, only four years from now.

And basically the big result here is that we get much more gas capacity
than we would in the base case and a little bit more renewables. In
addition, we have some significant retirements, as many as 50 gigawatts
in the 2006 to 2015 period, much of this is coal. We also get some
nuclear retirements during that period of time.

But the big story is that we will be much more reliant on natural gas
generation than we would be in the base case.

And this graph is like the first one I showed you, except this shows the
results in the most stringent case and now you see that coal falls
considerably instead of maintaining its roughly 2000 billion kilowatt
hour generation levels, it drops to about half of that by 2020.

And this has to do with the fact that coal is the fuel that contains the
most emissions when used in electricity generation, particularly carbon,
and we see a much larger growth in natural gas in this case. Plus we
also see some addition in renewables as well.

Electricity prices were a prime variable of this study and basically
they, again, cluster in two clusters. By 2020, the base case and cases
that just deal with NOx and SO2 tend to show prices at about the six
cents per kilowatt hour range. We\'ve assumed deregulation for those
states that have already deregulated continues and for those states that
have not, they continue as cost of service.

Nevertheless, we do see a price decline from something like 6.7 cents
per kilowatt hour today down to about six cents in 2020 under these
circumstances.

If you add carbon and the integrated cases to that mix, the price goes
up about two cents a kilowatt hour reflecting the allowance costs that
would be required to meet those caps and the additional cost to the
industry, particularly natural gas.

In the stand-alone NOx and SO2 cases, there are both additional revenues
because there is a slight increase in the price, but there are also
additional costs and in both of those cases, over the 2005 to 2020
period we do see both a slight increase in revenues on the order of
\$1.5 to \$3 billion, but also additional costs.

So basically, these cases tend to cost the industry the difference
between those costs and revenue for that period.

Another look at electricity prices and, as you can see, they kind of
peak out in 2010. In the CO2 stand-alone and integrated cases they\'re
over eight cents a kilowatt hour compared to the reference case of about
six cents.

This graph shows projected changes in annual household electricity bills
and along with the electricity prices, in 2010, there\'s about a \$200
per year increase in residential electricity bills in the integrated
cases and in the stand-alone CO2 cases. The cost of about \$150 by 2020
as prices begin to drop as the industry adjusts to the lower level of
emissions that are required.

Natural gas consumption for electricity generation, again in cases where
we\'re only looking at NOx and SO2, most of the mitigation strategy
could be done by retrofitting coal plants, but as you move to CO2 and
integrated cases, you need to switch to natural gas. By 2020, we\'re
using about 11 TCF in the base and SO2 and NOx cases. By the time we get
to CO2 in the integrated cases, we\'re up to 15 trillion cubic feet a
year for electricity generation, nearly a 50 percent increase in the
reference case.

And that\'s reflected in the price of natural gas as well. Base case and
NOx and SO2 cases, something about \$3 per thousand cubic feet in 2020,
but if you\'re going to be regulating carbon dioxide in an integrated
strategy, we can see prices above \$4 per thousand cubic feet as well as
by the year 2020, concurrent with a greater fall in natural gas
production.

Renewables play a significant role. In the reference case we see about
450 billion kilowatt hours being contributed by renewables including
hydroelectricity, but in stand-alone CO2 case, the number rises to 700
billion kilowatt hours. Most of that additional renewables is wind based
and biomass based, co-firing and dedicated. These are the renewables
that we see as having the lowest cost and the greatest ability to
penetrate in central station electricity generation.

We looked at employment and not surprisingly the coal mining employment
tends to fall, even in the base case. Productivity improvements over the
past couple of decades has been about 6 percent a year in the coal
mining industry. We don\'t perceive that great of an increase over the
next 20 years, but nevertheless, you can get a lot more coal out of the
industry even with employment being relatively steady.

In the cases where we see a lot of coal retirements that obviously
declines fairly percipituosly by the time you get to 2020.

On the other hand, natural gas employment will grow from a number of
around 300,000-odd employees today, we perceive growing to about 450,000
by the year 2020 under circumstances that help the emissions crisis.

Of course there are a lot of uncertainties in a study such as this which
I just pointed out. One is, what would the actual impact on technology
costs be? We tended to take our base case assumptions and carry then
through. There are impacts on new technologies, as more of them
penetrate the market the cost tends to fall. But if that didn\'t happen,
of course the results would be different.

The timing of the scenario is very important whether it is 2005 or 2008
remain a significant difference in the results. How market participates
respond to the higher prices and the lower caps of these emissions will
be important.

Reliability is an issue that we discussed in the report. What might
happen as units are taken off line to make the necessary adjustments and
the possibility of price volatility as well.

As I said, we\'re now working on part two of this study which is going
to add mercury and an analysis of non-renewable portfolio standards
within this strategy and that of course should be completed in about a
month.

Thank you very much.

DR. COWING: On the base case, what were the real price projections of
the future for coal and natural gas?

MR. SITZER: Natural gas, at the well head, was just over \$3 per
thousand cubic feet.

DR. COWING: For 20 years?

DR. COWING: Natural gas is part of what we model. So, of course in the
AEO 2001, which was the basis for this study, we had some adjustments
for short-term increases that weren\'t as high as they are now. If I
remember correctly, we were starting around \$2.80 in 2000 and 2001.

CHAIRPERSON CRAWFORD: In 2000 it was \$3.30

MR. SITZER: Was it \$3.30? Okay, and then it drops over the course of a
couple of years and rises up to about \$3.15 cents per thousand cubic
feet.

DR. COWING: In the base case?

MR. SITZER: Right. For coal, mine mouth prices I think we\'re about
\$12.70 per ton by 2020. Again, we have a model that is responding to
the demands.

DR. COWING: Okay, because I thought real coal prices were actually
falling.

MR. SITZER: Real coal prices are falling.

DR. COWING: And that\'s in your base case?

MR. SITZER: That\'s in the base case.

DR. COWING: Second question, and this may by old information, but I
thought Congress was talking about direct regulation of nitrogen oxides
rather than a market system based on permits. Is that wrong?

MR. SITZER: I think there are a lot of proposals in Congress that \--
I\'m not sure what you mean by direct.

DR. COWING: Well, kind of old fashioned regulation rather than \--

MR. SITZER: I\'m not aware of specific proposals on that. I know there
are some proposals that do this, that essentially \-- they don=t really
specify. That\'s why we had to make the assumption how we were going to
model it. We used the cap and trade system.

DR. COWING: Okay, the question that comes from this then, suppose
Congress did move towards direct regulation, rightly or wrongly, how
that would that affect your analysis?

MR. SITZER: I\'m not sure.

DR. MOSS: I had a question where in your graph where you have the costs
going up a lot more than the revenues as a result of these policies.
Does this feed back into your supply and the ultimate cost-to-volume
interest or the \-- actually, is this just electricity?

MR. SITZER: That graph shows the cost to the industry versus the
revenues against the base case over a 15-year period. And that feeds
back to the price, which in turn feeds back to the assumptions forecast.

DR. MOSS: Well I was just thinking, if this implies a return and this
industry is declining, then you\'d expect supply to pass the additions
to your targets. Is that built in to your \--

MR. SITZER: That\'s built in, in terms of they need to recover their
costs before they\'re going to put in new unit.

DR. MOSS: The other thing is, on your employment numbers, is that just
the U.S. or is that North America?

MR. SITZER: That was U.S.

DR. BERSTEIN: On your last slide, your uncertainty is responsive market
participants, I notice there\'s a slight decline in demand or
generation. One I suppose would expect more than that with a 2-cent
average increase. What do you think its impact might be?

MR. SITZER: Well, it\'s built into the model. I mean, that\'s a 33
percent increase in price and I am not sure exactly what the assumption
is. The elasticities tend to be fairly small for energy consumption for
a long time. So we\'re basically using the estimated elasticities that
we see in our data.

DR. MOSS: Right. So you see little response?

MR. SITZER: We see little response in the price.

DR. MOSS: And on the NOx issue, come back to it for a minute, since
right now it\'s regional based, I assume what you did here is a national
cap and trade?

MR. SITZER: Yes, the base case has the regionally based 19-state
program. The scenarios we did are essentially national cap, meaning 75
percent below 1997, the national cap for the electricity producer.

DR. MOSS: And you found NOx didn\'t have much of an impact on overall
costs?

MR. SITZER: It doesn\'t have much of an impact on price. It does have an
impact on needing to install additional equipment to mitigate NOx
issues.

DR. HAMMITT: I guess I\'m sort of curious about this substantial
disconnect between cost and price and I understand how that comes about,
but my question has to do with how confident should we really be of the
price-versus-cost differences. And it seems as if it almost hinges quite
critically on whether the marginal plants are buying permits or
installing capital costs.

And obviously this is one aspect that has undoubtedly attracted a lot of
attention.

MR. SITZER: Well I think the idea that you don\'t see much of a price
impact until you regulate CO2 is probably a significant finding.

DR. HAMMITT: They are also sort of claiming even though NOx regulations
increase costs, they won\'t increase prices.

MR. SITZER: Very much.

DR. HAMMITT: And that\'s \-- there are some reasons to doubt that if
long-run marginal costs goes up that the price won\'t respond.

MR. SITZER: That graph I showed was a 15-year graph and there wasn\'t a
whole lot of difference between the costs and revenues. There may be
some adjustments you\'d want to make if you\'re looking out beyond this
period of time.

I mean, basically we\'re saying industry would have to eat some these
costs.

DR. MONTGOMERY: Are the prices that are used to estimate the revenues
the short-run or long-run margin costs? Because it seems like they\'re
not in the long-run marginal costs, but the short-run and that\'s why
you\'re not picking up the capital costs.

MR. SITZER: It is short-run, but we also have long-run in the sense that
as generation requirements get closer to total capacity we have what we
call reliability price adjustment. So when that happens, the price is
going to rise.

So I think in that sense they\'re long-run. They\'re not 40 or 50-year
long-run, but I think midterm.

DR. COWING: I read a recent report that focused on the Midwest region
and electricity. And what they found, and as I recall, they were
modeling nitrogen, salt, and sulfur and particulate matter, and at the
end they also threw in carbon abatement.

What they tended to find that was that because the real price of coal
was falling and the real price of natural gas was rising, that you would
get very little conversion of coal to gas in a sense of building. When
you build a new plant, it was be gas fired rather than being coal fired,
which is the predominant fuel of use today I believe in the Midwest.

But rather, you get substantially more retrofitting, and the addition of
scrubbers and it\'s still coal fire. Is that consistent with what you
found?

MR. SITZER: If it\'s SO2 and NOx, yes. If it\'s CO2, there\'s not a
whole you can do except reduce use.

DR. COWING: And buy a permit.

MR. SITZER: And buy a permit. As long as you don\'t add CO2 to the
picture I think that\'s a reasonable assumption. And that\'s what we
found.

PARTICIPANT: In the scenarios where you used a lot of additional natural
gas and have a fairly high rise in the natural gas prices, I\'m going to
guess that that probably implies if there\'s a lot of investment in
natural gas infrastructure, which essentially displaces what would have
been there if we\'d continued with the coal fire capacity that was
replaced by the natural gas.

And so it\'s a form of investment that sort of begins to fall onto the
compensatory side of the ledger as opposed to being the kind of
investment that would support a broad-based economic growth.

What I\'m curious about is whether or not there\'s anything in the model
that attempts to look at sort of mix-up investment over time and has \--
to the rate of economic growth, GNP growth, and some dampening of GNP
growth as a result of very high demands on investment under some of
these scenarios.

MR. SITZER: Yeah. We have a macroeconomic model. It\'s a reduced form
version, it\'s a DRI model and we do have feedback. If you read the
report, we have a section on macroeconomics and just what you\'re saying
is what happens. There is a short-term decline in the GDP growth,
returning to baseline, I\'m not sure what year it returns, but once the
adjustments that have been made. There is some loss of economic growth.

DR. MOSS: I just noticed, it looked like you had nuclear increasing, is
that right?

MR. SITZER: I don\'t think so, not for the base case; however, in these
scenarios, because natural gas prices rise, you have fewer retirements.
So basically existing nuclear is competing with new natural gas and
they\'ve reached the end of it\'s licensing period and because natural
gas prices rise and you\'ll get a few more gigawatts of nuclear staying
on line but they\'re at the base case.

PARTICIPANT: Is it an assumption that no new nuclear power plants WILL
BE BUILT.

MR. SITZER: Yes, absolutely.

DR. HENGARTNER: What if that assumption wouldn\'t be true?

MR. SITZER: I beg your pardon?

DR. HENGARTNER: Did you cap test the sensitivity to that assumption?

MR. SITZER: I don\'t recall that we did in this study. We did in the
previous study we did which looked at the overall economy. First of all,
the Committee requested that we make the assumption that no nuclear
power plants will be built as they did in the previous study.

We also did a sensitivity case where we allowed nuclear to built and it
did come in under fairly stringent CO2. We didn\'t do a case study.

DR. SITTER: Was something similar done on hydroelectric?

MR. SITZER: Did we make an assumption about hydroelectric?

DR. SITTER: Yes.

MR. SITZER: We didn\'t make an assumption different from the \-- about
hydro. We tended to think that hydro hasn\'t grown in our projections
because of the environmental issues and the nonlikelihood that you\'re
going to get new dams. So we basically looked at the current \-- of
hydro. So we aren\'t \--

DR. SITTER: So in essence, you did a similar restriction \--

MR. SITZER: In essence, we did.

DR. SITTER: And did no sensitivity analysis?

MR. SITZER: We didn\'t do a sensitivity analysis.

DR. COWING: \-- the ability of importing power?

MR. SITZER: Imports are in the model, but we didn\'t \-- we basically
made the assumption that with these kinds of stringent requirements in
Canada, as well. So we didn\'t really allow the model to reflect it.

DR. COWING: So you don\'t see it as a possibility?

MR. SITZER: Well, you could see it, but even so, from what we know about
the situation there, it wouldn\'t really bring in a whole lot of
capacity. You might get some increase from their surplus, but not the
capacity.

CHAIRPERSON CRAWFORD: We\'re a little bit head of schedule, but I think
what I\'ll do is just extend our break because the presenters for the
breakout sessions aren\'t all here.

After the break at 10:25 we\'ll reconvene for breakout sessions and
they\'re How to Develop \"Emergency\" Surveys: Suggestions for Cutting
Corners will be in this room, called the Red Room. Look for Beth
Campbell and Stan Freedman to lead that.

In the White Room, which is the middle room down the hall there past the
refreshments, How Does EIA Measure the Impact of It\'s Data, and look
for Howard Magnas to lead that discussion.

And finally in the Blue Room, which is down at the end of the hall, will
be the Monte Carlo Analysis of Uncertainty in Greenhouse Gas Emissions,
and Perry Lindstrom will be leading that one.

Again, we\'ll reconvene then at 10:25.

(Whereupon, the meeting recessed for a short a break and resumed at
10:25 a.m.)

CHAIRPERSON CRAWFORD: I want to start the summaries from the breakout
sessions and I\'ll start with Beth Campbell, if she\'s ready, about her
discussion on how to develop emergency surveys and suggestions for
cutting corners.

What did we plan for, about seven minutes or so for each summary? And
questions. Come up here where we can see you.

VICE CHAIR BREIDT: Beth, you\'ll need to project \--

MS. CAMPBELL: We did have a very useful small group meeting here on this
topic. For those of you who are unfamiliar with it, EIA began this
winter following \-- began what we\'re calling an emergency survey to
collect data on natural gas interruptions and their impact on customers.

The selection began this winter and it\'s for January through April time
period. We are still in the process of collecting the data. We received
this as a directive or request in our Appropriations Bill which came to
use in a delay, so we didn\'t have it at the first of October. There was
a specified term that we do this as a biweekly survey beginning this
winter, and we had received this I think in November, I\'m not sure. We
missed the start of the winter already when we learned about doing this.

So the comments that we received, essentially we had some questions and
back and forth about what did we do going into the project, did you try
to say no, what did you do to narrow in on the specific data collection
needs given the general statement and the legislation, and urging us to
pay particular attention to the analysis needs, the data needs, and the
question of what is doable given the respondents.

In other words, urging us to stay alert to these issues and to try to
define scope proactively as we go in.

The other discussion areas were about the management of the project and
urging to learn from this experience. First of all, in the management
area, I think the thought was that this effort is probably not a
miniature of other survey efforts.

Something that we would do with business as usual approach is just
faster, you know, just faster. And we\'d be able to cut a few corners
here and there because we knew enough where we would be cutting these
corners. And it\'s probably a different beast, and that we need to think
about managing these differently, managing maybe with some parallel work
efforts in different places, using design formats that the populations
are familiar with, starting with our developing frameworks for our most
problematic respondent groups.

In other words, some of the sequences that we\'re used to, we might not
be following because we might learn that certain parts of the process
are harder, take longer, should be anticipated and begun first. We might
think about what we could contract out.

But I think probably the most telling comment was urging us to write up
this experience and take it as an after action effort on our part,
describe what we did, and then take it to the firms and contractors who
do this more regularly and ask for their experience. Take it to other
agencies who may do this more often. And try to be prepared to do an
assessment for how this type of request is different from our usual
surveys.

And that if we had that type of assessment and both our experience and
their advice to us, we might be better prepared to go back to the
originators of these requests to tell them what we thought we would be
more or less successful in doing. In other words, we could respond to
these directives with saying, we\'ll do this, but we have more success
in this area than we do in that area.

And so that we might in that way, early on, rather, before we begin,
warn them of where we thought we would be more or less successful rather
than just afterwards report out to them where we have been.

And it would perhaps help to shape future requests or help us to have
the latitude to make some small changes. How\'d I do?

VICE CHAIR BREIDT: That was good.

MS. CAMPBELL: In terms of capturing?

CHAIRPERSON CRAWFORD: Does anybody from that breakout session have
anything to add or clarify to her? Stan?

MR. FREEMAN: Well, I think if I could just put my management on the
spot, sitting up there at the front of the table that that\'s probably
\-- I personally think that the suggestion that the Committee made about
doing an assessment, and I\'m not talking about something that would
take years or anything, but this is what we learn and then seeking the
counsel of others that have been in a similar situation is something
that I think is pretty important for us to do for all the reasons that
the Committee suggested at the table.

So we don\'t go back and repeat the same mistakes again or so that if
this comes up with some other fuel area in EIA, there\'s always
something we can take off the shelf and say, when we did this back in
2000-2001 time frame, here\'s some of the issues that came up, here\'s
how we can deal with them in the future. And I think that would be time
well spent for us to do when I see at least half of my management
shaking their heads. I can\'t see the other half up there.

(Laughter.)

MR. FREEMAN: I think that would be a real useful thing for us to do and
something that is probably just as important as the data that we will
collect from this survey, what we learn from it, from you know, survey
management and methodologies. I felt the comments from the Committee
were very helpful and right on target for us. I appreciate it.

CHAIRPERSON CRAWFORD: Howard Magnas. He\'ll tell us about what his
breakout session discussed on how does EIA measure the impact of its
data.

MR. MAGNUS: Our topic was how does the EIA measure the impact of its
data and we started out with a discussion of the performance measures
that we do here in the EIA that focus a whole lot on customers and
what\'s needed.

Then we went into a little bit of a discussion on user surveys, how you
really find out about impact analysis if you talk to people who actually
use your data. And talking to them in some survey venue, is a
possibility.

Then the next discussion we had was on the budget and that pretty much
cut everything else off. We discovered that five projects were probably
going to have to be replaced because of budget cuts and that certainly
there wouldn\'t be any money to do these user surveys. So we got into
all sorts of discussions.

One of the comments was that could use conjoint analysis to evaluate
budget cuts and get in touch with customers about the cuts that we made.
So in other words, we weren\'t just only ones deciding on the projects,
but we would talk to customers about the projects and try to see what
sort of an order they felt it should take.

We spoke about the public sees the data more than analysis, so maybe
it\'s better to cut some of the analysis so that we could collect more
of the data. A lot of the people thought that our data all the more
important. If we stopped some of it, it would interfere with time series
and things like that, and they felt that the raw data was much more
important to the public than the analysis. I guess that could be
debated, but that was spoken about quite a bit.

Another issue was how much information should be provided, who should
provide it, public, private. When we know that certain data is being
provided by other sources by the public, should we just move into the
breach and collect the data that these private sources aren\'t
providing? And certainly that\'s another strategic angle to deal with
these budget cuts.

We had a discussion about categorizing the projects, although we spoke
about that in a little earlier one, but basically to categorize it so
that when we see how much money we actually get, we could be quicker in
deciding which ones we should do. And maybe the ones that we do, if we
show the public\'s opinion about it, that may help decide the issue.

Then there was a suggestion that we might want to cut modeling out and
focus on the data. A lot of the people here felt \-- there were a lot of
data people there, I guess we just didn\'t have enough model people to
make the scales a little bit more even.

Okay, and then there was a very, very long discussion on how, especially
the members of ASA, felt how important the data was. And they felt that
we should be in contact with people who use our data to modeling groups
who use our data bank and to data groups and other groups who use our
data, and they thought that we should talk to them and to try to build
some of their arguments up.

Our data, as we know, certainly is very important and very used, and we
should use that to buttress some of the cuts, as an argument against
some of the cuts we\'ve made. If we describe how many organizations and
how people and how much value is added by the use of our data, which we
pretty much provide free, the people who are cutting our budget might
understand it and might change their mind. And I think that about covers
it.

CHAIRPERSON CRAWFORD: Does anyone from Howard\'s breakout session have
anything to add, clarify, interject?

DR. PHIPPS: I have just a question. I just wondered what the focus
seemed to be so heavily like on newspapers and web sites in this and if
there had ever been any effort to look at things like journal article
citations and policy reports, things like that \-- it seemed to me that
that was a left out part of this.

MR. MAGNUS: Actually, we discussed at length. I just sort of picked
through for something to discuss, but a lot of people, especially
academics use the data to get citations and it is used and it has great
impact on their work, which impact on the country, and we did discuss
that at length.

DR. COWING: In response, I think one of the issues is how to do that at
a fairly low cost. Certainly want to do more, so if the question is
what\'s it going to cost, but we do if fairly efficiently.

DR. PHIPPS: Did you search, just like electronic databases \--

DR. COWING: Well that question came up, whether the citation indices
include references to data sources. They certainly include references to
related literature.

DR. PHIPPS: And bibliographies relating to \--

DR. COWING: Right. But do they \--

DR. PHIPPS: \-- from the articles.

DR. COWING: Right. But you don\'t want somebody simply reading every
bibliography on every published or unpublished paper.

VICE CHAIR BREIDT: It would have been worth trying to see what you get.

CHAIRPERSON CRAWFORD: Anybody else?

DR. COWING: No.

MS. KIRKENDALL: Yeah. The other suggestion that sounds like something we
can actually do, I think Howard alluded to it, but it didn\'t sound
quite as much like something we could actually start and do, would be to
make a list of important users of our data. We could start with the
federal agencies \-- to identify the important users of our data or
analysis, and they can be in federal agencies. We know the agencies that
use our data. We know some of the national labs use our data.

There are associations, Edison Electric actually had a representative
there and said that she\'d be willing to help us with this effort as she
was sure that other associations would be, to try to find out NASEO,
NARUC, some of the state regulatory agencies, to try to get a picture of
the kinds of users of our data. Maybe some of the folks that repackage
our data and sell it.

And find out exactly how they use the data, of what value it is to them,
perhaps what they\'d do if they didn\'t have it, to try to better frame
and get a better understanding of the data. And not only data, it was
also data modeling forecasts, it was our products, to get a better idea
of the value.

Another indication of the value might be how much they charge for it
when they sell it. Well, at least that\'s something to look at.

CHAIRPERSON CRAWFORD: Okay, thank you.

VICE CHAIR BREIDT: Thank you.

CHAIRPERSON CRAWFORD: Finally, our last breakout session was on Monte
Carlo Analysis of Uncertainty in Greenhouse Emission Estimates and Perry
Lindstrom is going to summarize that discussion.

MR. LINDSTROM: I guess I\'m quieter than the other two. As you might
know, this is my third ASA meeting in a row. Every six months I go to
the dentist and I go to an ASA meeting.

(Laughter.)

MR. LINDSTROM: And I\'m not going to say which is more painful. No,
actually, it was a very good session from my perspective. The hour and a
half went by so quickly, I was amazed when I looked up at the clock and
saw that we were about to finish up.

As you can imagine, this is a relatively complex subject I think,
compared to the other two breakout sessions in terms of it\'s more of a
technical topic.

We\'re essentially dealing with two sets of data that we combined to
make emissions estimate. The first is the activity data which we all
deal with here at EIA in terms of levels of fossil fuel consumption, and
the other are the carbon coefficients that we apply to that data in
order to come up with our estimates.

So within both of those sets of data, there are uncertainties that we
need to get a handle on and we need to understand how they interact and
give us a total uncertainty for the estimate.

One of the things that came out of this meeting is there are some
available data that we haven\'t really used yet in terms of, for
example, coal coefficients that we don\'t have to kind of invent the
shape or come up with it in terms of an expert guessing at it, but we
have the data and we can just build the shape from that data of the
density function.

I think we did know that was available, but we just haven\'t gotten to
the point of using that yet, but that certainly is a level that we want
to investigate more deeply.

Another issue was the relationship between variability in the data,
which is a naturally occurring variability versus uncertainty. And we
were trying to get a handle on that in the shape of these probability
density functions. And we\'re thinking about using a uniform
distribution to represent natural variability, which kind of stops at a
point and you got equal probability along that between the two values.

And I think the feeling of the ASA members is that\'s probably \-- while
it\'s conceptually good to understand the difference between variability
and uncertainty, really what we\'re doing here is using distributions
\-- or it\'s easier to use distributions that are normal or log normal,
but we may be limiting ourselves or putting too much stock in our own
judgment when we come up with a uniform distribution. So we should be
careful about using that approach.

And at this level of aggregation, it\'s less important to understand the
individual variability than it is to understand the kind of the central
variants. And because we\'re just looking at a national number, there\'s
a certain advantage in that.

And certainly there was a feeling I think from the Committee that we
should focus on certain areas, the big areas, petroleum and coal used in
electric utilities for example, that if we take care of understanding
that, those areas, that the others will fall into place and it won\'t be
that important.

I think they were unanimous, and in fact, admonished us that we should
use a minimum and maximum value because when you do a Monte Carlo
simulation, the min and max will go to infinity as the number of
iterations goes to infinity. So that\'s not a good situation to be in.

And so we are abandoning any discussion of that and instead just talk
about what you get at 97.5 percent interval and I think that was a
unanimous decision.

We need to probably work more on the uncertainty characteristics across
time and understand how trends may be changing in uncertainty. These
are, as people who have dealt with the data know, very complex issues
when you wonder how has deregulation in the natural gas industry, in the
electricity industry, affect the uncertainty in our data.

It\'s a nice thing to think about, whether we can quantify well enough
is going to be another issue, but certainly sensitivity analysis and
that sort of thing is a good thing to do in this area.

So for me it was a very helpful session. I started this project a year
ago and a year later I know how much I don\'t know about the topic,
which is an important thing to realize, and so I think I\'ve moved up
the learning curve quite a bit in the last year. Anybody have anything
to add?

CHAIRPERSON CRAWFORD: Thank you. Are there any other questions or
comments from the Committee or EIA people? Are there any questions from
the general public, if there\'s anybody from the general public here?

Before we adjourn for lunch, Bill has a few announcements.

MR. WEINIG: Thanks. Really only one, and that is, as all of you know,
we\'ve been dealing with audio problems all morning. I appreciate your
patience. I also appreciate all the guidance I\'ve gotten.

(Laughter.)

MR. WEINIG: And so what I was going to do is play a little trick on you
at lunch time. When you came back I was going to do a mirror image of
the room because I\'m told by lots of people outside of the room that
that would help, and I was just going to see your reaction to it. But I
thought, well, what the heck, we\'ll tell you in advance what we\'re
going to do.

We\'re going to bring the audience up here, bring the ASA participants
down at this end, shift the tables a little bit, and generally reorient
ourselves 180 degrees. We\'re going to do this while you and I are
having lunch.

So we\'ve got four sets of folks who are going to be looking forward to
starting as soon as we leave at 12:30 and I think I\'d take my purses
and stuff with me, but I don\'t think there\'s any need to take your
working papers.

You might tidy up a little bit if you\'re at this end of the table and
we\'ll go to lunch as soon as Carol says we can go to lunch down on the
sixth floor.

There are some signs down there that will make it a little easier to get
to where we\'re going. And the bathrooms and the fountains and stuff
that are on this floor are also available on the sixth floor in the same
locations. Thank you Carol.

CHAIRPERSON CRAWFORD: Carol says we can go to lunch now, and we\'ll
reconvene at 1:30.

(Whereupon, the foregoing matter went off the record at 12:26 p.m. and
went back on the record at 1:30 p.m.)

A-F-T-E-R-N-O-O-N S-E-S-S-I-O-N

(1:30 P.M.)

CHAIRPERSON CRAWFORD: I\'d like to reconvene and start our 1:30
presentation. We\'re going to receive an update in the result of the
cognitive testing of EIA graphics, and Herb Miller will be the lead with
additional participation from members from his cognitive testing team.

MR. MILLER: Okay, as Carol told you, the title is Update and Results
about Cognitive Tests and the members of the committee at the bottom are
Colleen Blessing at the back of the room, and Renee Miller, Howard
Bradsher-Fredrick is BLS this afternoon, and Bob Rutchik, and myself.

And the topics I\'ve planned on covering are introduction, goals, the
test results, conclusions, recommendations, and implementation plans.

The ASA Committee has been very helpful in the past in giving us advice
on graphs and acting as judges in the graphs contest. And as most of you
are aware, we\'re very determined to make sure that our graphs portray
the main message in the shortest time possible. And our committee began
conducting cognitive testing internally last fall, at the fall ASA
meeting, and a the NASEO Winter Fields Conference.

At that time, we made revisions to the graphs that we had selected, dual
Y‑axis graphs, stack bar charts, and a few other complex graphs. And
after we had analyzed the results of the tests last fall, we made
changes to the graphs. We tested again a couple of weeks ago at the
National Energy Modeling System Conference and that was attended by 500
consultants, economists, and other educators, and people with ties to
the energy industry.

And we tested at least 10 people on each of the graphs. And at the end
of this presentation, we\'d like your opinion on how we can use our
conclusions to get EIA to adopt best practices and any other suggestions
that you could give us on how to implement our findings.

According to William Cleveland, you should use graphs to see overall
trends, patterns, or relationships in the data, compare two or more
factors in general or quantitative fashion, present large data sets in a
comprehensive way, and to analyze data.

And another graph, Bob calls him guru, Tufte in 1983, says an excellent
graph is one that gives the viewer the greatest number of ideas in the
shortest time with the least amount of ink and the smallest space.

And I think in the EIA we\'re trying to do that and follow some more of
Tufte\'s advice on graphs should show the data without distortion, serve
a reasonably clear purpose, induce the view to think about the substance
rather than about methodology, present many numbers in a small space,
and encourage the eye to compare different pieces of data, to reveal the
data at several levels of detail, make large data sets coherent, and be
closely integrated with the statistical and verbal descriptions of a
data set.

And now for the results of our test using that as a background, I\'ll
start with Bob Rutchik, did a report of test on world demand and world
oil prices. And Bob began with a dual Y‑axis graph with the nominal
dollars per barrel on the left, and million barrels per day on the
right.

And we basically, all of us asked questions on, do you get the main
message, and they asked them to read, asked a question at a particular
point in time, and asked them for the answer.

And along with this graph, Bob showed them \-- have we got the other
one? Okay it is not on there, so you can back up. Bob showed one graph,
which I don\'t have on here, it had \--

MR. RUTCHIK: \-- from 1970 to the present time and had one line for
demand and one line for world oil price and it was cumulative, so what
happened here was built on the year before that, and on one graph where
you could see whether it went up or down from one year to the next.
That\'s the one we don\'t have up there.

MR. MILLER: And then at the NEMS conference, when we talked to the ASA
last fall that they said that the percentage change was not the best
alternative to show the results of a graph. So then Bob changed the
graphs and showed the percent change for the demand on one graph and
then the percent change and price on the next graph and asked them
questions on that.

Most of the people could see the big picture and they could see the
price and they could see the demands, but they didn\'t understand
percent change. So Bob concluded at that time that the dual Y‑axis
graphs confused the readers and percent change is not the best
alternative. Readers do look at trend lines.

The next test was done by Howard Bradsher-Fredrick on the Residential
Energy Consumption Survey, and Howard tested stacked bar charts and
basically he asked them the ability to read values from the figures and
if they could tell whether the values are increasing or decreasing. When
you look at that and try to figure it out for yourself, you can imagine
what the results were. And he tested this the same way.

And then Howard changed it to multiple bar charts and then he asked them
the same questions, the ability to read values from the graphs and the
ability to assess whether the values are increasing or decreasing, and
got much better results.

His conclusions were that regardless of the type of chart, that they
made numerous mistakes in reading data from the stacked bar charts and
the multiple bar charts, but the ability to discern trends was much
better for the multiple bar charts compared to the stacked bar charts.
And for stacked bar charts, even expert users tended to add the values
together rather than visualizing the quality separately.

And I think last fall we might have used the work dummied-down or
something at the meeting, but I think as a result of these tests that we
found that it probably \-- rather than look fancy and look nice maybe we
should make them a little more simple.

And Colleen tested the Crude Oil Production and Reserves Graph. And I
think when you see the graph that Colleen tested, Lower 48 Crude Oil
Reserves and Production was the title. And if you\'ll notice on the left
Y‑axis, crude oil reserves, billion barrels. And this was really
confusing to the people because on the left was 10 times the amount of
the volume on the right. They were both billion barrels, but one is 10
times the amount of the other.

And I think that if you look at what Collen did \-- I\'m not sure of the
title \-- she put a title on, Crude Oil Production Follows Proved
Reserves, Although Reserves are Ten Times Production Level. And then
you\'ll see how she stressed the point that she wanted them to see and
incorporated with one on top of the other.

And in the tests she found that at least about half of them got the
message from the title and the questions that she had asked. And she
concluded that the graph should have a descriptive title that includes
the message of the graph, or at least half of them. Dual Y‑axis graphs
are confusing, and use readable font sizes. I think when you get to be,
at least my age, a lot these young people creating graphs don\'t realize
that we can\'t, we don\'t even try to read them, we just skip over to
something else.

And that the bubble boxes on the first graph that Colleen had, these can
be helpful, but if you get too much on a graph, then it becomes
cluttered and they don\'t pay attention.

And Renee tested Ethanol Production and Corn Prices. Renee had 1995 corn
prices and ethanol prices side by side like this, price and production
for \'95, and then the next graph for \'96. She tested these graphs and
then the next graph, she had Ethanol Production Decreases as Corn Price
Increased \'95 through \'99.

And on the dual Y‑axis, she had dollars per barrel on the left and
thousand barrels per day on the right with production by line and then
the corn price by bars. Compared to the previous one with just a
straight line, that they preferred having those, I\'m not sure what you
call them, but spots where they could at least measure something for the
year to match the tick marks.

And Renee concluded \-- this is a little different than the others \--
that when they had something like price and volume, that dual Y‑axis
graphs could be useful in conveying the relationship between price and
volume. And for those of you that attended the NEMS conference, even
people outside of EIA are using the dual Y‑axis graphs more and more.

I know a couple of years ago we thought we could try to stop them from
doing that, but more and more people are using them and they\'re telling
us that it shows them what they\'re looking for.

Titles \-- again Renee and Colleen got the same results. Titles helped
users get the message some of the time. I think Renee found about 50
percent. And the same thing again on the fonts. Make fonts large enough
to be readable for users of all ages.

And Renee found that when she asked them, most of the people familiar
with our data, know that they\'re data behind the graphs, so they like
to look at the graphs to see trends, things like that, but if they want
actual numbers, precise numbers, they know to go to the data rather than
getting it from the graph.

And the last test I conducted, was Texas Oil and Condensate Production
and Texas First Purchase Price. And the graph that I tested first \--
this was the first graph I tested and last fall I even told people that
the MBOPD stood for millions of barrels of oil per day. I don\'t know
why I had that in my head, but it\'s thousands of barrels of oil per
day. But most of the people, when you asked them what it stood for, they
didn\'t know.

And another one that this graph had, it had a model on it. But the
people \-- what got their attention on this graph was the arrows. Can
you go back to the other one? Okay. The arrows on the production and
price decline in \'86 and again in \'98. So they got the message from
the graph from the arrows without even reading the title, and then if
they did read the title, they were asking me, what was Texas first
purchase price.

They also at the NEMS conference, naturally they didn\'t like the fact
the I took out the model, because since they were modelers, they
preferred to model and wanted to know did we tweak it along the way or
was it actually that accurate.

But they did pick up things on this graph which, like in the upper right
hand corner, what does have to do with the graph. And they asked me to
explain, you\'re trying to get me to see that the production decline
goes down, but yet when price goes up, production still goes down, so
explain that.

So they do \-- there are a lot of things in the 1 percent reduction,
little things on the graph that gets their attention and they want to
know why.

And the conclusions that I had \-- every time I did this I changed my
opinion on the dual Y‑axis graph. I know that it seemed like we thought
they were supposed to be bad, but everybody\'s using them and I think
Renee\'s is \-- if they\'re done right people can get the message. So
I\'m confused myself on dual Y‑axis graphs.

And the tick marks \-- could you go back Bob? The tick marks, you can
see at the bottom, when I got like 10 dollars on some and 17 on the
other, the arrows showed them, that they thought that January \'98 was
this tick mark, so I think that when people didn\'t know where to go to
find the tick marks that that\'s one of the conclusions and to convey an
accurate number, a table is preferable.

And the recommendations \-- at least we can come up with some guidelines
\-- I\'m not sure we want standards \-- but graphs should communicate
one major finding or concept per graph because graphs can portray a lot
of things. They can see trends, they can see other things in a graph
other than one message, but at least one main message.

Graphs should not contain acronyms or abbreviations. If you\'re in the
EIA, you know we use like NEMS, RECs, CBECs, RTECs, MECS, and we
understand them, but we\'re not sure people outside of EIA know what
we\'re talking about.

Consider other types of charts to replace stacked bar charts. Of course,
we didn\'t test pie charts, so there are probably other things that you
can add here that we didn\'t test.

Use of different colors. It\'s amazing how many people saw the graphs
and saw the colors like green with a line and then they said if we had
made the Y‑axis green then they would have automatically gone over there
with a price. And that\'s fine if you have color, but then if you Xerox
and you get black and white, it isn\'t going to be advantageous.

Use plain language, proper grammar. I don\'t know what you do with
something like Texas First Purchase Price unless you put a definition at
the bottom. And there again, we found that they don\'t read footnotes,
so don\'t use Y‑axis graphs when both scales are volumes and one is a
multiple of the other one, like Colleen\'s, when one figure was 10 times
the other.

Because then she asked them, when did reserves fall below production and
right where they crossed, even though one was 10 times the other, they
knew where they crossed.

And make sure ticks are readable and the graph scale should be zero
based.

And what we\'d like the opinion of the Committee, is if you could give
us advice on how we can use our conclusions to get EIA to improve their
graphs to adopt best practices, and are there any other suggestions on
how to implement the findings in EIA.

CHAIRPERSON CRAWFORD: We have two ASA discussants for Herb\'s work. The
first one is Johnny Blair.

MR. BLAIR: I am going to talk for just a few minutes about the
methodology of the cognitive testing. Those of you who have been around
instrument development and other kinds of forms development and such,
know that cognitive methods have kind of permeated the different federal
statistical agencies and you would also know that there is really no
uniform understanding of what constitutes cognitive testing, that there
are different flavors of it. People take different approaches. Some
good, some not so good.

I think that the approach that was taken here, and this is described
really on the web site rather than in the presentation that was just
given, but the approach taken here I think is a quite sensible one and
that it produced useful information with really a reasonably small
amount of effort.

But because there\'s not a uniform definition of these procedures, I
think that it\'s important in writing up the results of this kind of
testing, to provide a bit more detail about exactly what was done. One
guideline that I think is sometimes useful for these kinds of tests is
that the reader, if he or she wanted to, should be able to use the
description of the process of the testing to replicate it, if in fact,
that was desired.

But not so much providing it, because people are going to replicate it,
but I think that that\'s kind of a reasonable standard.

So to just give a couple of specific examples, to know exactly how the
purpose of the testing or the exercise was explained to respondents, to
see the actual script that was read to respondents and how they provided
their answers.

I think all of these kinds of things, if added to the description makes
it more useful to the readers, and also makes it possible then for
people to suggest modifications or perhaps improvements.

At the same time I should I guess interject that I\'m probably being
somewhat unfair to the researcher because that may not have been the
purpose of the write up that was put on the web site, and so I may be
judging it from a different standard.

But I think that when something is put up on the web site, it is
essentially published and even though it may be thought of as kind of a
quick informal description, that many of the people who look at it are
going to judge it by a different standard.

One other thing as far as the presentation of the results again, as on
the web site, and the description of the findings. At least in a couple
of the summaries, the researchers use percentages of respondents that
understood a particular task or had a certain difficulty, so there are
lots of things, like nine percent of respondents understood what the
main purpose or the main point of the graph was, and so forth.

This isn\'t in all of the reports, it\'s in a couple of them. I would
just suggest being fairly cautious and conservative about using
percentages with this kind of testing.

The sample sizes that were used are perfectly normal and typical for
this kind of work, but they are small. The ranges roughly, I think most
of the tests were done with 10 to 15 respondents. There was one with 26,
but most were in the range of 10 to 15.

And I think that when you start putting percentages on that
unintentionally, that you may imply a level of precision that really
isn\'t there, that the data really doesn\'t support, and I think those
were shown in some of the other reports that didn\'t use that approach.
These kinds of results can be explained without resorting to percentages
or other statistics that for these tiny sample sizes it would probably
be not really that useful or appropriate. But that\'s a small sin if a
sin at all.

The other thing that I wanted to mention, and this may be something that
the researchers are aware of, that there is a small literature that I
think is very applicable to this work and I think would probably
complement the kinds of things of Cleveland and Tufte that were
mentioned. And this is on cognitive aspects of statistical mapping.

A group of researchers at the National Center for Health Statistics,
guess this was about five to six years ago now, I did a fair amount of
work and publication on tasks and testing that\'s very close to the kind
of thing that\'s being done here.

Here\'s with presenting statistical information as part of maps, in
geographic areas. But to convey information, in that case, of course,
about health issues and epidemiologic kinds of data. But a lot of the
methods that they used I think might be worth reviewing and might be
applicable to the work that was done here.

They explored, for example, and then they just had to give up to give a
quick flavor to this. The use of combining cognitive interviews in focus
groups for evaluating graphics. They used warm-up exercises to bring
respondents up to a similar level of familiarity with the type of
graphics that they were going to be exposed to.

And finally, they also made some use of more general models of the
response process to examine possible reasons for difficulties that
respondents might have, so that this is really \-- and for those of you
that kind of followed this area in development \-- you know, the
standard response model of comprehension and retrieval and response
formation and so forth.

And what they did in their work was to kind of expand this and apply it
to the understanding and interpretation of graphical images or of forms
so that they were able to use their work to separate comprehension, that
is, understanding the graphics display and objectives from reasoning
that is drawing conclusions based on it, and decision making and sort of
taking that information and how would one apply it.

And these are things that I think complement the work that\'s being done
and I expect that you\'ll be doing more of this rather than suggesting
that this is something to do instead of what\'s being done. I think that
these are things that would complement it.

And I should mention that work from that group is published in a number
of places and the proceedings of the STAT meetings, the joint
statistical meetings, in a journal called Journal of Visual Cognition.

Then lastly, I just wanted to suggest one additional measure that might
be considered. The amount of time that it takes someone to comprehend a,
whatever, a graphic in this case, or to answer questions about it and so
forth, sometimes they\'re in the survey jargon called response latency,
that\'s often a very good measure and a fairly simple one of the
difficulty of dealing with a particular cognitive task and the burden
and so forth.

So again, I suggest this is something that perhaps in future work that
you might want to consider as a measure that could very well be
integrated with the kinds of things that you\'re already doing that
would provide a lot of additional information, potentially at very
little, almost no additional cost.

To sum up, I think that, again, the work that was done and the results
from it I think are a good example of using cognitive testing to inform
the design of graphics.

CHAIRPERSON CRAWFORD: Our second discussant is Nicolas Hengartner.

DR. HENGARTNER: While it fires up what \-- it turns out I\'ve downloaded
probably a previous presentation you made from your web page. And so
essentially what I\'ve done is I\'ve looked a little bit at the graphs,
and I want to discuss some of the aspects, essentially the visual
aspects.

And I have to say, a lot of the conclusions, all of the conclusions it
came to, I agree with them. And especially one of them, the one thing
I\'ve learned is that as a teacher, I need to teach my students to read
the labels.

I was shocked to learn that 50 percent of the people didn\'t read the
labels. And so this isn\'t something EIA should do, but rather us, the
professors in academia should teach the people about that. So that\'s
one thing I retained.

So essentially, after your presentation I asked myself, well what is a
good graph? And essentially, we have to ask ourselves, what are the
purposes of a good graph and you do ask these questions. And the purpose
is possibly to look at trends and changes in trends.

Also to look for the parts of familiarity. If the hypothesis is it\'s a
curve, it\'s going to be hard to see the parts from that, but if
something straight, you\'re going to be able to see that.

And finally, there\'s possibly wide effects seeing that one curve drives
another one. And the main regions, and here I\'m going to make two
distinctions, one is relevant data. If you don\'t have the right data,
there\'s now way you can tell a story.

And the other one is the visual display, and here in the visual display
I have three sets of things and one is the title, the labels and the
legend, the other one is the draft, and third is what I call crutches.
And those crutches are those little arrows or little indicator which
really helps the deficient people, like myself, to actually get the
message. And all three of these are important.

The one thing is that which graph should one present? Well that depends
on the story you\'re trying to tell. In fact, no single graph does it
all. And I think that\'s a little bit the story of those dual Y‑axis
graphs is that, in fact, you want to present two graphs.

And there\'s a lot of it which is \-- the same thing with those multiple
bar charts. What you want to do is, actually present more than one graph
at the time and in some sense one gets to that.

So what do I mean by relevant data? Well very often when we make a graph
we know much more than the reader does. And here\'s a good example. I
took that \-- they show a graph of percentage of OPEC share with prices
and then they say, for example, as oil prices rose dramatically,
production increased, thus reducing OPEC\'s market share.

Well, that\'s not true. I only see the change in percentage. I actually
don\'t see if the percentage of OPEC increased or decreased or non-OPEC.
And so this is actually \-- I know you\'re right because I\'ve seen some
other graphs afterwards and indeed this is what\'s happening. But
there\'s more information that you require to make the story go here.

So the other thing when I say, what\'s the relevant data, many graphs
are going to have the nominal dollar amount, and the question is, is
that what you want, or do you want the constant dollar. And I know this
is something that comes up all the time again in those discussions.

Titles and legends. Sometimes the titles are too suggestive. Sometimes
you\'re trying to lead someone with the title. I know I don\'t \-- this
type is very small. So I\'m going to read it for you. Crude oil reserves
are ten times production levels have followed the same trend over time.
And then I stopped to think about it.

So we\'re looking at oil reserves and I\'m looking at production, and
I\'m thinking, if I\'m producing something the reserves should decrease.
So why should those follow one another. I mean it\'s more like the
production level seems to be more like the slope of the change in
reserves given that there\'s no new discovery. So one has somehow to ask
ourselves if really that\'s the kind of thing we want to convey,
although it is true at the end of the graph there seems a small decay.

On the other hand, that is a good graph, there are good things on that
graph as well.

So let\'s go to the chase, those dual Y‑axis graphs. Here are a couple
of rules of thumb I think. They are, keep it simple. And what I mean by
that is, don\'t let those curves intersect. There are two curves, a red
one and a blue one, and they\'re on different scales. There\'s no reason
for them to cross each other and chase each other, because there\'s too
much action. You get distracted.

So there\'s no need for the lines to cross. As a matter of fact, I would
say, if you can segregate the ranges, you still can have the dual Y, but
as long as the ranges are segregated, that is a good thing.

For example, there\'s vertical ordering. Now, if we look at those dual
graphs, there\'s one on top typically, one on the bottom. That\'s how
they are presented. Well there\'s a reason \-- or you can think of which
one goes on the bottom. The one on the bottom should be the driver.
Because people \-- at least when I look at that, I\'m looking at bottoms
up. I don\'t know, that\'s how I\'m thinking.

And it might be something to test in the future. Which order of the
graphs should be to be able to tell such a story. And here the visual
crutches are good. They say, there\'s a point here and you see the other
one, and they really tell the story. So I like that example.

Then with the dual axis there\'s a danger. Not only do they cross, but
there\'s different variability in the curves. For example, in this one
here, one wanted to see what happened to world demand, and in the upper
part there was a story that there was a slowing in production in the
\'80s and another slowing in the end of the \'90s because of crisis, and
so you see those little changes and this is part of the story.

And at the same time you should be able to argue from the same draft
that this wiggly line down there had nothing in there, that was noise.
And so you look at this and they\'re different scales. And it\'s very
hard to see that when you have those dual graphs. So I don\'t have a
solution actually for this. But this is one of the pitfalls of doing
this.

There\'s also something else. Multiple bar graphs. I love them.
Honestly, this is the great way. There\'s one better thing, well one
more complicated thing which are mosaic plots, but most of the bar
graphs are good. But if you\'re going to compare different graphs
together, put them on the same scale.

It turned out, and also in your presentation, on this one there were 80,
88, and the other one was 66, the maximum. And so I know the software
just spits out the graphs. Force the scales to be the same because not
only do you want to read off the things, but you want to have visual
holistic view all the data and if they\'re not on the same scale you
might actually get wrong impressions.

So conclusions, I agree with all your conclusions. And the first
conclusion I have is I\'m going to need to go back and teach that to my
kids to read the labels. And I think that\'s a very good conclusion.

Then visual information in graphs. It has to be obvious. If it\'s not
obvious, you need too much explaining to do, that means you probably
don\'t have the right tools. Dual Y‑axis are okay if you segregate the
graphs, if they\'re in different ranges. Then I think you can make it
work.

And the other thing, you should not expect people to read off graphs.
They said people read 21.1 and that was good, at 21.3, and that was up.
I expect the 5 to 8 percent error rate when people are reading off
graphs.

I mean, I see now my little daughter, she\'s six years old, and they
teach them to guess. And I think that\'s what we\'re taught, just
looking at the graph and guess the values. Because you\'re not giving
the numerical information. As you said, if you want them, you go back to
the tables.

And finally, I think I had a lesson in humility, which is actually a
good thing because it shows that professionals who think they know
things, in their reports it is mentioned that statisticians got all more
or less fooled by one of the graphs and they were certain that they were
right when they were completely wrong. Where some other people with less
training said, well, maybe, but I\'m not quite sure.

So it was a good lesson in humility and I thank you very much for a very
interesting report.

CHAIRPERSON CRAWFORD: Herb, did you have any comments or questions or
anything you\'d like to add?

MR. MILLER: No, I appreciate all the help that you\'ve given us.

MR. RUTCHIK: I\'d just like to make one comment, and Herb touched upon
this. I\'m just going to elaborate on it.

Dr. Hengartner talked about really teaching his students to read labels
and a couple of our findings that people read certain things like labels
half the time, and Herb talked about that.

I think and I found this with my graphs. The last test I ran, some of
the people had problems with cumulative percents because they didn\'t
know whether I used cumulative or from a base year even though it was
described in the title. But the point is, I think these findings that
people don\'t read fit into other findings.

Tufte talks about putting things in a smaller space and a lot of people
talk about keeping it simple, and also in area forms design Don Delmen
talks about eye span. People only read certain amounts. And we found
this in cognitive testing of the EIA forms, that people just blow over
things. They don\'t read instructions. They don\'t read certain things.

So I think what we found in this particular instance about not reading
labels, fits into this larger context and to me and my colleagues, think
this is one of the major things we kind of re-found. And Cowling, Renee,
you have anything to add? Okay, thank you, and thank you both for your
suggestions. It was great help.

MR. MILLER: One of the things that Johnny said about that when we met at
the NEMS conference and got the data, then we had actually a week for
them to get the information to me and I was told that we were going to
give this information to the Committee to give them something so they
can \-- so I have never told the Committee any of this information is on
the web site.

So if we had any idea that that\'s where it was going, we probably would
have met and made sure that they were all similar, that we did the
things \-- we hadn\'t met to discuss it yet, about our results, so they
were all done separately and not as a team. We would have done that \--
we will do it. So I appreciate your help.

CHAIRPERSON CRAWFORD: Randy.

DR. SITTER: There\'s a lot of emphasis on use of color in graphs. Only
one of the graphs actually uses color to make the title stand out. I
think it\'s something, that if you\'re going to put the message in the
title, making it small and black point while you make the graph colorful
and eye catching, kind of draws you to the graph first and the title
second.

The one that you have that you thought was a good success story was to
combine the line graph with the bar graph on the dual Y‑axis graft, has
the title in red. At least for me, it\'s the one graph where it\'s
almost impossible not to read the title first. It just sort of grabs the
eye and I immediately read it.

CHAIRPERSON CRAWFORD: I have a question. Do you have any suggestions for
black and white graphs? Because most of the statistical journals at
least, they don\'t publish in color. So if you make those line graphs
and you have three lines that you want to compare, it\'s hard to do it
in black and white.

MR. RUTCHIK: This won\'t help in all graphs because it depends on the
spacing of the data lines, but if you have enough room, you could put
the label of the line, one\'s price, one\'s demand, and one\'s something
else, right on top of the line, that would be the best way. Because then
the label, the metadata is right there with the actual data and people
can see it all at once instead of having to go back and forth from a
legend and with all the cognitive problems that that entails.

CHAIRPERSON CRAWFORD: Thank you.

DR. HAMMITT: I want to say two things. The formal testing of this
business, I suspect all of us, in working real graphs all the time, we
all have our own opinions of what works. And this one where you applied
bar graphs with a line on top of each other, I don\'t like that because
it seems to me by using a bar in one case and a line in the other,
you\'re trying to tell me these things are different in some sort of
conceptual way.

Whereas both of them are either discrete measures, or they\'re both
continuous time series depending on which way you want to look at it. So
that\'s a kind of distracting inference I think.

MR. MILLER: Renee, do you have \-- that was the ethanol and corn
production?

MS. MILLER: Yes, that was the ethanol and corn production, and people
either really liked it or they didn\'t. I mean, some people said that it
made it easier for them to follow having the bar and the line, and
others said, why did you do that?

DR. COWING: You talked about comments for guidelines or rules on good
graphs in the future. I would hope that you don\'t adopt guidelines
about what I would call input, how to do things. For example, if you\'d
done this a year ago I suspect one rule would be no double Y‑axis
graphs, and now that turns out probably to be wrong.

And I\'m sure we can think of lots of other examples where we might have
rules that actually kind of miss the point. If we\'re going to have
rules, first of all, I\'d be a little more flexible and call them
guidelines.

And secondly, I\'d have them focused on what it is graphs try to do and
that is output to be clear that they present information, they have to
be eye catching, and then let people decide what works for them and what
works for this particular topic rather than having very strict rules
about what you can and can\'t do.

DR. WHITMORE: I was going to say the same thing. I\'m just going to add
just a little bit to what Tom was saying. You\'re talking about how you
get the EIA to learn from this as an agency, and I think certainly
thinking about and publishing some of the guidelines as opposed to rules
and regulations is one way of approaching it.

And then maybe having some requirement that people provide some feedback
that they actually looked at the guidelines or considered the guidelines
in developing the graphics. Kind of emphasize the importance of that.
But I don\'t think you can have any hard and fast rules and I agree with
that.

CHAIRPERSON CRAWFORD: Are there any other questions or comments? Okay.
Thank you very much.

We\'re going to take a break now and then pick up with our last set of
breakout sessions. Two logistics, we\'ll start the breakout sessions at
2:40 and we\'re going to end them 10 minutes early, so contrary to the
agenda, we\'re going to end the breakout sessions at 3:50 instead of 4
o\'clock.

Electricity 2002 is in the Red room. The Disclosure Auditing is in the
White room. And Verifying an Electricity Model is in the Blue room.

(Whereupon, the foregoing matter went off the record at 2:15 p.m. and
went back on the record at 3:52 p.m.)

CHAIRPERSON CRAWFORD: We are going to do this in different order. We\'ll
start with Mark Schipper who\'s going to talk about the discussions for
the disclosure auditing system.

MR. SCHIPPER: This was a year-long process. Last year this time, I met
with this group to discuss the development of an audit software that
deals with basic confidentiality and data tables. We have at EIA data
tables that we release and our protective with confidentiality, and we
suppress sales in those tables to protect the confidentiality of the
individual respondents.

We either do this because of promises to the respondents, or because of
legal requirements. But in those suppressions, we have no way to
evaluate the suppression pattern. To do that evaluation, we\'ve in the
past year worked on the audit software which determines the minimum and
maximum and of the average value for the suppressed sales that exist in
EIA tables.

It\'s a long process which uses operations research techniques to
basically qualify the suppression patterns that we use in EIA reported
data in tabular form.

With that model, we asked the Committee to look at certain attributes of
our data tables. We have independent rounding which exists in all our
EIA reported data tables that I\'m aware of, and that rounding process
provides a lot of protection. I don\'t have my slides up here, and we
went through a nice demonstration of ways in which we can break the
confidentiality of certain tables. And it was quite enlightening in
terms of the outcomes.

But the question I had for the Committee, which I got a lot of feedback
on, was how can we address this independent rounding without assuming
maximum independent rounding. It came to my knowledge through Jay, that
there is a distribution called the Beneford distribution, which Jay had
asked me a question.

He says, well if you went to a newspaper and looked at a beginning digit
or a decimal digit, what\'s the probability of picking a one? I said,
probably a tenth or actually one through nine, so it\'s the uniform
distribution.

He said, well, you\'re the perfect common man, because it\'s not.
There\'s a distribution which he related the experience of this \-- how
it developed.

But in terms of feedback, I guess without going through a demonstration
of this audit software for the Committee, which I don\'t think we have
time for, I guess I can give feedback that the inclusion of one, the
Beneford distribution, or at least employing some type of varying
rounding process. Will one improve the EIA audit, and I should mention
that EIA is a lead agency in this auditing process, but there are seven
other federal statistical agencies involved. If I could list them off
the top of my head, it would be the Bureau of Labor Statistics, the
Bureau of Economic Analysis, the National Science Foundation, the
Internal Revenue Service, the Bureau of the Census, and National Center
for Education Statistics. I hope I got them all.

They\'ve all contributed their staff or money, actually all of those
have contributed money for this project, and I guess to do all the
plugs, it funnels out of the Federal Committee of Statistical
Methodology which is also a subcommittee of the Confidentiality and Data
Access Committee.

If you would like to learn more about the committee, it\'s www.fcsm.gov.
And if you are interested in the audit software, I\'m more than willing
to provide demonstrations I guess with a larger group on how it runs. In
fact, Carol had mentioned doing something for ASA in an upcoming future
work. Any questions?

CHAIRPERSON CRAWFORD: Questions? Comments? Additions? Okay, thanks Mark.
Have a good trip to New York.

Our next summary will be provided by Dean Fennell on Electricity 2002.

MR. FENNELL: My presentation was involving the redesign of electricity
2002 data collection effort. And I basically had three questions for the
Committee. The first one was, in terms of appropriateness of data
elements, should we be collecting the information in detail?

And basically the Committee\'s comment is that they\'re really a
statistical group and it was difficult to make a comment on that,
particularly since they hadn\'t seen the forms. But after going through
the presentation and how we got to where we were, basically the thought
was that we had taken the right steps and we appreciate that comment.

The second question dealt with the appropriateness of data elements in
terms of disseminating and aggregate. Part of our project here is not
only to determine the data elements, but also which elements should be
considered confidential, and we walk a fine line between whether or not
we\'re going to get input from our respondents if we determine that the
information will be public and they don\'t want it public.

Some of the ideas that were discussed for this element, one of them was
a sharing agreement between different agencies, which we have been
doing. There was also discussion, BLS is doing that with some of the
states.

A lot of the state collections were dropped because EIA\'s data
collection effort was so good and now we\'re considering making
information confidential, so that brings up and issue that a lot of the
states have addressed. So that\'s one of the things we\'ll look into.

Also, another comment was, in terms of systematic information collected,
go out with maybe a survey to go out and ask different entities about
the issue of confidentiality. And we will certainly look into that
between now and August, and if we find that there isn\'t time for that,
we certainly will consider that for the next clearance in three years.

And certainly the last question dealt with, will these disseminated data
be sufficient for the data-using community? And we sort of talked about
that in the other two questions. As we mentioned, it\'s trying to find a
midpoint between what we can collect in detail and put out to users.

And basically it boiled down to, one individual brought up the issue of
mandating, who do you serve and EIA\'s legal responsibility as to who we
serve is the government and those folks that collect the data. So right
now the information is in the Federal Register Notice. It\'s out for
comment. The comments are due in by the middle of May and once we get
all the comments, we\'ll make further determinations from those
comments. That\'s it, thank you.

CHAIRPERSON CRAWFORD: Thank you. Any questions or comments, additions?
Okay, thank you. Our third summary or if you will is Ray Pang, who will
talk about some of the frequently asked questions about survey response
rates and that discussion.

MR. PANG: I presented a project for OMB. It\'s the frequently asked
questions about the response rate. And we talk about one estimate model,
which is a regression model being developed in 1986. And based on that
model, several factors that can be determined are what\'s their expected
response rate.

At the end, I have several questions for the Committee and we didn\'t
have much time to get into that, so we just got to the first one. When
is it important to get a high response rate, and when does a high
response rate matter? When doesn\'t it matter? And the Committee
provided a lot of help on this and I will summarize as follows.

First of all, we will look how different the respondents and the
nonrespondents, and maybe we can get into the late respondents versus
earlier respondents as a part of the solution.

Since we don\'t know what\'s a nonresponse and we may have to conduct
another small follow-up to get some information about the nonrespondents
and then compare with the respondents. Also, we can do some
characteristic in subgroups and set up the approach to evaluate the
response rate.

Third, the cost implications. We have to put the costs in the beginning
of the survey and then see how we can reach the response rate. And that
way in relation to the analysis.

Also, we will look at some special, hard-to-reach populations to see if
that\'s a special case, what would be the acceptable response rate. And
the model of the estimated response rate may be misleading because you
may get a response rate out of the range, which you cannot do anything
about. That summarizes the Committee input.

CHAIRPERSON CRAWFORD: Does anyone else from that breakout session have
anything to add? Clarify? Any questions?

Okay, thank you. Finally, Douglas Hale will summarize the discussion on
Verifying an Electricity Model.

MR. HALE: Hi. We had a little session on verifying an optimal power flow
model. I think as most of you know, over the last three years we\'ve
developed a series, an ever larger series of models of the eastern
electric power industry. We started with New England, then expanded that
to New York, New Jersey, Pennsylvania, and most recently in the series
of models, we\'ve developed one of the entire eastern interconnect.

In these models we basically represent the electrical system down to the
level of a 68 KV line. That is something you might see at a substation
close to your neighborhood.

We then also modeled essentially all the generators in the eastern
interconnect. The eastern interconnect by the way, is everything west of
the Rockies, exclusive of Texas. So we\'ve got a lot of stuff out there.

We\'ve used these models to do a couple of things. One is to try to get
a feel for how large particular markets are. It\'s not at all clear when
you say you\'re looking at New York and trying to do an analysis or
competition in New York, what the set of potential suppliers are. Are
they in Pennsylvania or are they just in New York or do they include
Canada?

The only way to tell is to actually have an electrical model to see how
much flow of electricity you may potentially have.

We\'ve also used the models to take a look at the opportunities for
raising prices above competitive levels. That\'s a fairly hot topic
these days. Generators have proven to be pretty good at it in the west,
and the question is, can they do it in the east. And the short answer
is, if you\'re in Florida, you\'re a very happy monopolist, and if
you\'re in certain other areas, it depends on what the supplies are like
in the rest of the east.

These studies have been encouraging. They\'ve been useful I think. But
the question still is, how accurate are they quantitatively. I\'m pretty
confident that we\'ve got the qualitative part of the equation right.
The directions of changes are right. The question is, are the numbers
all that good.

So to get at that, we selected one particular part of the country,
Pennsylvania, Maryland, New Jersey, and Delaware, and tried to compare
the prices coming out of the model with those prevailing in that market
on two days. Actually only one day so far, June 15th.

We picked June 15th because it was a relatively easy, easy day in which
prices change by hour, but prices within PGAM were the same everywhere.
Quite often electricity price literally two blocks away can be very
different depending on the state of the system.

We did our initial comparisons a week ago and we were able to follow the
pattern of prices very well. And especially in the early morning hours,
up to about 7 o\'clock, we followed actually prices within 10 percent,
something like that. And from probably about 4 in the afternoon back to
midnight we followed prices pretty closely.

It\'s about 10 hours in between where prices were increasing and we
systematically underestimated prices. So the question is what should we
be doing, what should we be looking at?

There are at least two suggestions that made an awful lot of sense. One
was to take a look at whether generators are actually withholding in the
day-ahead market which we were studying. We really hadn\'t made
provision for that. If they are withholding, the prices will go up, and
maybe we\'ll do better.

The second, after doing some more cleaning up of our data sets which we
need to do in any case, comparing, trying to also replicate the
real-time pricing market. What we\'re doing now is a day-ahead market,
and seeing how we do there.

In fact, if there\'s withholding in the day-ahead market, maybe under
our assumptions we may even do better in their real-time market
surprisingly enough. We\'re going to check that out.

The third suggestion was to get off my dead butt and model California
while people are still interested. There\'s a lot of pressure to do that
and we probably will. It just hasn\'t happened yet.

I would like to thank the Committee for sticking with me through these
years of development of these models. The help, and the encouragement
has really been critical. Because anytime you\'re doing this sort of
stuff, you end up in a closet basically a lot, just working problems
out. And coming here occasionally and finding that other people think
this is actually interesting and useful has been a great source of
inspiration. And the technical points have been very, very good, so I
want to thank the Committee for that.

CHAIRPERSON CRAWFORD: Does anyone else from that discussion group have
anything to add or questions for anybody? Thanks, Doug.

Are there any other Committee questions about any of the topics today?
Any other topics? Are there any questions or comments from EIA? Finally,
any comments from the public?

Before we adjourn then, I just have a couple of administrative things.
Remember our dinner tonight is at 5:30 at La Brassarie, which at 237
Mass. Ave. Northeast. So if you\'re not at the Holiday Inn Capitol, or
you wish to do your thing, just show up there sometime around 5:30.

For those of you that are at the Capitol I thought we could walk back
and maybe meet in the lobby at 5:15 to catch a cab to go there.

Last note, you may leave your materials here if you want. I wouldn\'t
leave anything valuable, so take your purses, wallets, laptops, and
those kinds of things with you and bring them back tomorrow.

Okay, thank you. This concludes our meeting for today.

(Whereupon, the above-entitled matter was concluded at 4:12 p.m.)