AMERICAN STATISTICAL ASSOCIATION

\+ + + + +

COMMITTEE ON ENERGY STATISTICS

MEETING

\+ + + + +

THURSDAY

NOVEMBER 19, 1998

The Committee met in Room 1E-246 of the Forrestal Building at the
Department of Energy, 1000 Independence Avenue, S.W., Washington, D.C.,
at 8:30 a.m., Daniel A. Relles, Chair, presiding.

[PRESENT]{.underline}:

DANIEL A. RELLES Chair

CAROL GOTWAY CRAWFORD Vice Chair

DAVID R. BELLHOUSE

CHARLES W. BISCHOFF

JAY BREIDT

R. SAMPRIT CHATTERJEE

JAMES HAMMITT

GRETA M. LJUNG

POLLY A. PHIPPS

SEYMOUR SUDMAN

[ALSO PRESENT]{.underline}:

ERIN BOEDECKER

STEPHEN CALOPEDIS

LYNDA CARLSON

SAM COHEN

JOHN COLLIGAN

CATHY DIPPO

STAN FREEDMAN

DWIGHT FRENCH

JAY HAKES

MARY HUTZLER

W. CALVIN KILGORE

PERRY LINDSTROM

HERB MILLER

RENEE MILLER

BETSY O\'BRIEN

LARRY PETTIS

ARTHUR RYPINSKI

HANK SATTLETHIGHT

JOHN SHERWELL

BILL WEINIG

[C O N T E N T S]{.underline}

[PAGE]{.underline}

Welcome 5

Opening Remarks 14

ASA Committee Questions

Summary of Committee Questions

Dan Relles 26

Handling High-Profile EIA Reports

Mary Hutzler 28

Questions from ASA Committee 55

Update on Greenhouse Gas Environmental

Impact Studies

Overview and Update 71

Contributions to Greenhouse Gases

Arthur Rypinski 108

Discussion

James Hammitt 125

Phil Hanser (prepared text) 132

Questions from ASA Committee 143

LUNCHEON RECESS

Electricity Restructuring

Status Report of CNEAF progress

John Colligan 146

Betsy O\'Brien 164

The Federal Statistics Working Group 184

Cathy Dippo 184

Cal Kilgore 210

Renee Miller 225

Questions from ASA Committee 179

EIA\'s National Energy Information Center 237

[P R O C E E D I N G S]{.underline}

8:40 a.m.

CHAIRMAN RELLES: Welcome to all of you. I have several announcements I
have to make, formal announcements. I\'ll read this. This meeting is
being held under the provision of the Federal Advisory Committee Act.
This is an ASA, not an EIA committee, which periodically provides advice
to EIA.

The meeting is open to the public. Public comments are welcome. Time
will be set aside for comments at the end of each session. Written
comments are welcome and could be sent either to ASA or EIA. All
attendees, including guests and EIA employees, should sign the register.
We are asking for your e-mail address this year for the first time.

In commenting, each participant is asked to speak into a microphone. The
transcriber will appreciate it and so will I. Also, committee members
and speakers at the head table need to speak clearly and into a
microphone. These microphones have an approximate range of one to two
feet.

Next, I\'d like to ask you to introduce yourselves and give your
affiliation. I\'ll start. I\'m Dan Relles with the RAND Corporation. Why
don\'t I move around the table first?

DR. SUDMAN: Seymour Sudman, University of Illinois. For a year I\'m on
sabbatical at the U.S. Bureau of the Census.

DR. BREIDT: Jay Breidt, Iowa State, but this semester I\'m at the BLS in
the Census Bureau as an ASA (inaudible).

DR. LJUNG: I\'m Greta Ljung at the IBHS in Boston.

DR. BELLHOUSE: David Bellhouse, University of Western Ontario.

DR. BISCHOFF: Chuck Bischoff, Binghamton University.

DR. PHIPPS: Polly Phipps, Washington State Institute for Public Policy.

DR. CHATTERJEE: Samprit Chatterjee from New York University.

DR. HAMMITT: Jim Hammitt, Harvard.

DR. CRAWFORD: Carol Gotway Crawford, Centers for Disease Control.

DR. CARLSON: Lynda Carlson. I\'m a Designated Federal Official and I\'m
at EIA.

MR. PETTIS: I\'m Larry Pettis and I\'m from EIA.

MR. WEINIG: My name is Bill Weinig and I\'m the liaison at EIA.

DR. HAKES: Jay Hakes, EIA.

MS. HUTZLER: Mary Hutzler, EIA.

MR. COHEN: Sam Cohen (inaudible).

CHAIRMAN RELLES: I\'m afraid the transcriber is going to get very upset.
If we could speak louder, please?

MR. FREEDMAN: I\'m Stan Freedman with EIA.

MS. MILLER: Renee Miller, EIA.

MR. LINDSTROM: Perry Lindstrom, EIA.

MR. CALOPEDIS: Stephen Calopedis, EIA.

MS. BOEDECKER: Erin Boedecker, EIA.

MR. JONES: Jeff Jones, EIA.

MR. SATTLETHIGHT: Hank Sattlethight, Aluminum Association.

MR. SHERWELL: John Sherwell with the State of Maryland and Power Plant
Research Program.

CHAIRMAN RELLES: Did we get everyone? Okay, thank you. Continuing the
announcements, there will a luncheon for the committee and invited
guests at 11:45 in Room 2250, immediately across the hall. We have
dinner reservations this evening at 7 o\'clock in the Oval Room at 800
Connecticut Avenue, N.W. I\'m told this is a very nice place.

Is anyone not going to dinner? What are people\'s plans? It would be
nice if the committee could have dinner together and it looks like \-- I
don\'t see any hands so it looks like we\'ll have a lot of people.
That\'s great. Maybe we can meet at the Holiday Inn about twenty minutes
of seven and take cabs over.

Breakfast for the committee tomorrow morning will be held here, again at
7:45, and our meeting will resume at 8:30 here in this room.

Now, this is the big announcement. For your information, Dr. Lynda
Carlson is the designated Federal Official for the Advisory Committee.
This official may chair, but must attend each meeting, and she\'s
authorized to adjourn the meeting if she determines this to be in the
public interest.

I\'m always fearful of reminding her of that. She must appear at all
meetings of the Advisory Committee and every agenda. She may designate a
substitute at her absence.

This is our first time meeting in the Forrestal Building. I\'d be
interesting in your reactions to this change and how you feel about
holding future meetings here.

I guess I\'d like to name a nomination for Vice Chair. The position of
Vice Chair has been open since Brenda Cox resigned and since I ascended
to the throne last year. I\'d like to nominate Carol Crawford for that
position.

Carol is agreeable. She\'s been a member of the committee since 1997.
She\'s made wonderful contributions and I recommend Carol to you and
that you consider her favorably for Vice Chair. If you agree, Carol
would become the Chair at the conclusion of my term, which would be next
year. So might I hear a second on that?

DR. BELLHOUSE: Second.

CHAIRMAN RELLES: David seconds it. So thank you. Are there any other
nominations? Would you like a chance to think about it? Okay, so all in
favor?

(Chorus of ayes.)

Okay. Congratulations, Carol, and my best wishes.

We have a lot of business to conduct before the meeting. I apologize.
I\'d like to set the date for the next meeting before we leave this one.
I\'d like to have you review the choices sent to you earlier and \"X\"
out any dates you can\'t make.

Our choices for the Spring Session are April 22nd and 23rd, April 29th
and 30th, May 6th and 7th, and May 13th and 14th. So those are four
consecutive Thursdays and Fridays and if you know now of dates you
can\'t make I\'d appreciate your \"X-ing\" them out and giving your
votes to me or to Bill at the first opportunity. And then we\'ll discuss
the outcome with Lynda and Jay and pick a date for the next meeting.

Okay, I\'m going to do something different this time and try to stay on
schedule. The committee is always in transition, so this year Jay Hakes
gets to say thanks and goodbye to the retiring members. He\'ll do that
in a minute as soon as I give up the floor.

But at this point I get to welcome the new members. So we have two: Jim
Hammitt is one and Jay Breidt is the other. Jim attended the last set of
meetings as a guest and has consented to join the committee for the
remainder of Brenda Cox\'s term, which will expire at the end of next
year.

Jim has a very long title in his resume so I truncated it to co-director
of the Program in Environmental Science and Risk Management at Harvard
and Associate Professor of Policy and Decision Sciences there.

Is that okay, Jim? Jim\'s interest has been in modeling, simulation,
global climate change, costs and benefits of carbon mitigation actions,
and health and risk effects of environmental damage. Jim and I were
colleagues and friends at RAND so I\'m especially glad to have him on
the committee from a personal perspective.

He and I used to run together too, but I\'ll have to tell you all about
my knee problems later.

Our other incoming member is Jay Breidt from Iowa State University. Jay
is a card-carrying statistician who\'s Associate Professor of Statistics
there.

He\'s visiting this year at BLS, which indicates his strong interest in
Applications, but his skills seem to be perfectly matched to the
committee\'s needs: interests in survey design, time series analysis and
methods, modeling and simulation, and he\'s done a few applications in
environmental statistics.

So we\'re very fortunate to have Jay on the committee and we thank him
for joining.

The last comment is that when you do want to speak please turn your
card, your tents, vertically.

Now I really should give up the floor but I\'m just going to say one or
two more words. I think it\'s important just to do a little
introspection of the committee and remember sort of who we are and why
we\'re here.

We\'re recruited by the ASA. I view the EIA as our customers; we\'re
here to help them. We are a bunch of scientists who bring skills in
survey methods, design methods, analysis methods, forecasting,
economics, but many of us don\'t know much about energy or energy
statistics.

There isn\'t a long list of statisticians belonging to an energy
statistics committee that we can actually run down to pick members for
this committee. So I think it\'s important to EIA to know that for many
of us, our world of energy statistics begins and ends with our meetings
here.

Advisory Boards aren\'t necessarily people who have to be able to get
down into the details of the research or the activities. I mean, we are
here for essentially two days of free consulting for you and we would
like to help \-- we would like our expertise to be of use to you.

But Advisory Boards in general can be just a forum for you to exchange
ideas amongst yourselves, so the benefits may be mostly internal, and
therefore indirect.

And finally, I think we\'re here as an Advisory Board partly because if
the integrity and the independence and objectivity of the Agency is ever
threatened by outside forces, we\'re here to stand up for an
organization we know something about.

It\'s a challenge and struggle to make sure that, with our lack of
knowledge of energy data and statistics that we always end up being of
help, but we\'re all here trying and the folks at EIA are trying too and
I appreciate that and thank everybody for their part.

Bill Weinig especially, just works tirelessly to put these meetings
together and I think we all owe him a great debt of gratitude for that.

So with that I really am done and I would like to turn the floor over to
Jay Hakes, the EIA Administrator.

DR. HAKES: Thank you, Dan. I\'d like to commend Dan for his leadership.
I think he would make a great college football coach because he\'s a
very good recruiter \-- bringing in a lot of good talent.

I\'d like to welcome Jay Breidt and Jim Hammitt. I think they\'re
examples of Dan\'s good recruiting skills and I think you\'re going to
give us a lot of help, and I\'d like to congratulate Carol on her
overwhelming victory as Vice Chair.

The newcomers will be filling the shoes of some major contributors and
we have some certificates to hand out. These are signed by myself \--
which my signature\'s fairly easy to get \-- but it also has the
signature of the Secretary of Energy.

Particularly the current Secretary, it\'s very hard to get his
signature. There was one day last week where he was actually in four
different countries on the same day. He maintains quite a schedule. But
not just EIA but the whole Department appreciates the contribution that
he\'s made.

Davis Bellhouse is first on the list. And Charles Bischoff. Samprit
Chatterjee. Greta Ljung. And then Bradley Sharpness is not here today.
All of these people have put in at least four years, in some cases six
years. So several of them have been here longer than I have. So they\'ve
had a chance to have a real impact on the EIA and I think we all
appreciate this very much.

Let me make some general opening remarks that are consistent with
remarks I usually made at this point and that is, sort of where we stand
on a resource basis. And the news actually, has gotten somewhat brighter
in the last year or two. You\'ve been hearing a lot of gloom and doom
for a number of years but it\'s actually getting better.

Back in 1995 we had a budget of about \$85 million, and in one fell
swoop we took a \$13 million cut, down to 72. And then we continued with
smaller cuts for several years. The budget for 1999, which the fiscal
year started on October 1, is the first year where we\'ve actually had
an increase. And we went from 67 million to 70.5 million.

Basically the way this happened was rather unusual. We got a million
dollar increase as part of the normal budget process. Both the House and
the Senate provided a one million dollar increase, which was very small
but at least it reversed the downward trend.

Then you remember in the last days of the Congress they had this
emergency appropriation for things that were so vital to the national
interest that they came outside the normal budget rules. And I got a
call from OMB while I was on the road saying, you got your
two-and-a-half million dollars and I said, what two-and-a-half million
dollars?

And apparently we had some money that was put into that appropriation.
So that\'s how we ended up with basically a three-and-a-half million
dollar increase.

It is the intent of the Department to request more money above that for
the year 2000, and that is now being worked within the Administration.

We have a number of needs that need to be addressed and I think that
certainly the work that the committee has done will be helpful in
shaping these needs. You have had discussions here about some of the
problems in our data series where industries have been changing \-- for
instance, natural gas \-- and the surveys are not picking up as many
respondents as they should be or have in the past.

So we also have the electric industry which is in the midst of a lot of
changes, and therefore we have to make sure that our data series there
don\'t get out of line. And we also have new requirements from people
who, for instance, want to know about the analysis of carbon emissions
on an international basis, and are wanting more detail and depth than we
are able to provide with our current systems.

So there are a number of things that we are looking at now and trying to
evaluate in terms of how these, still somewhat meager, additional
resources can be directed.

But I think it is fair to say that one of the ways that we have adjusted
to the budget cuts in recent years has been deferring some maintenance
so that we could continue to produce a large number of data series, have
a rich analysis program, and meet the needs of our customers. But at
some point we have to do that maintenance and I think that needs to be a
priority for us in the coming year.

Related to these budget developments have been developments \-- sort of
complementary developments in the personnel side. In 1995 we had a
budget ceiling of 483, and by 1998 that had gone down to 382 \-- a cut
of more than 100.

We basically have had now, a couple of years of very small increases \--
or, small decreases \-- and the Department has now agreed to level off
and we will not have a declining ceiling in future years.

This is sort of welcome \-- well, it\'s welcome news when you consider
that a couple of years ago we were working on contingency plans to go
down to a budget of \$53 million, and to go down to 320 employees. So
that pressure is off and I think we do have pressures to continue to be
more productive.

I think that the old days of automatic increases are not there, but I
think that the situation has certainly gotten more manageable. We were
able to go through this period with no involuntary reduction and were
able to handle it with attrition.

And we are now hiring for the first time in many years. We have had a
hiring freeze and a promotion freeze at certain levels, and that was in
force for about three years. And so we\'re now out where we can be
hiring and we\'re looking for good candidates. We\'re not doing a lot of
hiring but we are doing a modest amount of hiring.

So we\'re optimistic about the future. We feel we\'re in a better
position to sort of manage our resources, to be able to address critical
questions when they\'re brought to our attention in a way that we
probably couldn\'t be doing when we were getting these cuts each year
and just trying to survive.

One thing that I would point attention to that\'s not so much on a
budget issue but on a management issue is, we have tried within EIA to
move to a quality organization that\'s constantly trying to improve its
quality and its timeliness and its relevance to customers.

And as part of doing that we\'ve been convinced that we need to have
better summary statistics of how we conduct our business. And having a
lot of statisticians at EIA we have some advantages in doing that.
We\'ve been a pilot project in the federal government for the
development of performance measures.

We decided this year to go ahead and apply for a quality award based on
the Baldridge Criteria. This is administered by the Department of Energy
but has external examiners and is a very, extremely rigorous process.
And EIA received a Quality Achievement Award.

There are a few other organizations in the Department of Energy over the
years that have gotten to the Quality Achievement level, which is a very
high level. But no other headquarter\'s organization has even gotten
that high, so this is the highest award ever received by a
headquarter\'s organization at the Department of Energy.

A fairly good-looking award. And we sort of pass it around so the
different offices can get a look at it. And I think that you have \-- do
they have the Quality Award application? Okay, on the table.

I think, you know, applying for this award is not just to sort of feel
good to try to get an award. The process of filling out the application
and putting together an award application teaches us a lot about
ourselves, how we do our business and where the strong points are and
where the gaps are that need to be filled in.

I think a lot of things we\'re addressing now are things that we\'ve
learned more about because we have performance measures that track them.
And I think it\'s good for our outside examiners like the Appropriations
Committee or the Offices of Management and Budget, or this committee, to
sort of be able to get regular reports on how we\'re doing and actually
have some data to back it up, rather than just a few anecdotes. So I
would commend that particular document to your attention.

Another thing I would say, and it somewhat will be reflected in the
agenda that you have today, is that we\'re concerned to put out very
high quality material \-- both on the data side and the analysis side.
But there\'s really not a lot of point in doing this if no one uses it.

So we\'ve always been interested in for instance, data on how many
people on the Internet are using our information, and we track that
pretty carefully. But if you look back to why EIA was formed back in
1977 it was to assist policymakers who were dealing with confusing
issues and need someplace where they can go and get accurate information
and information that they know is not tainted by a policy agenda.

And I think that that is certainly the case in a number of areas. And
two that I would just mention that I think are the most recent in the
last few months \-- in September we put out a document on the 25th
anniversary of the oil embargo of 1973.

It was that \-- I don\'t think it was just the embargo in \'73 but the
events of that year that produced very high oil prices and oil shortages
where people drove up to pumps and couldn\'t get gasoline \-- sort of
created the modern energy era where energy was thought of much
differently than it had ever been thought of before.

And it sort of was a time to reflect on what we had learned from this
experience. So we put out a summary document which did not, in and of
itself, get a lot of attention but served as a building block for a lot
of the news coverage that took place.

As part of that exercise I testified before the Energy and Power
Subcommittee in the House of Representatives on what has been the major
trends of the last 25 years. And if you noticed, [The]{.underline}
[New]{.underline} [York]{.underline} [Times]{.underline} had a very long
story on energy that was related to this anniversary, and EIA was really
the basis of almost the whole story.

The reporter told me that the editors of [The]{.underline}
[New]{.underline} [York]{.underline} [Times]{.underline} are convinced
that energy is emerging back as a major issue, partly because of carbon
emissions, and want to do a lot more on it. He said jokingly before it
appeared, he said, this may be the longest story [The]{.underline}
[New]{.underline} [York]{.underline} [Times]{.underline} has ever
produced. And it was a very long story and a very, I thought, accurate
story, and EIA certainly helped that to occur.

The other thing that went on was of course, in the climate change area.
We were asked by the House Science Committee to do a study on the
impacts of the Kilter Protocol and that \-- I won\'t say a lot about
that because that\'s a subject on the agenda.

But I do think it is of note that we have testified before the House
Science Committee on that study. Mary Hutzler and I have probably done
about 20 briefings on the subject. We did a briefing at the White House
which we thought was going to be for one or two people. When we got
there 20 people showed up and the briefing lasted for about two hours.

And we had the Chairman of the House Economic Advisors, the Chairman of
the Council on Environmental Quality, and a number of very high-ranking
officials at that meeting. So I think that the work that we\'re doing is
being very widely utilized and I think, obviously that\'s no good if the
quality of work is not good. But if we can combine high quality work
with being relevant and being used by the various people who are making
the decisions, I think that\'s an ideal combination.

Usually I try to attend just about all of your events and I am usually
quite eager to attend the dinner. I have to apologize. I have a speaking
engagement on the oil crisis in New Orleans which I\'ve had for many
months. I was asked to do this by the former Chairman of the Senate
Energy Committee who handled my confirmation, so it was hard for me to
say no.

But I\'ll be leaving after lunch and we\'ll just have to get a report
when I get back on the deliberations. But I look forward to being more
fully available at the next meeting. I do plan to attend the lunch.

If there are any questions that you have, now might be a good time to do
that. If there are general questions that don\'t appear on the agenda or
things that I might be able to provide some perspective on, you know,
I\'d be glad to do that now.

Well, if there are no questions we can move on.

CHAIRMAN RELLES: Thank you, Jay. I\'d like to move into the agenda now.
We determined that I would try to do some transitions between meetings;
that is, when there was a topic that was carried over to a subsequent
meeting I would try to indicate what the gist of the issues were so that
we could have some continuity.

The agenda now gets us into sort of looking at the Kyoto Report. The
title of the talk by Mary Hutzler will be \"Handling High-Profile EIA
Reports: A Case Study of the Greenhouse Gases Protocol Service Report\".
Let me try to provide some continuity with what happened last time.

Last time we heard about two service reports. Service reports are things
that EIA has been doing for many years and they\'re very consistent with
what Jay talked about to try to give good information to the policy
debate.

EIA is uniquely qualified to do them because it knows the most about
energy and it has some longstanding models, such as NEMS, which know how
to take energy data and translate them into outcomes of policy interest.
Service reports are inherently very difficult to do because you just
don\'t get much time to do them.

Kyoto, which you\'ve all gotten a copy of, was six months from inception
to completion, and I think as all of us know who try to publish in the
scientific literature, six months is an obscenely short amount of time
for that kind of work.

In any case, the committee last time offered a large number of comments;
mostly maxims which everybody knows but comments that the committee
thought were worth emphasizing. I don\'t intend to summarize them at
this point because I think most of them have found their way into the
Kyoto Report and we\'re going to be listening to Mary talk about the
Kyoto Report and on whether or not the committee comments were viewed.

So please, Mary, would you \--

DR. CARLSON: I think what Mary means is that formerly EIA is responding
in three parts to the questions that the committee raised at our last
session. Mary is going to respond this morning; Renee this afternoon,
and Dwight French tomorrow.

CHAIRMAN RELLES: My intention was to summarize the comments from Renee
about the time she was going to speak, and the same would be true of
Dwight French tomorrow.

MS. HUTZLER: I was asked to talk today about handling high-profile EIA
reports. And what I thought I would do was to discuss this in three
different ways.

First, in terms of the process that we use in developing a surface
report. I essentially wanted to start with that because I felt that most
of the comments back in April were process-oriented. Then I wanted to
compare our most recent Service Report that we\'ve done with the two
that were briefed back in the April meeting. And lastly if there\'s
time, I\'ll summarize some of our results from the Kyoto Study.

To start with, I went back to the transcript in April to make sure that
I got the major comments from the committee. And I\'ll hand out this
little chart. The reason why I did this chart is because I wanted to
make sure that I captured all of the major comments that the committee
told me, and I just thought that this would be an interesting way to
represent it to you and to make sure that I\'ve accomplished covering
all of those.

To start with, as Dan has already said, doing work for non-EIA clients
is not new to EIA. We\'ve been doing this for as long as I\'ve been here
\-- and that\'s been since 1978 \-- and in fact, we do have an EIA order
that specifies what we\'re supposed to do when we do work for non-EIA
clients.

The last full order was dated October 15th, 1985. It was signed by Bill
Dorsey. It was amended on March 8th, 1988. The amendments were minor and
that was signed by Larry Pettis.

Essentially, the order calls for analysis to a service request be
published as an EIA Service Report and that the report must contain a
number of items. Those items include the name of the requestor, the
scope of the study, the assumptions provided by the requestor, the
methodology, the results, and a disclaimer.

Essentially, the disclaimer goes on the report in a number of places. We
put it in a box on the cover page and we also mention it in the preface.
And essentially the disclaimer says that service reports are prepared by
EIA upon request and are based on the assumptions of the requestor.

The order also says that the Administrator or the Office Director may
determine it prudent to obtain technical comments outside of EIA.

Now, the process we generally use to develop a Service Report is first
of all, a letter of request goes to the EIA Administrator or to the
Office Director. Then what we do is we meet with the client. In terms of
those meetings, what we try to do is figure out whether we can respond
to the request that we\'ve been asked to do.

And we also try to work with the client on the assumptions; trying to
direct the client in terms of the assumptions they would like to use for
the report, whether we think that they make sense, and the best way of
incorporating them into our methodology.

Now, one of our service reports that was briefed in April, the
assumptions essentially came from an inter-Agency task force group that
was working on climate change. And so we were asked to incorporate those
particular assumptions.

Once we\'ve determined the assumptions and what we can do we essentially
ask the requestor to provide them in writing. There are really two
purposes for that. One is that the requestor must take responsibility
for those assumptions, and putting them in writing makes them do that.

The other reason is that it ensures that we have a proper understanding
of the assumptions that they want, and in that way we can better ensure
that we incorporate them in the work we\'re doing.

Once we\'ve done the analysis we brief the client. Some of the clients
work very closely with us. They like to see results at intermediary
points within the project and we provide those.

And finally, after we\'ve written the report we put it in the public
domain. We do that by sending the report back to the client under the
signature of our Administrator.

We\'ve done a number of Service Reports in the past several years. I\'ve
put a few of these on this slide because I wanted to mention different
items about those particular service reports.

Starting at the bottom, back in 1996 we did a Service Report for the
Environmental Protection Agency. The report was called \"An Analysis of
Carbon Mitigation Cases\". We were reimbursed for this particular study.
EPA gave us anywhere from about \$350,000 to \$400,000 to do the work.

They stipulated that they wanted us to use the money to improve our end
use demand models, and in fact, we did do that. One thing about this
particular study is that we didn\'t believe in the assumptions that the
client gave us, and as a result we documented that in our report. As a
further result, we no longer have them as a client because they weren\'t
happy with our truth.

The next Service Report from the bottom was one that we did for Senator
Jeffords. FERC had a few orders out and they did an environmental impact
statement regarding those particular orders. And Senator Jeffords wanted
us to essentially see if we agreed with their results or not.

So we modeled them as well as FERC modeling them and we did come up with
results that were very similar. That was released at a hearing in
Vermont and Jay Hakes did present that work.

The next one from the bottom was study that we did for the General
Accounting Office. Their client was Congress and they were looking at
issues regarding reducing oil imports. They turned out to be a very
active customer.

We spent many, many meetings with them discussing assumptions and also
the results. They wanted to be able to write the results very clearly
for Congress so they could understand them, and so we assisted them in
all of that work.

Again moving from the bottom, there are two Service Reports there. Those
are the two that were briefed to you back in April, and I really don\'t
want to spend much time on them. I do again, want to mention though,
that the analysis of carbon stabilization cases was based on the
assumptions of the inter-Agency task force that was working on climate
change.

And our report actually did something very major and it pointed out that
this particular committee felt that there was an announcement effect;
that just because people met in Rio in 1992 and talked about climate
change, and because people met in Kyoto in 1997 that they would make
changes in their behavior, and as a result that there would be certain
things that would happen just because of that announcement effect.

We showed that in fact, the amount to change they thought would come
about because of the announcement effect was not able to be
accomplished. So I thought that was a major benefit of the work that we
did.

The second from the top is one that we did this year. Our customer there
was the Office of Energy Efficiency and Renewable Energy, here in DOE.
They wanted us to look at the Increased Diesel Penetration and
Transportation Sector. We were asked to look at the impact of new
vehicles sales being 10, 20, and 30 percent diesel in the year 2010.

And the top one is the one that was just released on October 9th before
the House Science Committee. Jay testified at a hearing and essentially
we produced two volumes. They were mailed to you before. I\'ll talk a
little bit more about that Service Report a little later.

One of the comments back in April was that NEMS requires EIA staff to
run it. Well in fact, that\'s not true. It does require a dedication
from a company to run it because it is a very large model. What I did on
this list was to provide you with a list of users of the NEMS model.

The Tellus Institute was our first user. That was back in \'95/\'96.
They used the 1995 version of NEMS. At that time, when we had a lot of
money when we were working on NEMS, we had a PC program and we provided
the different NEMS modules on diskette.

Tellus Institute took that, used the PC, but because they couldn\'t fit
the whole model on it they had to iterate manually between component
models and between iterations, and they did that. They briefed their
work at a Climate Change Conference in June of 1996. And actually the
conference was pretty interesting.

It was a 2-day conference and I was interested in noting that at about
50 percent of the sessions either people were briefing the results based
on using the NEMS model directly, or they were briefing results using
the AEO that in fact, was based on NEMS. So I thought we had gained
quite a bit of visibility by 1996.

The Office of Energy Efficiency and Renewable Energy also wanted to be
able to get the national laboratories to use NEMS for them and they
asked us and paid us to port our model to a work station at the National
Renewable Energy Laboratory and at the Lawrence Berkeley National
Laboratory.

We did that and we helped them use it; we helped them to set it up.
I\'ll talk more about the Lawrence Berkeley work in a minute.

The Gas Research Institute, when they were downsizing, decided they
needed to get models elsewhere and so they asked Energy and
Environmental Analysis to use the NEMS Residential, Commercial,
Transportation and Electric Models with their hydrocarbon model and with
EEA\'s industrial model to do their modeling work. So that\'s another
user of various components of our models.

The five lab study that was released last summer was done by a number of
laboratories \-- five of them \-- Lawrence Berkeley, Argonne in
Oakridge. Oakridge used a transportation model. They had been user of it
\-- they just updated to the latest version that we had.

Argonne used our industrial model more as a validation check, not as
their main modeling tool. And Lawrence Berkeley actually tried to put
all their results into NEMS. We weren\'t happy with their work so they
didn\'t publish it at the time that the five lab study was released.

But recently the Environmental Protection Agency did ask Lawrence
Berkeley to do that work and they have released a report using NEMS to
consolidate all of the five labs\' assumptions.

The Electric Power Research Institute has become a user in the past
couple of years. They are a pretty sophisticated user. They have the
entire model on the PC. They did that work themselves, and they can in
fact, update the model themselves. I\'ve been pretty impressed by the
work that they\'ve published.

On Location is a contractor with us and they have been asked to use the
model for work for both our Policy Office for electricity restructuring,
and also for EE in their quality metrics program. That\'s probably
another example of work we turned down.

EE had asked us to look at the impact of their quality metrics a number
of times. We did it twice, and after we found that they really weren\'t
using our work we felt it wasn\'t a good way to use our resources so we
didn\'t continue doing that for them.

I left out on this page the fact that the Canadian government is also
using parts of NEMS. They have our transportation model and they\'re
interested in using our building models.

Back in the April meeting one of you commented about Jay Edmonds and his
modeling capability and the fact that he has fairly quick turnaround
models. And he does. We\'ve used them. One of the factors about his
models though, is that he is funded by a large grant from our Office of
Energy Research and his model isn\'t documented.

That does create problems because it makes it very difficult to
understand the details of the model and make changes to it. So that was
the impetus for this particular slide. I just wanted to mention that.

We do try to make our modeling capability very transparent and we do
provide a lot of documentation. I think some of you are very familiar
with that documentation.

Looking at the second bullet, I counted up the number of volumes of NEMS
documentation we have. There are 31 of them. We update them annually.
They\'re provided on our Web site on our CD ROM and we also provide
photocopies to people who are interested.

We also have done a Model Developer\'s Report and in this report we
looked at the sensitivity of major model parameters within that model
and we document that, and that\'s also available.

Because we knew people didn\'t want to read all this detailed
documentation we also produced summary documentation. We have Short
Report; we update it every other year. And that\'s also available on our
Web site on our CD ROM and in publication form.

Now in April one of you commented that it would be nice to have a
200-page documentation \-- something in between this. And truthfully, I
agree. I think it would be nice. It\'s all a matter of resources. Back
when we had a lot more resources it might have been able to make our
priority list. Right now I don\'t think we\'re able to do that.

I do want to mention that our model code is available. It\'s available
on our Web site. And for instance, the AEO-99 reference case was
released on Tuesday and Tuesday afternoon it was put on the Web site. We
have two very active users: one is EPRI and the other is Lawrence
Berkeley and they wanted it as soon as they could get it.

And lastly I wanted to mention that we do do a forecast evaluation
comparing our results to history. And we\'ve published three versions of
that in our Issues, Mid-Term Analysis, and Forecasting in \'96, \'97,
and \'98, and we will continue to do that regularly since it is one of
our performance measures.

Another comment that was made in April was, I don\'t think NEMS is going
to be accepted by the scientific community if it remains static. Well, I
wanted to address that because I don\'t think NEMS remains static.

We do update it annually. We make sure we put in current laws,
regulations, and policies. We make sure we update it to new data. And we
also to the best of our resources, put in new modeling methodologies.

And whenever we get a study we always ask ourselves the question, do we
have the right things in the models, do we need to update any of the
modeling logic? And we do that first before we do the study.

We do have an active program of model development and maintenance
outreach. When we developed NEMS we produced component design reports,
we circulated them to the academic community, to industry, and also to
our independent, expert review program.

We have continued the IER Program. Recently we had one done on ethanol,
and in fact, the recommendations made from that IER were implemented in
the Kyoto Study. We also had an IER on competitive pricing and that was
briefed to you. That IER proved to be very valuable. They reviewed
copies of the report that we produced last August quite a few times.

We held an annual conference on NEMS in AEO. Originally we did this to
brief our modeling logic and let people know about NEMS. Now what we do
is, we cover energy issues.

We do that for ourselves and for other people because we need to keep
learning about what\'s going on in energy and we need to know both sides
of the issues and we try to cover both side, having a very balanced
program.

Another thing we do regularly is to hold focus groups, workshops, and
panel discussions. One of the major areas we confront ourselves with is
technology and it\'s very useful for us to bring in engineering firms
and manufacturers to let us know about what the costs are, what the
performance is, and what they think the use and penetration are.

We also hold working group meetings. We\'ve done that from the
beginning. They\'re at various frequencies depending on the interest
that other government agencies have in our work, but we cover our
modeling changes, we cover our cost and performance information, and we
also cover the results of major studies.

And lastly we have bilateral relationships with industry. For example,
we have one with EPRI. They give us technology data and we help them
with the model.

Another comment from the April meeting dealt with uncertainty analysis.
That\'s not a new issue for us but it is a very difficult issue for us.
Before NEMS was actually produced the National Academy of Science, the
National Research Council, was paid by DOE to make recommendations about
how we should build NEMS.

And in fact, they mentioned uncertainty analysis explicitly then and
they also talked about reduced-form modeling. These are two quotes from
their report; you can read those yourself.

Moving on to the next slide, I do want to mention that this was included
in our original NEMS requirements. We had a 4-year research program
covering these issues. We had written a couple of journal articles and
we did brief you on this, and I think that was mentioned in April as
well.

We did achieve some applications from that process. We did take our
transportation model and apply it so that we could get uncertainty
ranges for that particular model.

Unfortunately with our budget cutbacks we stopped there because we knew
once we integrated that in NEMS that there would be even bigger problems
because there were issues dealing with having more variables once you
put it into NEMS than when you use a particular model in a standalone
fashion. So the work essentially stopped at that point.

In terms of the reduced-form modeling we looked at a couple of
methodologies. One was dealing with the model aggregations and we did
apply that to two areas. They were the areas that took the longest
run-time: electricity module and the refinery modules. But we did gain
from that application.

And we also looked into localized kernal regression analysis and we just
applied that recently. We used it for our Kyoto Study but we started it
before we were asked to do that study. And so that\'s helped us quite a
bit because now we actually can look at the policy implications of the
macro-economic impacts within NEMS.

I did want to again mention that all of this in terms of development,
was terminated with our \'96 budget cuts. In that year the NEMS contract
budget was cut by 70 percent and our FTEs were cut by ten percent.

I wanted to move on to the issue of resources, and I find that resources
are really important because that\'s a decision that I have to
constantly make: how many resources to put on each of the different
requests that we get.

It turned out that I knew that the Kyoto Study was going to be very
visible. We had a very visible client. We also had other results out
there that said the cost would be very inexpensive, and we weren\'t
seeing those results with our own modeling capability so we knew that we
had to do a lot of work on this particular study.

I knew I wanted to cover a lot of areas on this study, so I knew I was
planning for a 200-page report so that we could make sure we put our
opinion on each of the issues in the energy area that were confronting
people.

Once we wrote the report people indicated the subject was complex and
that it was difficult to get through it, so we also did a summary
report, and you received copies of both the detailed report and the
summary report.

We also wanted a summary report for our client because we knew
Congressional people wouldn\'t have the time to really go through a
200-page report but they could get the essence from the summary report.

Also, I did want to include in the main report, comparisons with other
models. That was something else that you brought up in April and I
thought it was important because people are constantly wanting to know,
well, how does your analysis differ with other analysis that\'s on the
street?

There\'s a difference in resources between doing these studies. Kyoto
cost us a lot; about 20 FTEs were probably spent on it in a 6-month
period. That\'s probably about half my office working on it, full-time.

We also spent a lot in contract dollars relative to the other studies.
We did use a panel of reviewers. It cost almost \$90,000 for that panel.
Our reviewers were not comfortable with the DRI macro-economic model so
they asked us to look at other macro-economic models.

And I took the list and I went down them in order, and the second one
said that they could help us and that was the Oxford model. We got their
results. We paid \$25,000 for them and we found that they were not good
\-- they were not as good as DRI and the panel agreed with that.

Next, there was an issue dealing with industrial leakage. When you get
high carbon fees the question becomes \-- will some of your industries
migrate outside of the U.S. and if so, how many might do that \-- is a
question that people had posed to us.

There was one study out at the time done by Argonne National Laboratory
that said, maybe 25 percent of our industries would migrate. A lot of
people thought that was a high number and they thought that there was a
lot of controversy around that report.

So we asked Charles Rivers Associates to give us an estimate and their
estimate was much less. And we included both of those in our report.

Finally, in terms of editing and printing, we essentially paid about
\$53,000 in editing costs. There were a number of authors to the main
Kyoto Report and half of that went into editing for that report.

Also, the summary report took quite a bit of time because writing a
report like that is pretty difficult for that and trying to perfect it
took a number of cycles with the production people.

And lastly, we decided that this area was so complex that we wanted the
reports to be printed in color and that cost \$57,000. The summary
report is already in its second printing so it is a report that the
public and people are interested in.

The policy study and the Jefford Study took a lot less resources. These
are all estimates because we don\'t track these exactly, but I estimated
that policy took us about seven FTEs and Jeffords about three.

Now, interestingly enough, although things that were mentioned in April
were incorporated in our Kyoto Study I did want to mention that if I had
to do it over again I would make the same decisions for all three of
these reports. I would put the resources into the Kyoto Study and I
would put less resources into the policy and the Jefford Study. So these
are tradeoffs one has to make and they were made each time.

Moving on to more of the specifics on Kyoto, I wanted to show you where
our panel of independent expert reviewers were. We had John Weyant from
Stanford University. He heads up the Energy Modeling Forum. And he is
very well versed on various models that have been used in the climate
change debate.

We had Mike Toman from Resources for the Future. He was actually at
Kyoto and he worked on the protocol and knew much about it. We had Bill
Nordhaus at Yale University. He was our macro-economic expert and he
gave us a good deal of valuable advice.

And in fact, he even held the meeting at Yale for my macro people with a
good deal of Yale\'s macro-economists to try to help us, using the DRI
model and in dealing with the macro assumptions.

Lorna Greening helped us with technology, and Bill Hogan is certainly an
energy expert but he\'s been working a good deal in electricity.

These people worked very hard and they worked with us quite a bit. We
had three meetings on site. They turned out to be all-day meetings. And
one of the issues we had to contend with was that these people gave us
different comments. They weren\'t the same and we had to sit them down
at a table to try to get some consistency and some direction from them.
So it took quite a bit of time.

Now, even with all of this, did our Kyoto get criticized? Yes, it did. A
lot of them we think, were unfair. I\'ve listed them and I was going to
go over the responses that I have to each of these.

First of all they said, it doesn\'t include international trading. Well,
that was false. We had six scenarios: four of those scenarios did
include international trading.

Secondly, they said that it doesn\'t include policies that could reduce
carbon emissions. That in fact, was a true statement. The reason for
that was that our client did not ask for it and that we as an
independent, policy-neutral organization do not come up with our own
policies. And if the client isn\'t going to specify what a client\'s
standards to use, we\'re not going to make up our own. So we did not do
that.

As Jay likes to say, policies can make things go both ways, though. And
while some policies can reduce carbon emissions, others in fact, can
make them increase.

Another criticism was that we didn\'t include electricity restructuring.
This was another false statement. We did include in that particular
study and in AEO-98, full electricity wholesale restructuring. In terms
of competitive pricing we only looked at three regions having marginal
cost pricing. However, we did include in the Report of Sensitivity that
had all regions of the country with full, competitive pricing.

Another statement that was made was that it didn\'t assume consumers
would change their behavior due to the announcement effect. In fact, as
I mentioned earlier, that is a true statement. We don\'t believe that
just because people are talking about the Kyoto protocol that they\'ll
make massive changes.

For instance, in the residential sector builders of new homes tend to
put in and install heating equipment based on the costs of purchasing
that equipment. They don\'t do it based on lifecycle costs because
they\'re not paying for them.

So there are things built into our system that don\'t make people react
just because an announcement is made. And those are things that we try
to incorporate in our modeling system.

Another criticism was that we didn\'t include the new NO~x~ and
particulate rules for electric plants. And that was true at the time. We
did however, do an offline analysis dealing with these and we felt that
they would not impact the results much; that the costs once you put them
on the basis of electricity prices, would be very, very small.

And in fact, we did look at the NO~x~ rules in AEO-99 and that did
become the case; they are very small. Regarding the particulate rules we
didn\'t incorporate them yet because they still haven\'t been
well-defined. They\'re still undergoing revision.

Next we were criticized on not incorporating technology changes that are
already in the marketplace. Again, this was false. We did look at our
technology menu and we did make sure it was complete. We even made sure
we had a company, A.D. Little, look at our end-use data, and we did
update it for the Kyoto Study.

And lastly, we were criticized because we didn\'t use the appropriate
macro-economic model. We felt we did. We used the DRI model and we did
take steps to ensure that we felt comfortable with it.

Bill Nordhaus did help us in terms of making sure that we looked at both
the long-term impacts, which are in the potential GDP figures that we
reported, and we also looked at the transition costs that Kyoto would
result in, in looking at the actual GDP.

So at this point in time I think I\'m supposed to turn it back over to
you for questions. And I did bring my slides about the Kyoto Study so I
could tell you about the results if you\'d prefer to do that. But if
you\'d like to open it up to questions I\'d be happy to answer them.

CHAIRMAN RELLES: How about comments?

DR. LJUNG: I wanted to ask you about the collective (inaudible).
(inaudible) EPA and I would have felt that you should have a lot of
information in terms of the (inaudible) studies of this type
(inaudible).

MS. HUTZLER: We do collaborate with EPA in a number of different ways.
Number one, in our greenhouse gas emissions program, voluntary reporting
program, and in doing our estimates of greenhouse gas emissions, we do
work very closely with EPA and the next group on the agenda are people
from my staff who will be talking about that. So there\'s a lot of
collaboration on that side.

We also have a lot of collaboration with them in their work on NO~x~ and
other types of standards. In fact, one of the reasons why we felt
comfortable with not having the NO~x~ and the particulate rules in there
was that we knew what the results were from the EPA work and we thought
that they were right on.

So that we felt that the costs would be small. So we do talk to them
quite a bit and we do collaborate with them. They were also part of the
inter-Agency analysis task force and we certainly worked with them too,
in terms of their assumptions.

But in terms of that particular task force, the decisions were made as a
group within the government. They\'re not decisions that we can make
unilaterally.

Any other questions?

DR. CHATTERJEE: You mentioned some national institutions, national labs
which deal with energy, having used them, which is very, you know, very
good to have your things being tried out. Did they have any suggestions
or updating modification and so forth, or what is the general feedback
you got?

MS. HUTZLER: They do have suggestions. They\'re probably suggestions
that they would make even without actually using the model. We do
interact with them quite a bit and we do let them know of things that
they could help us with.

For instance, there\'s a component of the buildings model where we look
at miscellaneous uses of electricity, and that\'s growing at a very fast
rate and it\'s helpful to us to actually get information about that and
get more data on it.

And they\'ve done some work working with utility companies to get that
information. We\'ve incorporated it into our buildings model and we\'re
waiting on them to do some work, for instance, in commercials that we
can incorporate that as well.

So we do work with them and we do try to build on those suggestions
wherever we can.

Other questions? Yes?

DR. HAMMITT: I\'d first like to say, to commend you and the group from
the Kyoto Report. I think that\'s a really well done one that seems to
me to be quite completely documented, very clearly written, very
accessible. I liked the last chapter where you compare your results with
a variety of other models out there. So I think it\'s really an
excellent report.

I just wanted to ask if you could say a little bit more about how you
evaluate the accuracy of NEMS. And one thing you pointed to is these
comparative forecasts to realizations, which is a tremendously important
thing to be doing. So I\'d just ask about other activities that may be
dealing in that spirit.

MS. HUTZLER: Obviously, that\'s a very difficult issue to deal with when
you\'re dealing with the mid-term. What we do is, we try to make sure
that our data source is as good and as comprehensive as they can be. And
that\'s the reason why we have focus groups, we call in experts in
different areas.

Obviously, in any mid-term modeling activity there are zillions of
numbers out there and different advocates and we\'ve got to really try
to understand the area to make sure that you\'re putting the best data
that you possibly can in the model. That\'s one aspect.

Another aspect just deals with methodologies. We\'re constantly trying
to deal with issues of high technologies: change over time, how
penetration and adoption will take place. And that means that we have to
talk with other people who are dealing with these issues. We have to be
on top of those and we have to keep on working on making sure that we
have that within our modeling system.

Another area deals with looking at the results and talking with other
people who are pretty expert in the area. And we do this pretty
regularly in our working group meetings where we\'ll brief the results
and explain the results to folks. And of course, they will challenge
them.

For instance, an area of challenge is, on certain renewable technologies
that we had in our Kyoto Study the advocates of solar were pretty
disappointed that we didn\'t get much penetration with solar. But if you
look at the costs, and we tried to explain it in our report, they\'re
still non-economic, even under our the carbon fees that we had gotten.

So there are numerous ways that we tried to make sure that we\'re coming
up with as accurate a study as we can. Of course, will reality say that
we\'re right? Probably not. There are going to be laws, there are going
to be regulations, there are going to be changes. The real issues is,
are we as best as we can today, and that\'s what we strive to be.

CHAIRMAN RELLES: It\'s certainly a lot easier to just collect data. When
you do complex models of this kind you\'re generally goring somebody\'s
ox and as we can see, there are numerous criticisms of the study, many
of which or probably all of which, are unfounded and biased.

I guess, getting back to the committee\'s comments of last April, I
think we had three general classes of comments. One was that we would
have liked to have seen more uncertainty analysis than, we would have
liked to see more peer review than, and we would have liked to have
understood more about NEMS and seen knowledge of that promoted.

And I hope those weren\'t taken as sort of zero and one dichotomous
situations. I think we were just putting in a plea for more of each of
these things, and I\'m gratified to see that in the Kyoto Study we saw a
lot more.

Now whether those would have occurred without our comments is an issue I
don\'t quite know the answer to. They all have resource implications and
I was gratified to see that money was found in Lynda Carlson\'s area to
help pay for the reviewers.

So to the degree that those comments may have freed up some of that
money, I do feel like we\'ve made a contribution. I\'ll be the first to
admit that sometimes the things we say are not on target, pretty much as
some of the criticisms of the EIA Kyoto Study are not on target.

And in this little cartoon here that has our recommendations I say the
most. And I\'d rather err on the side of saying too much and being told
I\'m wrong than sitting back and not saying anything. So I just hope you
won\'t take offense when I say too much and tread beyond the bounds of
things I know about.

There were some \-- I guess there were some hard feelings about sort of
the way the messages might have been delivered last April, and for that
I apologize on behalf of the committee and myself.

DR. SUDMAN: Can we just follow up a bit on uncertainty analysis? We
heard from Jay that there\'s been a little bit more money now added to
the budget this year. Do you have any plans to use any of that for your
research programs?

MS. HUTZLER: Actually, I feel that our first priority really is to beef
up our international modeling capability, and Jay mentioned that. I was
a bit disappointed that when the Kyoto Study came about that I couldn\'t
deal with looking at international trading of carbon permits and I
can\'t do that directly. So we found a way to deal with it without
directly modeling that.

So that\'s my first priority and that\'s what I\'m going to be asking
money for. It would be nice to also do some uncertainty analysis and
maybe that\'s the way that Lynda might want to participate in some
activities, to carry on that program. I don\'t know. It\'s something
that EIA can discuss.

DR. CHATTERJEE: I have one suggestion because I think I talked about a
couple of years ago about international forecasting, international
modeling and so forth, and EIA, essentially a national agency, rather
than trying to you know, take the global issue.

What it might consider is collaborating with other national agencies and
international organizations which might do some of the things, so that
it might \-- you know, and incorporate them into kind of a
component-wide model rather than (inaudible).

MS. HUTZLER: In fact, we do participate with different international
agencies. The International Energy Agency has recently had a workshop
where they have asked for NEMS to be used as well as other models that
the Germans have, the European Union has, and we have participated in
that.

It turns out that they like to use us as being the model for modeling,
so we sort of stand out in that role and we sort of set the stage for
the kinds of things that people may be looking for in models.

Recently, the Asian-Pacific Economic Cooperation started a modeling
group, called the Asian-Pacific Energy Research Center. They\'ve only
been working together for a couple of years but we have a person from my
staff who\'s working with them, and we do comment on their forecast.

This is a good way to get data and to get more information about
countries, but it doesn\'t necessarily deal with the answer of a
particular forecast and what international permit trading might
accomplish.

And the reason for that is that it\'s based on assumptions and you can
use a lot of different assumptions in each different area but you can\'t
really bring them together unless you have one sort of framework to do
that.

Other comments or questions?

DR. LJUNG: I haven\'t seen any of the comparisons with forecast
activity. Could you tell us a little bit about that; how NEMS has done
short-term compared to like your STIFS model, for example? You do that
comparison?

MS. HUTZLER: Well, in terms of STIFS we\'re asked to actually
incorporate that result so that essentially EIA speaks with one voice.
We work with the STIFS people pretty closely to try to deal with issues
where our models can accomplish achieving something that they may have.

But the bottom line is, we do incorporate whatever results they have
through this collaborative effort of working together. When we compare
with other models we do that both with our forecast for the annual
energy outlook, and there\'s a section in the AEO that does that, and we
did it in the Kyoto Study.

Now, if you want us to specifically discuss those we can. In the Kyoto
Study we asked people for instance, to look at a case where they would
have to meet the Kyoto protocol target \-- the seven percent below
target \-- totally reducing emissions from energy-related sources.

And we also asked them to take a look at international trading with
Annex 1 countries. And we had a number of people participate. It turns
out in the seven percent below type of case in the commitment period we
had carbon fees that were higher than the other people.

In many cases we feel that that\'s partly because of the fact that we
are an annual model. We can track the transition effects, we look at
capital stock turnover and the penetration of technologies over time.

And so the end result is we essentially got a higher carbon fee in the
earlier years. But because we deal with technology, by 2020 our carbon
fee was lower than some of the other models who didn\'t deal with that
particular issue as well.

In terms of Annex 1 trading, since we couldn\'t represent that directly
we compared the cases that were closest. And it turned out that two of
our cases tended to be pretty close to what some of the other people
were saying in terms of Annex 1 trading.

Again, during the commitment period I think we had slightly higher
carbon fees but later on it turned out that we were a little bit lower.

DR. LJUNG: So all in all, you\'re happy (inaudible) seen so far compared
to actual \--

MS. HUTZLER: We\'re comparing to that particular situation. In terms of
our historical evaluation I would say that we\'re probably pretty close
on consumption numbers; we\'re probably pretty far away on prices.

But you have to realize that it takes you a number of years before you
can get data on what we\'re forecasting, so therefore we\'re comparing
to forecasts that were made many years ago, way before people realized
that technology is going to do a lot in terms of the production and the
cost of fossil fuels. And we were projecting fairly high prices before
but now we understand the effects of technology and we\'ve embedded it
in our models.

Any other questions? Okay, thank you.

CHAIRMAN RELLES: Well, we\'re finishing right on time. I want to thank
Mary very much for coming and giving us this update. The schedule now
calls for about a 15 minute break. So, thanks.

(Whereupon, the foregoing matter went

off the record at 9:58 a.m. and went

back on the record at 10:20 a.m.)

CHAIRMAN RELLES: One of the announced intentions of the committee \-- at
least I\'ve announced it \-- is sort of to continue to put on the table
issues that are placing EIA in the middle of the policy debates.

And no policy debate is hotter at the moment than global warming and
climate change, and so we\'re going to have two talks this morning on
updating us on what\'s happening in the area of greenhouse gas
emissions. First an update by Arthur Rypinski who spoke with us last
time; Perry Lindstrom and Steve Calopedis \-- did I get your name right?

MR. CALOPEDIS: No, Calopedis.

CHAIRMAN RELLES: I\'m sorry. And then Arthur by himself talking about
sort of, measurement issues in greenhouse gases. So I\'m not sure who\'s
supposed to take it over first.

MR. RYPINSKI: Before I start, I\'ve had a gross injustice called to my
attention which I would like to take this opportunity to rectify.

I\'ve learned that Greta Ljung has attended 12 straight sessions of the
Energy Committee of the American Statistical Association, which
certainly qualifies her for an award for valor.

Yet as far as I\'m aware, the EIA has taken no special cognizance of
this. So I have empowered myself to prepare the 12-session, Greta Ljung
Memorial Gyrocopter, which I would like to award to you as a small token
of our esteem.

(Applause.)

DR. LJUNG: Thank you. It reminds me of the helicopter experiment that I
used to teach in the time of experiments.

CHAIRMAN RELLES: What do you get for 11 out of 12?

(Laughter.)

MR. RYPINSKI: You get a gyrocopter with only one rotor. Art Anderson and
I have the pleasure of working in EIA\'s Greenhouse Gases Team. Art
would like to be here and sends his regrets. So what we\'d like to do
today is give you a little update on what we\'ve been doing. It\'s been
a busy year; just a few of the \-- I won\'t call them high points;
we\'ll call them points.

We did our first-ever flash estimate of emissions in June which we
posted on the Internet; which meant that we had early data on greenhouse
gas emissions four months earlier than it had ever been done, and we\'re
a year ahead of the EPA crowd \-- which always is my personal
performance metric.

We accelerated the reporting cycle for voluntary reporting. We\'ve done
two reporting cycles this year and we\'re finishing up our second one.
And we\'ve posted two short reports on the Internet and will be doing a
longer, written report.

We of course, did the large, published report on emissions of greenhouse
gases which was released on the second of November \-- maybe it was the
third; my memory deteriorates rapidly \-- which got modest press play.

And also since Kyoto there\'s been an explosion of interest in this
topic and my team has done something on the order of 12 or 15 public
presentations. So my colleagues have done an extraordinary job this
year, in my opinion.

I\'m real proud of them and I\'d like to introduce Harry Lindstrom who
is a perpetrator of the annual Emissions Report and will discuss some of
the things that he\'s perpetrated.

MR. LINDSTROM: Just the first thing to note is, this is a data report.
We don\'t actually do analysis or any sort of economic gyrations in this
report, so you\'ve already heard about that. But we do \-- you know,
this is the input data for some of the analysis that you do see.

Just a quick review. It\'s mandated by Section 1605(a) of the Energy
Policy Act of 1992. First inventory was in September of \'93 and just
came out with the most recent one. The gases that we track include
CO~2~, methane, nitrous oxide, PFCs, HFCs, and SF~6~, which are all
included in the Kyoto protocol.

The general findings of the latest inventory include that it was about a
1.4 percent increase between \'97/\'96 from 1767 to 1791. This is
million metric tons of carbon equivalent. Most of that is carbon, and
I\'ll show a slide a little later on to give you the breakdown.

They are about ten percent higher now than the emissions were in 1990,
and of course 1990 is an important year in terms of the Kyoto protocol
so we try to include that as the main reference point at this point in
time. And 84 percent of the greenhouse gases are related to carbon
dioxide. And most of this is from the burning of fuel.

A couple of adjustments have been made to keep with the IPCC guidelines
on greenhouse gas emissions. U.S. Territories, which are not included in
our basic energy data were added in, and bunker fuels which are excluded
from the IPCC inventory guidelines are subtracted \--

CHAIRMAN RELLES: Could you tell us what bunker fuels are?

MR. LINDSTROM: They\'re fuels that are used mainly in ships and things,
you know, that are going internationally \-- international trade,
transportation.

CHAIRMAN RELLES: Why the name \"bunker\"?

MR. LINDSTROM: Does anybody know the answer to that? I don\'t know. Art,
what\'s the answer to that? Why are they called bunker fuels?

MR. RYPINSKI: Well, 70 or 80 years ago ships carried coal in bunkers.
And the bin in which coals were carried in a ship was a bunker, so the
fuel you load in your \-- the coal you loaded in your bunker to drive
your ship around was the bunker fuel and the name stuck, even though
it\'s jet fuel you\'re loading in an airplane now.

MR. LINDSTROM: There you go. The net result of that is that the
emissions estimate is reduced by about seven million metric tons of
carbon.

Just to give a little graphic, as I said, 84 percent of the greenhouse
gas emissions are carbon dioxide. You can see that the other gases are
pretty much stable over time with increases in one category being offset
by decreases in some of the other gases, and you see that most of the
growth occurs in the carbon dioxide, especially since \'94 or so. We\'ll
talk a little about where that\'s coming from.

Just a couple of the uncertainty issues. I know uncertainty is a big
issue itself that people are interested in. The good news is that the
greatest emission source is the most certain, which is energy CO~2~.

There are some issues on that and I think we\'re going to be under even
more scrutiny nowadays with the activities that are going on throughout
the world. Arthur might talk a little more in detail on that.

As you get away from CO~2~ though, with methane and especially nitrous
oxide, the uncertainty increases and unfortunately we don\'t have the
resources to do the kind of research that would be needed to make the
nitrous oxide numbers better, so we have to rely on that being done by
somebody else in terms of, you know, looking at automobiles and that
sort of thing. So hopefully over the next few years some more will be
done in that area that we can benefit from.

Another big area is in terms of land of use issues. There\'s an
empirical issue there; it\'s called the \-- well, we\'ve used the term
\"missing sink\". And just to give you an idea, on the next slide, in
the magnitude of the land use issue it\'s about 200 million metric tons
of carbon absorbed every year by the U.S. forest land \-- which is about
13 percent, depending on how you calculate it.

And this so-called \"missing sink\" is 2 to 3.4 billion tons. And the
research is going on right now in terms of trying to figure out what is
happening here. And so the book is still open on that and we\'ll keep
you updated if we hear anything about it.

Now, the other area of uncertainty is not empirical but it\'s what would
happen under a Kyoto-type protocol. And under Kyoto the countable land
use sinks and sources are those that are: 1) anthropogenic; 2) derived
from afforestation, deforestation, and reforestation since 1990, and
measurable and verifiable. All of those things depending on how you
interpret them, will affect, you know, what you can count.

For example, grassland is included in IPCC guidelines but as it says
here, it\'s only forestlands that are included in the Kyoto protocol. So
this will be an issue that evolves over the next few years and I\'m not
quite sure \-- I haven\'t really had a chance to look at what\'s come
out of Buenos Aires to find whether they really did anything on this
issue.

Arthur might be able to \-- they did not do anything. So it remains an
issue that, you know, will be resolved in the future, hopefully.

These are questions that you\'ve seen in the packet that was sent out
earlier; hopefully you\'ve seen these questions before. What I\'ve done
is a couple of slides related to how we could slice some of the data and
how we could do some weather-adjusted emissions.

So some of the possible ways to slice the data include per capita
emissions or GDP emissions, emissions by regions and states, emissions
by SIC codes, weather-adjusted emissions, and possibly doing an
emissions profile of what a typical household would look like in terms
of where the emissions are coming from.

So what I did is, I did some \-- I took some of, both our data from our
report and then what was generated in the Kyoto Report that you heard
about earlier, and I came up with a little emissions profile per capita
for the United States.

So in 1996 you see the historical data, and you see oil is in the far
left, natural gas in the middle, and coal on the right. You see in the
baseline for 2010 if there\'s no policy changes essentially that
emission\'s profile remains the same. You\'ve got somewhat of an
increase in oil, increase in natural gas, and an increase in coal but
the relative shape of that stays that same.

Now, when you look over at the Kyoto 2010 projection based on our
analysis, you see that oil decreases but not by that much. It\'s
probably a reflection of the relative inelasticity of oil demand. Then
you see natural gas as you would imagine, going up. But the big hit is
taken by coal, and that kind of makes sense intuitively.

Another thing that I worked on is the international energy outlook and
one of the benefits of converting to a per capita emissions number is
that you can compare across countries that are different population
sizes and that sort of thing.

Recently looking at the data \-- and I noticed something interesting \--
and that is that the Netherlands\' baseline that we\'re working on right
now is the same in terms of total amount. In other words, about four
tons per person as what the U.S. would be under the Kyoto protocol.

So I thought it would be interesting to present what the U.S. looks like
under both our baseline and the Kyoto protocol and what the
Netherlands\' baseline looks like \-- that\'s before Kyoto \-- what we
project they would be if nothing changes.

And you see some interesting things. In the oil category for example, as
I mentioned you see the U.S. \-- which is our baseline again, on the far
left \-- going down in the Kyoto but again, it doesn\'t go down that
much as I mentioned because of the inelasticity.

And then you see in Netherlands today, you see that the oil consumption
is lower than what our Kyoto protocol would be. Well, if you\'ve been to
the Netherlands you know it\'s a relatively small country, it\'s densely
populated, they\'ve got a lot of people on bicycles and mass transit and
that sort of thing, so this again, makes sense.

Now you look in the middle category there, you look at natural gas, you
see that under the baseline there\'s less natural gas in the U.S. And
you see that increasing under the Kyoto protocol which of course, makes
sense since that\'s a low carbon emitter.

And in the Netherlands it\'s even higher under their baseline and
that\'s because they need more of it to generate electricity. They
don\'t have a lot of hydro or nuclear so they tend to be more dependent
on natural gas to generate.

Then you look in the coal category and you see again, there\'s the U.S.
baseline which is much higher. And then under Kyoto we get down to about
what the current level, or at least the projected level of the
Netherlands\' baseline is.

So it kind of gives you an interesting way to look from one country to
another and say to yourself, okay, this is the structure of this
country\'s economy in terms of, you know, the transportation
infrastructure in the United States relying more on oil than a country
like the Netherlands, but at the same token we have less dependence on
fossil fuels for electricity generation because of our nuclear and hydro
capacity.

The next issue, one of the other questions to the committee, was in
terms of weather adjustments to the emissions data. And the question is,
would it be helpful or could it introduce some sort of confusions to
people?

One of the issues is, when you\'re going \-- when you\'re
weather-adjusting just for energy demand it\'s less sticky than when
you\'re trying to weather-adjust for emissions, because the fuel
component is so important that you really have to try to capture those
changes much more importantly than if you\'re just doing overall energy
demand.

Some of the issues that would come up would be normalizing for hydro,
and this could include Canadian imports of hydro electric, normalized
for fuel switching that\'s not inherently captured in the model. that
would be maybe purchased power agreements and things like that, that
would affect things a little differently than if you could \-- than just
talking about natural gas to oil switching.

And there\'s also some current trends that are going on in coal
generation that might be happening anyhow. So you\'ve got to be careful
when you\'re looking at those trends.

But with those caveats I went ahead and, thanks to Dave Costello \-- he
provided me some data that came out of the STIFS model, to look back,
since the last few years, since \'94, have kind of been kind of
interesting as I showed in an earlier graph in terms of increases in
greenhouse gases.

And so he provided me some information based on re-running the STIFS
model and normalizing for weather. I must say though, any mistakes are
my own. I didn\'t give him a chance to scrutinize this. I just did it
quickly and put it up, because I think it is interesting to look at it.

You look at the normalized trend \-- well, first of all, \'94, they\'re
pretty close. The interesting thing starts in \'95. If you look \--
well, look at the right-hand one, the actual. You see a bigger blip. You
see the line is steeper there between \'95 and \'96, and then levels off
between \'96 and \'97.

And \'96 was a big year for weather in terms of its impact on emissions.
However, if you look at the normalized one, which is on the left, you
see that that line\'s almost straight between \'95, \'96, and \'97. And
what that would indicate is, the strength of the economy during that
time period was causing more a structural increase and emissions and
that\'s kind of masked by the weather blip-up.

So if you look at that, \'97 is actually \-- there\'s more emissions in
\'97 than you would think by not looking at the weather-adjusted. So
this is definitely a potential area to get into for the next report. We
don\'t have this in the current report but we have to look at the data a
little more, make sure we can maybe adjust for hydro and that sort of
thing.

And to look at the weather-adjusted in terms of fuel what you see is \--
I left the oil out of here because when I looked at the data oil wasn\'t
that interesting. But the way you read this is, if you look at the first
two on the left \-- that\'s natural gas, actual and normalized \-- you
see in \'95 they\'re fairly close. This is the growth from the previous
year. And then the coal is the two on the right there.

But when you get to \'96 what you see is the actual natural gas is much
higher than the normalized, and that is the weather phenomenon in terms
of increased demand for natural gas. Coal is fairly close in \'96.

Then when you look over at \'97 you see an inverse of what happened in
\'96 in terms of natural gas. You see that the normalized is much higher
than the actual. And then you see also that coal on a normalized basis,
is higher.

So you know, if these are not spurious results it\'s kind of interesting
and something we definitely want to do a little more analysis on and
think about for the future.

So that\'s it.

MR. RYPINSKI: Next I\'d like to introduce my colleague, Steve Calopedis,
who is perpetrating our voluntary reporting program, and Steve would
like to give you an update on what\'s cooking.

Oh yes, and one of the many things we did this year that I neglected to
mention, we\'re also working on our second GAO investigation this year,
and Steve has had to bear some of that as well.

MR. CALOPEDIS: Good morning. As you know, under Section 1605(b) of the
Energy Policy Act, EIA was responsible for developing and maintaining a
voluntary reporting system of greenhouse gas emissions reductions which
afforded the opportunity for entities on a voluntary basis, to report on
any actions which they feel have reduced emissions of gases or are
expected to.

Under Section 1605(b) the second requirement had to do with the
establishment of a public use database, which EIA developed and
maintains which affords an opportunity to present all the information
that was reported to EIA under the 1605, Section b, Voluntary Reporting
Program.

And it also gives an opportunity for the reporting entities and the
public to share information and benefit hopefully, from the experience
of the participants.

What I\'d like to cover today is several points. First, the issue of the
participation has shown a steady increase from the first year of
reporting, which was 1994, compared with the last year of the latest
available data for 1997.

At the same time, reduction claims have also increased, and for the
purpose of this presentation what I am going to focus in on in terms of
comparisons is the first year of reporting for 1994 versus the most
current year of reporting.

Also I\'d like to mention the OMB extension of the form which has
occurred. Additionally, as Arthur mentioned, the GAO had been requested
to perform an investigation of the program to look at various issues and
aspects related to perhaps the context of a potential early credit
system.

And last but not least, I\'d like to touch on a little bit about some
proposals that have been presented on possible early credit mechanisms.

My first slide presents some information on the increase in
participation; again, I\'m comparing 1994 to 1997 and as you can see
based on the slide, there has been a substantial increase, not only in
the total number of reporters but if you look at it by the type of
energies reporting, there\'s been a steady increase across the board.

Looking at the types of projects reported, again as reflected by the
participation, there has been again, dramatic increases when you compare
the first year of reporting versus the second year. A significant
increase if you looked at the increase in carbon sequestration projects
which increased from 78 in the first year of reporting to 299 for data
reported for 1997.

Again, increases occurred in all the other project types of categories
as well. For example, electricity, which covers generation transmission
and distribution, increased from 223 projects to almost double, to 383
projects for 1997.

For my next slide I\'ve got some information which presents the
composition of the projects by reduction objective. As expected, carbon
dioxide seems to dominate here. In 1994 there were 471 projects that
were reported with the primary objective being carbon dioxide
reductions. And in 1997, 721.

Also, again as mentioned, based on the previous slide, if you look at
carbon sequestration there was a dramatic increase as well, from 78 to
299.

Now, one of the pieces of information that\'s reported to us on the
Forms EIA 1605 and EZ, are claims of omissions, reduction of the
greenhouse gases. And if one were to look at it at the project level,
again, there was a substantial increase in reported claims of emissions
reductions when one compares 1994 to 1997.

Again, as expected, CO~2~ is the dominant mover here and another key
player is methane gas and nitrous oxide, to a lesser extent.

The next thing I\'d like to cover is the OMB extension. On March 10th
EIA submitted and OMB received, a request for the extension of the form,
and we requested that the form be extended to May 31st, 2001. On May
14th EIA received notice that in fact, that request for extension had
been approved and the current Form 1605 and 1605 EZ have been extended
until May 31st, 2001.

With respect to the GAO investigation there was a letter from
Representative Henry Waxman, a Democrat from California, to the General
Accounting Office requesting that they look into some aspects or issues
associated with the voluntary program. In the talk context of a
potential early credit system for greenhouse gas emissions reductions.

The aspects addressed by the GAO had to do with first, accuracy of data
reported and verification process; secondly, incentives for reporting
accurate information; identification of the types of projects reported
to the program and their contribution to reducing emissions; and lastly,
incentives for emission reductions and program participation.

Amongst the various findings of GAO\'s report which was submitted to
Representative Waxman on March 24th, 1998, was the fact that first, GAO
acknowledged that the program has been successful in meeting its
requirements under Section 1605(b) of the Policy Act.

And again, it had to do with the establishment of a public use database
and the establishment of a voluntary reporting system which afforded
entities an opportunity to report on their actions on reducing emissions
of greenhouse gases.

GAO also acknowledged that EIA has made some effort in the area of
trying to ensure accuracy and quality of the data through a variety of
review processes, and at the same time we have tried to place a
responsibility on the reporters to provide some form of self-assessment
of the accuracy of the information that they provide to us.

In addition, GAO did identify that DOE does have several incentives to
promote accurate reporting \-- the major one being public acknowledgment
\-- and recognition and attempts performing good will in keeping with
the President\'s Climate Change Action Plan, which is part of an overall
effort for the United States to show effort in trying to reduce
greenhouse gas emissions within the global aspect.

GAO also contacted some of the participants of the program to elicit
some feedback on why they chose to participate, and based on GAO\'s
results public acknowledgment for actions performed to try to reduce
emissions of greenhouse gases was one of the major reasons.

A second reason given had to do with sharing information and experiences
that the participants had in the various activities.

And last but not least, a number of the participants contacted
anticipate some type of early credit system for emissions of greenhouse
gas reduction.

And so a number of folks felt that this was their opportunity to get
their activities put down as a matter of record in case in the future
there is some type of regulatory environment and an opportunity for
getting credit for things that have already been accomplished.

And last in terms of findings, GAO \-- getting back to the issue of the
voluntary program in the context of an early credit system \-- GAO
determined that there were still some very basic issues that needed to
be looked into. Again, none of these issues I think, are new.

One had to do of course, with data accuracy and verification. Another
had to do with a reporting level, meaning whether it should be an
organization, whether it should be at the project level or some other
form of classification.

Another issue had to do with emissions reductions reporting baselines.
For example, whether the emissions reduction claim should be calculated
based on historical information, or whether it should be based on
projected emissions reduction levels.

And last but not least, ownership of emissions reduction claims, which
on the Form EIA 1605 is dealt with to a certain extent through the
notion of direct versus indirect emissions of greenhouse gas reductions.

Switching to the potential early credit systems, there have been several
proposals that have been put together and made available through the Pew
Center for Global Climate Change. This report was co-authored by Robert
Nordhaus and was made available on October 1st, 1998.

The report contains an analysis of proposals put forward by several
organizations: Resources for the Future, The Environmental Defense Fund,
Center for Clean Air Policy, Niagara Mohawk, and The Coalition to
Advance Sustainable Technology.

The purpose of the report was, in addition to analysis of the five
proposals, was to try to put together a set of general principles which
can be used as a guideline for policymakers to develop a workable and
effective, early credit system.

The report focused in on some of the key design elements of the program
which included for example, the amount of credit, the source of the
credit, the scope of the program, past versus future reductions.

At this point it was emphasized in their report that what was contained
in the report was preliminary in nature and many of the proposed
recommendations are under revision and advisement, and so there\'s a
very good likelihood that some of that will change.

In addition to these proposals there was a Senate Bill, S-2617, which
was introduced on October 9th of this year by Senator Chaffee\'s office.
And that Bill introduced the notion again, of granting of early credits
for emissions of greenhouse gas reductions.

And in particular of interest, the EIA\'s Voluntary Reporting of
Greenhouse Gases Program was cited in that Bill in terms of the context
of, who might be entitled to make a claim for early credit for
greenhouse gas emissions. And in the Bill it was indicated that existing
participants to the Voluntary Reporting Program would be entitled to
make a claim.

For your convenience I have included in my slide presentation, a small
set of Web site references. The first of which is being the EIA Home
Page for the Voluntary Reporting of Greenhouse Gases Program, which
contains all the documents pertinent to the Energy Policy Act, the
forms, instructions, the various publications, public use database, and
so on, related to EIA\'s program.

Also available is the report that was put out by the Pew Center on
Global Climate Change. Also, there\'s a Web site for Senator Chaffee\'s
proposal for granting credit for emissions reductions actions. And last
but not least I\'ve also included a citation address for getting access
to the report on the GAO review of the voluntary reporting program.

And as Arthur mentioned, currently GAO is undergoing performing a second
follow-up investigation for Representative Waxman to delve further into
some of the basic issues that would need to be addressed if an early
credit system mechanism would be developed.

Thank you.

CHAIRMAN RELLES: We have time for questions from the committee at this
point. Samprit.

DR. CHATTERJEE: Yes, I was wondering, do you think the increased
participation \-- you reported increased participation in \'97 as
opposed to \'94.

MR. CALOPEDIS: Right.

DR. CHATTERJEE: Do you think it came about in the hopes of future
benefits, credits given for reducing emission, or as more people got
familiar with the reporting procedure and was accepted?

MR. CALOPEDIS: I think it\'s a combination of both but my suspicion is
it\'s probably the anticipated economic benefits from an early credit
mechanism. Again, that\'s just my suspicion, but it doesn\'t seem to
appear that there\'s, you know, one specific reason that seems to
dominate over the other.

But I think that\'s where I would lean, is the anticipation of perhaps,
a potential, you know, great leaning towards environment which would
perhaps include an early credit mechanism. So I think it\'s sort of
leading to the second.

DR. CHATTERJEE: Can I have a follow-up question?

MR. CALOPEDIS: Sure.

DR. CHATTERJEE: In case the early credit is being given do you think you
might think about introducing some kind of an auditing system to get the
quality of return?

MR. CALOPEDIS: Well, that\'s what \-- I guess that\'s one of the major
issues that I guess right now GAO is sort of grappling with a little bit
further. And you\'re right. I mean, those types of issues would
definitely have to be addressed and resolved in some shape or form, and
EIA\'s role is not to make policy or decide what those positions should
be.

But you\'re right; there will have to be the issues of accounting and
there will have to be some kind of regular set of guidelines to make
sure everybody\'s playing by the same rules and so forth and so on.

But you\'re right. I mean, that\'s what I would consider one of the,
maybe one of the four or five major issues that would need to be
addressed. And again, GAO right now, at the request of Representative
Waxman, is looking into it further to see what kind of options might be
available.

DR. PHIPPS: Have there been any studies, either yours or GAO\'s \-- on
non-participants to this procedure, to the Voluntary Reporting \--

MR. CALOPEDIS: Not to my knowledge.

DR. PHIPPS: Because that would be \-- give you more information than
maybe a \--

MR. CALOPEDIS: I don\'t think so; not to my knowledge.

MR. RYPINSKI: The frame for non-participation is quite large.

DR. PHIPPS: Yes, I mean, you can sample, and I mean, I think it would be
interesting because \--

MR. CALOPEDIS: I guess one of \-- I guess one of the key considerations
here is just the nature of the program \-- I mean, being voluntary.
We\'ve tried to embed in it as much flexibility as possible.

And of course, you know, once you start talking about frames development
and identifying who should be reporting versus who shouldn\'t, then you
get farther and farther away of the voluntary notion and flexibility.

And at this point what we\'re trying to do is preserve a system that at
the same time, you know, encourages participation while also giving us
an opportunity to try to \-- you know, not only to enforce but try to
ensure some level of accuracy and quality in the data.

And of course, there\'s a drawback there. I mean, if you want
rigorousness you\'re going to have to get a little bit more strict. And
so at this point what we\'ve done is, we\'ve tried to stick and adhere
to the voluntary nature of it, and that\'s how it is.

Again, perhaps in the future, you know, there may be a shift in mood and
determination in that, and if that occurs then perhaps what you\'re
talking about might be an option. But right now it\'s not an option;
it\'s purely a voluntary program.

If they choose a report that\'s fine; if not \-- I mean, certainly we
try to encourage them, make people aware of the opportunity for them to
get public recognition and so on, but we\'re not in an arm-twisting role
here.

DR. PHIPPS: Yes, I didn\'t even mean it in terms of that kind of
compliance; just be interesting to know, maybe by major industry and
size class and things like that, who are \--

MR. CALOPEDIS: Well, Arthur\'s right. There are a lot of folks who
obviously are \-- I mean, we sort of \-- we try to do our homework and
identify potential participants, and you know, we contact them.

I mean, we have a very \-- I think a very thorough communication center
which, amongst its responsibilities is to try to evaluate and identify
who\'s not participating who might be a likely candidate, and we contact
them and explain to them and so on. And some of those folks say okay,
you know, we\'ll get on board; some folks say, well that\'s nice but
right now, no.

DR. LJUNG: In terms of, I guess there must be a reporting bias. I mean,
how often do those reports increase? You are just tracking reductions?
If you take an electric utility, I mean, they may have many projects.

MR. CALOPEDIS: Right.

DR. LJUNG: They will pick their best project and send it into you. They
say nothing about their other projects. What happens there.

MR. CALOPEDIS: As part of the Voluntary Reporting scheme we have two
forms: the Form EIA 1605 and the Form EIA 1605-EZ. The Form EIA 1605 is
a much more lengthy, detailed report. And that report provides a company
or entity to provide as much detail as they want \-- both at the project
level as well as we have a section of the report that solicits entity
levels \-- sort of company-wide information on emissions reduction.

Most of the folks that participate opt to respond to the project level
information more often than not. We do have I guess, around 40 or so
folks who report entity level information as well. But most of the folks
lean toward reporting project level information.

And within that they\'re afforded an opportunity for not just
reductions, but emissions as well. So we\'re not just collecting or
asking for information strictly on emissions reductions. We do provide
them opportunity to also report emissions, so that does occur.

DR. LJUNG: But they can choose to decide \--

MR. CALOPEDIS: Well, that\'s again \-- you know, that\'s correct. Again,
that\'s why we would admit that in terms of, you know, the accuracy and
the data quality there\'s more work that needs to be done.

But again, that\'s a function of the nature of the reporting system.
Unless there\'s some, you know, major changes in that, our hands are
somewhat tied. Again, we do go through a review process and again, the
responsibility of self-assessment is placed on the recipients.

The other thing is, I think in terms of data quality, at this point I
don\'t believe there\'s evidence to show that there\'s an incentive to
report inaccurate information. I think that what we\'re getting is
probably the best that we can get based on the available information.

And for some of the \-- for example, for electric utilities it\'s a
little bit easier for them, I think, because a lot of the information,
the operational information, is available. And so, you know, we have the
consumption of fuels, we have information on generation, and so forth
and so on.

But for some of the non-utility \-- when I say non-utilities, for
example, manufacturers and things of that nature \-- it\'s a little bit
more challenging for them because that information is not necessarily as
easily obtainable.

And so perhaps, you know, one can expect that perhaps the accuracy and
the data quality may sort of start to fall there. But as far as the
electric utilities which are you know, the largest share of our
participants, we\'re pretty confident that the information that we are
getting is relatively accurate.

DR. LJUNG: Thank you.

DR. BREIDT: I\'ve got a fairly naive question about units here. There\'s
a plot that shows 140 million tons of carbon dioxide equivalent claimed
reduction. Is that comparable to the 200 million tons of carbon absorbed
by U.S. forestland? Are they the same units or something \--

MR. RYPINSKI: No, that\'s part of the \--

MR. CALOPEDIS: Yes, right.

MR. RYPINSKI: So no, to make them comparable you divide the 140 by
3.666.

DR. BREIDT: By 3.666? Okay, so still, this is a substantial, big chunk
of claimed reduction?

MR. CALOPEDIS: Correct, correct.

MR. RYPINSKI: Another way of putting it in context is, it\'s \-- it
would be \-- I\'m doing arithmetic in my head \-- it would be on the
order of three percent of U.S. emissions.

DR. HAKES: You know, the early credit program makes a lot of theoretical
sense for several reasons. One is that, since carbon concentrates in the
atmosphere saving it early does just as much good as saving it in the
target period of 2008 to 2012.

Also, the more you make this a gradual process I think you can achieve
greater economic efficiency because you can take advantage of the
turnover of capital stock.

That being said, I think there are a couple of fairly serious hurdles
that an early credit system has to overcome. One is that the Kyoto
Treaty itself \-- and Arthur can correct me if I\'m wrong \-- but
basically does not allow for, in general, for early reduction programs.
It does under one of the trading mechanisms.

But what that means is, if you award credits based out \-- for early
action before 2008, that that\'s then subtracted from the target that
you have to meet in the period. So if this was done in any massive way
it really \-- you have to pay the piper at some point. And sometimes
this appears to be a free good if you\'re not aware of the requirements
of Kyoto.

The other is that EIA did not design the Early Climate System. We are
implementing it and we had input into the design, but basically the
design was given to the Department of Energy, not to EIA.

And the decision was made by the Department to not enforce a common
baseline system on reporters in order to make it easier to report. The
idea was that there was an educational value in an area like carbon
that\'s so new, so hard to measure, very mysterious; that to get a lot
of reporters would be very good as an educational device so there was a
desire not to create artificial barriers or great difficulties in
reporting.

However, if this is going to be monetized, the baseline issue to me is a
bigger issue than the verification issue. Because you know, for instance
you say, well, my carbon went down. Say you\'re a utility in New York
and your customers have moved from New York to Florida. Well, you as a
company have reduced you carbon emissions but that has not made any
measurable contribution to national reduction.

And there are a lot of things that could be reported that might have
been things that would happen under the normal course of business. And
all of them could have been verified and be true and we would still be
at the baseline of 33 percent more carbon in 2010 than we were in 1990.

So you could have this transfer of money going on without any
appreciable impact on national carbon levels. I think that baseline
issue is a very, very serious issue. It was pointed out at the time that
the program was put into effect.

And some of the people at that time who were arguing for early credit
were arguing for stricter baselines but sort of lost out to the argument
that broad participation had a value too. And then I think that\'s a
legitimate argument, but this will be a very tough, intellectual puzzle
to work out as we move further in this direction.

DR. HAMMITT: Can you clarify one thing you said? That if somehow we give
credit to agents for reducing emissions now that will require that even
greater emissions in the commitment period \--

MR. RYPINSKI: Right, the U.S. \--

DR. HAMMITT: \-- cap on emissions in the commitment period that\'s based
on our national 1990 emission?

MR. RYPINSKI: Yes. Basically when we went to Kyoto my understanding was
the U.S. position was that you ought to get early credit. But my
understanding is that, except for the clean development mechanism, you
don\'t really get early credit.

And so, if we say to Company *X*, we\'re giving you, based on your \--
you get so many tons of credit against your 2010 level, then we go below
the seven percent cut level.

DR. HAMMITT: That\'s within economy rather than a national level?
You\'re saying we don\'t get any credit under T&O for reducing emissions
now so if we get some firms credit for that we have to do emission
reductions elsewhere?

DR. HAKES: Right.

CHAIRMAN RELLES: I think we need to get on with the program. Oh, I\'m
sorry, did you have a question?

DR. SUDMAN: I just had a \-- on the two papers sort of combined. I guess
\-- I\'m sure the data are right. Id didn\'t quite see the data sort of
\-- one of the major sources of the greenhouse emissions by industrial
sectors, and then secondly, you know, for each of those sectors are you
keeping track of how this voluntary program is working?

Basically what I thought I heard is that at the moment it\'s mainly the
electric utilities that are participating in this thing and not much of
anybody else. Could you say a word about the sectors who are the
contributors and what they\'re doing?

MR. LINDSTROM: Well, we have on the Web site \-- and we can give you the
report \-- we have detailed information by sector. But it only makes
sense that electric utilities are the largest participants because if
you look at this number \-- department number \-- I would say it\'s
about 500 million metric tons of that are electric utilities. So that
gives you an idea. That\'s a pretty big chunk.

DR. SUDMAN: A third?

MR. LINDSTROM: Yes. So they would be the big three points and it would
make sense that they would be the largest participants.

CHAIRMAN RELLES: Perry had a couple of slides which were questions for
the committee and I don\'t think we\'re going to have a chance to get to
those. I wonder if we could try to do that offline after Arthur\'s talk?
Some of us will stick around and try to answer those or give you input
to those.

But I would like to keep this on schedule, so Arthur Rypinski\'s going
to be talking about measurement.

MR. RYPINSKI: When Bill asked me to present I spent some time thinking
about how we might best make use of this opportunity. And what I finally
decided after some dithering was that maybe the most useful thing that I
could do this morning is to spend a little time talking about, not so
much what we\'ve done in the past but the ways in which the advent of
the Kyoto protocol and the greatly increased public interest in climate
change in general, and U.S. emissions in particular, will be affecting
the EIA\'s work in the future and some of what I think of as emerging
issues that we may have to deal with in the next few years.

I\'d like to emphasize that this is a, to some extent a personal view,
in part because many of these issues haven\'t emerged yet so whether
they will emerge in fact, is a matter of opinion rather than fact, and
how one might deal, in the hypothetical case, with an emerging issue, is
also more than an opinion.

DR. HAKES: You don\'t have a model for emerging?

MR. RYPINSKI: Yes, but I used the model and it said, situation murky,
ask again later. That was always a problem.

As you probably know, the Framework Convention on Climate Change, which
is a treaty which the United States has ratified, obliges the United
States to prepare a periodic, national inventory of emissions of
greenhouse gases. It\'s a treaty requirement under the Framework
Convention.

EIA is required under the Energy Policy Act, as you\'ve heard, to
prepare annual estimates of U.S. emissions of greenhouse gases. If we
move into a Kyoto protocol world the requirements for emissions
inventories become more elaborate.

Under Article 5 of the Kyoto protocol, if ratified, would require that
each Annex 1 participant put in place a national system for estimating
emissions of greenhouse gases, and that this national system be in place
by 2008.

That\'s not entirely clear. I think it\'s \-- one could argue quite
legitimately that the United States in fact, has a national system in
place now and has had for some years. But there may be, when that
language is operationalized, or if that language is operationalized,
there may be additional requirements that flow from that language.

The Kyoto protocol would also require that countries provides
supplementary information for the purpose of ensuring compliance. And
once again, it\'s not entirely clear what that supplementary information
would be, yet, but I expect in the years to come that that clause will
also be operationalized.

And finally, it requires independent expert review of national emissions
inventories. So these are all things that one would be seeing in a
post-Kyoto world.

Now one of the problems is that there are lots of different kinds of
inventories floating out there and they have different purposes. And
when you talk about, well how much money you want to spend, how much
detail should be in the inventory, what is the acceptable accuracy?

That is frequently a function of the purpose of the inventory; how good
you have to be depends on what you want to do with the data. So it\'s
simply a specific application in that sort of general theme.

So what I would like to spend a few minutes doing is walk you through a
taxonomy of greenhouse gases emissions inventories and try to illustrate
how increasing complexity, increasing detail is a function of what it is
you want to do with the data.

So the first national inventories were not actually intended for climate
models. And these were done typically by atmospheric scientists. The
scale was global. The question they were trying to answer is, how much
CO~2~ is going up in the air and from what large sources?

The accuracy required is actually relatively low by comparison with what
came later because what you were doing with this basically \-- whether
the number was six billion tons or seven billion tons \-- from the point
of view of the atmosphere over 100-year time periods didn\'t make all
that difference; you had to be close.

And then the methods that you used to do this calculation tended to be
\-- stuff you could do at your desk was sort of the rule. And these sort
of global balances of emissions are perfectly satisfactory for the
purposes for which they were intended.

Long about 1990 we started seeing the run up to what I call, national
Framework Convention inventories. And these are national inventories in
which the analytic question is, what\'s the national total? That\'s
really all you need to know to comply with the Framework Convention.

What\'s the national total and is it going up or down? You don\'t
actually need, for Framework Convention purposes, to know very much
inside the box; what\'s going on.

For this kind of work, random errors are really bad because it blurs the
question of whether or not emissions are going up and down. You can have
\-- if your estimate of emissions is up three percent one year and down
four percent the next because of artifacts in your measurement, that
makes it hard to see whether emissions have gone up two percent or not.

Bias errors \-- nobody likes bias errors but they\'re not as big a
problem in this context because they are likely to be serially
correlated and will persist from one period to the next.

There is an unwritten rule for Framework Convention inventories that has
been pretty much observed by all the countries that have done them;
which is that they shouldn\'t cost too much. And they haven\'t.

Moving on is this concept for an inventory for policy analysis. Now,
inventory for policy analysis, the scale jumps down from the national
level down into sectors in individual sources.

But what you really need to know for this stuff \-- and you need to do
this because you need to start thinking about, well if you were going to
influence the outcome of the emissions inventory, what are places where
you might think about pulling levers?

And so \-- interesting thing about accuracy in this context is that it
may be more important to be right about the causal mechanism than the
absolute number. The absolute number is important but the causal
mechanism is important, too. These also should be relatively low cost.
How you do them: measurement where you can; estimation when you have to.

And that\'s pretty much the state of play today in the United States. In
other countries they\'re either in this \-- they\'re in this ballpark or
perhaps one step behind us.

And then the next category is emissions inventories for compliance
purposes. And there are lots of emissions inventories in the United
States for compliance purposes but they\'re not emissions inventories of
greenhouse gases. They\'re emissions inventories of criteria pollutants,
they\'re inventories for sulfur, they\'re inventories for emissions that
are regulated under the Clean Air Act or other laws.

The scale usually is by the regulated source, and what you really want
to know is the emissions for the guy you\'re regulating so you can
figure out whether he\'s in compliance. They tend to be really expensive
because there are many, many elements in the inventories.

In fact, actual compliance inventories are kind of funny things, where
they have high levels of details for some sources and they\'re very
general for other sources. And then where there\'s detail is really
mapped by the regulatory strategy. You have high levels of detail where
you must have it to implement the regulatory strategy and then you have
much lower levels of detail for others.

Compliance inventories are very expensive. I think I said that at least
four times. The methods incline much more towards direct measurement and
much less in the direction of estimation. So I should really say, direct
measurement for the regulated sources, and they were still frequently in
estimation for unregulated sources.

What we do in EIA under the greenhouse gases program is what I would
more or less characterize as an inventory for policy analysis. That\'s
really what our thinking is on who we\'re doing it for and the kinds of
uses we expect to be made for it.

However, in the future I expect that people will be thinking about
designing compliance inventories to support policies such as credit for
early reductions and emissions trading. And well ahead of whether
there\'s any legislation there\'s going to be a lot of thinking about
this. And I think that these kinds of things are already of increasing
interest inside the United States Government, and certainly outside it.

And that will have consequences since on the order of, as Perry
mentioned, 84 percent of U.S. greenhouse gas emissions come from carbon
dioxide, and of that, 83 percent is from combustion of fossil fuels, the
accuracy, the detail and the quality of U.S. inventories, at present and
probably into the immediate future, is going to depend on the accuracy,
quality, and detail of EIA energy statistics.

And so if there\'s only one sentence that I\'d like you guys to take
away from here, it\'s that there\'s going to be much more government and
public scrutiny of EIA\'s energy statistics in the future than it has
been in the past. So if I were to sum this whole thing into two words
it\'s \"get ready\", because I think it\'s going to come.

So moving from the general proposition to some of the special
propositions \-- we\'re already there, we\'ve already moved. Excellent.
I would list five themes. They\'re not exhaustive. They\'re just the
ones that happened to come to me on the specific afternoon when I
absolutely had to finish this report because Bill was screaming at me at
15-minute intervals.

The first one and the most important to me is, how do we characterize
the reliability of U.S. energy data? The second one is, how do we handle
individual respondent data? The third one is, more comprehensive summary
presentations of energy data. The fourth one is more inter-Agency
collaboration. And the firth one is, international verification and
technical assistance. And I\'ll talk a little bit about each of these.

As I\'ve said also four times, the reliability of U.S., and for that
matter, foreign greenhouse gas emissions estimates is largely a function
of the reliability of the underlying energy data. And there are various
pockets of the United States Government that are getting very interested
in quantitatively assessing the uncertainly in national \-- and for that
matter, in project level \-- greenhouse gas emissions.

There\'s an IPCC working group on uncertainty and emissions estimates.
That IPCC working group has produced guidelines in the past and will
produce more detailed guidelines in the future, which will require a
quantitative assessment of the uncertainty in emissions estimates.

And if the United States adheres to the Kyoto protocol the United States
is also obligated to, at a minimum, to pay attention to those IPCC
guidelines. In the slide it says the U.S. must follow. Reality is a
little more complicated than that but we certainly can\'t ignore them.

So I think that the United States Government will be asked to prepare a
quantitative assessment of the reliability of U.S. energy data, and I
anticipate that this request will likely find its way to the EIA.

Parenthetically, I went up to my office to make some copies of the
handout to give to you guys this morning, and the phone rings. And I
pick up the phone and I answer it and this guy says, well my name is
so-and-so and I work for the Central Intelligence Agency.

So I say, yes. One moment, I\'ll pull the blinds and I\'ll climb under
my desk. What can I do for you? And he says, how accurate is U.S. energy
data? And I said gosh, that\'s a really interesting question.

DR. HAKES: The best in the world.

MR. RYPINSKI: That\'s what I told him. I told him we have the best
energy data in the world. And he said, well does that mean it\'s within
five percent? So I gave him a long dissertation in the course of not
answering his question. But I think he may be back.

And I\'m not really worried about the guy in the CIA, frankly. There are
some other guys from some other places. I\'m also worried that if we
don\'t do it there may be some people in some other places who might be
willing to do it for us, possibly, without asking our opinion. And
that\'s something to keep in mind.

The second broad theme is individual respondent data. It turns out that
this discussion of emissions trading and credit for early reductions and
also the notion of moving in the direction of compliance inventories, is
going to create interest \-- in fact, it\'s already creating interest
\-- in using individual respondent data collected by the EIA for
compliance or verification of early reduction.

And this raises a number of sticky issues, which is why I\'m bringing it
up. And the first one of course, is the confidentiality question. A
second question that may be relevant is quality control.

It turns out that, you know, we QC our surveys mostly to get to the
total, and it may turn out that if somebody uses the dataset for a
purpose for which it\'s not intended they may get some anomalous
results.

There are also lots of definitional issues. We tend to design our
surveys as a way of getting to a national total and there may be lots of
sticky things associated with somebody using it for some other purpose,
using a definition that doesn\'t really match the definition used in the
survey. So there\'s lots of potential for people to misunderstand and
misuse data used in this way.

Also, support and dissemination for individual respondent data if we
were to go down that road, is like many, many times more \-- there\'s
much, much more of it. It\'s an order of magnitude more complicated and
there\'s just more of it than anything we contemplate today.

And so I don\'t have any answers. I just think that this is something
that we\'ll be seeing in the future and that we would profit from
thinking about: how can, how should we respond to such requests and what
are the issues other than my five minute scan of them that would be
associated with this sort of thing?

This is something that I think I can see on the horizon and it\'s coming
in our direction.

Five more minutes. Boy, I\'m going to talk really fast now.

Better summary data. I don\'t think that\'s so important so \-- it\'s a
topic close to my heart but we\'ll skip it.

More inter-Agency collaboration. I think that\'s another wave of the
future. Certainly on the inventory side, as I\'ve said now 12 times,
energy related carbon dioxide is 83 percent of the total but there\'s
another 17 percent of the total that\'s not.

This is doing the non-CO~2~ submissions; requires non-EIA data and it
frequently requires expertise that may be very thin in-house. It may
require research in fields in which we have no expertise. It doesn\'t
\"may\", it does.

And so the best way to go after this will be collaborative and we should
think about how we might do that better in the future.

Also getting back to this reliability issue, one of the ways of pursuing
bias error in the energy data may be to start systematically using
comparable surveys by other agencies as a QC check. And once again I
suggest that other parts of the American government are going to be much
more interested in EIA energy data in the future than in the past.

And you can see this as a problem or you can see it as an opportunity;
and particularly at some of these contexts maybe we should see it as an
opportunity.

Last, another subject that\'s not so close to my heart but I think is
interesting \-- it\'s another one of these cloud on the horizon things
\-- is that the Kyoto protocol requires these national systems emissions
estimation, it requires independent expert review of inventories, and it
requires all the supplementary information \-- whatever that quite means
\-- for the purpose of ensuring compliance.

It will turn out I think, that improving the quality of national
inventories is going to require reviewing, measuring, and improving the
quality of national energy statistics.

And another sort of strategic issue maybe we ought to think about is,
does the EIA wish to involve itself in participating in international
inventory reviews, or in technical assistance for improving
international energy statistics, internationally or emissions
estimation?

This is something that we do now in a fairly low-key way but I think
that the salience of energy statistics will rise as a result of Kyoto,
and technical assistance for doing energy assisted statistics abroad
will become increasingly important, and we should think about as an
institution, whether or not we\'d like to get involved and if so, how,
and what are the problems and issues associated with doing that.

That\'s all I\'ve got.

CHAIRMAN RELLES: We have two scheduled discussants this morning. Phil
Hanser couldn\'t be with us but did send some comments which I will
read. But Jim Hammitt is the other discussant. I\'d like to welcome Jim
to discuss this first.

DR. HAMMITT: Thanks. I think Arthur\'s done a really nice job of
identifying a lot of issues that we need to think about and it\'s really
nice to see proactive thinking: what are the issues arising and do we
want to try and deal with them?

So I\'d like to sort of highlight a few of those, I guess, and give some
comments on them of my perspective. One is the last thing you brought up
\-- the involvement in international activities. And I would think this
is a relatively important thing for EIA to do.

I guess \-- you didn\'t say, but on this IPCC committee that\'s drawing
up guidelines, is there someone from the IA that is part of that
committee?

MR. RYPINSKI: I attended one meeting.

DR. HAMMITT: Okay. Well presumably this is one of the stronger
collections of, in the world who know how to measure this kind of stuff.
So you have a lot to offer in doing it well.

Well, point one is, EIA can help those guidelines be written in a better
way just in the abstract, and then more specifically, they can be
written \-- what are their choices and some ways would be easy for the
U.S. to deal with and other ways would be hard for the U.S. to deal
with? You might as well try and write them the way that it would be
easier for us to deal with, or that\'s sort of an arbitrary choice.

Also, this idea mentioned in the paper: EIA could either review other
countries\' inventories or estimates or even if EIA doesn\'t do that it
will certainly likely to be called on to help whoever in the U.S.
Government is doing that review function, to assist them in the review.
So understanding more about other countries\' systems would be valuable
in that regard.

Understanding how to think about and assess the accuracy of systems is
clearly very, very important. I don\'t have a whole lot more specific to
say on that except to endorse your attention to that matter.

In talking about these things \-- oh, it seems clear that it would be
very valuable for EIA to just align its definitions to the extent
possible, with whatever the international energy agency or the IPCC
group are recommending.

So things like the inclusion of the territories or exclusion, and the
bunker fuels issues, and maybe you want to have two sets of accounts,
but you want to be publishing stuff. If there is an international
standard coming about it would be a lot easier to be consistent with it,
and I guess \-- I hadn\'t thought about this but does that mean we
should be using metric units? I guess we\'ve resisted that one for a
long, long time.

And on the unit questions, I guess all these total emissions that were
given are in \-- well, we\'ve already found out, sometimes they\'re
carbon, sometimes they\'re CO~2~.

MR. RYPINSKI: But they\'re always metric.

DR. HAMMITT: Okay. That\'s good. And when we\'re converting from
non-CO~2~ gases, is that global warming potential 100-year horizon \--

MR. RYPINSKI: It\'s 100-year horizon, yes.

DR. HAMMITT: And is that officially written in stone or \-- because its
conversions are very sensitive to that \--

MR. RYPINSKI: No, the choice of GWP is ultimately arbitrary. And by one
of those mysteries of fashions, the 100-year integration has won the
popularity contest on which column to use in that particular table, and
we are simply following along the lyrics of some ancient song, because
we\'re dedicated followers of fancy.

DR. HAMMITT: Right, but also throwing out statement like 84 percent of
U.S. emissions are CO~2~, if we use the 20-year GWP, that answer would
change a lot, I suspect. It wouldn\'t be 84 percent, it would be 60
percent or something. But maybe not that big.

MR. RYPINSKI: No, it would be more, actually, because the \-- let me
think about that.

DR. HAMMITT: I think \-- you know, in dealing with the climate change
problem GWP is \-- as you said, the horizon is totally arbitrary and I
don\'t know in fact, what Kyoto protocol says about what we\'re
committed to.

Its sum total basket of greenhouse gases and obviously the way in which
they\'re traded off is essential to defining whatever it is the, I guess
the (unintelligible) is not letting us commit to. So more attention to
how, not just signing on to GWP 100, it would be \-- for a start. I
think that\'s an issue that needs more attention.

If you get into, I guess EIA is responsible for emissions inventories
for all greenhouse gases, and obviously as you said, that requires
expertise and data on a whole lot of things you traditionally haven\'t
done: agriculture, land use changes, this missing sink stuff, carbon
going into and coming out of the soils, and all this stuff which is just
a tremendous challenge.

It obviously requires a lot of people from expertise that\'s not
represented here now. And I don\'t know exactly how you\'re going to
manage that, but good luck.

On the extent to which EIA collects data that are used for enforcement,
that seems like a big change in the character of the Agency and it could
have negative repercussions for getting people to comply with voluntary
surveys and so forth. And it also \--

MR. RYPINSKI: You mean the compulsory ones.

DR. HAMMITT: It seems like that would be a big sea change in operations
and how things are done and perhaps the kinds of people need to do them,
and would affect your ability to do some of the things you have done
relying on cooperation and a good working relationship with some of the
survey respondents.

MR. RYPINSKI: I would just point out that I would be the last person to
advocate such a thing. It\'s something that\'s in the air. The question
is not, do I think we should do it? No. The question is: It\'s out
there; what do we do now?

DR. HAMMITT: Well, is it clear that EIA is the group that should be
doing that as opposed to a regulatory agency?

MR. RYPINSKI: No. No, it isn\'t.

DR. HAMMITT: Well, I think I\'m supporting you and trying to do that. As
Dan said, we\'re here to help.

I think it would be useful to collect more information that \-- maybe
you already do this \-- but not just emissions but find out what people
are doing that influences emissions. Because for this policy analysis,
obviously it\'s important to look at trends and to be able to decompose
those into a bunch of factors.

So if you can find out from people sort of where they\'re shifting
technologies or process changes, and also what the costs of those
changes are or if they\'re buying equipment \-- like scrubbers or
something wouldn\'t apply here \-- but something analogous.

For analyzing these trends and evaluating how we\'re doing, we obviously
care about the costs as well. So if you can obtain more information on
those that would be very helpful.

These questions about what analysis should be done and the controlling
for weather and so forth, I think as Perry\'s example shows, that\'s
vitally important.

We probably need to do it both ways but we certainly need to be doing
analyses to adjust for unusual weather patterns and probably also
business cycles, and that what we care about is average emissions of
these things \-- gases over decadal periods.

And looking at year-to-year values is not even particularly relevant, so
anything you can do to focus people\'s attention on underlying trends as
opposed to, this year we\'ve earned a huge amount or not much, would be
very valuable.

I\'ll stop there.

CHAIRMAN RELLES: Thank you, Jim. Phil is actually quite sick with
bronchitis and apologized profusely and I made him pay by sending me an
e-mail message based on Arthur\'s paper, which I think Arthur was \-- I
didn\'t really see a lot of traces of it here today \-- so some of these
comments will be kind of on the paper but most of them are I think,
still relevant. So let me just read them verbatim. I left my bow tie at
home, unfortunately.

\"I must begin with my strong concern that measuring greenhouse gases
and developing an estimate of the nation\'s inventory of these gases is
a Don Quixote exercise at best and at worst, creates the potential for
international gaming of the highest order.

\"It\'s not very clear to me to what use they can really be put. If the
goal is to get countries to change their admissions as greenhouse gases
then what might be more useful are inventories of a very specific sort
\-- electric generation plants, transportation \-- that are most likely
to be amenable to changes in their capital stock in such a way as to
reduce their emissions of greenhouse gases.

\"My concern is we will find ourselves trying to get increasingly
accurate measurements of emissions for which there is little or no
likelihood of control. For example, methane emissions from agricultural
animals.

\"This is all a long-winded way of saying that not all information is
equally valuable and that the likely first step would be simply to do a
standard decision analysis calculation on the expected value of perfect
information on a source category basis.

\"I think that Arthur\'s taxonomy of emissions inventories is
interesting but I am at a complete loss as to where he derived his
estimates of required accuracy. Why is it the national emissions targets
require ten percent accuracy but climate models are okay with 25
percent? I don\'t really have a good a priori feeling about why this is
so and I find his explanations a bit lacking.\"

And that was in the paper and I think you probably decided the same \--

MR. RYPINSKI: I came to the same conclusion about 15 minutes after I
wrote it.

CHAIRMAN RELLES: Yes. \"The question of uncertainties and characterizing
them appropriately is very difficult if not intractable. I\'m not sure
that the problem is distinguishing between risk and uncertainty in the
way that Frank Knight used them.

\"In particular, risk estimates are what comes out of the usual
probability calculations that are done, while uncertainty is the
unquantifiable source of randomness. This is not to say that we cannot
be good basions and assign a probability to some form of uncertainty,
but that such an estimate does not flow out of the analytical model.

\"Thus I think that EIA can go a long way toward characterizing the
riskiness of its estimates and perhaps even get to the underlying
correlatedness of the risks of the estimates. However, there are some
uncertainties that are never going to be analytically unravelable.

\"My concern here is that only the policymakers would like to quantify
them and that the EIA might be coerced into an estimate that, while
educated in its reasoning, is still just a single, individual\'s
prior.\"

I didn\'t see the distinguishing nature of risk and uncertainty in what
we just discussed. That too, was eliminated from the paper. Did you see
these comments before you gave the talk? Okay. The RAND mail system may
not be that secure after all.

\"Arthur rightly raises the issue of data availability and
confidentiality but I have concern that some data that is crucial to
achieve greenhouse gas inventory is going away; in particular if the
electricity industry deregulates the data on generation is no longer
going to be available.

\"I believe that this issue has come up before but I don\'t believe
I\'ve yet heard a resolution of it. I agree that the EIA has much to
offer, both other agencies and even other countries in statistical
knowledge and experience.

\"I\'m a little concerned that EIA might not have the resources that are
implied by such collaborations. I leave it to EIA\'s management as to
whether creating such roles for EIA will necessarily bring the attendant
resources required.\"

And Jay, I told Phil I\'d get back, and so maybe you can answer that
question soon.

\"Finally, I would like to raise one additional issue that the paper
does not seem to address explicitly. I think that a significant focus
will be measuring the impact of a particular change in policy. This puts
us in what I would like to call, \'but-for\' land. What would emissions
have been but-for \'blank\'? Fill in the blank with the appropriate
remedy.

\"I think this is a very perilous country to be in. Much money was
spent, likely with very little results on utility energy conservation
programs. I fear that greenhouse gas emissions reductions will be much
like the phantom reductions of energy consumption will do to such
programs.

\"In the case of energy consumption programs the problem was the
inability to account for the impacts of energy price changes on both
consumption and investment behavior, and thus, correctly characterize
the \'but-for\' world.

\"I know that the EIA\'s policy models are in part, oriented toward
this. I don\'t have a way of resolving my concern though. I believe it
needs to come back periodically. I hope these remarks are of some use to
the EIA as it goes forward with its greenhouse gas inventory efforts.\"

I\'d like to thank Phil and Jim both, for their wonderful comments and
invite Arthur to respond and then we can throw the floor open for
questions.

MR. RYPINSKI: Let\'s see. I didn\'t actually know I was supposed to
respond.

CHAIRMAN RELLES: You don\'t have to.

MR. RYPINSKI: That\'s all right. The quantitative emissions \-- the
quantitative accuracy that was cited in the paper was, I thought \--
mature consideration was unwise. [Excise]{.underline} didn\'t mention
it, actually, so I agree with the comment provided.

In terms of whether we can pursue \-- I think that measuring uncertainty
in EIA energy statistics is an intractable problem and I think that\'s
one of the reasons why it hasn\'t happened 20 years ago. If it were easy
it would have been done a long time ago.

I don\'t think that actually lets us off the hook, though, and I would
suggest that there may be ways of thinking about the problem; that we
could never do a perfect job but there may be ways of doing something
and doing things that are better than we\'re doing now.

Let\'s see, what else were we? In terms of collaborative international
roles exceeding the Agency\'s resources, I think that\'s actually a very
cogent comment and one that we would need to think about hard before we
got into it, rather than after.

And in some cases such activities might be a source of resources but
that raises its own set of questions, like you know, nothing\'s free in
this world. I guess that\'s all I\'m here to say.

CHAIRMAN RELLES: His other question had to do with proposed remedies and
whether it\'s possible to measure their effects.

MR. RYPINSKI: Oh, yes. Well, I think that the EIA \-- I think that\'s a
big problem associated with the clean development mechanism and the
credit for early reductions. It\'s going to be an issue that people are
going to be wrestling with over the next few years.

And I suspect that it will appear on EIA\'s radar scope in the form it
has appeared on a small scale already which is, there are all these
voluntary programs and my colleagues who run the NEMS have to decide on
what are the effects of these voluntary programs to roll into the NEMS.
I think that was \-- that\'s a problem.

CHAIRMAN RELLES: Jim?

DR. HAMMITT: That was an area that I thought of commenting on; this
attribution of Phil\'s \"but-for\", and your comments earlier about how
much credit to give people for voluntary reductions and all that.

You know, the counter-factual results is always unknown. It can be
estimated in various ways. But I think the thing that\'s good about the
Kyoto protocol relative to another standard which says, reduce emissions
by 23 percent below what they otherwise would have been is, we have some
number which is our target which we\'re aiming at.

So I think you ought to be focusing effort on measuring emissions from
the various sectors and understanding the changes going on in the
sectors \-- fuel switching, industrial change in composition, all this
kind of stuff \-- in order to have an informed idea about our trends
kind of getting us somewhere near the target we want to hit.

Or are they high and are there some sectors where we\'re seeing
favorable trends and others that are unfavorable, and help us understand
what the factors are influencing those trends, and stay away from the
kind of \"but-for\" stuff \--

MR. RYPINSKI: Well, having said that \"but-for\" is difficult, the
\"but-for\" problem is in fact, the classic problem in project analysis,
cost benefit analysis \-- either ex-ante or ex-post. I mean, it\'s
another one of these things that\'s hard but if you\'re \-- you know,
it\'s bread-and-butter work for energy analysts.

So I think that any kind of policy assessment has to exist in a
\"but-for\" world. So I expect we\'re going to be doing this kind of
thing. The really tricky thing is when you start \-- is if someone were
to start thinking about paying for \"but-for\" cases. But that I will
leave to my colleagues in policy to worry about.

DR. HAMMITT: But if, let\'s say we agree to comply with the, you know,
protocol, you have plenty of interesting and challenging things to do
just to sort of help us see how we\'re doing on the road to that. So
I\'d be leery of detracting too much from that side in order to
evaluate, you know, is this really the right target, should we be having
more stringent or less stringent reductions, and so forth.

MR. RYPINSKI: The way I think about it is that the \-- and in 1605-speak
we refer to it as \-- a wonderful euphemism \-- a modified reference
case. Which means a comparison between an actual outcome and a
hypothetical, what would affect it.

What modified reference cases are actually \-- you actually have to use
them for is to measure the impact of a particular action, in isolation.
And in fact, if you\'re trying to measure the impact of a particular
action you wind up forced, willy-nilly, into using a hypothetical
baseline.

So one way of thinking about it is these absolute baselines, a total
like the Kyoto protocol which you must hit, measures outcome and
doesn\'t care about why it happens. Whereas if you go into hypothetical
reference cases and hypothetical baselines you\'re measuring the
consequences of individual actions without reference to what the impact
on the greater scheme of things is.

So one analytical technique aims at particular actions; the other
analytical technique aims at large-scale outcomes. And what you want to
do of course, is have the right tool for the right job.

DR. CHATTERJEE: I think I want to bring a slightly optimistic assessment
of a very difficult situation \-- this measurement process. There may be
measurement processes or analysis which are highly inadequate in trying
to estimate the actual level, but you might be able to get a very good
assessment of the change in levels.

And I think the main problem here we\'re looking at is trying to
estimate the change in level as accurately as possible. And I think that
may not be as difficult a task.

And after, in order to \-- I think that\'s the basic question \-- and
after that if you have set target levels for measuring changes, what
policy tools are you going to use to bring t out is a very different
question, I think.

I think we are at present, really grouping under the case of trying to
get an idea of assessed change of level rather than the level itself.

MR. RYPINSKI: I have the opinion that the \-- and it\'s opinion because
I don\'t actually know this \-- but my opinion is that the random error
in EIA energy data is rather low and that the errors that we don\'t \--
the ones that we don\'t know about, because all the ones we know about
we fix, right?

The unknown errors are probably mostly bias errors; that is, it\'s an
error we make in the same way every year. But I can\'t, you know, I
can\'t demonstrate that. I just think it\'s probably true and there may
be other people who know the EIA energy data systems in more detail and
may have other opinions.

DR. CHATTERJEE: And I was actually not just referring to the EIA energy
data as even thinking of estimation of the carbon emissions.

CHAIRMAN RELLES: Are there any more committee questions? Questions from
the audience? As asked earlier \-- oh, I\'m sorry, Lynda?

MS. CARLSON: Actually, it\'s not a question. I just wanted to note that
some of your last slides, the ones on some post-Kyoto challenges for
EIA, especially when you talked about the data quality of EIA\'s data
and confidentiality and the issues of summary presentations, are in fact
themes that you\'re going to be hearing the next two days and things
that EIA is constantly looking back at \-- not just in terms of the
emissions but overall questions facing them.

CHAIRMAN RELLES: That\'s one of the things we can help on the most, I
think.

Jay actually apologized earlier for leaving town but he volunteered to
be available either now or offline afterward to answer any questions
that may arise. Would anyone like to ask Jay a question in a large group
or in a small group? I guess it will be in the small group.

So it\'s 12 now. We\'re due back here at one o\'clock, and lunch will be
right across the hall. thank you very much.

(Whereupon, a brief luncheon recess was taken at 11:57 a.m.)

A-F-T-E-R-N-O-O-N S-E-S-S-I-O-N

1:06 p.m.

CHAIRMAN RELLES: I\'d like to reconvene the meeting of the ASA
Committee. This afternoon we have two talks, two different sessions: one
involving probably three different talks.

And they\'re mostly about data acquisition \-- data acquisition into the
electricity restructuring era and activities going on in statistics
working groups now that interact with other agencies and to improve
EIA\'s consistency definitions so that when other agencies ask about
this or that you have some answers for them.

So let me begin by introducing Betsy O\'Brien and John Colligan. Did I
pronounce that name right?

MR. COLLIGAN: Yes, you did.

CHAIRMAN RELLES: Good. I really screwed up this morning. And invite them
to come up and give their talk, please.

MR. COLLIGAN: Good afternoon. My name is John Colligan and I work in
EIA\'s Office of Coal, Nuclear Electric Power, Alternate Fuels, and I
worked on this product of confidentiality for a little while.

I do feel a little bit like the mosquito who flies over a tall wall and
finds himself in a very well-populated nudist colony. I\'m delighted to
be here but I hardly know where to begin.

We\'ll talk a little bit about some of the things that are involved in
confidentiality. I\'m going to talk a lot about the regulations. Of all
the things that have been discussed and published these are the least,
and frankly, these are the things that govern what we do.

The electric power industry is an environment that is changing; an
industry that is really in a state of flux. Our electric utilities that
are moving away from the closed service area markets to open retail
competition. Currently, the restructuring is taking place in the retail
generation sector.

EIA is still operating the same but we\'re operating under different
conditions. Electric power industry changes by necessity require changes
in EIA data collection, data dissemination, data disclosure.

Confidentiality is not necessarily a new issue. Hints of the issue have
been raised in the past from time-to-time; principally I think, with the
non-utilities. However, it started \-- our current interest in
confidentiality started in earnest in the spring of 1997, with some
respondents seeking non-disclosure on specific data elements they
consider confidential.

And they did this when submitting the forms under normal course of forms
reporting. And of course those data elements were not confidential and
the forms were very specific about what was confidential \-- in this
case, nothing \-- and what wasn\'t confidential.

We also got a lot of unsolicited comments when we went out for a forms
clearance extension and some minor changes in 1997. EIA has been very
proactive in their approach of this issue, as has been raised.

Two things that we did in 1997: one was, we asked for a single year
extension on the forms clearance and on the time extension of the forms.
That was not necessary. The normal is a 3-year extension. There\'s
little doubt in my mind that if we had gone with the normal 3-year it
would have been granted, but we had this confidential issue in mind.

We also submitted to OMB a proposal and they accepted it, that we have a
single issue solicitation of comments through a *Federal Register*
Notice on confidentiality.

The reason for the single issue is very clear: we didn\'t want it
embedded in anything; we didn\'t want it to be confused or in any way
hidden. We wanted to make sure the confidentiality of all the people who
were involved were aware of the problems and give us their ideas and
thoughts on a possible solution.

And that first solicitation was made through the *Federal Register* of
January of 1998. And we started essentially with, should we be changing?
And we had a couple of questions that we asked people to respond to.

What electric power data should be treated as non-confidential? Explain
how release of the data is in public interest. Why should it override
considerations of the likelihood of competitive harm to the energy
providers?

The other question was: what electric power data should be treated as
confidential? Explain how release of the data is likely to cause
competitive harm to the data providers?

We received over 115 comments; some single commenters represented large
group of the industry and stakeholders, so the multiplier effect there
was more than 115. But there again, numbers really \-- this isn\'t some
sort of a popularity thing. It\'s a question of whether the case is made
and how the case is made and if it makes sense.

Confidentiality decisions do not affect what is collected. I mention
that primarily because we\'ve been misquoted and there were times when
people think if the data becomes confidential it will no longer be
collected.

Confidentiality goes hand-in-hand with dissemination. It has nothing to
do with data collection. The reason to collect a data element stands on
its own merits. It makes sense to collect that data element.

We\'ve studied the comments we\'ve received and used the laws and
regulations governing disclosure issues as a basis to develop a new
procedure on confidentiality. We\'ve made several announcements using
the *Federal Register* as the vehicle to publish the information.

The first, as I mentioned a moment ago, was in January of this year \--
January 13th \-- and it was a first request for comments on
confidentiality as a single issue, not as I say, embedded with some
other activities that we were involved in.

Then on the 30th of June we published a *Federal Register* Notice
advising what we were going to do with our forms. There\'s some minimal
changes. We\'re asking for a 3-year extension to keep the data series
going. Betsy O\'Brien is going to talk in more depth about the forms
themselves and what will be happening in the redesign.

On the 17th of July we published our Proposed Confidentiality Procedure.
This procedure is based on what we had been doing, the information we
collected on the first *Federal Register* Notice, and as it applied to
the laws and supplies of DOE and the regulations that we operate under.

To be published Monday the 23rd, I hope, the Final Forms Clearance and
Confidentiality Policy \-- there will be a notice in the *Federal
Register* that that is going to OMB.

Actually, the support package is at OMB right now and they are supposed
to publish the *Federal Register* Notice on the 23rd, and that\'s the
day that the time starts to click and gives us 30 days for them to
approve or make whatever adjustments they want.

The authorization statutes of EIA really are what EIA\'s right to
collect surveys \-- to collect survey data and collect data itself \--
is in fact, a legal obligation to collect data. The statute details what
EIA must do.

The reason I go into some of these things, if anyone reads a lot of
comments you\'ll find on both sides of the issue people are claiming
that EIA is not doing what they\'re supposed to or they are doing what
they\'re supposed to. And these are same laws, same basis, same
everything \-- two opposite views. That\'s why we have lawyers.

The data dissemination equally as well is a legal requirement, and part
of the law says that the data collected must be made available to the
public except for exempted matters.

Freedom of Information Act, FOIA, is our principle vehicle for giving
out data, for rejecting claims of people who want data that we hold
confidential; all the aspects of that. FOIA requires disclosure of data
collected by federal agencies unless the information sought falls within
one of the exemptions of the Act.

The exemptions potentially relevant to EIA\'s data collection and
dissemination are exemption 3 and exemption 4. Exemption 3 is
information specifically exempted from disclosure by statute. Exemption
4 covers trade secrets, confidential, commercial information.

Let\'s look at exemption 3. It specifically exempts from disclosure by
statute, provided such statute requires that the information to be
withheld from the public in such a manner as to leave no discretion on
the issue. It must be straightforward. It\'s not a question of, well
maybe so and forth.

Or the statute establishes particular criteria for withholding, or
refers to particular types of data to be withheld. They call these,
amazingly enough, withholding statutes. Therefore, if such a statute
sets forth specific criteria, that criteria must be followed.

The statute must clearly state that the data gathered under its
authority is not to be released to the public. A mere statement that
information is to be kept confidential as specified in FOIA will not
suffice to create a withholding statute.

Withholding statutes need specificity. EIA\'s enabling legislation under
the statutes on which we operate do not contain any exclusions data, and
therefore there\'s no expansion of that exemption 3 that applies to EIA
to offer non-disclosure.

Is a Trade Secrets Act a withholding statute? It\'s a very good
question. A Trade Secrets Act again, is one of those that we hear much
about, or we\'ve heard much about in the comments, of people defending
or trying to make a particular position regarding data to be released or
not to be released.

For purposes of non-disclosure under exemption 3, is that legitimate?
And case law has indicated that does not meet the requirement of
withholding statute. So the idea of the Trade Secrets Act is not to be a
viable one.

The only possible place that we have seen so far where it might apply
would be if a utility \-- because people all generate electricity pretty
much the same way. They may be more efficient via equipment and so
forth, but it\'s not something necessarily new.

If a utility entered into a Trade Secrets Act with a manufacturer of a
boiler or a generator or some means, that probably would allow them to
\-- you know, might give them protection under the Trade Secrets Act.
But that\'s highly unlikely and probably doesn\'t exist.

Exemption 4 of FOIA protects from disclosure, trade secrets and
commercial financial information, data obtained from an organization,
and privileged or confidential data. But protection requires all three
of these elements to be present. It\'s not a two-out-of-three or
anything of that sort.

There is no definition of these elements per se in FOIA. If you go
through FOIA you\'re not going to find these defined as such, but the
concepts are defined. That\'s where it gets to be a little more fun.

However, when applying FOIA concepts the elements of this extension are
narrowly defined. There\'s evidently a broad definition of these. The
courts have rejected the broader definition when looking at the FOIA,
and required a much narrower definition.

There\'s been some dispute of information that qualifies as a trade
secret for protection under exemption 4, but the fact that these
requirements have been narrowly defined by the courts has pretty much
rejected that.

A trade secret for FOIA purposes is a commercially valuable planned
formula, process, or device used for making, preparing, or processing
trade commodities that can be said to be the end project of innovation
or substantial effort. The definition excludes the type of information
that we collect at EIA.

Commercial financial data is considered to be confidential and thus be
exempt if the following conditions are met. Is disclosure of the data
likely to impair government\'s ability to collect future data or cause
substantial, competitive harm to the data provider?

The answer to number one is, our data is all mandatory data so there\'s
no harm in collecting it. And you really can\'t look at the fact that
people might drag their feet because that\'s illegal and that\'s not
what we\'re talking about.

Disclosure could impair \-- there\'s been some point made \-- not at EIA
\-- and I don\'t know, this came out in our research \-- that some
agencies feel that disclosure of the data has hurt their \-- or,
non-special data has hurt their analysis program. I think almost all of
our analysis programs rely on the data that we collect on the forms
themselves.

The second prong of the test: will disclosure cause substantial,
competitive harm to the data provider? General rule as stated by the
courts is, there must be actual, competitive harm \-- a showing if you
will, of such \-- and a likelihood that substantial competitive injury
would result from disclosure of the data by EIA.

If the information is available through another source it will not be
considered confidential. FOIA exemptions do not act as a real block but
they give the agency \-- in this case, EIA \-- a discretion, a latitude,
to consider withholding information that would otherwise be
automatically subject to public disclosure.

DOE FOIA regulations take these factors into consideration: Are data
held in confidence by the parties to whom it belongs? Are data of a type
customarily held in confidence by the data provider? And is there a
reasonable basis for the withholding of this data? Was the information
transmitted to and received by DOE in confidence; i.e., essentially a
notice-type thing \-- we\'re on notice, they\'re on notice? Is the
information available in public sources?

Then there\'s the balancing test. Courts have examined these positions
and conducted a balancing test of the public interest in favor of
disclosure versus the rights of the submitter in privacy and
confidentiality, and to protect sensitive information.

Courts ultimately concluded the strong presumption in favor of
disclosure, trump the data provider\'s interest in individual data
confidentiality. Again, this is not some EIA thing; it\'s a general
ruling in some of the case law.

I mean, perhaps I failed to mention it and I apologize, the overriding
concept of FOIA is, data should be disclosed; then we look at the
reasons not to disclose it.

The balancing test has been applied differently by some Courts dealing
with exemption 4 issues. They consider public interest a third factor of
the competitive harm test, along with competitive harm and likelihood of
competitive injury.

DOE\'s regulations look at the public interest, specifying that to the
extent permitted by law, DOE will make records available even if it may
be authorized to withhold them under FOIA, whenever public \-- DOE
determines that disclosure is in the public interest.

Such regulation provision is consistent with the balancing test in many
cases that have gone before the Courts.

Burden of proof. The burden of proof, proving that information is in
fact, confidential, rests with those seeking to prevent the disclosure
of the data.

It is not an exact test as to whether the burden of proof has been made
or not, and the rule seems to be that since it\'s not articulated that
heavy preponderance of evidence theory seems to be, what guys? If you
have heavy, strong evidence this is such you may well prevail; if you
don\'t you may not \-- if one does, may not.

This is consistent with FOIA exemptions which do not act as a bar to
disclose, but allow an agency the discretion as we mentioned a moment
ago, of withholding or not withholding data.

Trade Secrets Act. We talked about that some. This Act prohibits \-- and
there are some \-- I remember not only, I think, in print but in some
phone calls people suggesting that an individual employee may be legally
responsible, or things of that sort.

This Act prohibits the release by any government employee of any
information qualifying as a trade secret except as provided by law.
Courts have interpreted the Trade Secret Act as we said, in a very
narrow way. It has to be co-extensive with FOIA. This is a very key
issue: the co-extension position of The Trade Secrets Act and FOIA.

Therefore, the data that might be considered to be a trade secret under
the Act but not falling within exemptions 3 or 4 of FOIA, and would not
be protected against disclosure under the Trade Secrets Act.

The reason is that the Trade Secrets Act prohibits release of data if
there\'s an authorizing law and the FOIA is in a sense, the authorizing
law and regulation that allows that to happen.

You can make a legitimate case for Trade Secrets Act but if it doesn\'t
fall within the exemptions of FOIA that data may be released and in
fact, is.

Administrative Procedures Act. There is no provision for lawsuits under
FOIA. If a company believes that information submitted to the government
should not be released because that information falls within an
exemption to FOIA, the company must first go through the authorization
review process as provided by FOIA.

We also have an obligation. The obligation to the agency is proper
notice, time, opportunity to respond \-- all these different things. If
the administrative review process is not resolved satisfactorily the
company may institute a reverse FOIA action.

A Court then examines the actions of the Agency to determine if the
Agency acted in a manner that was arbitrary, capricious, and abusive
discretion without observance of procedural required by law, or
otherwise not in accordance with the law. There is however, a
presumption in favor of validity of the Agency\'s actions in most
instances.

I\'m going to finish up, okay. Sorry about that. Okay, then I\'m going
to kind of slip to the conclusion. There\'s a couple of things here
really. These are the data elements that are going to be held as
confidential. And the main thrust of those who are seeking
confidentiality is that substantial, competitive harm will be done to
them if all the data is given out.

Now you have on one side \-- you have extremes, as in most instances.
One extreme says, give everything out all the time, never hold anything
back. The other side says \-- we even had one that said, we can\'t tell
you what it is that we don\'t want given up because by telling you that
will give up the secret.

Well obviously, there\'s got to be something in between. And I think the
facts specific is really the key to the whole question of
confidentiality.

And Betsy I\'m sorry I took some of your time. Well, thank you all for
letting me come and giving me your attention. Thank you very much.

CHAIRMAN RELLES: Should we \-- I guess we should wait until Betsy\'s
given her presentation to ask questions. I\'d like to have a copy of
that. How sure are those likely to be the items that are suppressed?

MR. COLLIGAN: These have been submitted to OMB \-- it\'s final approval
of the OMB \-- and I see no reason to think that they won\'t be, because
I think we\'ve done our homework. But that really is up to them.

They are obviously, the final ones and we consider a lot of the element
\-- frankly, the most response we got from our procedures was pretty,
you know, there was still complaining but it was pretty positive. And
I\'ll give you a copy of this.

MS. O\'BRIEN: Good afternoon. I\'m Betsy O\'Brien and I\'m happy to be
here with you. It\'s the second time that I\'ve talked about our forms
redesign. I\'m focusing on our longer-term project of redesigning the
electric power forms when the industry is competitive.

I was a little concerned about the time slot that we drew after lunch so
I decided to use bright yellow cards, and I also have some bright colors
on these slides. Can you hear me without the microphone? Okay.

First I thought I\'d start by giving you an update of the status of
restructuring in the states. Since last spring there has been some
movement. There are now 13 states that have passed legislation to
implement restructuring. There are five states that have issued
regulatory orders and one state, Ohio, has legislation pending.

It\'s important to note though that even though a state has actually
passed legislation to initiate retail choice for their electric
supplier, it doesn\'t mean that it\'s going to start soon. There\'s a
wide range in the legislation. It has a timeframe associated with it.

For example, California actually started retail choice for all their
consumers in March of this year, and Virginia, which is one of the
states that passed legislation, will not be starting sole retail choice
until January of 2004.

What I\'ve done here is try to show how it really is going to be
phased-in over time. For the 19 states that I\'ve talked about that have
had legislation passed or regulatory order, when I look at their sales
of electricity in 1997 and when their state was going to have full
retail choice, it is really phasing in and only is going to reach \-- it
won\'t even reach 40 percent \-- it will be close by 2002.

And that\'s all sales in the state. Now of course, not all retain
customers will choose to select a different supplier than their
traditional utility, so many of our data surveys will still be touching
the main supplier of electricity to retail customers.

In our long-term project for forms redesign we targeted it to 2002 as
the date that when we would have our new form cleared through OMB and
ready to send out, mail out, to our survey respondents and have our
computer systems ready to process the data.

And given this timeframe right now of where the states are, it looks
like a reasonable time for us to have those forms ready. We are doing
some interim changes also as we go along, as needed.

And I put this up to just explain a little bit about the concept of how
we see the industry changing in the future. We see power plants and we
tend to focus on generating companies in the future rather than the
terms \"utility\" and \"non-utility\".

There really won\'t be that distinction and we\'re already seeing it
being blurred right now; that power plants are being sold from one type
of company to another, and it really is data about the power plants
themselves and not what kind of owner that makes a difference in what
kind of information we want.

The general trend is that there will be a company or an organization, an
independent system operator that will run the transmission network. The
company will post on the Internet the capacity that\'s available to use,
post the tariff prices, and also schedule transactions on a day-to-day
basis.

And so we\'re then moving forward in our redesign with the concept that
there would be some kind of an operator of the transmission system
itself.

And then finally, there will be local distribution companies that will
still run the wires to the homes and businesses and although they may
not be the company that\'s providing the energy, they\'ll still be the
company that\'s bringing the electricity into your home.

And it\'s very similar to the idea of the telephone and the fact your
long distance carrier may be different than your local telephone
company.

As we designed our forms and been working on our data requirements,
we\'ve classified them into four general areas: generation,
transmission, consumption and retail sales, and the hotel market.

And the first one on generation is, in some ways is a gift for us
because the types of data requirements that have been identified are
very similar to the types of data that we collect today on our survey
forms.

The primary difference is that today we have a distinction between a
utility and a non-utility company, and that distinction will go away.
And in fact, we\'ve already moved toward that with the forms that we put
in for clearance this year with OMB on our monthly survey of generation
and field consumption and stock.

We\'ve made the two forms that we have, identical; whether there\'s a
form that goes to utilities, a form that goes to non-utilities, that the
information it collects will be the same.

The other data that we plan to collect in the future is exactly the same
as what we collect on a form right now called the 860. It\'s about the
generating capacity \-- the power plants themselves: what type of plant
it is, what type of primary, secondary fuel it will burn, what
ownership.

That type of information will be continued to be needed and we\'ll just
standardize it so that all plants report the same.

The third area is information about the environment, and it\'s about the
kind of equipment that\'s on a power plant \-- very detailed information
on the fuel that\'s burned and the characteristics as far as sulfur and
ash of the fuel that\'s burned, to come up with estimates of
environmental emissions and also waste products.

It\'s extremely expensive and burdensome right now, and it only goes to
electric utilities. And in our framework of having comparable data from
all generating units, what we plan to do is work with the Environmental
Protection Agency which is one of the sponsors for this form, and try to
reduce the burden on the form \-- reduce some of the questions we\'re
asking but then we would be expanding the universe that we collected
from.

And the last two areas concern information about cost of producing
electricity. The costs that a power plant owner is paying for their fuel
and also the cost to build or buy the power plant and operate it.

And the reason I have question marks there is because there is some
concern that when you\'re in competitive markets should you be asking
questions about costs or should you be focusing on the price of the
product instead?

And it\'s still on our list however, because many people that
participated in our focus group findings and as we\'ve gone out to visit
different states and industries in the states, those are \-- that\'s
information that people have told us they still would find useful and
would like.

And it may be that right now, because the industry isn\'t fully
competitive, it\'s hard to see a chang in philosophy \-- the move from
cost to price. But we are designing with the idea that we would collect
it at this time.

Now I\'m going to move into data on transmission. We collect some
information right now and we would continue on the lines themselves: the
size of the lines, their physical locations, and the ones that are being
planned for the future.

The remainder of the data are items that are trying to get to the
essence of data uses. People told us that they were concerned about the
effectiveness of the open access rules that the Federal Energy
Regulatory Commission passed and whether or not they are leading to
additional use of the transmission lines.

And also whether there\'s congestion on the lines and how people will
know when there\'s need for new lines to be built. So in going out and
talking to the various states, especially in the Northwest, we heard
information about kinds of statistics that the independent system
operators or transmission line operators will be collecting for their
own purposes that we can tap into.

And those are data on congestion pricing. On the Internet it will post
\-- when there\'s congestion on a line the price of electricity will
vary from one region to another \-- for example, in the state of
California \-- and that\'s an indication that there was congestion.

The same type of idea is posted by the PJM ISO. It\'s in Pennsylvania,
New Jersey, and Maryland. So what we would be doing is collecting those
statistics from the Internet of the frequency and locations of when the
prices were different, as an indication of where congestion occurred and
how frequently.

The other is on cost of transmission. Tariffs will be posted. There is a
system, it\'s called OASIS \-- Same-time Information System, and I
forgot the OA \-- but those are on the Internet. You can only see the
tariff if you have a password, so we will have to make an arrangement
with the different organizations to be able to go in and see their data;
that it would save them from reporting to us.

The last two are ideas on the utilization of the line and it\'s looking
at all the generation that goes on the line and a \-- for example,
during a particular hour versus the amount that\'s needed to serve the
customers in that service territory of the owners of the transmission
line.

And when that \-- when the amount of generation is greater it means that
the transmission lines were used for wheeling power through to another
region. And it\'s another indication of how open the access is and how
much it\'s change \-- or, is changing over time.

And the area of consumption or retail sales, again, we collect
information right now on the sales by customer class and the average
revenues per kilowatt hour. We collect it from a sample on a monthly
basis and from everyone on an annual basis.

We would continue to do that in the future only it will be from
different companies, and also there\'s an interest in having more detail
on the components of the price now. There will be an energy component
because that\'s where competition will be taking place, as well as the
transmission and distribution and the strand and cost recovery
components.

So we\'re looking at how to maintain what we have but also have enough
new detail to be able to answer a question in the future of, did
competition raise or lower my prices? And to be able to answer that we
need to know what kind of service and did they change the kind of
service \-- they went to interruptable or time-of-day pricing or if they
chose the green power supplier. So we need to ask additional information
beyond what we\'re getting right now.

Another area that we\'re interested in, the consumption, the area of
reliability. I\'m sorry for you in the back that can\'t read those
letters. These last three are related more, primarily to the idea of
reliability.

And in our visit to the states we found that states actually \-- the
state utility commission had performance statistics that the
distribution companies were reporting. That when there was an outage
they had to report the number of customers and the time of the outage
but it wasn\'t standard across all states. So one service that we could
do is to standardize the type of reporting on outages so they would be
comparable across the states.

And the last area is really about planning the future, and it\'s data
that we collect right now about projected demand capacity, to see that
they\'re in the frame of perhaps the next five years that there\'s a
plan for the expected demand.

I\'m now moving into the hotel market which is the most complicated.
There\'s three parts really, of the transaction. There\'s the
generation, then there are a whole lot of hotel transactions that take
place, and then there are retail sales.

And this actually represents sort of like a white tornado because we
found that the most changes are occurring in the hotel transactions. The
duration of contracts are shorter, there are multiple transactions,
buyers are saying that they have found that one part of their company
sold power and another part bought the same power.

There are terminologies coming about. There are things called sleeving
for when there are financial risks, and many layers to transactions. So
the difficult part was how to collect data without duplicating or having
\-- to collect something that\'s meaningful.

And we thought that we should focus on either the front-end or the
back-end. And the first was to focus on the first sale, so looking at
wholesale sales from the generating company. And they would be selling
either into a power \-- exchange of power tools, or through some
bilateral contract with another company.

And that sounded like a good idea but it still \-- it\'s not going to be
purely a first sale because some of the power that generating company
has really was purchased already as a wholesale market and you can\'t
distinguish.

We also looked at the idea of the last sale; that the energy service
provider that actually \-- the company that\'s selling to the retail
customer would also be purchasing power in the wholesale market. And the
blue lines are representing what we would be trying to collect as the
last sale.

But again, that\'s not going to be distinct because they may resell some
of that power back in the wholesale market. At this point we really
opted on the last sale and we were convinced primarily from this summer
and the Midwest price spikes that occurred, because we did not have the
data on the wholesale transactions to really look at what happened at
that time.

And if we tried to collect the first sale we would have seen a price
spike. It was only the last sale where you actually would have seen it.
So we thought for our data needs focusing in on the last sale seems to
make sense at this time.

And quickly, what we would be doing is look at the wholesale purchases
by the energy service providers, and we\'d also look through the
Internet. There\'s a lot of data coming out there now in the future\'s
market and power that builds through a power exchange or a power pull,
that we can pick up.

And there\'s another kind of wholesale transaction that\'s occurring.
It\'s more of a trade \-- a tolling where you trade fuel for power, or
it may be fuel delivered *X* and power delivered *Y*, but we have that
on our list as something we want to observe to finalize as far as data
requirements.

To wrap it up, the next steps are, now that we have our list of data
requirements \-- there\'s much more detail than what I have on the slide
\-- we\'re working on designing the layout for our forms. Once that\'s
completed we\'ll go into cognitive testing and based on the results,
doing modifications and proceed to pilot test the new forms.

And then during 2001 go to OMB for clearance and be designing and
developing our computer systems to make it all happen. And be ready to
go by January 2002.

CHAIRMAN RELLES: Thank you, Betsy. I\'d like to ask a couple of
questions. To what degree have these utility companies bought off on
your forms design, and in particular I\'m wondering \-- you know, some
of the things that you want to suppress I can understand, like heat
rate. But might things like heat rate be derivable from things that are
not being suppressed right now?

For example, energy inputs and outputs, I believe those are heavily
needed by other groups we hear about all the time, like the global
warming people. And I just wonder whether, when some of these things
actually come down you\'re going to get some strong reactions from
people in the generation side of the house.

MS. O\'BRIEN: Heat rate \-- there\'s actually \-- it\'s not one, single
number, it\'s actually a curve and it\'s based on the utilization of the
plant. What we\'re suppressing is a specific point on the curve where
it\'s at full load heat rate. On our one form it asks for full load; on
another form it asks at 50 percent load; and 100 percent load.

We\'re not suppressing an average heat rate, and that\'s why we\'re
still revealing generation of fuel consumptions, which in balance of the
public interest, people concerned about the environment were very
concerned to make sure that those data remained in the public domain.

CHAIRMAN RELLES: Oh, I agree they have to remain in the public domain
but I think you\'ve given people just about all they need to know by
telling them that. The average heat rate is a pretty good proxy for
what\'s going to happen with the generator. And that\'s why I asked sort
of, to what degree have you gotten this past the utility companies?

MS. O\'BRIEN: In responding, the utility companies would like us to hold
fuel consumption and generation confidential, which is not a big
surprise.

And we\'ll see in the end \-- I mean, I think so far \-- we\'ve also had
some utility companies say that the average is not as critical; that
could be calculated from that as knowing the curve or specific point
from the curve.

It was a balance. Those two items were the most controversial and the
most comments on one side or the other.

CHAIRMAN RELLES: My last question: can you conceive of people not
responding to those questions because they hold them \-- you know,
because they disagree with your decision to release the information?

MS. O\'BRIEN: Well, right now they\'re being released by the Federal
Energy Regulatory Commission, so they\'re already in the public domain.
I don\'t expect that we\'ll have problems collecting it because of that.
And, I guess \--

CHAIRMAN RELLES: I do.

MS. O\'BRIEN: You do? That should make 1999 interesting.

CHAIRMAN RELLES: I didn\'t turn my thing vertically but I don\'t have
one, so. Are there any questions from the committee? Okay. Thank you
very much. So 1999 should be interesting.

I\'m a user of those kinds of data and especially in this competitive
environment I\'m now a potential user of data from the company other
than the one I\'m working for, and I know these numbers are going to be
scrutinized very closely. There\'s a lot of profit to be made by
understanding what your opponent\'s or competitor\'s costs are.

We should move on to our last session of the afternoon, which is going
to be chaired essentially, by Herb Miller of EIA who will be introducing
himself and the other speakers: Cathy Dippo from BLS, Cal Kilgore and
Renee Miller.

Today\'s talk actually, will follow-up on something that happened last
time, and I had promised there\'d be a transition if we needed a
transition. So if that\'s the case Renee should point to me at the time
she\'s about to start and I\'ll try to summarize what I thought was said
from last time, but I think she can probably do a better job than I can
on that.

So the title of the talk is, the Federal Statistical Working Group: A
discussion on data topics on the 21st Century. So Herb, thank you.

MR. MILLER: The title of our next presentation is \"Creating a National,
Statistical Information Infrastructure\". And why I think you will find
this interesting as statisticians and users of government statistical
data, you recognize the need for government agencies to work together,
to integrate their systems and their data collections.

And the agencies will benefit from the economies of scale that this
collaboration affords. And after reading \"Towards the Digital
Government of the 21st Century\", a report from the workshop on Research
and Development Opportunities and Federal Information Services that was
co-chaired by Sal Stolfo in Columbia and Herb Schorr at USC, and I was
amazed at the recommendations in this report that was held in May of
\'97 \-- that are already being implemented.

And the FedStat\'s working group that I am part of, that\'s also chaired
by Cathy Dippo at BLS, that some of our goals were to solicit the
cooperation and participation of the federal statistical community in
the development of the digital government of the 21st century, and to
encourage the development of pilot project proposals.

And I think the last one, very important, is to foster academia,
business, government partnerships that could carry out the research and
development work necessary to make the pilot projects reality.

And now I\'d like to present Dr. Catherine S. Dippo who has a Ph.D. in
Mathematical Statistics and is Associate Commissioner in Survey Methods
at the Bureau of Labor Statistics.

DR. DIPPO: Thanks, Herb. Let me give you a \-- I\'ve done various
versions of this talk, long and short, so if I skip over things that you
really don\'t understand, raise your hand, keep coming back up, because
different audiences understand different parts of this.

First, I\'ll get started with the definition of what the NSII is. And
one component of that is a library \-- and I\'ll go into a little more
detail on what we mean by that \-- explain the vision and implementation
approach that we\'re using for actually bringing this about, spend a
little time on metadata, discuss the projects we have, working on now,
the proposals that Herb referred to that we\'re seeking funding for at
the moment, and then have a brief summary.

To put this in context \-- I hope you can read it in the back \-- but
we\'re sort of down here \-- FedStat. This working group that I chair,
that Herb\'s a member of, is co-sponsored by the Inter-Agency Council
and Statistical Policies \-- that\'s the Agency heads \-- and by the
Federal Information Services and Applications Council.

The Council, along with those five boxes to its left, are sort of under
the umbrella of the National Coordinating Office of the Computing
Information and Communications R&D Subcommittee: all of the Committee on
Technology, all of the National Science and Technology Council \-- which
is part of the Office of the President of the Office of Science and
Technology Policy.

These agencies over here, which do include DOE, receive money under the
Next Generation Internet Bill, and that funding which actually is then
distributed for the five boxes over here, okay, and this group here just
sort of decides along with Agency representatives, how it\'s going to
spend on high-end computing, large-scale networking, high confidence
systems, human failure systems, and education, training in human
resources.

In addition, this Next Generation Internet Act also calls for a
President\'s Information Technology Advisory Committee. And this group
just issued in August its interim report and their final report is
expected in the next week or so.

Now, what is the National Statistical Information Infrastructure? This
builds on the whole concept of the National Information Infrastructure
which basically what brought us the first generation Internet and then
sort of the foundations of what the Next Generation Internet\'s all
about.

And it\'s really looking at an infrastructure that has to do with
statistical information. And I think to sort of set yourself sort of a
metaphor to think about this, think about a room that\'s filled with
books \-- just books. Just you know, you sort of walk in; there\'s just
floor-to-ceiling, nothing but shelves of books and they all have exactly
the same binding.

And you walk into that room and finding information is very difficult.
There\'s something more to a library than just books. You need lots of
tools to use whatever the materials are in that library. They have to do
with binding information, extracting the tools that you need to re-use
the information that you find there, and then transform that knowledge
into intelligence.

And so when you think about an information infrastructure you can\'t
think about just what\'s there but how you\'re going to use it.

Now, what is the information that\'s in a Statistical Digital Library?
There are sort of two ways to look at it. One sort of goes back to the,
sort of the artifacts idea of, in the library community of documents and
databases and those kinds of things.

But I think that\'s not really what you need to look at in terms of
statistical data; that we have many different forms of kinds of data and
you need to look at a lower level than what the artifacts are \-- what
the complements of those artifacts are.

And what we have are microdata \-- that\'s sort of the idea of
relational databases. I have answer from person *X* to question *Y* for
survey \-- you know, something else. I have lots of those kinds of
little pieces of information floating around.

Then we have macrodata. We have the actual estimates and the kinds of
things we\'ve produced from those little pieces of information. And then
we have metadata \-- all about that process of how we actually collected
those pieces of information and then put them together into aggregate
statistics.

And it\'s important to realize that you need all those little pieces
when you start talking about what a digital library is in terms of
statistical information.

So what is our vision? So we have common user interfaces, data access,
and searching tools that are useable by persons with different levels of
computer and statistical literacy.

So thinking about, you know, Internets and all kinds of electronic
access information of people who are anywhere from K through 12, college
people, people trying \-- you know, just your usual, everyday citizen
\-- trying to use statistical information.

We need methods for enabling appropriate users of the data, enabling
analysis within and between databases, and in fact, one that\'s not on
here but it\'s an important one, is that\'s integrating legacy data in
with current and data as we go forward. Because one of the major issues
is, once we develop systems it\'s sort of fairly easy for us to get
what\'s going on now into them, but getting all that stuff in the past
in is another major.

So how are we achieving this vision? There in fact, is another status
group and they are the ones that actually created the Web site for
FedStat. They\'re sort of working on the everyday basis. They work on a
contract year from the ICSP.

They decide what they\'re going to do for the year, have a contract, the
agencies put in money, and they go forward. And they\'re sort of dealing
with, sort of now and sort of tomorrow \-- near-term.

Then there\'s our group which is much longer term. You know, we\'re
talking about developing those tools, the things that we need five, ten
years out.

And then we\'ve sort of co-opted and infiltrated ourselves into a lot of
other existing efforts that aren\'t directly related to statistics. But
are very important to creating this kind of an information
infrastructure.

If you\'ve not seen the Home Page for FedStats it\'s www.fedstats.gov.
It\'s basically a locating service at the moment. We are going to be
adding in the next year some additional kinds of things like a kid\'s
page, some direct, sort of a help you to get into various different
specialized tabulation systems like DADS and FERRET and CDC Wonder and
etcetera.

I\'d like to turn a little bit to the idea of statistical metadata,
because if any of you are from a computer or information science world
statistical metadata is sort of a different animal in that it includes
not only the kinds of things that computer science and information
scientist think in terms of, what is a group created, where and when was
it created. You know, a typical graphic kinds of things.

Another more computer science, well, what\'s this unit in and how\'s it
stored, and all kinds of information about that. But it\'s important to
have with it the things that we consider important with respect to how
the processing was created.

You need the form; you need to know how the process of collection \--
was it mail, was it CATI, was it CAPI, you know, all those kinds of
things, and what was the process that you went through? How were the
interviewers trained; you know, how experienced are they? What are the
concepts behind that?

What kind of manuals and other kinds of definitions existed in terms of
helping interviewers or the respondents decide how they were going to
answer? And those are the typical kinds of thing associated with
editing, coding, estimation, etcetera, and quality descriptors like
variances, response rate, coding error rates, and etcetera.

And then there\'s another kind of metadata and it has to do with usage.
And that goes to the idea that well, I use this particular data to try
to create this kind of regression and I\'ve found you need
relationships.

But when I went further and tried to prepare those relationships that I
found in that particular case to, say if I was doing it on the current
population survey using the March CPS, and then I want to look at those
relationships to what I found in SIPP, the Income Program Participation
Survey \-- you\'d find things that would be contradictory and things
that would be similar.

And I\'d want to know, well, if I investigated that further how do I put
those kinds of data together? Or if I\'m looking at employment data on
current population survey we\'d ask how \-- and we\'d come up with
employment numbers.

We\'d have the current employment statistics survey which is an
establishment survey and we\'d come up with other kinds of employment
numbers. Those numbers aren\'t the same, they don\'t track the same. So
there\'s other kinds of information that help people use these numbers
to reconcile what they mean.

Given the scope of metadata it\'s very important to think about various
aspects of how we\'re going to achieve making this really,
electronically available. And there\'s sort of two aspects to this.

One of them has to do with how we\'re going to do this \-- more in the
computer science kinds of aspects of interoperability. And then there\'s
the sort of, what, and the definition kind of thing that Renee is going
to be talking about. It goes more into the, what.

Okay, you know, if we say these words they mean such-and-such, and if
I\'m going to talk about a questionnaire I\'m going to use this standard
terminology about the questions, the response categories of various
different parts of that instrument.

I\'ve listed some things here that are actually going on and exist, and
I won\'t go into a lot of detail on this.

There are a number of activities that are directly related to survey
work that are important, and one of them\'s the Data Documentation
Initiative; this is an international group that\'s sponsored by ICPSR.
And that is how is it working on micro database? That\'s where they\'ve
come up with a standard data type definition, and we just recently
started working on aggregate statistics.

There is, in the computer science world, ISO 11179 and X3.285; we\'re
defining data elements. And here\'s where you might want to take the
kind of thing that Renee and company are doing on data definitions and
create a metadata repository with that information using these
standards, okay.

And then you want to hook them up with some other kinds of information
that would come from the metadata standard here which defines what are
the components of various surveys and do some tagging such that you can
link within a metadata repository to what those standards are.

There are a number of areas where we recognize that we need to do
significant amounts of work in terms of actually pulling together all
these different pieces. One of them has to do with information seeking.
How do people who come searching for statistical information on the
Internet go about it?

What kind of things are they looking for, what\'s the task like, how can
we then define design interfaces that will help them in their task?

A big issue has to do with ontology and thesaurus and etcetera. You say
electricity, your labor statistics, collect producer price information
on electricity. We can collect retail information and the Consumer Price
Index on electricity. How do we compare \-- if somebody wants to put all
that data together on electricity, how do they do it?

We need things like ontology as the source; ways that help you
manipulate or find your way around some common language, and what\'s
different and what\'s the same, can usually find the same words.

We need tools on knowledge acquisition \-- sometimes referred to as data
mining but other kinds of things also. Some statistical methods, if we
start to actually want to put together data from one survey and another
survey, how do we actually do that?

Information visualization. This is more on the information science
network but it also can come into statistical graphics. Information
visualization is actually, how do you show a picture common \-- a term
and the term sets in ontologies and thesauruses so people can understand
the relationships of words.

Expert systems for helping people actually use data, and human-computer
interface kinds of things. When you\'re getting people who don\'t know
anything about statistics and trying to use statistics.

Some of the projects that we have in progress, going actually to the
FedStat site we started \-- a year ago we had 14 agencies that were
indexed in the site. We upped that; there\'s now 14 more so there\'s a
total of 28 of the 70 statistical agencies are now indexed in FedStat.

We\'ve for two years been conducting user studies using contract
information scientists who specialize in this kind of thing. We\'ve been
doing some cognitive and HCI kinds of studies where we\'re in the
laboratory working with things like variable labels.

You take a survey like the current population survey, the SIPP, where
you have thousands of variables there. And you\'ve got these short
little names and we\'ve got probably several hundreds of them all use
the word \"income\". How do you really come up with labels that somebody
can make sense out of in short?

Issues on navigation; again, with Home Pages in dealing with some of
these tabulation projects like DADS and FERRET. Then how you use some of
these other tools that are available and go the environment to actually
help promote the use of metadata?

Knowledge and organization for statistics. This is sort of what I
mentioned before in terms of ontologies and thesaurus. We\'re just
starting some work on that in the area of labor statistics.

The University of Massachusetts Center for Information Retrieval has
been doing some work on actually finding tables within texts. You know,
the search algorithms that you use are mostly Boolean. One of the
exceptions is AltaVista does use some other kinds of things.

But typically the only thing that\'s indexed there is text. And text
within text. You hit a table in text, forget it. The labels that are \--
you know, the words that are in the labels \-- the title of that table,
the column labels, or the row labels are not indexed.

So one of the things is, how do you actually recognize the table, pull
out words that make sense, and index them? And then other kinds of
alternative information retrieval methods. That\'s going beyond simple
Boolean searches and how do you actually promote with things that make
sense for statistics?

There are three NSF programs that we\'ve been working towards trying to
get some funding for some of the projects that we\'ll talk about. One
was the Knowledge and Distributed Intelligence. We weren\'t successful
with any of the two projects we put in for that.

The Digital Libraries is still in its final phase review at NSF but we
don\'t know whether we were successful there. We only have one proposal
that went into there.

In Digital Government we have four proposals that are in under the
research grants out of 15, and two in the planning grants of about 35.
And the planning grants, the review panel will be December 11th and the
research review will be finished by the end of January. They\'re just
being bailed out actually now, to the proposers.

So what\'s in that package of material that are being reviewed right
now? As Herb mentioned, the original holders of the digital government
workshop, Sal Stolfo and Herb Schorr, the ISI and Columbia group has
since that workshop, actually created a Center for Applied Research and
Digital Government Information Services \-- CARDGIS.

That group has put in two proposals. One has to do with automated
ontology from the kinds of things that I was talking about, and it\'s
primarily focused at the moment on EIA. So I think Cal will talk a
little bit more about that one.

And the other one is on survey instrumentation. And this again, is
really relevant to most of the federal surveys. If you\'re actually
designing a computer-assisted instrument at the moment we\'re limited to
a few pieces of software; most of which are in DOS; most of which are
very difficult to use.

Currently, the Census Bureau has over 100 people who actually author
instruments \-- full-time. This is a heavily, labor-intensive operation
here, and we would like to reduce the labor costs on that and one of the
CARDGIS proposals is a method for doing that using off-the-shelf
software that is very much GUI-based and really puts it more into hands
of the actual instrument designer rather than having to go from an
instrument designer to an author to, etcetera.

The second one has to do here, is the National Institute of Statistical
Science, Los Alamos National Lab, Carnegie-Mellon, Kansas State. We
probably put together some the names of those, who are the people
involved: Feinberg, George Duncan, Sally Keller McNulty.

And that one is on, can we develop programs for remote access to
confidential data such that someone says, here\'s a dataset, I want to
do this kind of a regression, the regression is computed off-line, the
tabular output is reviewed for the confidentiality disclosure rules of
the particular agency that holds the data, and only tabulations are
passed back to the requestors that pass all the disclosure rules.

The next group are information scientists, and it\'s really focused on
tabular data and can we build metadata into tables? So that when
somebody actually mouses over that row label on the left that says
something about gender or age or something else, you get a definition,
or maybe you get the question that was actually asked to go with it.

So with that, you know, it will help the person actually, get closer to
the metadata while actually looking at the table. And it could even
bring up things if you\'re going over the number in the table.

Things like the standard area or the response rate and some other kinds
of things that we really would like people to use but are so difficult
because who knows where they are on the Web site? Probably many, many
clicks away.

Interactive statistical graphics. This is something that we had started
at BLS and the person we had working on it left and now works for SPSS
and they\'re about to issue a new piece of software for interactive
statistical graphics, but there\'s a whole lot of issues associated
with, actually how we build metadata into those graphics and how do we
use them.

And the proposal is primarily for useability testing and development and
actually how do you take this kind of software and actually use it, and
how do people use it out there if they\'re allowed to use it?

The ICPSR Michigan group is heavily focused on geography over time. You
try to take a particular county or city \-- a place is probably better
in the 1990 census \-- and you wanted to look at any kind of variable
and compare to 1980 and 1970 \-- forget it.

The geography\'s changed. There\'s no way you can actually do that given
the kinds of things we have now. So can we develop various algorithms
that actually deal with changing classifications over time?

It\'s going to be very important as we change from the SIC to NAICS; as
we change from the old SOC to the new SOC, as we change race and
ethnicity from the old procedures to the new procedures. We need ways to
help people bridge data over time as the classification structures
change.

And the last one has to do with geospatial images and data collection.
Can we actually put maps, can we put other kinds of images on the data
collection, tools that people are using such that they can actually spot
where things are, or no particular kinds of things in structures where
they need to know those kinds of things, and actually pass that
information back instead of writing long kinds of descriptions.

That\'s sort of a brief agenda where we are. There will be a workshop in
\-- has now been set for February 9th and 10th. The Computer Science and
Telecommunications Board and the National Academy of Sciences will be
holding a working group on federal statistics to help us develop a
further research agenda.

And so in summary let me say that it\'s important that we consider
statistical information, both data and metadata, and that if we expect
people to really use the kinds of statistics we\'re producing, we need
to help them with statistical literacy, with the kinds of statistical
methods that are really necessary to actually put databases together or
do things over time.

And obviously, if we\'re going to do that we need the statistical tools;
we need to invest in actually, how do you do this? Therefore our goal
has been, well how do we use information technology to facilitate those
statistics? And metadata is the vital infrastructure component in order
to bring this all about.

For those of you who are interested in other agencies, this is our
group.

CHAIRMAN RELLES: Can I ask a quick question?

DR. DIPPO: Yes.

CHAIRMAN RELLES: I appreciate the vision and I recognize it\'s out into
the future, but is there sort of a more intermediate a set of steps that
you can identify that could put some of these tools in our hands?

For example, just \-- I mean, doing the data acquisition stuff alone
would be of immense value. Okay, so you can\'t do the statistical
methods on it that I want but it\'s okay, I\'m happy to get it and run
with it.

That\'s a less visionary statement, it\'s more intermediate, but still
it might be nice to identify sort of phase-in steps so that the vision
can kind of be improved by real applications. And I wonder what your
thoughts are on that.

DR. DIPPO: To some extent, things like the statistical graphics package
and some other of those things, they are going to be there in the fairly
short time horizon, and it\'s more a case of, how do we use them?

Again, we have zero money here for this kind of thing. What we\'re
dealing with is trying to put things together and really using NSF to
fund kinds of projects. Basically the four research projects that we
have in under Digital Government add up to 16-and-a-half million dollars
over five years.

That\'s a lot of money in terms of, the Agency is actually putting in
money to fund that kind of thing. So we\'re sort of out there sort of
with our tin cup in trying to balance between, here\'s what we\'d like,
here\'s what\'s good science, here\'s what\'s cutting edge, here\'s the
kind of thing that NSF would actually fund in terms of giving us
something that then we can take back and actually give us something we
could use.

CHAIRMAN RELLES: So you\'ve got to argue with cutting-edge in order to
get NSF to be interested?

DR. DIPPO: Yes. That\'s part of the issue.

DR. CHATTERJEE: Just to slightly restate Dan\'s question or comment. I
think what you\'re attempting to do is magnificent and great and so
forth. But I as a user of statistical data might be moderately satisfied
if you make that basically the data, you know, collected data more
accessible, whereas I could choose my own statistical graphics or
methodology which I want to use.

But if the access to data was centralized and we could get it easily
I\'d be content.

DR. DIPPO: Well, it is centralized in FedStats. I mean, you can go into
FedStats and you will get all kinds of information you want, from every
single agency.

DR. CHATTERJEE: No, but the problem which you said, you know, which I
actually, these are good students trying to extract tables from them,
they have found it extremely difficult. So at least have you even
satisfied that that moderate (unintelligible)?

DR. DIPPO: Right. I mean, there\'s major pieces of software here that we
need. The problem is finding somebody who, number one, will find it
commercially viable to do that, or scientifically cutting-edge such that
NSF will pay for it. Those are our two options at the moment.

CHAIRMAN RELLES: In either case, we don\'t get to use it for quite a
while.

MR. CALOPEDIS: Cathy, this is all tremendously impressive, but the
thought occurs to me, are there parallel user groups that are working in
the same area? For example, one thinks of COPAFS, or maybe just college
students of some sort.

I mean, I know you\'re doing some user studies but it seems to me that
in parallel with all the sort of providers, one would be interested in
having the users kind of helping to shape this in a more direct way. Any
thoughts?

DR. DIPPO: Certainly in the user studies that we\'ve done we\'ve asked a
lot of questions about what kinds of things people want. The problem is,
we do end up with people who are in colleges, we end up with people who
are in industry.

The significant poll that we\'re having is: that everyday person out
there, is how do they do these kinds of things? Now we\'ve done kind of
follow-back surveys, you know, from users of FedStat. We sent out a
reply to them saying, you know \-- especially if they gave us feedback
so we had an e-mail address.

We limited it to people who gave us feedback. We asked them, would you
be willing to participate in answering a few questions about things? I
mean, we had a very poor response on who was willing to even talk to us.

And you know, it wasn\'t a case that they had to fill out an Internet
survey. We had our contractors call them and actually go through a
protocol call trying to \-- so it\'s difficult to get at this one group
of people.

We\'ve done a lot \-- fair amount with intermediaries; you know, the
people in the libraries \-- those are the people, you know. We\'re
working on it is all I can say. Any other suggestions for getting at
these kinds of people would be useful.

DR. LJUNG: I noticed you had NOAA as (unintelligible).

DR. DIPPO: Yes.

DR. LJUNG: Have you contacted FEMA and NIST GS or \--

DR. DIPPO: Oh, these people are also involved in \-- but they\'re not
involved in this particular project. They\'re on their own separate
track over there. Mostly they\'re concerned with networking and parallel
computing and some other kinds of things.

CHAIRMAN RELLES: Thank you. That was extremely interesting. Herb, did
you want to pick up and continue, or am I to introduce Cal Kilgore now?

I guess I\'m to introduce Cal Kilgore.

MR. KILGORE: I\'m very pleased to be here with you today. What I\'m
going to try to do is to narrow the focus of this discussion a little
bit \-- to EIA in particular \-- and what our experiences have been here
in this transition to this great new world of the Internet, which for us
is a fairly exciting exercise. We\'ve been in it for two or three years
now so we at least have some war stories to tell.

Up until recently, the way that EIA has communicated with its customers
is through printed publications. At last count we\'ve got 5510 people
who subscribe to one or more of our printed publications. As a whole
it\'s a fairly expert group.

They\'re either in the energy industry or deal with those who are. So
they have some knowledge of the subject matter before they even find out
about us and look at the information that we have to provide.

And our publications have served us well in addressing that community of
expert people in that the depth of the definitions are there for them to
take a look at, footnotes are there to explain any peculiarities about
the data that we\'re providing, and they\'re expert enough in using it
so that they can make sense out of all this.

There are two things that are going on though, over the past two or
three years. First of all, the subscriptions to our publications are
going down and our customer base is absolutely exploding by use of the
Internet. Most of our customers now are on the Internet. We\'re dealing
with between \-- as John said, between 6,000 and 7,000 people or user
sessions per day. Actually, we hit 7500 yesterday.

So it\'s not only \-- and just to put that into context, that equates to
about two million user sessions per year. So we\'re dealing with a
different thing entirely here.

User sessions are certainly not the same thing as subscriptions to our
publications but I think by any measure here, we\'re dealing with quite
an explosion in the people that we\'re able to reach and provide our
information to.

The average person on our site stays around for about six minutes for
one of these sessions, so that\'s about the attention span of the
average person for our information.

The growth of the site is very rapid. As a group I got 50 percent last
year. This chart shows the growth in the site since we put it in play in
the middle of 1995. You can see that we are experiencing exponential
growth, so this is \-- we\'re reaching more and more people every day.

This is \--

DR. BELLHOUSE: Sorry. What was the drop-off? There were a couple of
drops there.

MR. KILGORE: Well, it even makes sense. The troughs that are noticeable
are either summertime or Christmas. These are daily statistics and we do
track this on a continuous basis.

It\'s being tracked right now as I speak. So you can see when basically
people have other things to do besides sit in front of their computer
and look at statistical data.

Here\'s another indication of the growth in interest in what we do.
These are basically sign-ups to be notified via e-mail whenever we
update something on the site that they\'re particularly interested in.
At the present time we have in excess of 60,000 such ongoing, pending
requests for, when it\'s updated send me an e-mail so I can click on it
and take a look at it.

Moving into this brave new world of people who were I think on average,
less expert in the subject of energy or statistics about energy than we
had traditionally dealt with in the past, communications is a much more
difficult problem for us to tackle than we\'ve had to deal with before.

Units of measure differ around the world, I mean, we\'re big on English
units and the rest of the world likes metrics. We\'re big on measuring
oil in terms of volumetric measures, such as barrels or gallons or
things like that. When you get to the rest of the world they talk about
weights \-- tons, things of this nature.

So even when you\'re talking about the same subject, even among experts
units of measure can be significantly different even on the same
subject. It makes it very difficult for us to deal with other
statistical agencies on energy statistics in other parts of the world,
because they talk weights and we talk volumes.

Units are not understood. I would venture to say that most people here
in the building don\'t know what a quad is, and when you go outside of
the building don\'t try to explain.

Barrels are a little better because people think, you know, they at
least have a clue as to what you\'re talking about if you talk about a
barrel. If you talk about gallons they really know what you\'re talking
about. They buy those all the time so now you\'re making sense.

But the ability to communicate that we\'ve been able to use in the past
with statistical measures is just \-- I mean, we provide numbers but we
don\'t provide things that people really understand.

Geography, as Cathy mentioned, we can\'t really keep it straight as to
what our statistical area is in the United States. Well, it\'s even
worse than that: the U.S. can\'t decide what that is. When we report our
own information internationally, we define the United States as being a
different place than when we report data on the United States
domestically in our own publications.

And when you combine the difference in the geographies \-- whether you
include U.S. Territories or not. And when you combine the difference in
the geography with the difference in the units going from weight to
tons, even statisticians outside of EIA can\'t figure out that these two
numbers are supposed to represent the same subject. Very difficult.

When we move out of the experts who at least understand there might be a
problem, and we move into less expert and we start talking about energy
consumption and the United States of America, and they look at one set
of data and get one thing and one set of data and get the other and they
look totally different, you\'ve got real problems on your hands.

Definitions of energy terms \-- I say they differ around the world but I
mean, they differ \-- even within an industry they differ in the
countries that we deal with. They even differ here in EIA. If you go
from one of our forms where we collect data, to another one, the
definitions of the terms are a little different.

So we don\'t even have standardization in our own organization, much
less in the industry or in the country or in the world. Dealing with
those kinds of issues when we\'re trying to convey this information for
people do to analysis with, is fairly important.

Breaks in time series data go undetected. I mean, they\'re in our credit
publications too, but if you read the right footnote and think a little
bit you can at least find out that that might be the case.

But the truth of the matter is, from a communications point of view,
even in the credit pubs the footnotes are in large part, ignored. On the
Web they\'re flat-out invisible. They\'re just like they\'re not there.

To show you that that\'s not just a minor point, here for example, is a
fairly significant thing that we, as the Energy Information
Administration might want to convey to somebody. The red line is EIA\'s
published data on energy consumption in the United States.

This by the way, is our definition of the United States which is the
50-state area as opposed to the one we tell the rest of the world which
includes Puerto Rico, Virgin Islands, and our Territories.

How come our reports have the red line and the black line is reality?
Well, it\'s because we didn\'t collect that stuff \-- the black line \--
so that\'s really an estimate. What is it? Well, it\'s distributed
renewals. It\'s renewable energy consumption in the United States that
was accomplished by someone other than an electric utility, who did in
fact, report to us.

It\'s a lot. It\'s about three quads. That\'s one-and-half million
barrels a day. It\'s a very large number. Now, if you took the black
line off of this chart and you went to our site or our publication, or
wherever we wanted to get it, and you plotted the red line, looks like a
pretty darn good tread line. Doesn\'t look anything wrong with it, but
it\'s not right.

And somehow or another, getting that across to people who are less
expert, who don\'t really understand that there was a footnote somewhere
or somehow that indicated that in that particular year we decided to
start reporting something that we had never reported before \-- and not
only that, it was an awful lot of energy \-- it makes a big difference
in what\'s going on.

So a lot of these things in communication are not all new. It\'s just
that by the advent of the Internet and by the advent of dealing with a
much larger group of people, and in general a less expert group of
people who deal with our statistics, they\'re much worse than they were
before.

And the idea and the tasks were basically one set of data which is what
constituted one of our publications \-- sort of tried to at least deal
with it consistently within the context of that one publication \-- is
kind of out-the-door too, because when you go to EIA\'s Web site it
isn\'t one publication anymore, it\'s EIA\'s data.

So if you\'ve got inconsistencies between the publications \-- and
we\'ve got loads of them \-- and people say, well I\'ll take this one
from that \-- I\'ll take one in column A and two in column B and one in
column C and I\'ll add up A, B, and C and think that my answer means
anything \-- which it would logically do so from the terms that are used
\-- you can make big-time mistakes.

All right. So a lot of problems, a lot of whining. What are we doing
about it? Well, we\'re doing as much as we can. A question was asked
earlier about, how do you find out what users want to know? Well, I
agree. Trying to get their e-mail addresses by asking for it and asking
them questions, if you even found one-half of one percent who sent you
their e-mail address that would help a whole lot.

What we\'re trying to do is look at behavior on our Web site in as much
detail as we can. And we do have beaucoup information on that. I mean,
you can tell quite a lot just from their IP addresses and what they do
when they get there, what tense \-- you know, what leads them around,
what works in the way of them being able to find things, what doesn\'t,
and so forth.

And I think, we\'re just kind of beginning that but we\'re beginning to
see an awful lot of things that we wouldn\'t have been able to see in
the past. It\'s an awful lot of data processing because the amount of
data associated with something like this is horrendous.

We\'re trying to work on the query systems. I haven\'t seen one yet I
like. We\'ve got several but they basically have got to become much more
intuitive as to their use \-- for people to be able to use these
effectively.

We have a committee to kind of, at least inside the house here \-- share
war stories and experiences and what works and what doesn\'t. It\'s
called the Information Products and Services Committee. So we\'ve got a
lot of people, basically, trying to solve this problem as we move
forward in time.

The definitions issue, we have another committee, the Data Definitions
Committee \-- I think Lynda chairs that \-- which is at least trying to
straighten out the act here at EIA. So when we say something like
\"gasoline\" we all agree on what we\'re talking about \-- which is no
easy trick.

And we also \-- at least we have a proposal with the NSF that Cathy
mentioned earlier, along with several others. And there is some hope \--
well, assuming that we get it, which I guess is still a pipe dream \--
assuming that we do get some NSF support I think that the work that
we\'re trying to do with Columbia University and University of Southern
California might very well produce some near-term real results.

I mean, basically what we\'re trying to do is get them to, not only try
to figure out how to address some of these issues but to actually give
us something that we can put onto our Web site that will enable users to
be able to take advantage of the outcome of that.

So if we can move forward on that NSF thing I think at least there is
some hope that we will see some near-term results rather than just
statements as to why they can\'t solve it unless they get a lot more
money.

That\'s about it unless you\'ve got any questions.

CHAIRMAN RELLES: Jim.

DR. HAMMITT: I was just kind of curious about your Web usage. You said
the average session lasts six minutes, and you\'ve obviously done some
work trying to figure out what people are doing. Can you tell \-- I
assume the time people are there is very skewed. There must be a bunch
of people that click on the site, look at it for a minute, decide
they\'re not interested and go away. What \--

MR. KILGORE: If they come in by \-- we don\'t really count them as
visitors if they just kind of come to the site and say, oop, nobody\'s
selling applies here, and they go away. We don\'t count that.

DR. HAMMITT: So they don\'t click on something afterward \--

MR. KILGORE: They\'ve basically got to penetrate into the site to get to
some information before we recognize them as a visitor to the site. I
think \--

DR. HAMMITT: But six minutes seems pretty fast to get anything unless
you know exactly where you\'re going and what you want.

MR. KILGORE: Well, a lot of the things that we have on the site are
aimed at \-- not at users of statistical tables but are aimed at people
who are looking for fairly succinct summaries of what\'s going on.

So what we find is that if we get \-- if they come to the site they can
basically find that fairly easily and just either read it or print it
off or download it or whatever they want.

I haven\'t looked at the distribution function on this six minutes yet.
I know it varies a lot by product and I don\'t know much more about it
than the fact that it is in minutes, not in seconds \-- which is what I
was expecting when I did this.

But I think one reason is that we just kind of scrape off casual \-- if
you come and look at the EIA Home Page and go away it\'s like you never
showed up.

DR. HAMMITT: The other thing is, how do you know how long somebody
stays? Or you only know the time at which they last downloaded
something?

MR. KILGORE: Yes, you know \-- when they clicked on something is what
you really know.

DR. HAMMITT: Right. And then if they spend two hours reading what they
clicked on you don\'t know that?

MR. KILGORE: Right. A user session \-- I don\'t want to get into these
technical definitions, but basically if there\'s no activity for 30
minutes, whether they\'re there or not they\'re gone, in our view. And
so essentially once 30 minutes has gone by and we haven\'t detected a
click, they\'re no longer here, is what it amounts to.

CHAIRMAN RELLES: Thank you. Samprit, did you have your hand up?

I\'d like to now welcome Renee who will talk about the data definitions,
team efforts. And this is a talk that I think follows up some discussion
we had last time where Carol Caldwell \-- Crawford, sorry \-- made some
I think, really very useful comments about taking your definitions and
making them more hierarchical to try and resolve some of the
inconsistencies.

Fortunately, Carol was talking to Lynda so she didn\'t hear my
mischaracterization of what she said.

DR. CRAWFORD: She\'s asking for a definition of (inaudible).

CHAIRMAN RELLES: Definition?

DR. CRAWFORD: Status.

MS. MILLER: Good afternoon. Can you hear me in the back? I know there
was some problems previously. As Dan said, I\'m going to give an update
on the Common Data Definitions project. We\'re going to try to solve the
problem that Cal mentioned about the definitions not being standard at
EIA.

And we\'re working as a team and I think the first overhead has the
members of the team. One of them this morning asked me if the reason we
were in this session, which is entitled \"Data Topics in the 21st
Century\" is because we were still going to be working on this project
in the 21st Century.

But actually the tie-in is, as Cathy mentioned, definitions are
metadata. And also, Arthur this morning pointed out the importance of
definitions in data quality for respondent level data. I think we have
Arthur\'s overhead here.

As more data become available on the Web it will be easier for you to
access data from different sources, and so data quality and definitions
will become more of an issue over time.

Okay, going back to what happened at the last meeting, I spoke to you
about common data definitions, the study on the feasibility of surveying
end users, and the problem of standardizing the computation of response
rates.

This time I\'m going to focus on the work on the common data
definitions. The feasibility study on end users, we\'re basically going
to consider that part of our natural gas redesign work because natural
gas is the area that the problem is most apparent in. Also, recently we
put in a proposal to study getting data from energy marketers. This
would be a long-range feasibility study.

So we are taking the comment Seymour made at the last meeting about
relying on a combination of suppliers, marketers, and end users into
consideration. Seymour pointed out that it might be a messy combination,
and we hope not too messy.

On the issue of common response rates, we don\'t have too much to report
there right now. We did convene a small group of EIA statisticians and
we decided to try to get information from other agencies, but the Bureau
of the Census and at Statistic Sweden we heard had standardized
procedures. We\'ve had more luck in getting more information from the
Census Bureau. Statistic Sweden\'s best practices are in Swedish and we
haven\'t gotten a translation.

The Census Bureau had procedures for both weighted and unweighted
response rates. They don\'t quite address all the issues that we brought
up last time such as how to handle mergers, but we do think that we\'ll
find them helpful. So we will be reviewing them.

Okay, onward to the work on common data definitions. Where we were at
the last meeting was \-- I had told you that we had reviewed and
discussed the charter for the Common Data Definitions Team. As a test
case we had reconciled definitions, finished motor gasoline and gasohol,
we had started prioritizing problems to resolve, and we had asked the
committee\'s advice on, what is a definition?

And Carol gave us a suggestion that a definition consists of the terms
to be defined, the general category to which the term belongs, features
that serve to distinguish this term from other members of the category,
and as you may recall, she used forks as an example. And she said that
we should try to define fork to differentiate it from a spoon but that
we shouldn\'t get all tied up in worrying about salad forks and desert
forks and so on.

We think that we followed the spirit of the advice but this is EIA and
we are into details, and so we may have actually gotten into talking
about salad forks. But we did try.

We\'ve grouped the terms by areas, such as coal and petroleum, and then
within coal we grouped the terms by rank, such as sub-bituminous,
bituminous, anthracite. And in this way we were developing definitions
in an integrated manner, rewriting definitions where needed.

As shown on the next overhead, all of our definitions begin with a
generic, non-technical definition. And this is where the team has really
been struggling. How generic and how non-technical can we be?

Afterwards we have technical information and then where needed, a note
at the ends clarifying related concepts or the scope of the data to be
collected.

Now, the reason that we\'re having these problems is that we want to
serve a broad customer base. Originally, our definitions were developed
for survey respondents who we thought needed technical definitions. And
also, we have technically-oriented customers who know the terms
intimately.

Now, some of us have spent the last few days at the Census Bureau being
trained in Cognitive Interviewing. And we got to see how respondents
have been interpreting our instructions and forms and definitions, and
I\'m not so sure now that all the respondents really want or need
technical definitions.

But nevertheless, that\'s the assumptions that we\'re working with; that
we do have a group of technical users and we also want to reach out
though, to other users of varying degrees of technical sophistication,
and foreign customers, and the general public.

And that\'s why we have this struggle here. We want the definitions to
be technically correct but yet we want it to be understandable. And
that\'s where we really think we could use your help.

Okay. We developed new definitions for coal, crude oil, motor gasoline,
and aviation gasoline, and we\'ve sent those definitions to you. And
we\'d really be interested in what you think.

This has been an open process. We\'ve posted the definitions on bulletin
boards. All of EIA gets to comment and so far we\'ve gotten pretty
positive comments. In fact, we have had people eager to use our
definitions.

We\'re currently working on terms pertaining to distillate fuel oil and
then we\'re going to go onto the residential end use sector, which
should be interesting.

I wanted to show you an actual definition. We have crude oil. This was a
situation where we had several different definitions; some of which
included lease condensate and some didn\'t. And what we tried to do was
to unify the definitions. So we started the definition with the part
that all definitions had in common: a mixture of hydrocarbons that exist
in liquid phase and natural underground reservoirs.

And then in the note we went on to explain that different program areas
have definitional variations, and we explained that for instance, the
petroleum supply data include leased condensate, whereas the data on
reserves don\'t. So we tried to tie it all together rather than having
two different definitions.

DR. CRAWFORD: Did you find that \-- I wouldn\'t say easy \-- but did you
find that doable, helpful? Tell the truth.

MS. MILLER: Yes. I wouldn\'t say easy. We had several meetings with
various subject matter experts. We had lots and lots of e-mail, as we
have a couple of team members in the audience \-- Dwight and Perry can
attest to \-- like in other words, there was a lot of going back and
forth, but by the time we actually posted the definition for EIA review
we didn\'t get any more comments on it.

DR. CRAWFORD: Okay, so then just to reiterate, so you didn\'t find it
easy to do because it faced some of the potential problems that you were
having?

MS. MILLER: Well, we didn\'t find it easy to do but we think that we did
it \-- at least for the time being. We hope to have as an end product, a
file containing new definitions that would be available as a paper
product and on the Web.

Looking into the future, we\'d like to see the development of technology
tools to help users access the definitions, such as Window-type look-up
files and a database containing the terms, the survey forms, and the
instructions. And this is where the research proposals that Cathy
mentions, would help.

We\'d also like to see cognitive tests to see the best way to set up the
definitions on the Web, and possibly cognitive tests of the definitions
themselves.

So I\'ve told you about the past, present, and future of common data
definitions, and I would like to hear your comments. Thank you.

CHAIRMAN RELLES: Comments? Dwight.

MR. FRENCH: I would make one comment to supplement what Renee said. She
said in the areas of crude oil, etcetera, etcetera. It isn\'t that
we\'ve only done four definitions. What would you say, Perry? We\'ve
done maybe 30 terms, or something like that? This out of 170 or
something?

MS. MILLER: Yes, we have about 200 terms with multiple definitions.

MR. FRENCH: So we\'re into it now and it\'s going to be a long process.

CHAIRMAN RELLES: Is this metadata now on the Web site?

MS. MILLER: No, it\'s not on the Web site now.

CHAIRMAN RELLES: Have you \--

MS. MILLER: It\'s on our internal \-- it\'s on our intra-Net now. But
we\'re still in the review process.

CHAIRMAN RELLES: Have you made plans to have it integrated \-- sort of,
where you want to put the links that are going to get you to these
various bits of information?

MS. MILLER: Yes. We haven\'t gotten that far.

DR. CARLSON: Do you have some suggestions? That is the next problem
statement.

UNIDENTIFIED: My suggestion is, that\'s the next problem. I don\'t have
any solutions to that. You know, I think how many Web sites \-- I mean,
just go play around on a lot of Web sites and find the ones that seem to
work best. I guess I\'d log onto FedStat and try the other agencies
first and see what they did.

MR. KILGORE: It\'s also part of the staff proposal that we (inaudible).

DR. CHATTERJEE: I have a slight question. Clarification and simplicity
or clearness of definition of course, is highly desirable. But I\'m just
thinking of who the man on the street who might be interested completed
in energy figures so that we have to simplify it to that level.

We expect people who would be \-- to have some working knowledge in the
field. So I think, how clear you want to be is a very hard question. You
know, like I think Lichenstan said, I can make nothing clear but only
some things clearer.

CHAIRMAN RELLES: Yes, Polly?

DR. PHIPPS: This is a technical comment on your testing and your
cognitive testing \-- is it in these definitions. I would really
encourage some kind of usability testing of, you know, people on the
Internet who are looking at the definitions of things, just in looking
\-- and I don\'t think you probably set these up in how you would have
them come up on the Internet.

But if you could break any of this text out, you know, and put it into
more bulletin lines, people recall it a lot better and they \--
particularly when you have some of these definitions that have things
you might want people to know, like breaks in the time series you were
talking about, or how different surveys do different things.

If you have them out as a single line people who will read them \-- when
they\'re embedded in a lot of text like these paper definitions are now,
people are going to be less likely to go into the detail. So I think
usability testing would give you that ability to \-- or even forms
design testing of people; seeing what they remember in definitions.

I know in experiments that I did several years ago, just breaking the
definitions out into bulletin lines with what the definition included
and what it excluded, increased the respondent \-- less was mentioned
there in the long run.

CHAIRMAN RELLES: Okay. Let me call this session to a close. It\'s time
for our break. We\'re meeting the NEIC \-- EIA has various ways to deal
with its user community methods of getting information out, and the
National Energy Information Center is one and we\'re going to get a
chance to tour it.

So how do we get there? We\'re supposed to be there in 15 minutes?
You\'re going to lead us now or in 15 minutes?

MR. WEINIG: We\'ll need to change the configuration of the room, so I\'d
like folks to bring their things over to some of the chairs and then
we\'ll walk down to the NEIC and stop at the restrooms for a minute on
the way. And as soon as you invite comments from the public if there are
any, we\'ll leave.

CHAIRMAN RELLES: I\'m going to do that. Due to the nature of our
walkthrough, we\'ll go off the transcript recording and resume tomorrow
at 8:30. Before we conclude and move to the NEIC, are there any comments
or questions from the committee or from the public?

Okay, we\'ll resume the meeting tomorrow here at 8:30. Thank you very
much.

(Whereupon, the meeting was adjourned at 3:07 p.m., to resume on Friday,
November 20, 1998, at 8:30 a.m.)