AMERICAN STATISTICAL ASSOCIATION

COMMITTEE ON ENERGY STATISTICS

\+ + + + +

FALL MEETING

THURSDAY, NOVEMBER 2, 2000

\+ + + + +

The Committee met at 8:30 a.m. in Room 8E-089 of the Forrestal Building,
U.S. Department of Energy, 1000 Independence Avenue, S.W., Washington,
D.C., Dr. Carol A. Gotway Crawford, Chair, presiding.

[PRESENT]{.underline}:

CAROL A. GOTTWAY CRAWFORD, PhD Chair

JOHNNY BLAIR Guest

F. JAY BREIDT, PhD Member

THOMAS C. COWING, PhD Member

JAMES K. HAMMITT, PhD Member

NICOLAS HENGARTNER Guest

CALVIN A. KENT, PhD Member

W. DAVID MONTGOMERY, PhD Member

WILLIAM G. MOSS, PhD Member

POLLY A. PHIPPS, PhD Member

RANDY R. SITTER, PhD Member

ROY W. WHITMORE, PhD Member

[ALSO PRESENT]{.underline}:

STAN FREEDMAN Designated Federal Official

MARK MAZUR Acting Administrator, EIA

LARRY PETTIS Deputy Administrator, EIA

I-N-D-E-X

Welcome, Carol Gotway Crawford 3

Opening remarks, Mark Mazur, Acting EIA 11

Administrator

A Briefing on EIA\'s New Approach to Long-Term 25

International Energy Forecasting:

Mary Hutzler, EIA

Summary of Three Break-Out Sessions 52

Invitation for Public Comment 64

New Ways to Process, Store and Make EIA 65

Data Accessible: Bill Underwood

Estimating Procedures to Reduce Data 100

Reporting Lags: Inderjit Kundra and

Roy Kass, EIA

Interactive Session: Cognitive Testing 151

of Potential EIA Graphic Redesigns:

Colleen Blessing, Howard Bradsher-Fredrick,

Antoinette Martin, Herb Miller, Renee

Miller and Robert Rutchik

Invitation for Public Comment 167

P-R-O-C-E-E-D-I-N-G-S

(8:43 a.m.)

CHAIRMAN CRAWFORD: I would like to go ahead and get started. We have a
few announcements and some introductions that need to be made before we
get to the heart of the meeting.

This meeting is being held under the provision of the Federal Advisory
Committee Act, and this is an ASA, not an EIA, committee which
periodically provides advice to EIA.

The meeting is open to the public, and public comments and welcome. Time
will be set aside for comments at the end of each session. Written
comments are welcome, and can be sent to either ASA or EIA.

All attendees, including guests and EIA employees, should sign the
register. We are also asking for your e-mail address.

In commenting, each participant is asked to speak into a microphone.
Microphones are spread around the table, and for those of you sitting,
there is a microphone there that you should speak into. The transcriber
will appreciate this. All committee members at the head table need to
speak clearly into a microphone. The microphones have a range of about
one to two feet.

I would like to introduce \-- Actually, there will be two guests today.
One is Johnny Blair who, I think, you know from last time. The other one
is Nicolas Hengartner from Yale University. They will be appointed to
the committee beginning next \-- Their term starts in January 2001. So
they will be officially committee members next year.

We are not sure about Mark Bernstein. He is a formal member, but he had
a family emergency, and so we are not sure if he will be here or not.

Now I would like each of us to introduce ourselves, and I will start. My
name is Carol Gotway Crawford, and I am with the National Center for
Environmental Health at the Centers for Disease Control.

DR. WHITMORE: I am Roy Whitmore, a statistician with Research Triangle
Institute.

DR. BREIDT: Jay Breidt, statistician, Department of Statistics, Iowa
State University.

DR. SITTER: Randy Sitter, statistician, Department of Statistics and
Actuarial Science at Simon Fraser University in Vancouver.

DR. HENGARTNER: Nicolas Hengartner. I am a statistician at Yale
University.

DR. PHIPPS: I am Polly Phipps. I\'m a research analyst with the
Washington State Institute for Public Policy.

DR. BLAIR: Johnny Blair. I am a survey methodologist at the University
of maryland.

DR. MOSS: Bill Moss. I am an economist with The Brattle Group in
Cambridge, Massachusetts.

DR. COWING: Tom Cowing. I am an economist at Binghamton University.

DR. HAMMITT: Jim Hammitt, Harvard University.

DR. KENT: Cal Kent, Dean of the College of Business at Marshall
University.

DR. MONTGOMERY: David Montgomery, I\'m an economist with Charles River
Associates here in Washington.

MR. FREEDMAN: Stan Freedman, Energy Information Administration.

MR. MAZUR: Mark Mazur, Acting Administrator, Energy Information
Administration.

MR. WEINIG: Bill Weinig, Energy Information Administration.

MS. HUTZLER: Mary Hutzler. I am Director of the Office of Integrated
Analysis and Forecasting, Energy Information Administration.

MR. LATTA: I am Robert Latta. I am with the Energy Information
Administration.

MR. KHAN: I am Ehsan Khan in the Office of Science at DOE.

MR. KYDES: Andy Kydes, technical advisor to the Office of Integrated
Analysis and Forecasting.

MR. HALE: I am Doug Hale, economist, EIA.

MR. DeMOUY: Lou DeMouy, Energy Information Administration.

MS. PAIK: Inja Paik, DOE Office of Policy.

MR. DIEDRICH: Roger Diedrich, Energy Information Administration.

MR. KAPPELO-COHEN: Barry Kappelo-Cohen, Energy Information
Administration.

MR. GRILLOT: Mike Grillot, Energy Information Administration.

MS. BLESSING: Colleen Blessing, Energy Information Administration.

MS. MILLER: Renee Miller, Energy Information Administration.

MR. BROENE: Tom Broene, EIA.

MS. NESPOLI: Kristen Nespoli, economic statistician with the Census
Bureau.

MR. GAFFEN: Mike Gaffen, Datum International.

MR. YILDIZ: Orham Yildiz of Macro International.

MR. KNAUB: Jim Knaub, EIA.

MS. SPENCER: Linda Spencer, Edison Electric Institute.

MS. SAMUEL: Rachel Samuel. I am the Deputy Advisory Committee Management
Officer here at Department of Energy.

CHAIRMAN CRAWFORD: After Mary Hutzler\'s briefing and questions this
morning, we will adjourn into three break-out sessions, and please hook
up with the leaders of these sessions, Doug Hale, Jim Knaub, or Robert
Latta, so that we can get to the rooms as quickly as possible.

Committee members assigned to each break-out session are listed on the
agenda, so that you will know which one you are in. You should have
received an e-mail from me assigning you to one so that you could
prepare, and I don\'t think there\'s been any changes there.

Lunch for the committee and invited guests will be at 12:15 in Room
1E-226 on the first floor of this building. We will resume after lunch
in this room, and again let\'s stay together.

Our dinner reservations this evening are at 5:30 at McCormick &
Schmick\'s at 1652 K Street, N.W. We have reservations for the Board
Room. So we can go directly from the Forrestal either by cab or by
Metro. I guess it will depend on the time the meeting actually ends.

We will have breakfast for the committee again tomorrow here again
beginning at 7:45. Hopefully, security will be notified of that, and we
can get in there on time. Again, for the committee members staying at
the hotel, let\'s meet at the lobby at 7:30 and walk over together. The
meeting will resume here tomorrow at 8:30.

Some more technical information: For your information, Stanley R.
Freedman, who is sitting over here, is the designated Federal officer
for the Advisory Committee. In this capacity, Dr. Freedman may chair but
must attend each meeting, and he is authorized to adjourn the meeting if
he determines this to be in the public interest. If you want to chair
the meeting, Stan, let me know.

DR. FREEDMAN: Please be my guest.

DR. MONTGOMERY: Can we lobby you on adjourning?

DR. FREEDMAN: No.

CHAIRMAN CRAWFORD: He must approve all meetings of the Advisory
Committee and every agenda, and I assume that you have done that. Also,
he may designate a substitute in his absence.

Now some of our scheduling for today and tomorrow is rather tight. So I
will try to keep us on schedule. I\'m that way anyway. I\'m a stickler
for keeping us on schedule. So your cooperation is greatly appreciated.

Finally, one administrative item for the committee. We are without a
Vice Chair, and I have considered various members for this position, and
I would be glad to hear from you today or this evening regarding your
thoughts on this, too. If you would like to volunteer, that would be
great. If you think of someone that is particularly well suited to the
job, please let me know that, too.

I would like to appoint someone first thing on Friday to serve that
capacity.

Since the term of Chair is limited to two years, it is important, but
not urgent, and actually I think it is fairly important so that this
person has about a year to sort of be the Vice Chair and work with me on
some of the matters in terms of filling committee positions and things
like that. So please give it some thought, and let me know your
thoughts.

Finally, as an important meeting mechanic, committee members, please
turn your name tags vertically like this when you have a question, and I
will call on your as your tents pop up. Then members of the audience,
when you have a question if you could just raise your hands and then
proceed to the microphone to ask your questions.

I think that that concludes all the administration for this. Now it is
my pleasure to recognize Mark Mazur, Acting Administrator for the Energy
Information Administration.

ACTING ADMINISTRATOR MAZUR: Thanks a lot, Carol. This is my first
meeting for the ASA Committee on Energy Statistics. So while I think
this is a valuable group that provides us with good advice and ways to
improve the way we do business, part of this session for me is going to
be a learning session, basically, to try and determine in my own mind
what you all do, what is valuable, and what we can build on in the
future.

I\'m pretty much a novice at this type of advisory committee. So it is
very much a learning experience for me.

I do kind of note, interestingly, that all the economists are sitting
over here, and I\'m hoping that\'s not an indication that there\'s not a
lot of collaboration going on.

DR. COWING: There\'s a green line.

ACTING ADMINISTRATOR MAZUR: I am going to attend as much of this meeting
in the next couple of days as my schedule permits. So, hopefully, it
will be big chunks both of the days, and I\'m hoping that that will
provide a fair amount of insight to me as to how best we at EIA can
utilize the collective set of talents and wisdom that you all have.

I want to welcome the two guests here, and hopefully, you will be
appointed in January and be quite valuable members of the group. It\'s
good to have you here.

What I want to do this morning is just touch on a couple of overall
issues for EIA, and just to kind of tick them off, they are our budget
situation, a couple of short term challenges that we have facing us
dealing with restructured industries, and then long term challenges,
premier long term challenge that EIA faces, how we are going to
replicate the workforce over the next decade or so.

We have a fairly older workforce, just like much of the Federal
government, and we\'ve got to create a new generation of EIA employees
so we can build a valuable organization for the future.

On the budget situation, I guess my take on the overall budget situation
is that EIA has turned a corner on budget issues, that in the mid-1990s
we took a fairly significant budget hit. The last couple of years we\'ve
seen gradual increases in our budget of a couple of million dollars a
year.

For Fiscal year 2001 we have a budget given to us by Congress of
\$75.675 million. That\'s an increase of about \$3.3 million over last
year, but I think, more importantly, it\'s \$675,000 more than we asked
for. It\'s fairly rare that Congress gives you more than you ask for.

Typically, what happens is the House gives you something, gives you some
amount, and the Senate gives you some amount, and they compromise
somewhere in between. The compromise here was above what the Senate gave
us by more than \$1.5 million.

So I think I take that as a good sign that the folks in Congress value
the work that we do here, that there is a fair amount of support among
the Members and staff for the data we collect, the information we
provide, the briefings that we do, the analyses that we put forth.

For the next fiscal year, I\'m planning to ask for a significant
increase. We have a number of urgent needs we need to address. When you
deal with deferred maintenance over the last few years, we\'ve worked on
\-- well, we\'ve eaten up a fair amount of our capital, and we need to
spend a little bit of time going back and dealing with that deferred
maintenance.

We need to improve some of our data information systems, improving our
coverage and improving data quality, and we need to continue to upgrade
our analytical ability. That\'s one of the key areas that we provide a
good service to our customers.

Of course, the upcoming election and new Energy Secretary are going to
have something to say about this, but whoever is Energy Secretary next
year will surely place a high value on the information that we provide,
the credible information about energy issues. Given the experience of
the past year or so, I just don\'t see that as an issue at this point.

Short term challenges that we face: Probably there are none that are
more important or more challenging than how we deal with restructured
industries. In particular, we see the natural gas industry shifting from
a regulated industry to a competitive structure, and we see the
electricity industry similarly switching from a very heavily regulated
industry to one that is much more competitive.

In the regulated situation, it\'s fairly easy for us to collect data. We
just ask the regulated utility to send us the information. A lot of it
was the same that they sent to the regulators. They are not too unhappy
to send it to us, and there are really no issues about competitive
advantage or disclosure of any proprietary information, because
essentially this is all publicly available information in one form or
another.

Under more competitive market situations, companies get concerned about
providing this information about their operations that might harm their
competitive position, and in the competitive industry it is often
difficult to figure out exactly where in the distribution chain you go
to collect the information that you need.

In the case of a regulated industry you dealt with the utility. They
could tell you what the volume was, what the price was. In the case of
the natural gas industry in a competitive market, you go to a local
utility. They may be able to tell you the volume. They can\'t tell you
the price. You have to find someplace in the distribution chain where
you can collect both price and volume information.

Now we are working through these issues in both natural gas and
electricity. In natural gas, we have a program called NG^2^ Next
Generation Natural Gas. That\'s our plan to improve our data systems so
they function well in a restructured industry.

Stan has been working with a number of folks in this group to put
together a survey that we hope to field early in 2001 to provide us
information about the natural gas industry, I think, in Georgia and
other competitive markets.

DR. FREEDMAN: Five states.

ACTING ADMINISTRATOR MAZUR: Five states. So that\'s going to be a big
step forward for us. We have been field testing this. So I will be
keeping my fingers crossed as we enter into 2001 that we are actually
collecting high quality data from these folks.

The comparable for electricity is Electricity 2002, which gives a hint
about our time frame here. In this effort we are going to be updating
our surveys, our survey frames, changing the collection points, and
bringing the treatment of utilities and NG a lot closer together.

We have also begun collecting wholesale electricity prices and making
these available on our Website in a systematic way.

Basically, all these steps recognize that we have a much different
industry, electricity industry, than we had, say, a decade or so ago. We
are hoping that the lessons we learn in natural gas we can apply in
electricity and vice versa. That\'s part of the work that Statistical
Methods Group has been \-- the work on natural gas. We are hoping that
they can provide similar services to the electricity folks and will be
able to share the lessons learned there and be able to make these
changes in as efficient a way as possible.

Now long term challenge: As I said earlier, our preeminent long term
challenge is staffing. Basically, like a lot of other Federal agencies,
we have a fairly senior workforce. This has been an asset for us,
really.

We have an incredible amount of job specific human capital at EIA. The
downside, though, of course, is that we are facing a wave of potential
retirements as we go forward, which is going to be much larger than we
have ever experienced before.

Our challenge is to build the next generation of EIA workforce that is
every bit as good as the one we have right now. We\'ve begun tackling
this in a couple of different ways.

First, on the staffing side we\'ve been hiring at a rate that is much
greater than we have hired in previous years. This past year we hired 25
employees, which is large for an organization the size of EIA. Next
year, we are expecting to hire another thirty.

We think we offer an attractive set of job characteristics at EIA that
will help us get high quality potential employees. But if any of you
guys have students that you want to steer in our direction, we would be
happy to talk with them. Really, it is important for us to rebuild a
workforce as we go forward.

In addition to hiring, we are also working on \-- We are beginning to
embark on a plan of transferring knowledge from one generation of EIA
employees to another. This is still in the early stages, but we want to
establish a norm of knowledge transfer, so that when folks approach
retirement, they take it as part of their job to pass on to their
colleagues the information they know.

We are not as deep at EIA as we used to be in terms of having several
people being able to do the same job. In fact, in many places we are
exactly one person deep. It makes it all the more important that we are
able to engage in that knowledge transfer. So any ideas that you folks
might have in that regard also would also be much appreciated.

That\'s pretty much all I wanted to say today to you. I view this
meeting, as I said, much more of an opportunity for me to learn about
you than anything else, and I\'m looking forward to being able to
utilize the longstanding relationship between the Committee on Energy
Statistics and EIA as best as we can. I would also be happy to take any
questions at this time or, alternatively, to move to the presentations
today. But if anybody has got any questions, I\'m happy to answer them.

CHAIRMAN CRAWFORD: Calvin?

ACTING ADMINISTRATOR MAZUR: You didn\'t even put your tent up, and she
knows.

DR. KENT: At the opening of your \-- At the onset of your comments you
talked about three areas, and you only mentioned them generally. Would
you be more specific about what you are going to be looking at in all
three of those areas?

ACTING ADMINISTRATOR MAZUR: What do you mean, three areas?

DR. KENT: Well, you talked about that you had three major areas of
concern.

ACTING ADMINISTRATOR MAZUR: Budget, restructured industries, and long
term challenging of hiring and staffing.

DR. KENT: Well, no, I was talking about under \-- when you were talking
about restructuring your industries, specifically, what are your goals
in each one of those areas that you mentioned? I should have written
them down.

ACTING DIRECTOR MAZUR: In restructured industries, basically, what we
want to do is we want to have good coverage of the industries we are
looking at.

In the case of natural gas, on the industrial side the amount of
coverage we\'ve had has dwindled to some pretty small number. It\'s just
really difficult for us to get good information about quantities and
prices of gas delivered to industrial customers.

So the coverage ratio is low, maybe 25 percent or something like that,
really low. So the idea is to get that up to some much larger number so
we have better coverage.

On the residential side, unless we go to you as a residential customer
and ask you what you are paying for your gas and how much you are
consuming, it\'s tough for us to get that information in a restructured
industry.

So what we are trying to do is figure out a way that we can get that
comparable amount of \-- a comparable kind of information for
residential customers as well. That\'s the survey that we have been
field testing the last month or so.

So we want to get good volume information, good price information in our
competitive environment, which is just much more difficult than we\'ve
been able to do in the past.

It\'s been easy just to ask regulated companies what\'s your revenues
from your different sectors, how much did you deliver. We can do the
arithmetic. So we\'re trying to get comparable information going
forward, and it\'s tough, tougher than usual, because gas marketers
segment their customers.

If we look around the table and all of us have a contract with
Washington Power and Light, we all could be paying a different price for
gas that we are getting this summer or this winter. So that makes it a
much more difficult situation than it\'s been in the past.

So what we want to do is we want to get the coverage up. We want to get
the quality of information to levels that we are comfortable with, that
we\'ve historically seen, and then we will be able to knit the data
series together, stuff we\'ve had in the past with stuff we will be
getting in the future, in a reasonably good time series. It\'s the exact
same situation in electricity as well.

DR. MONTGOMERY: I was interested and, I guess, concerned about pointing
out that we were all getting older, including the EIA staff. I wondered
if that would be something you would be interested \-- if you put some
plans together \-- scheduling for a presentation here at the next
meeting. I think both on the hiring issue, I think we probably represent
here your major competitors in universities and kind of consulting, and
I\'d be interested in kind of thinking about the issues about how you
transfer what people know now into \-- you know, people who are here to
the people who are coming.

ACTING ADMINISTRATOR MAZUR: Very good idea. Could we put that on the
agenda?

CHAIRMAN CRAWFORD: Yes. Well, I\'ll suggest it.

ACTING ADMINISTRATOR MAZUR: Who do you suggest it to? Stan?

CHAIRMAN CRAWFORD: No. What I do is \-- I guess it\'s Stan, but I make a
list.

ACTING ADMINISTRATOR MAZUR: That\'s a good idea. I think that would be
helpful. Presumably, you are facing similar situations in terms of
we\'re all looking for the same high quality people.

DR. MONTGOMERY: Exactly.

ACTING ADMINISTRATOR MAZUR: And figuring out a way to turn them out and
get them interested would be a real plus.

DR. MONTGOMERY: And I\'ve noticed that, although I think we pay a great
deal better, we tend to lose people to other Federal agencies, because
they offer \-- well, the Fed pays well, too, but the Fed and the Justice
Department and FTC. They are quite competitive in getting economists in
there.

ACTING ADMINISTRATOR MAZUR: And while we pay less, we think we have some
characteristics to make adjustments.

DR. MONTGOMERY: Exactly.

ACTING ADMINISTRATOR MAZUR: Anything else? Thanks.

CHAIRMAN CRAWFORD: Thank you. Okay, we will move on. Mary Hutzler will
give us her briefing on EIA\'s new approach to long-term international
energy forecasting.

MS. HUTZLER: I just want to correct one thing that Carol said. I\'m
going to talk to you about our international forecasting, not our
general approach to forecasting. We are still using the same models that
we built back in the early Nineties, International Energy Modeling
System, to do our U.S. forecasting, and we will continue to do so. This
is to really enhance our international modeling effort.

Back in April I gave you a status report at that time, and you all asked
me to come back to update that status report. When I was here in April,
I did a bit uncomfortable, because I was telling you that we had changed
our minds about our direction in terms of doing international
forecasting.

We had started out with an approach of using the second generation model
and building upon that, a model that Pacific Northwest Laboratories
developed, and it\'s the Computerized General Equilibrium Model.

We had made the decision this spring that that was the wrong approach
for us, for many reasons which I told you about before in April, and we
had decided to go to the Markal model at that time.

My discomfort was that we didn\'t really have a lot of time to look at
Markal. We knew that that was a choice, and it was probably the only
internationally oriented interview model out there. So that was the one
we wanted to start with.

I feel a bit more comfortable today, because we\'ve actually had some
time to look at Markal and understand it better, and that is the
approach we do want to take. What I am going to do today is to tell you
what we plan to do with it and what our schedule is.

What we have done over the past six months is that we have provided
training. Brookhaven National Laboratory has come on site and provided
training for my international staff. We have also worked with the ETSAP
community. That\'s the Energy Technology Systems Analysis Program.

There are a number of countries, probably \-- countries or regions, but
there are probably about 30 Markal models out there right now, and the
country representatives participate in the ETSAP community. They meet a
couple of times a year, and they met in may in Paris and then again in
October in Switzerland. I had one of my lead analysts, Barry Cohen,
attend both those meetings where he talked to them about the fact that
we wanted to build a world Markal model and that, like all our other
models, it would be part of the open domain in terms of modeling, that
we would provide it to the public, and tried to get their support in
helping us.

So we are starting out in that tact, so that we can try to build on some
of their experience. So besides this, we also developed a number of
issues papers. There were ten of them, and they were provided to you. I
don\'t know if you wanted to read them or not, but you have the
opportunity to read them.

They dealt with different issues that we felt we needed to deal with in
terms of the Markal funneling. Some of them deal with the macro-economic
component that we might use, depletable resource issues, foresight
issues, the basic question of data, what data we might use.

So we produced those back in June, and then we met in July with our
independent review committee. The committee essentially endorsed our
approach of moving on with the Markal model

So that\'s where we are right now. What I want to do is to talk about
Markal a little bit, and then talk about our schedule.

There are a number of versions of Markal, and we found that this
confused a lot of people when we talked about markal; because a person,
when you mentioned the name Markal, would think of one of these
versions, but they are actually a bit different.

The basic Markal, standard Markal model, is a bottom-up model. It\'s a
dynamic linear programming model. But what you do is you specify
exogenous service demands to it, and it has no feedback loop. It does
look at technology in detail, and it does look at it by the various
sectors.

Then there is Markal-ED. ED stands for elastic demand. What this does is
to add a price elasticity to the service demand, so that when you get
prices feeding into your energy service demands, you can get feedback
that changes your service demands.

Markal-Macro actually has a macroeconomic component to it. That
macroeconomic component does apply service demand so that they don\'t
necessarily need to be exogenous.

The component that\'s there now is probably a very simplified
representation. It was something that we developed very quickly, and
there are people that use Markal-Macro, but not every single country
that has a Markal model uses that approach, and probably most of them do
not.

Markal-R deals with the multi-country or multi-region representation. So
this allows you to look at several regions and see how they might
interact in trade. There are only a few Markal-R models out there. There
is one that deals with the Canadian provinces and that adds on different
countries to it. The U.S., of course \-- There\'s another version that
has India.

If you look at the different versions, you wonder why the various
countries are there. The real answer is they were able to obtain those.
So that\'s where the state of the art is right now. There is no global
country or global region model.

Then I mentioned TIMES to you before. We really didn\'t know what TIMES
was about at that time in a lot of detail. We just knew it was an
enhanced version, and it had a multi-region capability.

That model is still undergoing debugging. It is not ready for any energy
community to use at this point in time. TIMES stands for The Integrated
Markal Ephram System, and I\'ll talk to you a little bit more about that
later on in my presentation.

So as I said, different people will think of different versions of the
Markal model, and what our proposal essentially is is to start out with
Markal-ED but have a multi-regional capability so that we can deal with
the entire world.

I have a few slides that go into a little bit more detail on each of the
different versions of Markal. I\'m not going to go over these in that
detail. You can read it.

Here I just wanted to point out that there are resource technologies,
process technologies, conversion technologies, demand technologies, and
then here\'s end-use service demand. That can even be exogenous or
obtained from the Markal-Macro version of the model.

Now I talked about Markal-Ed already. Again, this is where you provide
price elasticity so that you can get feedback, and you can change your
service demand. Essentially, what you have to do is, instead of a fixed
exogenous energy service demand of the standard Markal, you use
piece-wise linear energy service demand functions. That\'s part of the
development process that we will be undergoing for each of the regions
that we\'ve decided to work on.

Now on the Markal-Macro, one thing I want to point out is that the U.S.
models that the Department of Energy\'s Policy Office uses is a
Markal-Macro model. it is a nonlinear programming model. So it takes a
lot longer to solve than the regular standard Markal model which use a
linear program.

One country or one region can be solved still in a reasonable amount of
time, but if you start adding lots of regions, you really blow up the
matrix so that instead of like 11,000 rows which the U.S. model, I
think, has, you\'re talking about way over 100,000, if you have ten
regions, let\'s say.

So you get into an issue of being able to solve the problem, and that\'s
one of the issues we have to deal with when we go further and want to
add on a macroeconomic component.

You can see down here that the macroeconomic component for our work, we
see as a research problem. We see it for two reasons. One is to develop
the model itself and to deal with many issues, issues of how do you deal
with developing countries versus developed countries, plus how do you
solve the problem; because we are going to have a multi-region model,
and we are going to have to figure out what kind to use.

There will be more discussion of this in the breakout session later, as
well as the issues such as data.

Now I just want to summarize some of the benefits we see with the Markal
approach and why we chose it. It\'s technology rich. That\'s one of the
issues we wanted to deal with.

If you remember, one of the reasons why we decided to build this model
was to deal with the issue of carbon emissions trading, and with that
issue comes issues of CDM and other issues of trying to reduce carbon
emissions that deal with technology. So we did want to make sure we had
a technology rich model.

Another benefit of Markal is that it\'s capable of handling the
different amounts of data that you get with the different economies. Not
all countries have the same amount of data. The U.S. is very rich in
data, but there are lots of countries that aren\'t. Africa, a lot of
Asia are not, and we are going to need a system that can be adaptable to
the amount of data that we can get from a particular country or region.

Another issue deals with how we are going to manage the process. Using
Markal we can set up different countries and regions and use a team
approach where we can have different people on the staff working on a
different region of the world.

Lastly, because there are thirty-plus models that exist, we can build on
the work that other countries have done.

Now it\'s a Catch 22. Because we are in the public domain when we
release this model, many countries don\'t want to give us their models.
They may have proprietary data or they may not want us to know exactly
how their economy works.

We\'ve started working with some countries. I\'ll be talking about that
a little later in my presentation, but that\'s going to be a major
effort for us to get their expertise and have them work with us in some
sort of collaborative fashion.

Now what we are planning to do is to have 15 regions. First we will
build single region models, and the 15 regions are as they are listed
here: U.S. \-- and as I said, the U.S. model does exist right now.
Policy has one, and it is calibrated to our annual energy outlook that
we produce using the National Energy Modeling System.

We did break out North America, since Canada and Mexico are major
trading partners with us. A provincial model of Canada does exist, but
we need to make it a one-country model. A model of Mexico does not exist
at this point in time. So that would be new.

We are looking to build one for Annex I Europe. A European Union model
exists, and many countries of Europe have a Markal model. Italy, France,
Germany are some examples.

The former Soviet Union: I believe one is being built right now. I
don\'t know when it will be ready, but we need \-- Some of the same
people that built the Canada model are working on a former Soviet Union
model.

A model of Eastern Europe doesn\'t exist. There is a four-region model
of Baltic States that includes Estonia, Latvia and Lithuania with the
countries.

A model of Japan does exist. Again, being able to get the model and use
it is something that will be a challenge for us.

South Korea doesn\'t have a model right now, I don\'t believe. China
does have one. India has one that a student developed. It probably needs
additional work.

There is not a model of the rest of developing Asia. So that\'s
something that we will have to deal with. Also there is not a model for
Latin America, though there are many countries right now in Latin
America that are having models built, just as we speak. I believe there
is one being built for Ecuador in Central America and maybe Brazil in
South America.

Australia has a model, and that\'s about it. Africa and the Middle East
do not. We do want to break out the Middle East for questions dealing
with oil, not so much the carbon issue. That would be an oil issue,
because we do intend to use this model for other policy issues.

A schedule has us building each of these single region models by July of
next year.

Now there are some areas of software development that we are working on,
and we already have contacts out to the individuals working in these
areas.

ABARE \-- that\'s the Australian Bureau of Agriculture and Resource
Economics \-- has a software system called ANSWER. ANSWER is a system
that allows you to more easily use Markal. It\'s sort of a file
structure system. What you do is you use ANSWER to build your regional
model or your country model.

What we have asked that organization to do is two things: One, to come
up with a common new technology data repository, so that we can put all
of the single region databases into one location. Then we wanted them to
change the ANSWER system from a single region operation into a
multi-region operation.

Our basic schedule for that is next March we\'ll get a beta version of
the common technology data repository, and then that will be followed by
the multi-region capability in April, with a preliminary user\'s guide.
That work will be finalized in July of 2001.

Another software development area deals with ANALYST. ANALYST is a tool
that is already under development, and it\'s a tool to help you take a
look at the solutions as you de-bug the model, and also to help you look
at the results.

Essentially, we are helping to fund the development. A company in Canada
is already working on the ANALYST software, and some of what we will
obtain is the ability to easily get information out, both in a printed
version and a graphical version, and also be able to look at various
scenarios and to compare them, so to help us look at the results and
figure out more quickly what the results are and how we may need to
change the model.

It also will provide an archival capability, which is important to us,
because every time we do a study with any model in EIA, we have to
archive it.

The schedule calls for a version to be available this month, and we
actually have a training session scheduled November 20, and we\'ll be
testing the version of ANALYST we get, and we anticipate getting a final
version in January.

Now another important issue that we see with this whole model
development is peer review. We have two ways of handling that. One is
working with the community, and we have already tasked the operating
agent, who is Tom Kram, and he essentially has done the development for
the European union to review our models and to get other countries to
review them. This will take place next August after we finish building
each of the single models.

The other thing we want to do is to work independently with each of
these countries so that we can talk to them during the year to
understand better how they have built their own models and how they see
their energy system represented.

We have started talking to a number of people. Last month I was in
Beijing, and I talked to the Tsinghua University folks that built the
China model.

We have working relationship with the person in Italy who built that
model, and he will actually be working with us later on this year. We
talked to Canadian folks, and they are going to be working with us on
the Canadian model that we are going to be building later on this month
and, as we said, we have already talked to Tom Kram.

Now areas of future work that we see that we need to do after we have
the initial system built deal with four areas. One is the macroeconomic
area, which I already mentioned and which we see as a multi-year effort.

Another area deals with just doing the model enhancements that we will
be learning about and that will need to be done based on building the
single region models and making the new multi-region model.

Also we want to do a lot more in the area of fossil fuels and resources
to do more in terms of building the capability in the model. Then
lastly, we want to keep abreast of the TIMES development to see if that
is something that we want to use later on as we proceed with the Markal
development.

I\'d like to end \-- Well, I forgot one thing. I do need to mention
documentation. We do have to document our models in EIA, and we are
planning to do so here as well.

We intend to use Markal for our International Energy Outlook 2002, which
will be released in March of that year and, therefore, will need to have
documentation developed shortly thereafter to meet the standards that we
have in EIA.

So this does follow EIA standards, and it\'s just a list of what you can
expect from us in terms of the documentation.

My last slide just is a summary on the schedule. I wanted to show you
how all the different parts come together. One of the things that we are
working on right now is a common structure for building regional models.

One of the problems with just getting any of these models in is they all
have a different structure, because they are working on different energy
systems. So we just can\'t bring them in and use them.

We need to define a common structure, common naming conventions, and
have each of the models built to that specification. That definition
construction is being worked on right now, and we anticipate having it
done this month.

As we also told you, we are going to have training this month on the
ANALYST software. We are having the other software work done in time for
the people who are going to be building the models to start work.

Most of those folks are working on the current international energy
outlook. So they are going to be tied up sometime through January and
March, and then they will really get involved in building these models.
They will be done in July.

We will have the review in August, and then we will look at the
multi-region capability that we have to do, which also means that we
have to take a look at the regional trade flows and figure out where
those trade flows should be, and if there are any balance on those trade
flows.

Preliminary forecasts will be available November of 2001 for our
international energy outlook. As we said before, it will be out in
March, and then the documentation in April.

So that will be version 1 of this model. As with any model we build in
EIA, we\'ll be making enhancements to it and working on it for many
years to come.

We have a few minutes for questions. Yes, Cal?

DR. KENT: There\'s just dozens of questions that come to mind.

MS. HUTZLER: Yes, and you have two hours in the breakout session to ask
them.

DR. KENT: Let me one overarching question. What is the major driver
behind this effort? The last time when we were familiarized with it, it
was so that we could make estimations of carbon discharges.

Is that still the major driver behind this, because you did make a
fleeting reference in your presentation, well, we\'re going to want to
use this for other policy issues? But if that is your driver, how is it
coloring your whole approach to this, and how is that going to limit its
usefulness for other policy issues?

MS. HUTZLER: Well, we are trying to keep a broad view in mind so that we
can try to deal with other policy issues. That is one of the reasons why
we are breaking out the Middle East as a separate region so we can deal
with the resources in that particular part of the world.

The climate issue is the major driver right now, and that came about
when we did the study of the Kyoto Protocol. The one thing that we
couldn\'t do is to determine the international price for carbon trading,
and we actually had to use for our analysis the U.S. trading price,
which, of course, is not the global international price.

So that\'s the main driver to all of this. Hopefully, we are going to
keep other policy issues in mind so it doesn\'t color the detailed work
on this. But one of the main reasons why we wanted the technology rich
model is due to the carbon issue, because, of course, more efficient
technologies can reduce your energy consumption and reduce carbon. So
that is a mean driver to why we selected it.

DR. KENT: What is driving your timeline, because this is an extremely
ambitious \-- I mean, I would almost say incredibly ambitious.

MS. HUTZLER: Probably more the budget issue and the fact that this was a
multi-year effort in the budget. When you do have the budget to look at,
they do want to see accomplishments. So we may not make this, but it\'s
a goal, and it\'s a target.

In EIA we went through a lot in terms of quality management, and we do
have stretch targets. This is a stretch. We might not meet it, but it\'s
something that we are targeting. David?

DR. MONTGOMERY: I think Tom was ahead of me. I\'ll be happy to defer to
him.

MS. HUTZLER: Yes, Tom? I should mention that Tom did read our issue
papers and provide us comments. I thought that was quite nice that he
took his own time to read all ten of our papers and comment back to us.
So \--

DR. COWING: Well, I will second many of Cal\'s comments. This is an
incredibly ambitious project, and I\'m glad that we are all aware of
that.

First of all, I thought the project as you have set it out is very
impressive, but I, too, have some problems about the timing. Maybe the
answer is, well, we can be flexible there, too, but let me refer you to
the \-- The way I want to phrase the question is: Let me refer you to
the slide on the ten or dozen regions, and ask you to comment on each of
the \"not availables.\"

The \"availables,\" I assume, probably can be put together by July,
which is less than eight months away. But I\'m not too sure about the
\"not availables.\" I wonder if it wouldn\'t be reasonable to have some
kind of priority in mind, that you take what you could get, and you work
on what you can work on, and if you to leave some things aside, which
might be a lot of the world but it\'s the tough part of the world to
model, then you do that and, as you have time in the future, you start
bringing these slowly on line.

That raises some questions about how you are going to structure this. So
that you could start with a core set of regions and then plug in
additional regions in kind of a modular fashion later on as they become
available.

MS. HUTZLER: That is an excellent point, and what I want to mention is I
will intentionally put certain benefits down on this chart. One is that
the model has the capability of handling different amounts of data. We
don\'t have to do the same depth for all regions.

Obviously, the U.S. is going to have a lot of depth, and a lot of other
regions are going to have very minimal depth, especially in starting
out. You know, we may be able to add later as we find more data or even
as different countries of the world build more Markal models.

The other thing is that it is a modular approach. So we can put in
different regions or make changes to them. I was of the opinion that we
should start small, and our original version of the regions were less
than you see here.

It turns out that my folks who were working on it \-- For instance, we
didn\'t have South Korea broken out, but my expert in the Asia area felt
that she preferred to have it broken out, because the economics of the
country were so different than the rest of developing Asia. So that\'s
one reason why that is broken out.

The other area we broke out was Eastern Europe and the former Soviet
Union for the same reason. I had thought it might be better timewise to
merge those, but the analysts in charge of these areas just have
different opinions, depending on the data that they have and the way
they view that region of the world. But I should have mentioned when I
put this slide up, we need to be very flexible with it.

What actually comes out may not be these 15 regions, depending on the
data and depending on analysts\' judgment as they work with the models.
But your point is a good one, and is one that we\'ll have to be dealing
with.

DR. COWING; If I could ask a follow-up question: If these models turn
out to be fundamentally different, particularly some of the regions
where it\'s not clear how they should be modeled, it\'s not clear what
the behavior is, there are real data availability problems, that raises
a whole set of issues about how you plug in a module like that into a
framework that you\'ve already gotten constructed, designed and
constructed, unless that framework is incredible flexible. I guess that
would be point number one.

Point number two is it may well be that the models differ at terms of
the variables that are modeled, but if you have an overall integrating
framework that attempts to make some kind of global sense of all of
this, then there\'s a set of issues there about, you know, you can\'t
add things up that aren\'t available in some of the models. So what do
you add up?

MS. HUTZLER: Well, you have to be flexible with another region of the
world in dealing with that kind of issue. So it\'s going to have to be
some of \-- We would have to change our regional structure, probably,
and we are trying to deal with flexibility in terms of building the
integrating structure.

Barry, I know you want to \-- I have staff back there who wants to speak
in more detail on this, but since we are already late, I think it\'s
best if we talk about this in the breakout session. But they will have
more specifics to tell you about it at that time.

Dave, did you want to do a quick question before we break?

DR. MONTGOMERY: Yes, I\'d be happy to talk about this in detail in the
breakout session, but could you just say a little bit about how you see
the Markal strengths from the energy demand side versus on the energy
supply side, because I have the impression that it\'s very much better
on one end than the other, and I just wondered what your thoughts are on
that.

MS. HUTZLER: Is your view that they are better on the demand side than
the supply side?

DR. MONTGOMERY: Yes.

MS. HUTZLER: That\'s what we are seeing, too, but that\'s why we have
the program in the future to work on the supply side some more. At this
point in time, though, for the first year we are going to accept what
they have, because there is really not enough time.

I had in the first year wanted to do more on their supply side, but
we\'re just not going to be able to have time to do it. We\'re just too
bogged down.

CHAIRMAN CRAWFORD: Before we adjourn for breakout sessions, I need to
ask those people that have not signed in at the register to do that
before leaving for the breakout session.

(Whereupon, the foregoing matter went off the record at 9:38 a.m. for
breakout sessions.)

CHAIRMAN CRAWFORD: I would like to go ahead and reconvene and start with
the summary of the three breakout sessions. Doug Hale was in charge of
summarizing the long term international modeling breakout session. So we
will start with him.

MR. HALE: Okay. The first thing, my condolences to those of you who
missed our session. I thought it was terrific. Our discussants \-- we
were very, very fortunate to have four extraordinarily knowledgeable
discussants to help us think about this whole issue of long term
modeling and how to go about it.

I don\'t know if you all received them, because the papers were
delivered in one case just this morning, but there are two other papers
other than my own which I highly recommend to you, the first by Andy
Kydes where he deals with the goals of the long term international
modeling, how he is using this effort to improve the quality of the
international energy outlook, and his view of what the newer challenges
are.

In addition, there is a paper that, quite honestly, I would recommend as
a reference paper. So hold onto it, if you can get your hands on it.
That\'s Lou DeMouy\'s look at the data challenges in international
modeling.

I think he does a fine job of very systematically going through what the
nature of the international data is these days and talks about where
it\'s strong, where it\'s not so strong, and a little bit about how we
an build the data over time. Quite honestly, I wish we all had had a
paper like that six months ago. It\'s quite a good guide.

The discussion revolved around three major issues. The first is how do
we go about representing economies such as China, the former Soviet
Union, and India, which to various degrees are hard to describe as
Western style economies.

Does it make sense to talk about cost minimization in those areas? If
so, what is it that is really being minimized, and what sort of prices
should we use?

The next sort of issue had to do with the issue of data, for precisely
the same countries. We know the data is uneven. Is it so uneven to put
constraints or to heavily influence what we are doing, how we are
approaching the modeling?

The third area was how, if modeling isn\'t, you know, something we can
do in every case, how should we approach it?

I think with all the economists out there, we got exactly the answers
you would expect. One solid answer is cost minimization is just fine.
Another answer was it\'s not good at all.

In the case of data, one answer was there\'s very good data. In another
case, the answer, I guess, is \"says who?\" You know, for what?

I think one of the things that came out very strongly is a big
endorsement of the idea of flexibility and that we can\'t model every
area just like every other area, that in some cases where the data is
particularly weak, where our understandings of the economy are not the
same as they are for, you know, OECD, we are going to have to do things
differently than we would for OECD.

In particular, I think David made a very strong case for trying in those
cases to be simple rather than complex, and to think in terms of not so
much in \-- much more in terms of changes from baseline and trying to
model the behavioral changes as opposed to putting all our emphasis into
getting the baseline.

I think that we had very good discussions of the problem of regions,
which regions to take on, in what order, what criteria for regions, and
also in the case of the data, I think very illuminating discussion as to
how we might think about that data.

I found it to be a very worthwhile session. I\'m very, very happy to
have been able to participate in it. So that\'s where we are.

CHAIRMAN CRAWFORD: Thank you. Jim Knaub will summarize the second
breakout session.

MR. KNAUB: All right. In our session we were looking at some \-- With
the electric power data we have some highly skewed establishment
surveys, and we generally collect some annual censuses and try to
collect some monthly samples, too.

The problem with collecting monthly samples is that, with the highly
skewed establishment surveys, you\'ve got a lot of very small
observations. We could spend a lot of time processing a lot of data for
small observations, and I\'ve found it useful to do cut-off model based
sampling, getting results that were fairly compatible, I think, with
what we\'ve gotten before with some design based sampling, and I figured
it would help with the fact that with some of the smallest observations
the non-sampling error would be possibly quite high.

Of course, we were reminded once again how many people don\'t like
cut-off sampling, but it also reminded me of the fact that for testing,
besides the fact that we in testing look at two annual censuses and
pretend like one of them is a monthly sample and then see how the
results then compare, that there are some other things I thought about
doing before and have done to some extent in some surveys.

One thing that occurred to me was like maybe trying to do a special case
of collecting some extra data to be able to get a design-based sample in
some of the areas maybe, and then do studies like that along the way,
comparing what you get there with what you got with the model-based
sample, including comparing the mean square errors.

Let\'s see. States-based modeling was mentioned as a way of improving
the variance estimates. Let\'s see. And of course, there\'s always the
concern with misspecification of the model, regardless. But with this
highly skewed establishment surveys, I think that that\'s sort of
minimized.

The most testing we\'ve done with this, I reiterate, is that we have
looked at \-- pretend like an annual census is a monthly sample, draw
the gas from there, use the previous annual census for the regressive
data, see what kind of results we get and compare it to the results that
were actually observed.

But there are concerns about that, not being complete enough testing,
and we could look into that some more. Let\'s see. Maybe to look into
some other kinds of testing for the mean square error.

I may have left out some people\'s concerns. So if anybody from our
committee would like to bring something up \--

CHAIRMAN CRAWFORD: Jay, do you have anything?

MR. KNAUB: Thanks.

CHAIRMAN CRAWFORD: Thank you. Then finally, Robert Latta will summarize
the third breakout session.

MR. LATTA: Is it working? I assume so.

We were talking about the first-stage design for a residential energy
consumption survey and a commercial building energy consumption survey.
One of the questions that we asked of the committee was: Should we use
the same first-stage design for both the residential and the commercial?

They replied, not necessarily, that there may be enough differences that
it would be advisable to use a different first-stage design, and we are
definitely going to be looking into that and making our own assessment
and not going into it with the idea that there will be a first-stage \--
the same first-stage design.

One of the reasons for that is that people live in different places than
they work, and we would want to be able to take that into account and
take into account what data we can get of where people live and where
people work when we even do the first-stage design.

The second one was: Within the first-stage design, we would be breaking
up the country into primary sampling units and then grouping the primary
sampling units into strata. We were \-- had the object, we want to be
able to produce more state-specific or more geographic-level estimates
than we can under the current designs.

In the previous design we had made an effort to maintain a parity in the
way we treated Metropolitan Statistical Areas, and this would have to
change if we wanted to get more state-level estimates.

The question is: Do we really want to just totally ignore the definition
of Metropolitan Statistical Areas or do we want to kind of just
half-ignore them?

The kind of suggestion was that there is no fixed rule on how to do
this, that you\'re going to have to almost treat each large metropolitan
area separately, because they each have their own characteristics, and
that in some cases the commercial activity is very centralized, in other
cases it is more spread out.

The committee suggested that we go out to exterior data sources and try
to get data on how much this is spread out, and that when we do the
design, in some cases the central county of a metropolitan area will be
a separate primary sampling unit. In other cases, we will be putting the
inner suburbs in with the central city, and you will essentially not
have one rule that fits all.

We are going to have to be essentially creating maybe the \--
essentially looking at each large metropolitan area and make an
individual assessment how it best works for that area.

We didn\'t spend a lot of time discussing about other exterior data
sources that would be useful to our sample design and how to use them.

CHAIRMAN CRAWFORD: Is there anything to add from members of this
breakout session? Thank you.

Are there any comments or questions from the committee on any of the
breakout session summaries, anything to add?

DR. MONTGOMERY: It\'s interesting. After our session broke up, Doug Hale
and Barry and I started discussing another topic that I\'d just like to
suggest, and maybe we should look into more thoroughly at some point.

It\'s the issue in the Markal model, but the markal model \-- and it was
kind of at the very end of our session. It\'s just something I don\'t
think we had a chance to get to but, I think, might be worth more
attention than we paid to it.

The Markal model takes a set of technologies, figures what the optimal
choice of those technologies is, and uses that to predict energy
consumption.

We know that there is this huge debate that\'s always been going on
about is the economy as energy efficient as it should be. Well, the way
the Markal model solves that problem \-- This is what came out in our
very last discussion \-- is by saying we know it\'s not choosing exactly
what it should be choosing, and we adjust the discount rates that are
used in these optimal choices so that the model will choose in its
baseline \-- you know, in history \-- what is used in history. So we\'ve
got to get it to match up, say, for the year 2000 with what we actually
see happening in the year 2000.

Well, that may not be the best way. I mean, that certainly is an
effective way of doing something, and if you carry that forward all the
way in the future, it gives you a reasonable result.

I think we maybe should address whether that is a good way of having the
model work, because it leads to a set of potentially troublesome issues
in the future, if someone comes in and says, well, if we change that
discount rate to the social discount rate, we would see that there are
huge energy savings that are possible all around the world.

I think there is some things that you may be getting into with the
Markal model we haven\'t yet fully explored.

CHAIRMAN CRAWFORD: So I would just summarize that be saying something
like the Markal model issue of adjusting discount rates to \--

DR. MONTGOMERY: To basically calibrate the model with history.

CHAIRMAN CRAWFORD: Calibrate the model?

DR. MONTGOMERY: Yes.

CHAIRMAN CRAWFORD: Okay. Any other? Calvin?

DR. KENT: I was just going to pursue. Does the Markal model \-- Do you
need to use those same discount rates?

DR. MONTGOMERY: I believe that\'s what Barry said, but that is a \-- I
mean, I think that \--

DR. KENT: Then that really is worth inquiring, and that is a major \--

DR. MONTGOMERY: But if they don\'t continue to use them, then we\'re
going to have suddenly the world change, the world becoming more
efficient than we had originally \--

DR. KENT: That is a major thing.

CHAIRMAN CRAWFORD: Okay. Any other questions or comments from the
committee? Questions or comments from EIA people?

If there is anyone here from the public, any questions or comments from
them?

If not, then I will adjourn the meeting for lunch. I do want to take a
head count for dinner form the committee. So if you are on the committee
and you are planning to come to dinner, could you raise your hands. If
you are bringing a guest, could you raise both hands?

DR. FREEDMAN: Carol, do you need the EIA number as well?

CHAIRMAN CRAWFORD: I think Bill has got the EIA numbers. It was the
committee\'s numbers.

Okay, thank you very much.

(Whereupon, the foregoing matter went off the record at 11:52 a.m.)

A-F-T-E-R-N-O-O-N S-E-S-S-I-O-N

(1:43 p.m.)

CHAIRMAN CRAWFORD: I\'d like to go ahead and get started. For those of
you that weren\'t here this morning, there is a place to sign in
outside. Please just sign in for the afternoon session.

We are going to start this afternoon session with Bill Underwood, and he
will be talking about new ways to process, store, and make EIA data
accessible.

MR. UNDERWOOD: Good afternoon. Can everybody hear me? Is everybody
awake? You know, I always hate to be the person that does the speaking
right after lunch, because I know everybody is going to nod off.

You know, this feels pretty good standing up here. You got to
understand, I\'m the son of a Southern Baptist minister. So when you put
me in the pulpit, you know, I start wanting to preach a little hell,
fire and brimstone here. And being a good Southern Baptist, we\'ll pass
the offering plate, but remember, I\'m a government employee. So no more
than a \$25 contribution per person.

I was just talking to Mark a little bit ago, because I haven\'t been at
EIA but \-- what? \-- three months now. Seems like about three years,
but it\'s been an interesting three months. I told him personally I
think I have the most challenging and difficult job in all of EIA.

This organization \-- IT is really its business. In order for us to
produce the information that we have to produce, IT is a reality. On the
other side of that coin, the reality is most of the systems that we have
relied upon for numerous years are very old, very archaic, and very
expensive to maintain, because we are basically Band-Aid-ing Band-Aids.

In a lot of cases, we are running systems and software and operating
systems that are no longer offered. We can\'t even get support on some
of the things. So we keep our fingers crossed, because if it breaks, we
have no way of fixing it.

Yet on the other hand, through the numerous last ten years in government
here, obviously, the government has been in a downsizing mode, human
resources and financial. So that has left a challenge to how do we take
old, archaic systems, leverage what little bit of resources we have,
financial and human, to come up with new systems that are more efficient
and cost productive.

That\'s where I kind of fit in as the Chief Information Officer. I also
get the nice luxury of being dual-hatted, because I am also the Director
of Information Technology.

That may sound like to a lot of you the same job. It\'s not. The Chief
Information Officer is a person who is a businessman, who understands
technology and how to leverage technology for the business.

The Director of Information Technology is a technology guru that is one
that tells you what all the great, wonderful, new things out there are.
That side of me likes to play with the toys. The other side of me has to
tell you how to use the toys.

As such, EIA is held together right now primarily because we have
tremendous work ethic and a lot of people that\'s been here a long time
that work very hard to make the deadlines and get the things out in our
reports and our publications.

What we have been doing is using human extra workload and efficiency to
make up for our lack of efficiency in our automation. We are in the
process right now of putting together a three-year IT strategic plan
that says how we will support EIA\'s strategic business plan.

Those two things have to go hand in hand. I don\'t know. Did you all get
the paper that I wrote that was a white paper? It was kind of generic.
It doesn\'t really apply directly to EIA as you would think, but it does
actually, because it applies to the industry as a whole.

That paper was written, well, I guess, about six months ago, and it was
presented at a conference where I was the master of ceremonies for all
of government CIOs. But everything in that paper holds true.

I am not going to go through that, but it\'s true. The old model is
outdated. The old model basically was information technology drove your
business. That\'s wrong. Business should drive your information
technology.

What are your business needs? How can I take technology and leverage it
to give you that business need quicker, faster, and cheaper? That\'s
what it\'s all about.

That\'s a hard thing for IT people to understand. They have been in the
position so long of being able to dictate to you what you want. I mean,
usually they are responsible. This isn\'t Burger King. You don\'t get it
your way. We thought this is what you wanted. This is what we gave you.
Now make it work. That no longer holds true.

We are no longer a support arm anymore. My job is not just to support
the business. It\'s to make the business and to make the business
happen. The United States is moving from an industrial nation very
rapidly into an information business.

As such, there\'s a critical shortage of IT professionals. All you have
to do is listen to the news, watch the newspapers. We are lifting and
raising the number of people that can come in under visas who have
technology backgrounds.

I can tell you, when you go to hire people, whether they\'re a
contractor or people that want to come to work as government
professionals, very few of them are American citizens anymore and very
few of them fit the white Anglo-Saxon look that you are used to seeing.

The Asian community and the Indian community are the two largest
communities that\'s providing this technical support, and that\'s a fact
of nature.

What also is happening is the price for this labor has now shot through
the roof. It\'s one reason why we in the government have gone almost to
outsourcing, because we can\'t afford to compete half the time for these
resources within the salary structures that we have.

So it creates a lot of opportunities for me, and as such, I\'ve taken
that goal on in FTA \-- FTA, boy, that\'s a Freudian slip \-- into EIA.

There is no panacea here. I mean, five years ago, how many of you
thought that the World Wide Web was going to be where it is today? I can
tell you, I didn\'t, and I was in the business.

That\'s why, as I mentioned earlier, we are only doing a three-year
strategic plan for IT. Even at the three-year mark, I\'m crystal balling
the last year pretty good. In the old days we used to do five-year.
Five-year is such a swag, it\'s not worth the paper to put it on.

The other difference is EIA will live by this document, because it will
be a living document. It\'s not one of these things that you do, it\'s
pretty, and everybody looks and blesses it, and it gets genuflected, and
you stick it on a shelf someplace and it collects dust.

That document will be updated every year. There will be goals and
objectives underlying this every year that says where we are going to
go. We are in the process right now of trying to standardize EIA on two
primary databases, Oracle and Sequel Server.

We are in the process of trying to put common systems design methodology
in place so that all our systems are designed the same, and we can reuse
code, which is cheaper. You don\'t have to keep rewriting the routines.

I\'ve initiated that we are having a common data dictionary. We have a
data administrator now in my organization who is setting up the remnants
of the first data dictionary that will be used for all our systems
development from now on into the future.

This is not being done just by my people in a vacuum. The user community
is very much involved, because as a technologist, I really don\'t own
any system unless it supports my organization exclusively.

I maintain the systems that are owned by EIA, and as such, as the
owners, they should be telling me what they want those systems to do,
and I should come back and tell them how I can make that happen in
technology.

I can tell you that what we are heading toward is that we are not going
to do any systems at this time and place that are not Web-enabled. We
are in the survey business.

We should allow our customers to be able to access all our surveys
through the Web, fill in the information, and it feeds right into our
system. Get out of the paper business, out of the mailing business, all
those other things.

That is some of the things that\'s going on here. We are also trying to
leverage some of the existing technology we have in-house. We have a
nice mainframe, an S-390, downstairs. We are starting to move more and
more of our stuff over there to make it a huge file server and let it
run either in the Unix partition side.

Unix type databases are on the MBS side, the Oracle type databases. One
reason we do that is when you go into what we call client server mode
where you put up servers, every time we build a system I end up putting
up about six servers per system.

Right now we have, oh, about 50-60 servers in-house. Just through
standard life cycle, I\'ve got to replace those about every two years.
At \$25,000 a pop, take that times 50, you can figure out what that
costs, and in declining budgets that is very difficult to do.

The other thing that we are focusing on that is kind of a nasty word to
a lot of people is security. But it doesn\'t take very much of listening
to the news or reading the newspaper for very long to realize that the
real weakness of the World Wide Web is security.

How many of you in here online bank? Do you use Quicken? Do you use the
software given to you by the bank? It\'s a base on Quicken. Did you know
that, when it transports across the Web back and forth, even on secure
terrain, when it goes across the Wide Web, it\'s in what we call
\"natural,\" and some really smart, sharp people can intercept that and
know everything about your banking business they want to know.

I didn\'t know that, and I\'m paid to know that. I went to a security
conference about three months ago and listened to a couple of very nice
PhDs on the subject talk about this. It scared the life out of me,
because we\'ve been doing online banking for several years now.

Same thing on your credit cards. We are quickly trying to do that. Here,
you know, we have confidential information that comes from all different
parts of industry. It\'s very important that we keep that secure, that
it only gets out to those who are supposed to have it and need to know
it.

It gets much more beyond \-- I mean, even individual persons like us \--
I don\'t want my Social Security number floating around the world for
anybody to use. It probably already is, but I\'d like to think I have a
little bit of personal security, that not everybody knows more about me
than I do.

These are the things that we are looking at and we are trying to change
in EIA. In EIA we have to make these changes, because the last figures I
saw, between now and 2004 60 percent of this agency can retire.

That is a major core dump of information and historical records that\'s
going to walk out the door. We have not done a lot of hiring except for
a few great people like me. As such, are we going to be able to get
these people hired and get them up to speed when the other person is
already gone?

There\'s some major challenges facing us. We are spending most of our
money now trying to maintain our existing infrastructure and systems. We
have no money, for the most part, that we can use to dedicate to putting
in new stuff.

So now the challenges to me and my fellow peers is how we leverage what
small amount of discretionary funds we have to keep the old stuff going
while we build something new to replace it.

Why do we go toward something like what we call a data warehouse, which
is what we are heading toward slowly but surely? So that we can leverage
the information we\'ve got. We can data mine it. Our analysts can use
it, massage it and get the information out in a more timely manner
without having to put 0 hours in after they get it here trying to
manually manipulate it.

I think Mary talked to you all earlier. They spend a great deal of time
in modeling their statistical models. That\'s nothing more than
programming, because what you are doing is trying to get the answers
that you want that are appropriate.

E-business is here to stay. In fact, I think we haven\'t even scratched
the surface on what e-business is going to be, the same as cellular
communication. It\'s one of the fastest growing markets.

I figure at the rate we are going, within five to ten years the old type
of telephone that sets on your desk or hangs on the wall in your house
is going to be obsolete. You already see satellite TV, digital cable,
and you are starting to see now that e-business is going to slide into
that market where you can sit at your TV, buy things right there by
punching it in on a little keypad. They have your credit card on order
or you bank account, whatever you want, and it shows up at your house
through probably UPS.

The next ten years in our lives is probably going to be the greatest
change that we\'ve faced, and for all of us who are Baby Boomers like
me, we thought the last 10-15 years was rapid. I think the next ten or
15 are going to make those look slow.

I\'m going to try to make sure that EIA is as modern as possible within
the next three years, because in all honesty, I figure three years is
about all I\'ve got. If I don\'t have it done by then, it\'s moved past
me, and I\'m playing catch-up.

One of the things we are going to do within EIA is we are going to make
the IT organization take its rightful place out in front, leading EIA in
what\'s out there and what way to go, rather than sitting back and
trying to maintain old.

One of the difficult problems we have is what they call integrating the
front office with the back office. In e-commerce, that is one of the
major key issues people have.

You put a Website up. You offer a product to the market. You get 10
million hits within 72 hours that want to buy this product. So you\'ve
got a lot of widgets you need to get rid of.

Does that information tie into your accounting systems, your inventory
systems, your shipping systems? Do you even know how much you even got
on hand to even fill the orders?

That integration piece is the piece that is missing in most of the
business world today, and it\'s the same thing for us. We have the same
problems. How we look at that is how we take the survey stuff and get it
to where we go to publication and what\'s in between. That\'s what we
are working on right now, trying to resolve those issues.

You guys all look so serious. Smile. This is the silver lining. Anybody
have any questions while I\'m down here? Lunch still with you or \-- Do
I see a hand raised?

CHAIRMAN CRAWFORD: Yes Calvin?

DR. KENT: Yes.

MR. UNDERWOOD: Oh, okay. I don\'t know the rules.

CHAIRMAN CRAWFORD: Are you finished?

DR. KENT: Or do you just want questions?

MR. UNDERWOOD: I\'m pretty much finished right now. I\'m ready \-- I
figure the questions will make me answer what you guys really want to
hear.

DR. KENT: Well, I have two that have arisen so far out of your remarks.
The first one is: You talked about the confidentiality of the data, and
this, of course, is sacred to EIA.

If what you said about the data not being confidential in its neutral
space, how are you going to solve that problem? Now that the dirty
secret is out, how are you going to assure all these people who are
doing electronic filing that their information is not going to be
pirated, and particularly relate this to the problems of data collection
in a deregulated industry where confidentiality of data is right up
there with trade secrets in terms of competitive advantage?

MR. UNDERWOOD: There are several fairly new technologies on the market
now, and the one that the Federal government is embracing the most is
what they call PKI, Public Key Infrastructure.

What that amounts to is it encodes your data, ships it in codes. When it
gets to here, it gets re-translated and ungarbled. You have to have a
special little diskette with a memory chip on it. It has all the key
information, and also does the encoding, and we have a special server
here that receives the data.

That is becoming very much standard. That\'s one of the ways, and
that\'s pretty much \-- It\'s been accepted as a government-wide
standard. That will probably be the first thing.

So that means that our user community who decide to go to the electronic
submission will end up having one of these sent per user. Now what that
doesn\'t do, obviously, it doesn\'t keep you from handing it to the guy
next to you and giving him your password and letting him use it, because
we have no way of authenticating except that we know it\'s your disk.

DR. KENT: That is the direction that EIA is heading on all their \--

MR. UNDERWOOD: That is where \-- We are proposing to go to a PKI
standard, yes, which is pretty much universal throughout the government,
a lot of the places, and even a lot of the banking institutes are going
that way now.

There\'s different versions of it that are much more in depth, and it
gets into technology garble. But basically, what it does, it\'s just
like when you make a secure phone call.

It scrambles the stuff, and then it unscrambles at the other end. Same
thing, and we won\'t accept it unless you have that special diskette,
because it transmits your personal data that says you have a right to
deal with us.

What was your other question, sir?

DR. KENT: Carol, did you want to follow up on this before I ask my other
one?

CHAIRMAN CRAWFORD: No. Mine is kind of different. So go ahead.

DR. KENT: Okay. My second question is: Three times you said about
putting EIA out front. What is going to be necessary to put EIA out
front?

MR. UNDERWOOD: Hard work, money, creative mind, and a super great CIO.

DR. KENT: Okay. And what will hard work, money and creative minds buy?
What are we going to see? Is it systems? Is it some hardware? I mean, be
specific with me.

MR. UNDERWOOD: Okay. What you are going to see, you\'re going to see (1)
a vision that says this is the direction we are heading. Instead of its
being a free-for-all, multiple choice, whoever has the idea is what we
are going to do.

It means that you are going to hear things that a lot of people don\'t
like to hear, words like standards. We are going to use a common design
methodology for all systems.

At this point it\'s what we call a three-tier architecture, Web,
intermediate server, database. We are going to support two basic
databases. That\'s how you are going to get there.

Then once you do that, because you have now created a common data
dictionary where everybody uses the same definitions to define something
in their systems, systems can share information. That\'s how you
leverage that. That\'s how you get out ahead.

Right now if I go into one of the programmatic offices, they have a hard
enough time sharing the information from one system to another system
within their own office. Don\'t even try to go up and share with another
office like CNEAF, because they have done their thing. Oil and Gas has
done their thing.

That\'s been, you know, one of the fallacies and weaknesses we\'ve
always had. Where we have made up for it is we\'ve had people who have
worked the extra time and manipulated and manually worked on the data so
that, when we did the common publications, it had all of it in there,
that we could compare the apples and oranges and get all oranges.

Does that answer your question?

DR. KENT: To a degree, but go-aheads.

CHAIRMAN CRAWFORD: My question was: In your opinion, what\'s the future
of the wireless technology, and how does the PKI system fit in with
that?

MR. UNDERWOOD: The weakness of the wireless right now is that basically
it broadcasts at one-half of a watt.

CHAIRMAN CRAWFORD: But assuming that will change very quickly.

MR. UNDERWOOD: I\'m not sure if it\'s going to change on what the
broadcast is. It\'s just I think you are going to see more cells, and
they have gone to digital more and more. You are seeing less and less
analogs. You get a lot less noise.

The technology is going to change. The last I looked at it, right now in
the wireless technology I think about 2400 baud is about the standard
modem baud rate for electronic transmission. That is going to change
with technology, with more research, just like it has in your other
modems where, you know, now a 56K b modem is very standard, you know.
Shoot, ten years ago, if I had 9.6, I thought that was fast.

So that is going to happen. It doesn\'t make any difference if it
broadcasts across a wave length or it broadcasts across a fiber optic
cable. If the transmission is encoded and it can only be unscrambled at
the other end by a special piece of software and hardware, it doesn\'t
make any difference. It\'s just like the U.S. military has been sending
encrypted messages across wireless for years, and the same type \-- it
will apply when it goes to the public sector, and it already has, to a
certain extent.

You will see within the next three years major changes. What you are
going to see \-- Five years ago, the biggest boom in technology was the
proliferation and explosion and the usage of wide area networks and
hooking your local area networks to a wide area network and giving
everybody basically around the globe, if you wanted, total connectivity
and file sharing.

Now what you are seeing the biggest boom is going to be in your wireless
and in your Web technologies. We\'ve pretty much software-wise have
saturated \-- You know, fourth generation languages have been around a
long time. Yes, they have matured them a little bit. They have produced
better report writing software for them and things like that, but that
hasn\'t changed that much.

The changes now are coming elsewhere in technology. The PCs, the same
thing. I mean, big deal, you got a Pentium now. It\'s a 1,000 megahertz.
Show me where it runs your word processor any faster than the 500-600
megahertz one. It doesn\'t.

It is not tied to the speed of the chip. It\'s tied to the inefficiency
of the software that was written for it. Historically, hardware is
always ahead of software.

CHAIRMAN CRAWFORD: Thank you.

MR. UNDERWOOD: Anymore questions? Ah, I see you, sir. I\'m sorry. I got
to get used to this.

DR. BLAIR: I\'d like to come back for a moment to the issue of security,
but from a somewhat different angle.

You mentioned PKI as a shared standard within government and between
agencies and so forth. In the breakout session this morning one of the
things that we touched on was the future of data collection by
electronic means, some of that data collection being from businesses
where there would be concern about the security of the information that
is being transmitted.

In a world like that where there perhaps is not a shared standard, what
thought has been given to sort of how data might be collected but kept
protected in that process of bringing it over from a business survey
respondent to the data collector or is that something that is on the
drawing board?

MR. UNDERWOOD: I\'m not really sure if I followed exactly what you were
asking there. Are you saying if we take an electronic submission, how
are we guaranteeing the confidentiality and security of the information?

DR. BLAIR: Right. During the process.

MR. UNDERWOOD: Between there and getting to EIA?

DR. BLAIR: Correct. Yes.

MR. UNDERWOOD: Basically, at this point there is not a lot of security
on that. When you come into Web technology, there is very limited. It\'s
hooked to pretty much password control and that sort of thing.

When we go to the PKI technology, before you as the person who is
submitting the data ever broadcasts it, before it leaves your PC, it is
scrambled. It goes across the media scrambled until it gets to EIA.

EIA, the hardware and software will unscramble it, and it will become
the meaningful data that you had originally.

ACTING ADMINISTRATOR MAZUR: Right. Distribute the PKI technology to \--

MR. UNDERWOOD: Oh, yes. Yes. I spent two years on a special project with
GSA on this, and we gave out the special keys to the user community to
use.

It looks just like a diskette, a lot of these. You stick it in the
diskette reader on your drive, and type in, and it logs you in, and once
that does \-- because, first of all, EIA at the other end wouldn\'t
receive the data until the header comes first that says this is who I
am, here\'s my special credentials and my password and all those other
things.

Then we know what follows thereafter is your information.

DR. BLAIR: No, I understand, if the survey respondent accepts and uses
the same encryption technology. My question is: Is that a likely
prospect, that you would get a population of businesses that you are
trying to collect data from to agree on the same standard?

It probably can\'t be answered, but I think that\'s an issue, if you are
going to collect data, that businesses are concerned about keeping
proprietary. They may or may not agree with the use of a particular
standard.

MR. UNDERWOOD: Well, that may be true, you know. Everybody is going to
probably have their own personal opinion. But I would think that if the
Federal government is basically adopting that as the standard, and it is
in the CIO Council \-- all the meetings that I have attended, that\'s
the direction they are heading.

There\'s multiple versions of that, but that is something that, you
know, if an outside entity is not comfortable with that, that obviously,
EIA would have to work out with them in personal meetings and that sort
of thing.

You know, we will do everything that we can in our power to assure that
it is as secure as we can make it. Now, you know, I\'ll be honest with
you. There is no such thing as 100 percent guaranty. There is not now.

I mean, when you send to U.S. Postal to mail a survey back, there is
nothing that says that that doesn\'t get intercepted or lost or chewed
up in a sorter or whatever. I get mail every now and then that doesn\'t
have my address on it.

So you know, it just depends. And I understand that, you know,
industrial espionage is very real.

Anything else, sir? Yes, sir?

DR. HENGARTNER: I have two questions. One has to do with the
neutralization. As an end user, if I am going to access data, I also
want to relate it to other governmental agencies. In particular \-- I
know that is probably discussed to death, but I would like it to go to
Census, to the EPA, to here at EIA or whatever. I would like to link
them together, because data that I am not able to link will not be
useful to me.

Now I know there is an effort doing it here, but wouldn\'t it make sense
to actually have an overarching effort within the different agencies to
bring together all these different data such that they are compatible?

MR. UNDERWOOD: Yes. Is it happening? No.

DR. HENGARTNER: Should it happen?

MR. UNDERWOOD: Yes.

DR. HENGARTNER: Will it happen?

MR. UNDERWOOD: Eventually? Yes. When? I haven\'t the faintest.

DR. HENGARTNER: Okay. So that\'s not within the next three years \--

MR. UNDERWOOD: What you\'ve got to understand \-- This is the fourth
department in the U.S. government that I\'ve been in: Department of the
Army, Department of Agriculture, Department of Transportation, now the
Department of Energy.

We don\'t even do a good job on sharing information that pertains to
ourselves. Prime example: I\'ve been here 90 days. I still haven\'t had
my financial or my leave balance transferred from my old agency. I\'m
still waiting.

You want me to tell you how we can use all the data we collect, the
single largest collector of information in the world, and we are all
going to make it so you all can use it and share it? What a pipe dream.
A great one, and it should happen, but it doesn\'t happen, and it\'s not
going to happen, in my mind, until the Federal government realizes it is
one entity and not a bunch of separate kingdoms.

We can\'t even get EIA to agree on definitions half the time on our data
within-house. How do you think I\'m going to get the rest of the world
to agree to that?

One of the things that they have talked about, because there\'s a
necessity in this, they feel, and the verdict is still out, is whether
or not there needs to be a U.S. Federal government CIO that dictates
standards and policies for IT for all the government.

I mean, one of the things that is unique here is we use two different
word processors. We have Microsoft Office, and we have WordPerfect. Why
doesn\'t the Federal government make up its mind where it is going to go
and say we are all going to be Microsoft Office or all be WordPerfect or
we\'re going to use Lotus Smart Suite? I don\'t know. Who cares? Just
standardize on something.

E-mail \-- Every time I go to a different department, I\'ll guaranty
you, because they will have, you know, ten, 12 different organizations
like us in a department with 100,000 people, you will find six to eight
different e-mail systems.

Then we buy what we call an x.500 Gateway that does all this fancy
interpolation so that you can exchange messages. We spend a bloody
fortune keeping those things up. Germantown has got one. Every
department I\'ve been in has it. Why don\'t we go there?

I sit on CIO councils, and we all talk about it and say it\'s a great
and wonderful thing, but I don\'t see any movement to do that yet. Until
we do that, what makes sense that you are basically wanting there isn\'t
going to happen.

Now all I can tell you is I\'m going to try to make that happen so that
at least the data we bring into EIA that we can share it equally amongst
ourselves by setting standards that says, you know, if we are going to
call this crude oil that, when you want crude oil, this is what it means
over here. But as far as the government, to be honest with you, I think
I\'ll be long gone before that day comes.

I\'m sorry to say that I consider that a pathetic way of looking at it,
but it\'s basically reality.

Do I see anymore cards? Have I run out of time?

CHAIRMAN CRAWFORD: Do you have \-- I thought \--

MR. UNDERWOOD: You had two questions. I\'m sorry.

DR. HENGARTNER: Well, I had another one which goes a little bit in the
other direction, which says: If we data warehouse and put all the data
in one pot, don\'t they create security risks by a red flag. Come on,
here\'s the data, come and break in and get it?

MR. UNDERWOOD; No.

DR. HENGARTNER: Microsoft got broken in.

MR. UNDERWOOD: First of all, a data warehouse is not a single database.
A data warehouse is basically a composite of databases.

You may have 15 systems that support all of EIA. Each one of those is a
separate database, and they all feed a common database on information
that needs to be shared together, not all the data.

I mean, first of all, if I tried to build this huge database, Larry is
going to be yelling at me, because I\'m saying I need millions of
dollars more to buy X amount more mainframes, because we are now going
to start rivaling out here at Ft. Meade for the underground facility,
because I need that much computing power. But no.

So you suck out what you need out of these pertinent databases into
this. Usually, what that is used for is for what we call executive
information systems where you need to know something about this,
something about this, something about this. It all comes together to
give you a rolled-up answer, not everything.

Yes, you do suffer that problem. By the very nature of how we go about
doing our business \-- I mean, when I started in this business 20 years
ago, it was an old mainframe. Those were so easy to secure and lock up,
you know. If we didn\'t want you in, you didn\'t get in.

Then came \-- You know, in \'84 the PC came about, and we put it on
everybody\'s desktop. No longer a dumb terminal. We started giving you
the capability to do more things. Intelligence at both ends of the
stream, we call it.

Then we started saying, gee, wouldn\'t it be nice if we network all
these together and be able to share files and information, and then
e-mail came about. Personally, I think e-mail is the bane of our
existence, but that\'s another subject.

Then we said, now we got all these local area networks; wouldn\'t it be
great if we could tie all these local area networks together across the
nation, you know, so that the office in San Francisco can share with New
York.

Well, the very nature of building these types of infrastructures that
share information and allowing more and more people access creates a
security nightmare for a security officer. They go crazy, because it is
extremely difficult, especially when you get in the user community and
say, all right, every 90 days you got to change your password; it must
be nine characters long, alphanumeric, and it can make no sense.

You walk in the offices, and you see all these people with little yellow
stickies stuck up on the monitor, because who can remember these damn
passwords. I mean, every system in the universe has got one.

If you want to secure a system more, as you go down into the system
farther and farther, based on your access rights, the more passwords you
got to have. I\'ve been in some systems where to get to where you wanted
to be, you had to log in six different places, and you had six different
passwords.

So that\'s \-- but that\'s where we are going, because we want to share
the information. By the very nature of sharing it, you are opening
yourself to chance for intrusion.

I\'ll tell you the truth, firewalls like you have now, like we have here
and most places, those are only meant really, to be honest, to keep the
honest people honest. If you are really a devious hacker that knows your
business, you can break almost any one of those you want anytime.

First of all, if you surf the Web much, you can find out all the
information about any firewall in the universe on the Web, and there is
also code out there. You can download it, and it tells you here, use
this and you can break into these things.

The hacker community is smarter than we are. They come up with how to
get in faster than we can come up with ways to keep them out. It\'s a
risk, and what a security officer really does, where he earns his money,
is they do a risk analysis on how much is it going to cost to tie this
down versus what is the risk you\'re willing to accept. That\'s where it
comes down to.

We\'ll do the same thing here and anyplace else. It comes down to the
bottom line: How much do you want to give people access to, and how much
money are you willing to spend to try to keep people out?

Last question, right? Is that it? Thank you.

CHAIRMAN CRAWFORD: Thank you, too. So if our next speakers are ready,
our next presentation is estimating procedures to reduce data reporting
lags, Inderjit Kundra and Roy Kass.

MR. KASS: We are going to tag team. I am Roy Kass. That\'s Inderjit
Kundra, and we are going to talk about some experiments we have been
doing to solve a problem that we have in the Natural Gas Division of
getting our publication, [The Natural Gas Monthly]{.underline}, out in
the same month that the cover says it\'s been published.

We basically have a big problem with responses to one of the major
surveys, the EIA 857, that drives estimates for consumption by the
residential and commercial and the industrial sectors.

We have done a number of approaches to try and solve the problem of late
response and nonresponse. Last year, we came with an approach to
estimate for unit nonresponse, and then we got a blinding flash of the
obvious, that why don\'t we try and estimate the aggregate basis, which
is what we need to get the publication out in time.

So we are going to give a brief description of some alternative
approaches that we have developed. Currently, in order to increase the
timeliness of the data, we estimate using the gas module of the STIFS
model for the three months between the last survey month and the
publication month.

The lag on the surveys is either two or three months, depending on the
month. What we are talking about now is consumption. There is a
three-month lag.

In order to run the STIFS module, we have to have the most recent survey
month, and that\'s where the problem comes. If we don\'t have those data
until the third week of the month, we can\'t run STIFS. Because of
operational difficulties, it takes another month to get the publication
out. That means we are in the 28th to 30th to 31st of the month before
we are in publication. Sometimes it is first, second, or third, fourth
or fifth of the month.

Our customers really like to have a warm, fuzzy feeling that when they
are reading an EIA publication that says October, it was published in
October.

So what we have here is an alternative to estimating on a micro level
\-- that is, we are estimating in a macro level. We\'ve got some
approaches, and Inderjit is going to talk about the technical aspects of
them, but our basic question is two, and now we are ready for the next
one, I think.

We developed three estimation procedures, and we are going to be
comparing the results. But as I say, our basic questions really are two.
First, are we doing a job that\'s at least as good as we do now? Can
these estimates come at least within the error level that the STIFS
estimate is doing?

Then another issue that arose after Inderjit and I started this has to
do with the fact that steel is now published on the Web monthly. If we
incorporate a new procedure, EIA is going to be hitting the street with
two alternative numbers representing the same thing at approximately the
same time.

This initiative at EIA is going to have to deal with it, and I\'m going
to give you a heads up, and I\'d like to know your gut response as a
consumer of statistics what implication that might have.

Before Inderjit starts, let me just talk briefly about the data that we
use. The survey, the EIA 57, started in 1984. We have final data, final
monthly estimates, from 1984 through 1998.

Now our final estimates are benchmarked to our annual numbers. Our
monthly estimates are based on that survey. It\'s about 400 companies
nationwide. You all are statisticians. You know about sampling error,
other sources of error, and so forth.

On our annual survey we monitor the entire universe of companies that
deliver gas. We accept the results of that as truth, and approximately
November of the year following we benchmark. So you add up 12 months.
You get an annual number. It\'s these final numbers that we use to fit
the equations.

We are going to be talking about comparisons of estimates with 1999
numbers. The 1999 numbers are those that were accepted as reality for
the April 2000 publication.

If you follow our natural gas publications, you know we are not
completely idiosyncratic in our revision policy. What we do is take a
look at consumption by sector by state by month, and when, due to
revisions on the part of the respondents or filling in of late responses
or corrections or whatever, that cell differs when calculated from the
current numbers in our database with what we have published \-- if that
difference is more than ten percent, we revise.

Now on a state basis, that ten percent \-- usually, it doesn\'t
percolate to anything on a national basis. Sometimes if it\'s a very
large state or a very large revision, it might percolate to one-half or
one percent.

That having been said, I\'m going to turn the discussion on Inderjit,
who is going to talk about the models and the results.

MR. KUNDRA: I think Roy has explained the data that we have. What I did
when I looked at the data, it looked like the timely data. So in order
to make the maximum use of the data which was available to us, I tried
to pick some sort of a model which actually worked.

We actually developed three models. There is one of the transfer
function model. I just want to explain the models. The next one is the
dummy regression model, and then we run this regression model through
the PROC ARIMA, and we evaluated the residuals.

We found that for the commercial and the residential model, we should
add some other \-- So with the result that the residential model did not
\-- residential model, sorry \-- for the regression and \-- for the
commercial and the industrial model we should add some of the other
terms.

The residential model did not need any refinement, but in the case of
commercial we needed to add one autoregressive term, and in the case of
industrial we added the two terms, b~1~ and b~12~. That is the seasonal
plus.

These are the three models which are explained. But before fitting these
transfer function models, we had \-- I polled the data to find out what
model was suitable, and we came up with the 011 model, which actually
fitted to all of the sectors. But I didn\'t use the heating degrees data
at that time.

The results of that model when we computed forecasted for the years like
\'96-97 and compared with the so called actual results published into
the [Natural Gas Monthly]{.underline}, the results were well within the
accepted error. That\'s within ten percent.

Later on the question was raised, why we did not use the heating degree
days. We did find that the heating degree days were highly correlated
with the commercial as well as the residential consumption, but the
correlation was not very high in the case of industrial model. But
still, when I used the heating degree days with the commercial model
also, the results did improve.

So what we did, as a result, we developed the transfer function model,
because the model was not getting all of using the heating degree days
unless we used the transfer function model.

That was the Model 1. That was transfer function model which was used.
Then we used the transfer function \-- Before we actually even came to
this model, we actually compared the data, the results from those two
models like the time series and the \-- The time series model, Alex
talked to you about that.

Anyway, the transfer function model and then when we looked at transfer
function model, we found that the results which we obtained from this
model were good as what we had industrial model is concerned.

We actually used the data. The data was available from us from 1984 to
1989. When we had the full data, 1984 to 1989 \-- 1999 \-- we got
results which were very good for the industrial model, but for the
commercial consumption the results were really off the wall. We could
not accept that model as the transfer function model.

Then because what we found there from there, that if we cannot use this
data, then we should use some sort of a dummy variable second model,
Model Two.

Dummy regression model \-- then we used this model. We did find that it
was \-- as far as the commercial is concerned, as well as the
residential is concerned, because the \'94 data we included, the
transfer function model did very good so far as the industrial is
concerned, but it did very bad so far as the commercial is concerned.
The residential, it was so-so. It was giving good results at some
months. Some months, it was not good.

Then we were looking at that. But then I ran the regression model from
1984. The results for the industrial model were not good \-- The results
from the industrial consumption were not good.

So then I said, well, if that results are not good, let us come back to
reduce the data. We shrink the data. Instead of using from 1984, if we
change the year from 1984 to 1990 or \'95, what will happen?

We ran the model for 1990 and \'94. The tables which are provided to you
shows the year we ran, like 1991, \'93, \'94, \'95. For \'95, the
results which were obtained from the regression model for the industrial
consumption were not bad. They were actually better when we ran the two
models at the same level, \'95, the regression as well as this one. But
for the commercial it gave us very good results at the 1990 and \'95.
Residential also, the results were good.

So there was a bit of comparison. Then I said why don\'t we shrink the
data for all the three models and see what happens. So we compared the
two models, the regression model and the transfer function model.

We came to the conclusion that we can shrink the database, but when we
drove this model to develop the third model, we only ran results for the
third model from \-- We did not use the third model for the 1994 \--
1984, because, in our view, this model would probably produce the same
results which are produced by the time series model, the first model,
the transfer function model.

So we only concentrated up to \'95, \'90, \'91, \'92, \'93, \'94 and
\'95 for the three models. We compared the results. The results formed
in such a way that we came to the conclusion that model 1 and model 2
are \-- If you look at the attached table, first you should find a
mistake. It is one, two, three. We don\'t have the models for these
slides. We don\'t have for the tables.

I just want to mention one thing. The conclusion was that Model 1, if we
have the full dataset, probably will do much better, so far as the
industrial consumption is concerned. We can get the results within the
accepted level of error.

Then if we run the model like Model 2 and Model 3, Model 2 and Model 3,
they probably will do equally good if we use the data from 1990-\'95 and
we use them for the residential model, but still Model 2 is probably
preferred one. But we conclude with that, if we want a simple and
understandable model, which is the regression model, we have to shrink
the database, preferably either to \'90 or \'95. I would rather see it
at 1990. It will be better for getting the residential and the
commercial consumption, but for the industrial consumption we cannot use
the Model 2 unless we put the data back to 1995.

If we use two different models like the time series model and the
regression model \-- If we use the time series model for the regression
\-- I mean for the industrial consumption, it probably will outperform
every other model for all the years. But if we have to come to the
regression model, then we have to get the database from 1990 to the end
of year 1998, September.

The other task was that we developed this model. We can conclude on what
model we can use. Now the question was that how does it compare to the
steel.

Before we compared with the steel model, we have to keep one thing in
mind, that although the steel estimates with the forecasts obtained from
all the steel estimates are not altered every time [The Natural Gas
Monthly]{.underline} is published, they are carefully examined current
monthly data when estimates are available.

That means when all the data comes, we just look into it. We may or may
not revise it. The other is less than \-- within ten percent, then we
will not revise the estimates, but if it is more than that, then they
will be revised.

Still, I feel that the subsequent first published estimates are directly
or indirectly influenced by the current data, whereas the forecasts made
by the proposed models are one to 15 months ahead.

The model forecasts are shown in Table 4 are obtained by using the
starting of 1995. Now we have to put those graphs for the residential.

Now we added, as is shown here, 1995 time series. That is the transfer
function model, regression, times, ARIMA and the steel published. As you
can see, if you look at the residential model, that the regression model
is probably doing quite good.

Even the steel model is not behind. I mean, there doesn\'t seem to be
much that we can gain from using one of the other, in the case of this
residential.

Go to the next one, commercial. In the commercial model, we see that, as
explained, the time series model is useless. It doesn\'t work at all. I
will not use that, but still I think I have to use it. It doesn\'t make
any \-- I mean, good result.

The regression is still doing better, and even regression is doing
better than the ARIMA model \-- I mean, the regression model run on
ARIMA, which has got the autoregressive terms. The regression is
probably peripheral with regression and steel.

If you look at both the models, you probably see regression is that
line, and steel is the black line. They are probably competing with each
other, and they are still within the \-- leaving aside one or two
months, they are right within the bounds of the ten percent.

MR. WEINIG: Roy, would you use the laser for some of these things where
color is important here?

MR. KASS: Where is the laser?

MR. WEINIG: Thank you.

MR. KUNDRA: This is \-- The black is the steel. That line is the
regression. That is the time series. That\'s number one, the top one,
and the number two is the \-- ARIMA is this one, the blue line,
something like that.

CHAIRMAN CRAWFORD: The original data, is that what you mean by \--

MR. KASS: No. Steel is the estimate that we publish. It\'s not the data.

MR. KUNDRA: No. It\'s for the year \'98 \--

CHAIRMAN CRAWFORD: So these are just your forecasts?

MR. KUNDRA: Forecast. We are comparing forecast against forecast. The
1990 \-- These are the months from \'98 \-- 1998, October, because when
you publish the December [Monthly]{.underline}, [Natural Gas
Monthly]{.underline}, the results, it contains up to September, seems to
be data on actual, what they call, and the three months, October,
November and December, they are forecasted, whether they use steel or
they use something else.

What we feel here, that comparing all these models, they are zigzagging.
One month steel is good. The other month is good, something like that.
So there doesn\'t seem to be any preference. But if you look, the
closest result which you find, and the next one, the industrial
consumption, that\'s really starting \-- I mean, when you look at that,
those results are really close.

If you look at on ten percent, they are all under ten percent, almost.
If I had used 1990 \-- I mean, if I used 1984, I would have said that
the time series model probably would have \-- so far as the steel, would
have done much better job compared to this one, because we were getting
much better results there.

The time series industrial model is \-- industrial consumption probably
is already well forecasted, which is within bounds.

Then there are the questions for the committee. What do you think of the
proposal which model we should use?

The second question is do you have suggestions for the improvement? I\'m
sure you do.

What do you think of the prospect that EIA represent two different
series of short term estimates for volumes of natural gas delivered to
consumers?

The third one, should we shrink data for all the sectors, and to which
year?

Should we have two different models, one forecasting industrial
consumption, Model 1, and the other to forecast residential and
commercial forecasts? Commercial consumption \-- that is Model 2.

Thank you.

CHAIRMAN CRAWFORD: Thank you. Our first discussant is Tom Cowing.

DR. COWING: Okay. So Roy and Inder went over a number of parts of this
paper. So I won\'t go over that material. I thought it might be useful,
since I didn\'t know what table 3 looked like from the [Natural Gas
Monthly]{.underline}, to give you this look.

Essentially, what we are doing \-- The problem here is, as I understand
it, these three numbers down here. This is the September issue, I think
the most recent. I took it off the Website, which actually demonstrates
something, namely, I also can use the EIA Website. So it must be pretty
simple.

So the numbers, I think, to look at to demonstrate the problem are these
three numbers, monthly numbers, at the bottom right here, which have to
be estimated the three months prior to this. So my understanding is
that\'s the problem, basically.

Now before I start, actually, I should give you \-- I should lay my
cards on the table. I am not a time series person. I know nothing about
time series analysis. It\'s actually worse than that. I don\'t believe
in it.

What I am \-- but I could be wrong. I\'m an applied econometrician, and
what I like to see are structural models with economic content. Okay?
That\'s okay, because actually, if you look at the past recent two
winners of the Nobel prize in economics, they are both applied
econometricians. So I assume that\'s okay to be that kind of person.

DR. KENT: Are you predicting something here, Tom? Are you predicting
something in your own future here?

DR. COWING: No, no. Well, we use the time series model, since I wrote
under one of them. Maybe there\'s something there.

What I want to really do \-- I\'m not sure we get all these on, but we
get enough of them. I want to look at some of these tables, which I
think will be a useful complement to the previous discussion.

The tables are in the paper. In order to understand them, first of all,
the start date for the estimation period is given right here. So \'95,
\'94, and \'93. So you move down the table here. We have increasingly
longer estimation periods. Okay?

The ending date is, what, September \'98, I guess. Right? So the first
period forward \-- that is, after the estimation period for the forecast
\-- is right here, October \'98, and the monthly forecasts go all the
way up. I think, actually, over there somewhere it\'s December of \'99.

Then arrayed down in rows here are the three models. I\'m just going to
call them one, two, and three, time series, regression, and regression
with autocorrelation \-- I mean serial correlation correction, the ARIMA
correction.

Now there\'s some interesting points, I think. And by the way, this is
also residential. That\'s the other thing to point out. This is the
residential. There are separate tables for consumption industrial. We
will, hopefully, have a minute to look at those.

So what can we see here? Well, except for two what I would call
anomalies here and here, the two non-time series models, models 2 and 3,
clearly dominate, although it\'s a little had to tell at times. It would
have been useful, I think, to have had some averages of some kind,
three-period averages, six-period averages, something; because there is
a fair amount of bouncing around here. But it kind of looks like that
these two models in general \-- you kind of eyeball down your way down
the line here \-- tend to perform a little better.

The second thing that\'s interesting is that \-- let me be sure I get
this right here \-- that for shorter estimation, the shorter estimation
periods produce a better forecast than the longer estimation periods. So
these percentage forecast error, which \-- I should have pointed this
out. This is published, which we assume to be the real number, I guess,
minus the forecast over the published. So we have a percentage error.

So that if this number is positive, essentially that means we are
underforecasting. Is that right?

MR. KUNDRA: I think it is otherwise. It is forecast minus the \--

DR. COWING: Well, the title that I talked to Roy about this said it was
published minus forecasting.

MR. KASS: We\'ll do it one way or the other.

DR. COWING: But we don\'t know which. On Mondays and Wednesdays we do it
one way, and \-- Well, it\'s a random variable.

MR. KUNDRA: Forecasted.

DR. COWING: So if it\'s positive, we are overforecasting. Okay. Well,
actually, that would be the natural way to do it, I think. And if it\'s
negative, okay, we are underforecasting. Okay.

So just so you understand the signs, although interestingly enough, in
this table you don\'t see a lot of signs. That strikes me as a little
odd, because I think, if I were looking at forecast errors and I felt
that I had captured the structural part of the model well, wouldn\'t I
tend to see a random distribution of signs, apart from the issue of the
magnitudes?

I don\'t see a lot of negative signs here. So that\'s one point that
bothers me a little bit. Okay? I\'d like to see a little more of a
random nature of the errors. So I think what\'s going on here is we got
to work with the structural part a little bit more.

Now I think that\'s not surprising, because the only variable in this
model or these models that\'s driving it other than time is heating
degree days. heating degree days is only related to the heating service
demands, which is perhaps half of residential in the north and probably
\-- certainly less in the south, and probably less, I would assume, for
commercial and very little for industrial.

So it\'s not surprising that only a single explanatory variable other
than time doesn\'t do what I think is real well. Okay.

So at the end of this, I\'m going to propose that you really ought to
consider a few more explanatory variables, although I understand that
what we want is a simple model to kind of get the job done, and any
complications are simply going to make it more complicated, but clearly,
there\'s a basic tradeoff here between accuracy and complication.

So all I\'m saying is maybe we ought to move a little more toward a
slightly more sophisticated model, even if it costs us more, in a sense,
to run, because maybe there will be some big payoffs in increased
accuracy.

Interestingly enough, in the residential table here the forecasts tend
to do worse in the winter. Can we see that? Look at these big errors
here in December, and fairly sizeable but negative in January.

I think, if I had showed these numbers, they are pretty big over here
for \'99. So there\'s a sense in which, again, averages would have been
helpful. But there\'s a sense in which the models tend to do worse in
the winter and better in the summer. But if it\'s being driven by
heating degree days, wouldn\'t you expect just the opposite? So I found
that a little upsetting.

Finally, overall these errors \-- some of them don\'t look too bad.
Others look pretty bad, and I would \-- When we compare them to the next
two sectors - quickly, because I\'m probably using more time than I\'m
supposed to \-- I would have thought that residential would be the
easiest of the three sectors to model.

It\'s certainly the more homogeneous of the three, and industrial is the
least homogeneous of the three. Right? So if I were to predict ahead of
time, I would say, okay, you might get residential looking pretty good;
no way, you\'re going to get industrial. In fact, it just turned out the
opposite. Right?

So I don\'t know how to explain that, but that would bother me.

So let\'s look at commercial. Here\'s Table 2, which is the same kind of
framework, five different estimation lengths \-- six \-- the three
models, and this is now for commercial over the same \-- well, one
period out, out to \-- what is it, 15 periods out? I can\'t see the last
column. That\'s December.

So what do we have here? Well, generally, model 2 outperforms 1 and 3.
Now I told you up front I\'m not a time series economist. So it\'s okay
if model 2 performs better than model 1. But model 2 performs better
than model 3. That\'s very surprising.

Model 3 is the more general model, I think. I\'m ignoring Model 1, which
didn\'t do very well here. So let\'s just think about Model 2 and Model
3.

Model 3 is Model 2 plus autocorrelation structure. So Model 2 is a
restricted case of Model 3. Set the autocorrelation coefficient to zero.
You got Model 2. So since Model 2 is more restrictive than Model 3, I
would expect Model 3 to do better, and it didn\'t. A little surprising,
I thought. Okay?

Here in terms \-- The other thing I talked about in residential is,
remember, in residential that the longer estimation periods, I think,
did better than the shorter ones for residential. I\'m sorry. The
shorter ones did better. Okay. I got that wrong.

In this, it\'s just the opposite. For commercial, the longer estimation
periods do better. So as we scan down here, again ignore the time series
model. I mean, one reason to ignore the times series model is look at
that number. That\'s a percent error, 60 percent. We can flip a coin and
do better than that, I think.

Now on the other hand, it is 15 periods down. So we\'ve got to be a
little careful there. But in general, the longer estimation periods do
better than the shorter ones, just the opposite of residential outcome.
I think there are a lot of questions to be explored here is, I guess,
where I\'m coming down here.

Another thing that I thought very curious, and this table really points
out this: With one or two exceptions, there isn\'t a single negative
error on this table. I think there are two or three. They are all
positive.

So there\'s a real systematic problem here, and I strongly suspect there
is something \-- that says something about the structural content of the
\-- if I may say so, the economic part of the model, and I think it
really shows up here.

Now let\'s see. Industrial is the model. Remember, industrial is the
most heterogeneous sector, has given energy modelers for decades real
problems, and yet here everyone does very well.

I find that very surprising, and the time series model actually does
pretty well here. If you look at the regression model, I guess what you
see is, in general, Model 2 does better for shorter estimation periods,
Model 3 for longer. That seems a little strange. Okay, I think we will
leave it at that.

Okay. Then finally \-- unless anybody has any questions here \-- Then
finally, what this last table is, which they showed in graphical form,
is if you only take \-- what would this be? \-- a four-year estimation
period starting in \'95, compare all models to that across the same 15
months of forecast periods, and then add STEO.

So these three rows here are taken off the other tables. They are not
new. So the only comparison is to the STEO estimates, but as was pointed
out, this is a bit of an unfair comparison. The STEO estimates have an
unfair advantage, because they occasionally get updated, whereas the
forecast models don\'t, and this you showed in that last slide, I think.
Right?

Okay. So I guess to kind of summarize this, I would have liked to have
seen some more statistical detail in this paper. For example, none of
the estimation results are given. I would have liked to have seen those
from the regression models and the time series model.

Secondly, I think it\'s a little hard to see what\'s going on in these
kinds of tables. It would be nice to kind of have some averages across a
few time periods, just so you could make \-- kind of see what\'s going
on a little better.

I become very concerned about using these models in the future, given
what is currently happening in energy markets. So we\'ve got rising
prices. We\'ve got very unsettled conditions. To the extent that these
markets track from the past, I would guess that in the next year or so
they are going to do worse than they are doing here, and that would
bother me a little bit, because I think whatever time and variance
structure there is, that\'s probably going to change or already has
changed.

So that makes it actually more important to have a better structural
model.

What kinds of variables would I like to see in a structural model? Well,
as an economist I always want to see a price, not just heating degree
days, natural gas price. Might even want other fuel prices to
accommodate whatever kind of fuel switching is going on, particularly
when relative fuel prices start changing. Now if they are all going up
or going down, that\'s not going to be a big problem.

We only have heating degree days here. So what my interpretation is that
we are all from the north. Had we done this in Atlanta or Houston, no
way. We would have had CDD. So the point is maybe we need both, heating
degree days and cooling degree days.

In fact, it might not be a bad idea, although this again increases the
complexity of the model, is to do some regional \-- have some regional
specificity, since I assume down south cooling degree days is going to
be a better explanatory variable; up north, obviously, perhaps heating
degree days.

Okay. So overall I thought it\'s certainly a good idea. I think it needs
a little more work. In terms of forecast error, the one thought, I
think, I would leave you with is there are certainly errors in survey
numbers as well.

So in a sense, what we ought to be doing when we look across these
various procedures, including survey results, is there\'s an error in
each of them, and so we have to somehow compare errors across all of the
alternatives and ought not to assume that this published number is the
number; because there is certainly error in it as well.

CHAIRMAN CRAWFORD: Well, I am a time series person or at least I could
be, and I do believe in them. Surprisingly enough, though, I did reach
some of the same conclusions Tom did, although my approach is more going
to be why are these models being different, and why, in particular, are
they not working when we think they should?

I did go through your questions and try to give you my best answers to
them, based on the information I had. So the first thing was what do you
think of the proposal?

Well, I thought it was good. I thought it was well written. I could
understand it. The models were clearly explained. I found one of the
formats that had the equations written so that I could tell what they
were.

I thought you did a great job of defining the issues, both the
philosophical ones, why are we doing this, and what problems are going
to occur if we do it, and also some of the statistical ones.

What would have been nice for me \-- if I would have more time, maybe I
could have found this in [The National Gas Monthly]{.underline} \--
would be to see some of the published series so that I could see what
the data looked like that you were working with. That would have given
me an idea of what kind of seasonal and long term trends you had, and
maybe given me some idea why the models weren\'t working.

\"Do you have any suggestions for improvement?\" I assumed you wouldn\'t
be asking if you didn\'t want one.

First of all, they are \-- All the models, I think, are time series
models. So instead of calling one a regression, one an ARIMA model and
one a time series model, let\'s call the last one a transfer function
model like you did in your paper.

I also think they are all special cases of each other. The regression
model is a simplified case of the ARIMA model, which is a simplified
case of the transfer function model. So why the transfer function model
wasn\'t working, I don\'t understand that at all. It should have been
doing the best out of all of them, I thought.

I agree with Tom. Perhaps one of the reasons it\'s not, or they are not,
is the identification of the systematic component or the structural
component of the model. I notice in the transfer function you used the
log of the heating degree days, whereas in the other two models you
didn\'t take a lot. I don\'t know if that mattered. Maybe there are
other variables, as Tom suggested, that could also be useful.

I think before you get too much farther down this course, you may want
to head toward some kind of automation, because otherwise you\'re going
to spend the rest of your career fitting these models if you are going
to adopt this approach.

They are always going to require extreme user interventions and
refitting as things change. Also, I think you want to start thinking
about some kind of control. How are you going to tell when your model is
no longer fitting the data or your forecasts are really out of whack,
and you need to do some adjustment.

So for those reasons, I guess I\'ll stay on the soapbox that I got on in
this morning\'s session and suggest some kind of sequential updating or
common filtering, which again these models are all special cases of
that. That will also help you bring in some of the measurement error in
your forecasts and allow you a little bit more flexibility.

I also think that you need to focus not just on the bias, which you call
the accuracy of the estimates, but also on the measures of their
uncertainty.

I think that the regression model and the ARIMA model, they might give
very similar estimates, but the uncertainty that is attached to each of
those estimates is going to be very different for the different models,
and perhaps that\'s equally as important.

It certainly is going to be important if you are going to do some kind
of a control thing that says, well, when my predicted error \-- my error
gets too large, then I\'m going to intervene. So some things to consider
there.

Some questions that Tom didn\'t address is: \"Should EIA present two
different series of very short term estimates?\" Oh, what a sticky
question this is.

The answer really depends, I think, on your audience and their use of
the estimate, who is reading it and who is getting these estimates, and
what are they going to use them for.

I have gotten the idea that users of EIA data are fairly sophisticated.
I think that producing two estimates of the same quantity isn\'t going
to bother them, if you explain to them briefly in a footnote how they
were derived. But if you are a newspaper person that writes for a sixth
grade education level, they are not going to like that, and I\'ve heard
people call that voodoo statistics.

People don\'t believe in statistics when it gives them two estimates of
the same thing that are very different. So you do have to be careful.

I think you need to balance, and I think you know this, too, the desire
for a more timely publication against potential confusion, and also the
time and effort involved in producing these new estimates.

I assume that it took you a while to do these, and I can see a lot more
continued work for you in the future to get them. So I think those are
the tradeoffs that will have to be made.

I think that the ultimate question is: \"How useful would earlier
estimates be to those who use them?\" If you get answers like
\"extremely\" or \"varied,\" then it\'s probably worth it. Estimates
like \"somewhat,\" \"unclear,\" or \"not at all\" would tend to imply
that maybe your time might be better spent elsewhere.

Then the question you posed on the sheet I had was a little bit
different than what you showed on the slide. But should you shrink the
data for all sectors? That means, instead of using all the historical
\-- as I understand, you\'re used to the term shrink \-- it means you
will use all the data from \'84 to \'90, whatever, or should we just
start at 1990 and go forward from there? Tough question.

I think the answer depends on what the long term trends are and how
effectively they can be modeled, and that was why it would have been
helpful for me to see the series. Perhaps they are really complicated,
and it is this major structural component that you\'re missing, and that
is what is causing you to do better with shorter series.

I can\'t advocate, though, ignoring relevant data. I think the
statisticians don\'t like it when you throw half the data away and just
use part of it.

So maybe there\'s some compromise to say, based on local smoothing and
fitting where you actually use all the data, but it\'s only a local
neighborhood that actually goes in \-- so this common filtering type of
approach would work really well there, because you could smooth your
historical data based on maybe local one-step-ahead forecasts and then
project from there.

At least, that way you\'re using all of the data to inform on any long
term trends, but your forecasts are done locally.

\"Should we have two models, one for industrial and another for
residential and commercial?\" I had some questions here. Is it possible
just to have one model? Could you put another dummy variable in your
regression model that was a sector variable?

I know the models might look a lot different. So it might be kind of
complicated, but that was my first \-- and maybe you already tried it,
but that was my first instinct, is can you sort of have one model that
maybe even sort of even has an interaction term that lets things be a
little bit different from sector to sector, but there is one model, or
you could use a multivariate time series approach. Then again it\'s one
model that uses all your data.

It seems to me that a transfer function model should somehow fit all
three sectors, since the last two models are special cases. Maybe some
more work needs to be done there to see what\'s going on with that model
and why it didn\'t fit.

I guess I would say that before you decided on two seemingly different
models, just be completely sure that there isn\'t a single model or at
least a type of a model that can be used for all sectors. Not any
statistical reason for that, just for publication reasons. It\'s easier
to explain, well, this is the form of the model, and then I fit it to
this data, or maybe your coefficients change, but at least the type of
the model is the same.

If you say, well, I\'m going to use a regression model for one sector
and ARIMA model for another sector and a transfer function model for the
third sector, then it leaves the reader wondering why did you have to
use three completely conceptually different models for the three
sectors.

So in conclusion, I think that it\'s very interesting work, and I like
the paper a lot, and I think that it can have a big impact on the
timeliness of the publication.

The statistical models that are used are all valid and appropriate, and
I have no quarrel with them. But I do think that it\'s best to decide do
we want to have two different estimates now before all this work is
done.

I would hate to see them go to a lot more work and even incorporate some
of these comments, only for EIA or whoever makes these decisions to come
and say, hey, publishing two different estimates of the same quantity is
really bad, and then throwing this out. So the place to start is with an
answer to that question.

So I have only eaten into our break by about a minute. Are there any
questions from the committee on this paper or on Tom\'s discussion or my
comments?

DR. HAMMITT: Two. One is kind of triggered by the reporting two
estimates question. I couldn\'t see the tables clearly, but do you
somehow say something about the standard error of this estimated value?

I mean, if you report two different estimates which are sort of within
their respective uncertainty bounds, that\'s obviously very different
than if you report two point estimates and implicitly insert that each
one of those point estimates is accurate to however many digits are
reported.

MR. KASS: We report out the standard errors for the survey driven
numbers. Those have long been reported in [Natural Gas
Monthly]{.underline}. I have never gotten a question from a customer
about standard errors.

DR. HAMMITT: And I guess these things are \-- It looks like from the
table Tom put up there\'s a footnote or something that says these things
are estimates. So that is making the point that they are meant to be
treated with less certainty than the other values.

MR. KASS: Yes.

DR. HAMMITT: But there\'s no quantification of that.

The other point is sort of related to all these questions about why do
the restricted models forecast better than the more general ones. Isn\'t
that just \-- Isn\'t it true that within sample a more general model
will always fit at least as well, but out of sample a more general model
may project less well, because you sort of overfit within the sample?

Isn\'t that \-- I mean, both of you are saying it\'s so surprising.

MR. KUNDRA: I don\'t think we can answer that question. We don\'t know
what will happen.

DR. COWING: I\'ll defer to people who are more up on the topic than I
am.

DR. HAMMITT: As sort of a general point?

DR. COWING: Yes, you\'re right. My comments were really related to the
estimation period. That\'s right. And I don\'t know what the answer is.

DR. HAMMITT: But what we were looking at was forecasts out of sample.

MR. KUNDRA: That way the economic work is not out of sample.

DR. HENGARTNER: But the model has about three parameters. So you know,
for three more parameters I wouldn\'t expect forecasts out of sample to
be that much worse. So that I\'d suggest that something else \-- You\'re
right, but for three more parameters, you know? That\'s about right,
that three more parameters for the model 3 over model 2.

MR. KASS: Yes.

DR. HENGARTNER: You put in Na-2 on the residual of Model 2.

MR. KUNDRA; Right. In one case it was one. In another case it\'s two.

DR. HAMMITT: But how many observations were we fitting to, in the first
place?

MR. KUNDRA: \'95 to \--

DR. HAMMITT: So about five or ten years of monthlies?

DR. HENGARTNER: About sixty numbers.

MR. KUNDRA: Because we did use the whole data, 1984 to 1989, and we
didn\'t get good results. So the question was that we were looking for
and what dataset we should use. That\'s really \-- because \'90 and
\'95.

The question was that, if we use only regression, which is easily
understood model, dummy variable \-- you are right, that is part of the
same model, subset of the \-- That model actually is a case of that
transfer function model. But we are getting good results from that point
of view from the regression model a well as the residential, especially
commercial was concerned, because the commercial \-- if you look at that
table, they are blown out in the case of the transfer function and even
the ARIMA.

Where we started, we did not want to develop a very complicated model,
the econometric model that you suggested. He is absolutely right, what
he says, but we were not in any mood to do that, because it would have
complicated the structure and taken much more time.

We were simply looking for a model that easy to do it and sure to give
results, because this is only three months forecast. We are not going to
make forecasts. The only thing was that, because in the case of the
regression model, the results were coming very good.

With industrial, you are surprised. I don\'t know. Industrial model,
when you say that, industrial consumption was definitely being done very
well for the transfer function model, where you used the results of the
1984.

The model that you are arguing prove that the forecasts indicates the
industrial consumption. But the problem was only in the commercial and
the residential.

Residential answer not that bad, because the residential answer was \--
you can say that some were acceptable, but the commercial are very bad,
because you are getting 22 percent, 50 percent error, which is not
acceptable anyway.

We did not work on the \-- what you call the errors and averages and the
other things, because we didn\'t have time to do all those things. But
the main thing is that making the forecast only for three months, not
more than that, that\'s what they need for every publication.

So we simply \-- What we found, that the time series, the transfer
function and the ARIMA function, they are really effective with the
current data. If you add one month data, December of \'98, January \'98,
then \-- January \'99, the results would change. They reflect the change
and the impact of the new data which you added there. But in the case of
the regression, it remains the same. It doesn\'t make much difference.

That\'s what we found, instead of using every month data, but if we
implement or if we do something as you suggested, we are looking into
this, that we have to actually follow the pattern of that estimates
coming up.

If there is too much of a blend or if they are getting too far apart
from the actual results, then we have to change. We have to see which
data we have. Again, we investigate the data as well as the model, which
is useful.

We are not going to stick to one model, we all know. Once we have that,
that\'s it. No, that\'s not going to happen. It\'s going to change. But
I really appreciate the suggestions. They are very well taken.

CHAIRMAN CRAWFORD: Thank you. Tom?

DR. COWING: Yes. I want to come back to this two estimate question,
because I think you\'re right. That\'s a very important question in
terms of EIA products going out to the public.

There are actually two dimensions of it, though. One is that in the same
publication or in two different publications there could be two
different numbers for purportedly the same sector in the same time
period. So would that bother one?

There is another issue. That is, if the estimating methodology changes
over time \-- that is, which model does better \-- or as we\'ve seen
very clearly across sector, then there is an additional point of \--
dimension of complication and that is that you might could use one model
in one place and another model in another place, and then the reader
would really be confused. Okay?

When I think about this, my first take on it is I would really like to
see one number, because I get a little upset when I see two numbers for
the same thing. You know, my first question is, well, which is the right
number.

Of course, the answer is none of them are right. They are all
approximations. So then I think, well, it must be a typo or maybe it\'s
two typos. Who knows?

Then the other way of looking at this is, on the other hand, if this is
\-- if the audience is sophisticated researchers who could read
footnotes, then in a sense it doesn\'t matter. In fact, it\'s even
better, because they can choose whichever number is better for their
particular purposes. Right?

So maybe the question boils down to who your audience is. Here it\'s the
sixth grade mentality versus the more sophisticated people. You know,
who do you want to use these results, these numbers, these publications?
And you know the answer better than I do. I don\'t know.

CHAIRMAN CRAWFORD: Calvin, your tent went up and down.

DR. KENT: Oh. Well, just let me make a few comments. Number one, I would
go back to the original purpose of the paper, and that is unless there
is some compelling reason to do it, I\'m not sure I would do it. Unless
we can find that there is some real reason why the public is demanding
it and has to have it, then leave it the way it is, because we had this
debate ten years ago, and settled it that there wasn\'t a compelling
reason to try to do it.

I am yet to hear what the compelling reason is to try to make sure that
the data is exactly concurrent with the \-- In fact, we even used to \--
When I became Administrator, the publication was dated three months
after its release. So if it was the December edition, it was actually
the September edition, and that\'s what the name was.

I didn\'t see any compelling reason to do it. The other thing is I would
always be very reticent to say we are going to pick our methodology on
the basis of which methodology gave us the best results this time
around.

I think you need to have an underlying logic behind the methodology and
a reason above and beyond this just gave us the best fit for doing it. I
certainly wouldn\'t use different methodologies for different time
series unless you have some compelling explanation for doing it.

I think that you can use a single number, get by with it as long as you
are putting out standard deviations and everything like this. A
sophisticated person can pick it up from there, and whatever use is
being made of it, I think that would be a more advisable way to go, is
my thoughts.

CHAIRMAN CRAWFORD: So very quickly, Inder and Roy, do you have any, I
don\'t know, rebuttal or comments or anything that you would like to
add?

MR. KASS: Well, as everybody was speaking, I was writing my
hieroglyphics. First, Carol, you made a point about head towards
automation. That, clearly, is what we are going to have to do.

The issue is can we develop something that can be plugged into our
publication system, and then we\'ll monitor what goes on afterwards.

Ed, the comments I\'m hearing about the sophistication of our audience
is dissonant with what I hear from our audience. They want a number to
plug into their spreadsheets. I\'ve had people tell me that, when they
have a number of 1500 and then they put it \-- 1500 is the digit.
Doesn\'t matter what it means. Then it\'s revised to 1510. That\'s
completely unacceptable. Their clients won\'t accept it and so forth.

You know, the 1500 had a standard error of 25 on it, but they don\'t
look in the back, and a lot of our customers are not sophisticated, and
I really am concerned about the point that several people made. If we
talk about the basis, is it going to be lost on our clients?

DR. HAMMITT: Well, that raises an interesting question, whether you
should give them the forecast at all if you are sure they are \--

MR. KASS: WE have had that discussion. EIA in its infinite wisdom has a
strategic goal of being more timely, and we in the Natural Gas Division
decided one way of reaching that strategic goal is to provide estimate
clearly labeled as estimates, because we couldn\'t speed up the survey
processing.

CHAIRMAN CRAWFORD: Finally, last question from Jay.

DR. BREIDT: This is a quick and slightly facetious comment. Maybe
instead of reporting standard errors, you should report expiration dates
on your estimates.

MR. KASS: EIA \-- our data are never final.

CHAIRMAN CRAWFORD: Thank you, Tom. Colleen, are we staying in here?
Okay, all right, thanks.

Let\'s take a quick break, maybe ten minutes, and I hope there are
cookies, and get a cup of coffee and come back in about ten minutes.

(Whereupon, the foregoing matter went off the record at 3:26 p.m. and
went back on the record at 3:44 p.m.)

CHAIRMAN CRAWFORD: Okay. We have saved the best for last in a lot of
ways. We are going to do an interactive session on cognitive testing of
potential EIA graphic redesigns, headed by Colleen Blessing.

MS. BLESSING: Thanks. I\'m going to have a real brief introduction, and
then, like Carol said, we are going to do something fun. You guys are
actually going to be able to participate.

EIA is in the numbers business, and often times we use graphs to tell
our story. As many people \-- If you know the history of ASA Committee,
the Committee has helped us a lot with graphics. We have a long history
of sort of standards and guidelines and taking a look at how we are
preparing our graphics.

In the olden days in EIA, we had sort of a rigorous enforcement of \--
we\'re very careful in this wording \-- a rigorous enforcement of
graphic guidelines. I was corrected. They are not standards. They were
guidelines, but rigorous enforcement.

There\'s a little bit of an adversarial position where you create your
stuff and you would create your stuff and you would send it to this
group that reviewed, and they would say this wasn\'t right and send it
back.

There was kind of \-- It was a function of the review process that got a
little adversarial. I was on the other side. I was on the side of I\'m
turning in my stuff and they are telling me it\'s not right. So I
remember that.

Also in the olden days, as the people who are old-er will remember,
there wasn\'t a lot of sophisticated equipment. When I came to EIA in
1978, there was this machine that did graphics. They would send me and
they would say, Colleen, you go draw this graph.

I think it was called like Plot-a-Matic or Veg-e-Matic or something, and
you typed in your X\'s and your Y\'s, and like you had a title. Then you
put the pens in. Does anybody remember those little pens? Then the arms
would go like this, and they would draw the thing. Okay, I used to do
that.

Okay. Now let\'s skip ahead. Let\'s skip ahead to now, the newer days.
The newer days, we have a lot more of \-- The relationship with the
reviewing and the approval of graphics is much more collaborative.

There\'s a lot more flexibility, which \-- In addition to that, as we
all know, we have all this fabulous technology that can draw all these
fancy graphs and put all this stuff on it and all these lines on it in
five axes and titles and subtitles.

I think sometimes people think \-- and this is a problem with the
Website, too \-- it\'s cool, we can do it. So why not? Let\'s do it.

So we get all these comments junked up \-- I won\'t say all of them, but
sometimes you end up getting a graph that\'s got too much stuff on it,
and it\'s really pretty, but it\'s not exactly clear what the message
is.

So a couple \-- A group of us that I\'ll introduce in a few minutes,
because we are the ones that are going to be doing these little
interactive sessions, we decided that what we wanted to do is take a
look, to think back to the original purpose of a graph.

The original purpose of a graph, what we thought, is not just to \--
It\'s to convey a message, but to convey it clearly and relatively
quickly. If somebody has to have an advanced degree and it takes them an
hour to figure it out, then the graph conveyed the message, but that
wasn\'t what we thought the goal was.

So we wanted to figure out \-- wanted to take some graphics that were in
use right now and figure out if people could understand the message,
whether they were getting the message, whether they were getting the
wrong message, and if we were seeing some issues, then we felt like we
needed to take some steps and make some changes.

You guys have heard in a different context. We had \-- This is the same
team that did the cognitive testing of the Website, and we reported to
the committee several times. It\'s myself, Renee Miller, and Herb and
Bob and \-- oh, and Howard. There\'s Howard, Howard Bradsher-Fredrick,
and Antoinette Martin who is on maternity leave.

So the five of us decided that we would do some cognitive testing of
graphs. Cognitive testing is a fancy word for just sitting down with
somebody and showing them something and kind of talking them through it
and asking them some questions about it.

Now we are not asking people did they like it, is it pretty, because a
lot of times people will say yes. We are asking them to actually find
real information on it.

We did not \-- I want to stress this. We did not go through publications
and say, ah, here\'s a bad one. We asked the program offices for them to
nominate graphs that they thought \-- they thought \-- might be
confusing for us to test to see if our customers actually understood it.

These are graphs that are on the Web now, that are in publications now,
and that people from EIA actually submitted to us, rather than us sort
of trawling for the worst or the most confusing.

We are going to do these little interactive exercises, and we would like
feedback on two things. One is the graphs themselves, and also on the
process.

We created a little protocol, and we are going to do the protocol with
you to show you how we did the tests that we have already done, and then
you guys can give us feedback on whether you think that is working and
how we should change it.

We did do some testing up at the Winter Fuels Conference a couple of
weeks ago. We recognize that those people, as is true of yourselves,
were not your normal, average customer for EIA. Winter Fuels people are
people that work for [The Oil Daily]{.underline} and work for the
American Petroleum Institute.

We tested them, and we got some very interesting results. So we also
wanted to go ahead and test the Committee and see if we found kind of
the same results.

So here\'s how it\'s going to work. We are going to break up into groups
of two, two, two, two, two, and then the last group will have three, and
there will be one group here. I\'ll work with Roy and Jay. Then Randy
and Nicolas can move down to that corner. Okay, so you guys move to that
corner.

Polly and John, you guys \-- if you guys could go to that corner, then
you two guys could come to this corner.

Then the last three, Cal \-- I thought there were three. I guess
there\'s just Cal and Carol. One more? If you guys could go to the room
where the apples were, past the cookies over there, there is going to be
somebody over there.

Then we\'ll briefly do the test. What we will do \-- we thought \-- At
first, we thought we would have you guys all move from station to
station. We will move. So after I finish with this group, I\'ll move to
this group, that group, and then that group.

So we will all move, trying to just do five minutes a test. It\'s very
short, and see what we get. Does anybody have any questions? Is that
going to work? Okay. The cognitive people need to sort of grab their
people.

(Whereupon, the foregoing matter went off the record at 3:51 p.m. and
went back on the record at 4:24 p.m.)

MS. BLESSING: Some people are still working. I just want to thank the
committee. This was a very rushed exercise. We really wanted to take
advantage of the brain power that was here and move quickly. I think
normally we sat for between ten and 15 minutes with some of our people
that really wanted to go ahead and talk when we did the live testing.

I can only summarize my results. Everyone in the room except the people
that were outside saw the one where I asked you in what year did
reserves fall below production.

Two of the four groups, who will remain nameless, identified a year when
we did fall below production. Reserves are ten times production, and
there\'s a double y-axis. It\'s a confusing chart.

My fear is not that a statistician could make that mistake but that a
reporter might be looking at this and say, ah, reserves are falling
below production, and write that in a story; because that\'s what the
graph compels people to think and say.

One other person in my group came up just now and said that we got some
really good data from \-- There was a lot of discussion about the types
of things that were good about the graph, the types of things that
weren\'t so good.

One of the reasons why \-- I didn\'t say this at the beginning \-- that
we started this project is \-- I think I didn\'t say this \-- because
two of the four graphs that won in the ASA graphics contest last year
were double y-axis graphed, and we were \-- When we looked at those, we
thought, you know, gosh, EIA traditionally hadn\'t done that kind of
graph.

I guess, now that there\'s some fairly \-- at least this one that I
tested that one last year was pretty much okay. It had a good message,
and people could really understand it, but what that ends up leading to
is the graphs more like this that really confused people.

So I think we have to \-- I think we are going to have to think about
what we heard, and I think we are going to do some more testing of some
of these graphs and come up with some conclusions about having two axes
and how they should be labeled. They are maybe using colors or something
like that.

Did any of the other people who did testing \-- do you guys want to do
any summary? I know what we are going to do is probably in the spring, I
guess, we\'ll summarize what it was we found and, I guess,
recommendations.

MR. RUTCHIK: Right. Not only summarizing this test, but I guess the
previous tests that we had done and any subsequent tests. So you\'ll get
a much fuller picture of what we have done and the conclusions, you
know, that we have drawn, and also conclusions across possibly very
different types of audiences, from experts and people who are very
familiar with energy data such as yourself to maybe we\'ll get,
hopefully, tests with people who aren\'t so familiar with energy data.
So conclusions across groups.

MS. BLESSING: I think in closing, what I\'d like to say is I\'m a true
believer in cognitive work, and I\'ve seen it with our Website, that you
can sit around and think for days and weeks and months about how you
should change it, but if you sit down with people and watch them work,
they are going to show you where the mistakes are and where they get
confused a lot quicker than you could figure it out yourself.

I think it\'s the same way with the graphics. We have learned more in 45
minutes than we would in six months of meetings about, you know, the
graphics standards and all that. It\'s just much better to show people
and watch them work and ask them specific questions, and not \-- I bet
if I would have asked everybody what do you think about this graph, most
people would say its pretty, oh, it\'s neat, oh, it\'s cool. Yes, it is,
but it\'s misleading.

So that\'s why we are just going to continue to test and, hopefully,
come up with some recommendations.

MR. MILLER: One of the things, I think, was interesting was the group
was able to give us in five minutes what. The other people we test would
spend five minutes trying to figure out the main message of the graph.
So I think that was very helpful to us.

MS. BLESSING: Well, I think this group, on my graphs anyway, is much
better at figuring out the main message than the people that I had
tested before.

Does anybody else have any comments or questions?

DR. COWING: I think some of these issues \-- This is an issue that came
up earlier in another context. Some of these issues or confusion depend
on who you think the audience is. So that one type of people \-- and
this is not necessarily correlated with education \-- might have
problems, and another group might not.

So where you come down on that depends on who you are trying to target.

MS. BLESSING: I guess my personal opinion on that is that it seems like
ten or 15 years ago, before the Web, we had a more limited audience, and
I think they were more academic and more industry people, business
people.

They may have had an easier time understanding this. I like to look at
our customer base as a pyramid. Now that we are out on the Web and
anybody in the world \-- and they are getting on our site \-- we really
have to make sure that we are making our information understandable to,
I think, the broadest number of people we can.

DR. COWING: But does that imply dumbing it down?

MS. BLESSING: Some people in EIA would say that, and I think that\'s why
some people in EIA do say that, on the Website, too. So that\'s why
testing with true customers is important, I think.

DR. MOSS: Actually, I don\'t think dumbing it down is that bad, in the
sense that graphs are supposed to show quickly a fairly straightforward
message. You know, if somebody who is not that facile with them gets the
message, then somebody who is is going to get it that much faster.

So I don\'t think you\'re giving up anything by dumbing it.

DR. SITTER: I would even say that people that are qualified to work
their way through one of those graphs are probably less likely to spend
the time to work through one of those graphs.

I mean, if it\'s so confusing that I have to spend the kind of effort I
did on some of these, I just wouldn\'t.

MS. BLESSING: Then we\'ve lost the whole purpose of the graph.

DR. WHITMORE: I\'ll admit to being one of those who, when I had looked
at it quickly, I got the wrong answer on that production reserves thing.

MS. BLESSING: I\'m going to count you.

DR. WHITMORE: And I think the key is that, if the two vertical axes are
both the same type of scale like gallons of oil or billions of gallons
of oil, whatever, it is very likely that people are going to \-- the
same quantity was two different scales for that same quantity, that you
are asking for troubling.

When it\'s totally different scales like barrels of oil and dollars,
then it\'s much safer that people are going to interpret correctly.

MS. BLESSING: Cal, you had a comment?

DR. KENT: I think you always ought to go back to the rule of only one
message on every graph, that you\'ve only got one point that you are
trying to make.

MS. BLESSING: Also, I have to say, I forgot to mention that, before we
tested these graphs, we pulled in every author. Not only did they
suggest these graphs to be tested, but we pulled them in and we said
what do you think the message of this graph is?

I think that the author of this truly believed there was one message.
The problem was not very many people got it. Then when I asked them to
pull specific information off the chart, it was not easy.

Actually, in the general part, nobody in ASA made the mistake of going
\-- reading off of each line and going over to the same y axis.
Everybody here knew to go to this one for one line and this one for the
other line.

With the other people that we tested, I had a number of people that read
both off of the lefthand y axis or the righthand y axis. So they got one
answer right and one answer wrong for the questions that you know I
asked.

DR. HENGARTNER: Just visually, this one, if one axis would be blue and
the other one would be red, already you would reduce by half the errors
you would be making by reading it off.

MS. BLESSING: Yes, it\'s possible, and there was a graph \--

DR. KENT: And if it corresponded to the colors of the line.

MS. BLESSING: That\'s what he means. No, let\'s do it the other way.

DR. WHITMORE: On this point, on the idea of having a single issue or
concept that you are trying to show in each one, I remember one thing
that I was taught in a proposal writing training is to have something
called an action caption. That is a caption that describes what that
message is so that you don\'t \-- it\'s not hidden there somewhere.

MS. BLESSING: Right. And I think that that \-- A number of people that I
tested, when I said what\'s the message of this graph, they saw these
two boxes and said, well, there were two peaks.

So they looked at the squared boxes and read what was in there, and said
the peaks are the message. The author thought that the message was the
relationship of the two lines and that reserves follows production \--
or production follows reserves. I\'ve heard them both. But you could
have put that in the title. Then that would have helped people out.

Right now, you\'re assuming that they are going to try to interpret
that, rather than just putting it right there for them.

DR. WHITMORE: Yes. Some action caption that really does the message
doesn\'t work well as a title. We were told, you know, you always get
that action caption in there somewhere.

MS. BLESSING: I like that. Gotcha. Thank you. I need to run.

CHAIRMAN CRAWFORD: I know you do. That\'s okay. Thank you.

Anyone else have any comments or questions?

DR. SITTER: Are we allowed to leave all this stuff here or should we
take it all with us?

CHAIRMAN CRAWFORD: We can leave it. I\'d like to invite the public to
give their comments, if they have any, at this time.

If not, then I will adjourn the meeting until tomorrow.

One thing I\'d like to do is talk about dinner. I think we do have time
to go back to the hotel and leave from there. In case we are all leaving
separately, would you write down the address, the name and the address
of the restaurant. it\'s McCormick & Schmick\'s at 1652 K Street, N.W.

Probably a group of us will meet in the lobby of the hotel, say, at 5:10
and catch a cab to go over, or cabs, depending on how many there are. If
you want to go separately, that\'s up to you, but that\'s the address.
We need to be there at 5:30.

DR. COWING: So that\'s 16th and K, K between 16th and 17th?

CHAIRMAN CRAWFORD: I think so, yes.

(Whereupon, the foregoing matter went off the record at 4:34 p.m.)