AMERICAN STATISTICAL ASSOCIATION

\+ + + + +

COMMITTEE ON ENERGY STATISTICS

\+ + + + +

FALL MEETING

\+ + + + +

FRIDAY

OCTOBER 17, 2003

\+ + + + +

The Committee met in Room 8E089 in the Forrestal Building, 1000
Independence Avenue, S.W., Washington, D.C., at 8:30 a.m., Jay Breidt,
Chair, presiding.

[PRESENT]{.underline}

F. JAY BREIDT Chair

NICOLAS HENGARTNER Vice Chair

JOHNNY BLAIR Committee Member

MARK BURTON Committee Member

JAE EDMONDS Committee Member

MOSHE FEDER Committee Member

JAMES K. HAMMITT Committee Member

NEHA KHANA Committee Member

NAGARAJ K. NEERCHAL Committee Member

POLLY PHIPPS Committee Member

[ALSO PRESENT]{.underline}

NANCY J. KIRKENDALL Designated Federal Official

CALVIN KENT Invited Guest

BILL WEINIG EIA

JAY CASSELBERRY EIA

DOUG HALE Lawrence Berkeley Laboratories

JOHN WOOD EIA

A-G-E-N-D-A

Open the meeting, Jay Breidt, ASA Committee

Chair 3

EIA and the Confidential Information Protection and Statistical
Efficiency Act (CIPSEA)

Presentation by Jay Casselberry, SMG, EIA 4

ASA Discussant: Calvin Kent, Previous ASA

Committee Member and Guest of the ASA

Committee 22

ASA Discussion and Questions 30

Breakout Sessions: Data Edits for the EIA-920 41

Relative Standard Errors: ASA Summary by

Moshe Federer 80

Electricity Transmission Data Needs. Presentation by Douglas R. Hale,
Senior Economic Analyst, EIA, DOE, currently on detail to the Lawrence
Berkley Laboratories 83

ASA Discussant: Mark Burton 111

ASA Discussant: James K. Hammitt 120

Suggested topics for the Spring 2004 Meeting 148

Seek public comments and questions 148

P-R-O-C-E-E-D-I-N-G-S

8:31 a.m.

MR. BREIDT: Good morning. Welcome to day 2 of the fall meeting of the
ASA Committee on Energy Statistics. First I would like to ask any
committee member, guest, or EIA staff or member of the public who was
not present yesterday to identify yourself and your affiliation at the
microphone, please.

MS. SPENCER: I\'m Linda Spencer. I\'m with K(inaudible).

MR. BREIDT: Thank you. Is that it? The other thing is if you haven\'t
already done so, you are asked to sign in at the front desk there,
please, either at the break or before leaving.

Lunch for the committee will be held at the conclusion of the session
and this is a session on suggestions for the spring 2000 meeting \--
2004. I\'m off a few years here. Lunch will be in the new building \--
remember our change of venue? \-- after the break.

What we will actually do today is at the break grab your refreshments
and start walking so that we can try to keep on time. We will be fairly
tight during this morning\'s sessions. For example, some of our
discussants need to catch trains almost immediately after so we\'ll try
to keep on time.

We are going to begin with EIA and the Confidential Information
Protection and Statistical Efficiency Act, CIPSEA, presented by Jay
Casselberry.

MR. CASSELBERRY: Thank you. I\'m going to refer to it as CIPSEA through
most of the presentation but it is the Confidential Information
Protection and Statistical Efficiency Act. What I want to do is start
out by giving a little bit of background.

I have already briefed you at the previous session about this law so
just a little background to refresh it in your memory to tell you about
with the passage of this new law what confidentiality options we have
with the survey data we collect.

What we did was we did some research into the pros and cons of CIPSEA
for EIA surveys and then we decided on what surveys we would start
collecting under CIPSEA because it is a stronger confidentiality law,
what steps we\'re taking.

Then with regard to CIPSEA there are some issues that came up for EIA
when we were looking at what to do with our surveys. There are some
things that we need to do a little bit more investigation of, some
things that we\'re sort of unclear on we would like your advice on it.

CIPSEA was passed as part of a larger law, the E-Government Act of 2002,
back in December. CIPSEA is Title V of that Act and there\'s a Subtitle
A which just talks about confidential information protection and it
really has three major problems and the problems are sort of what lead
us into some of the issues that I will discuss later.

It does have a higher degree of confidentiality protection for the
information we collect but it does place limits on how the information
could be used after it\'s collected. It also allows us to share but only
for statistical purposes.

Just briefly, under CIPSEA any federal agency, not just the statistical
agencies, can collect information and if they pledge confidentiality and
that the information will be used for exclusively statistical purposes,
then they are in that box and they cannot get out of it.

Any information that they collect they have to follow those constraints.
It has to remain confidential. It has to be used for exclusively
statistical purposes and it can only be used for any other purposes if
the respondent then gives some type of informed consent saying, \"I
agree for other uses.\"

CIPSEA goes on to specify what is statistical purposes and
nonstatistical purposes. Statistical purposes just using the information
to describe or make estimates about the groups or subgroups of the
population that was surveyed. Nonstatistical is pretty much anything
else.

If the information is going to be used in identifiable form and legal,
regulatory, or administrative, or any other way they could affect the
rights, benefits, or privileges of the people who provide the
information to you.

And CIPSEA does have very large penalty provisions. We don\'t currently
have these same type of penalty provisions for our confidential
information. Anyone who violates CIPSEA either by improperly disclosing
or improperly using the information for nonstatistical purposes could be
tried and put in jail or receive a very large fine or both.

The Office of Management and Budget was given the lead role in CIPSEA
for the Federal Government and what they have done is they brought
together an interagency committee right now that just has
representatives from the major statistical agencies and we are
developing guidance that is going to be for the entire Federal
Government on what they should do with regard to CIPSEA.

If they collect information, what they should do and what we\'re looking
at is the wording for confidentiality, agents if the information is
shared, safeguards, and any security that should be on the information.
That guidance should be drafted. It\'s going rather slow but hopefully
it will come out in 2004 and it will be put out as a Federal Register
notice for public comments.

And OMB will also be looking. We have to send in our surveys for OMB
clearance. Before we do them they are going to look at what
confidentiality provisions we have on the surveys, why we\'re using
different levels of confidentiality for different surveys.

Also CIPSEA does require when statistical agencies collect information
if there is any possibility of nonstatistical uses of the information
the agencies must be very explicit to the reasons of the people who are
going to provide the information about those nonstatistical uses.

What we\'ve done in EIA we established a way to look at trying to do it
across the agency. We determined first what our confidentiality options
are. With CIPSEA there was some question as a statistical agency do we
have to move everything into CIPSEA.

We also then looked at the individual surveys to decide which one should
best hit under CIPSEA once we realized that CIPSEA was just an option
that we had. Then we work with the staff to do this and we are still
working with the staff because we are going to be implementing this in
2004.

What our options are for our survey information, we have really
confidential and nonconfidential information. If it\'s confidential and
we collect it under CIPSEA, then we are limited to using it for
exclusively statistical purposes.

Our existing confidentiality says that we can maintain confidentiality
but it is available for other official uses by other federal agencies
and by Congress, committees of Congress and such. That is not limited to
statistical uses.

Under existing confidentiality we do not publicly release identifiable
information but, as I\'ll talk about later, we do publish some tables in
that where it might be possible to identify the information submitted by
individuals respondents for some surveys.

Our law requires us to share the existing information with other federal
agencies and Congress for their official uses. It\'s also subject to the
Freedom of Information Act request. The Freedom of Information Act was
passed so that most of the information held by the Federal Government
would be publicly available and we have to defend each time if someone
requested it, why we\'re withholding it.

We don\'t have that same requirement under CIPSEA. CIPSEA overrides FOIA
in that respect. We do collect some nonconfidential information where we
make it publicly available in identifiable form.

The benefits of CIPSEA are that we can begin sharing outside the Federal
Government. Right now we limit ourselves, our sharing to other federal
agencies.

We set up data sharing agreements with those agencies and we define that
they will keep the data confidential, how they are going to use it. With
CIPSEA we\'ll be able to start sharing outside the federal confines but
it will have to be for exclusively statistical purposes.

Confidentiality has a higher level of assurance under CIPSEA than it
does under our current one. As I mentioned, we still have to defend our
decisions under our existing confidentiality. We have to defend them
against FOIA requests.

We have to explain why there would be competitive harm if we release the
information or one of the other FOIA exemptions that allows us to
withhold it and maintain its confidentiality and it is possible.

We do have very high response rates because we have a large number of
mandatory surveys but it is possible our respondents would be even more
encouraged to respond if they knew the information. The confidentiality
had a higher level of insurance and they didn\'t have to worry about the
information ever being used in some official way against them in
particular.

The drawbacks are for a number of our surveys we don\'t use statistical
disclosure limitation when we come up with the tables and do our
information products. When you do that, you\'re going to have some sales
where there may only be one or two respondents or they are dominated by
one extremely large respondent so there can be some disclosure through
those types of tables.

If we put it under CIPSEA it limits our ability to share for what some
people might consider nonstatistical which might fit in the
nonstatistical area. I\'m going to talk about each of these in just a
little more detail.

With statistical disclosure limitations there are certain surveys and
information products where we decided not to use statistical disclosure
limitation. It tends to be where you have \-- our users prefer very low
level information on energy supply, energy product supply. They like it
at the state level, at the PAD level.

Usually because you have a relatively small number of respondents
usually dominated by a few very large respondents, it\'s very easy for
sales to disclose information reported by a respondent. It does happen
in cases.

It\'s not happening all over the tables but it does happen. So we made a
decision that the information products we would continue to do that.
This was made back in the \'80s. We would put out some products where we
did not use statistical disclosure limitation.

It allows us to put up a large amount of information out there for our
users. But if we did not use statistical disclosure limitations, there
would be the chance of violating the confidentiality which would then
turn around and violate CIPSEA.

If we did then use statistical disclosure limitation, it would cut down
the products but also the systems that aren\'t using it would require a
large amount of resources just to develop the disclosure limitation
methods to implement them, to test them, to make sure that they were
working properly and there wasn\'t any change of any disclosure.

With sharing information, we do have a number of sharing agreements that
we are currently involved in. They come up more often. There are some
ongoing uses like the Strategic Petroleum Reserve with the Department of
Energy.

We provide them information from our petroleum supply systems so they
know what refineries are running, what kind of crude streams they
process, what types of products they put out and so they can do planning
so that in the event of a supply problem they would know how they could
get their crude from the strategic petroleum reserve at the different
refineries.

In other examples we provide the Department of Defense. They want to
make sure that the \-- they want to look at the refineries, what
products they\'re producing to make sure that they are going to have
adequate products to supply the military forces.

These kind of uses, they are looking at individual survey data,
individual respondent survey data. They are making decisions on those.
They are planning, anyways. The question comes up are those statistical
uses since they are really focusing on the broader picture but they are
also focusing on the operations of individual survey respondents.

We don\'t really have a good read on that right now and that\'s why
we\'re a little bit leery of putting some surveys into the CIPSEA
because later on then we would not have the capability to release it if
someone determined that those aren\'t valid statistical uses.

We can get informed consent to release the information for
nonstatistical uses. We do not have a lot of experience with informed
consent. We\'ve used it in a few very limited cases to get authority
from some survey respondents to publish very low-level data but we
don\'t have any real good experiences with using it in large surveys to
do that. Most of what I\'ve read, at least initially, is that it\'s used
more in survey with persons, in survey research and statistical
settings.

We have developed a informed consent agreement but the question is would
our survey respondents be willing to sign off and give us informed
consent to use it for nonstatistical purposes. Would we do that at the
initial time of data collection when they really wouldn\'t have an idea
of \-- wouldn\'t have a definite idea of what the information was going
to be used for.

If we try to push it back to the time that there is an energy supply
emergency, then you\'re in a real quandary because really the people who
need the information for emergency response or planning aren\'t really
in a position to wait while we send paperwork out.

Maybe it has to go through some legal people in the survey organizations
or whatever and then come back in before we could actually release the
data. We are looking at informed consent. It might be viable but we are
really just not quite sure right now.

So when we looked at our surveys and deciding which ones we would put in
CIPSEA, as I already talked about, the main reasons some surveys aren\'t
going to be put in CIPSEA is we don\'t use statistical disclosure
limitation methods. That\'s the resource issue and then we would have to
cut down our information products. We would have to suppress more
information.

Our sharing commitments, we do have a certain responsibility, at least
we feel, to support official federal uses of the information that really
aren\'t harmful to individual respondents and that gets into a gray area
in what are the statistical purposes.

But we don\'t want to move some surveys into CIPSEA until we have a real
strong read on whether or not the purposes that we consider legitimate,
the official purposes, are also considered statistical purposes. There
are some surveys that we do where confidentiality has been very
controversial. Our electric power is one area.

We have decided to wait until we go through the forms clearance process
which we do once every three years and that\'s a way for us to actually
go after the data users and the data providers and try and get their
input because it does affect it. Once we collect it under CIPSEA or
don\'t collect it there\'s limits on how it can be used. We would like
to get their input before we make decisions.

After considering all that, we have 58 surveys that are collected with
some confidential information. There are also certain ones that are not
confidential. But of the 58 that have any confidential information we
are moving 10 under CIPSEA for 2004. You can see in certain areas we
aren\'t moving any in those areas. A lot of them don\'t use statistical
disclosure limitation or they have significant sharing efforts underway.

Surveys we are going to move are consumption surveys which we have
always done by contractors outside of EIA do up to now because we wanted
to be able to protect the confidentiality of all ten of these surveys.
There are ones where the statistical disclosure limitation wasn\'t an
issue and we were comfortable that there wasn\'t any real need to share
the information for official nonstatistical purposes.

The consumption surveys will be done under CIPSEA are financial
reporting system which is the major energy companies giving us financial
data. In the petroleum area we have a frame survey in that we do two
weekly price surveys of motor gasoline and on-highway diesel fuel. And
in the uranium area all the surveys will be moved under CIPSEA.

Our next steps are we have to come up with confidentiality wording so we
can inform the respondents the OMB committee is coming up with the
CIPSEA wording so it will be used universally across all agencies when
they collect CIPSEA information so if the EIA is collecting CIPSEA
information respondents will see similar wording as if it\'s collected
by the Bureau of Labor Statistics.

Also the ones that aren\'t going to be under CIPSEA we have to put in
some wording to tell the people that there is the possibility of
nonstatistical uses in the information for official purposes.

We are looking at whether or not we need any additional safeguards and
security. We already have safeguards and security for all the
confidential information we have been collecting to make sure that it\'s
properly used, that people can\'t get it from the outside, that the IT
systems are secure, that the physical security is in place.

This is just another set of confidential information. We are deciding if
we need to do anything in addition for CIPSEA data.

We do need to train our staff because, as I mentioned, there are some
pretty strong penalties for misuse of CIPSEA information. We need to
make sure that people are going to understand that, that they don\'t go
\-- we have been doing sharing. They\'ll go ahead and do some kind of
sharing where there\'s the possibility of nonstatistical uses.

We are developing some informed consent forms. At a minimum we would
like to have them on the shelf so in case there are problems like the
blackout that just happened in August or there\'s a major supply
disruption in one of the energy sectors that we at least have it
available where we would be able to send it out and ask for their
consent to use the information for some type of emergency response.

We also need to develop some agreements, some protocols for sharing
under CIPSEA to make sure that we can put those into place so if we are
going to share outside of the Federal Government we have the appropriate
things.

Right now the Census committee \-- not Census but the OMB committee
that\'s looking at CIPSEA across the Federal Government is going to come
out with some guidance on dealing with agents who have access to CIPSEA
data.

Our request to the Committee is I have provided you with a copy of our
draft informed consent agreement. We are just sort of feeling our way.
We would like your opinions on it if you have any experiences, if you
have any opinions on whether or not you think this would work with the
energy industry.

Also the sharing that we\'re doing we want to be able to share for
legitimate official uses. I\'ve given you a couple of examples of areas
where we do share. We would like your opinions on whether or not you
think they are statistical and any other opinions you have on that
because we\'re sort of in the gray on that. Thank you.

MR. BREIDT: Thanks, Jay.

Our ASA discussant is Calvin Kent.

MR. KENT: First of all, I apologize for having a bit of a cold. I stayed
up and watched the ball game last night. From our position, the wrong
team won. That\'s happened twice so at least I don\'t give a hang who
wins the World Series. I apologize for it.

I am reminded of the old statement that we use in church all the time,
and that is, \"Be careful what you pray for because God may answer your
prayers.\" I feel very much about that same way after getting Jay\'s
report here about CIPSEA because, as you probably know, we have been
asking, we being the EIA, for more protection and better capability to
protect information since the 1970s.

Now that we have it, we don\'t seem to quite know what to do with it.
That, in a way, bothers me. Let me just begin with making some comments.
First of all, I need some clarification about the actual processes that
are being used to answer these questions.

We know that there are certain organizations and interagency task force
and so forth are set up. But it\'s not clear at all to me what the
process was that was used to make the decisions on the 10 forms that are
going to be protected.

I have just some limited idea about what the criteria is. I think that
it is extremely important for EIA at the beginning to be very
transparent in this process of what they are doing and how they are
doing it. Jay used the words \-- he didn\'t in his paper but he did in
his presentation, \"We are comfortable with this and we are comfortable
with that.\"

I need to know what made them comfortable and what would make them
uncomfortable. One of the things that we asked as a committee last time
was that EIA needed to come up with a tighter definition than the one
that\'s in CIPSEA of what is statistical data. I don\'t know if that has
yet been accomplished. I couldn\'t find that in the paper.

CIPSEA does give EIA a great opportunity to revamp the way that it does
some of its business. I think it gives it an incredible opportunity to
increase the reliability of some of its statistical information, even
that which is collected under mandatory provisions, you should be
getting a much higher response rate when there is not concern on the
part of respondents that this information is going to be used against
them either legally or commercially.

I\'m not sure how significant the problems are that Jay listed in his
program, or in his paper. I need to know whether or not these are going
to create any major issues or whether or not these are just minor issues
and to the extent that EIA has looked into the impact of each of these.

I do know that there is probably going to be some additional burdens on
folks but that by itself does not necessarily create problems. Let me go
to the three questions that we were asked to comment on. The first one
is on the informed consent agreement. I do not think that the current
agreement is adequate or will produce the desired result.

First of all, it seems to have been drafted with the idea that it would
be used only in emergency situations but it does not define what an
emergency situation is.

I have a feeling that an emergency situation will be like the poet Keats
described, \"Beauty is in the mind of the beholder.\" I think that needs
to be clearly defined because what may be an emergency to a very zealous
prosecutor may not be viewed as an emergency by the individuals who are
providing the data.

Since we don\'t know really what use is going to be made of this data
and it is not spelled out in the form either, I\'m not sure that this is
going to assuage any negative feelings that respondents are going to
have. I think that there\'s three things that need to be included that I
did not find in there. The first of these is specifically what of their
information is going to be released.

Secondly, what is the time period that\'s going to be covered for the
release of this information, and specifically what uses will the
recipient agency be allowed to put to use that information. Those three
things I think would increase the level of good feelings that the
respondents would have.

I think that another place to look is in the Section 524 of CIPSEA where
it does spell out the criteria as to what should be in the agreements
for sharing between agencies. I think that these also ought to be
included in the form that is being asked here and those criteria there
are in 524(a) 1 through 4. I think those would be useful to also be
included in this particular agreement.

I\'m also worried about the fact that once data leaves EIA whether EIA
really has any control over the use of that data. Once it\'s out there
what could be done to protect it if it all of a sudden becomes used for
purposes that are of either regulatory or of a commercial nature that
would damage the firms.

I think that the form itself has a way to go before it\'s going to
produce the response that you want and that is to have the flexibility
beforehand to be able to release data in emergency situations.

The second question that we had to deal with is that EIA is required to
share data with other agencies for official purposes and how do we go
about doing this without raising concerns from respondents. EIA has only
limited experiences, as Jay pointed out, with doing this. But I would
like to know what lessons have been learned from that experience that
could be transferred to this particular area.

I am a big believer in transparency, and that is that we make it
abundantly clear to all of the parties involved what it is that we are
doing. In my opinion, the best way to approach your respondents is to
absolutely let them know up front what uses will and will not be allowed
so far as the data that they are supplying.

To the extent that you make it transparent, then they can make an
informed decision themselves. I would encourage you not to try to put it
in the small print at the bottom of the page or anything like that, but
I would be very forward in terms of what you are looking for there.

Then my third comment is about the sharing with other agencies and we
were given two descriptions or two examples here. In my opinion, EIA
ought to be in the position where if this data is truly statistical data
and these agencies are apparently going to be using it for
nonstatistical purposes, and just from the short descriptions that we
had it does appear to me that these agencies will be using it for
nonstatistical purposes.

I think that EIA ought to consider whether or not they perform these
functions for these agencies rather than having the agencies do them
themselves. That would, of course, mean that EIA would be given the
resources which is likely to lead to resistance from those other
agencies.

I would certainly think that the best way the EIA could protect itself
and protect its data in these situations would be for EIA to take over
and do those functions on a contract basis.

My suggestions at this point is that EIA develop and make very
transparent their protocol for deciding which forms will and which forms
will not and which data and which data will not be protected under
CIPSEA.

This, I think, to an extent is being done but I think it even needs to
be more obvious to the public what the criteria are that are being used
and why those criteria were adopted. And then, of course, I think it
needs to be consistently applied throughout the whole range of the EIA
products.

I also think that there needs to be extensive stakeholder input. I\'m
not sure I\'m 100 percent comfortable waiting until OMB comes up with
its rules and then we respond in the usual bureaucratic way to the OMB
notices. That that is really the best way to get stakeholder input on
this.

I certainly would encourage the EIA to begin developing a formal
training program for all of those on their own staff, contractors and
people in other agencies who are going to be handling this data because
that needs to be done as almost a first step.

MR. BREIDT: We now have some time for discussion but first I would like
to hand it back to Jay to see if he has any responses to that.

MR. CASSELBERRY: Sure. Thank you. Our criteria were basically when we
went to the program office and went back to CIPSEA is can you ensure
confidentiality. If you cannot ensure confidentiality, you don\'t have
the processes in place to ensure confidentiality, the main statistical
disclosure limitation, then you shouldn\'t be considering CIPSEA at this
time and you can consider whether or not you want to put those processes
in place.

But until you do, you really should not consider CIPSEA because you
can\'t ensure the confidentiality if you aren\'t doing what you need to
do to make sure that the information products you put out won\'t have
some inadvertent disclosure.

The next step, if they were doing those, we asked them, \"Are you
currently doing any sharing or do you foresee any sharing? If you are,
or foresee it, is it going to be first for statistical purposes? If you
aren\'t sure that it\'s going to meet that criteria, then for now you
should not go into CIPSEA and we need to do some more investigation and
make sure we understand what the statistical purposes are.\"

I heard your comment that some of the ones that were currently going you
don\'t view as exclusively statistical. I think that is a legitimate
thing. I think some are waiting for the OMB guidance and since this is a
legal question and it would come up \-- we are trying to get our office
of general counsel involved in both our confidentiality provisions and
that so we are working on those kind of things. Those are really just
decisions we will make internally within EIA because if there is a
question, other people will get involved.

With the informed consent you saw, it was written pretty broadly. The
one that was given out was basically for the event of an emergency
situation or for emergency planning. I mean, we could change the wording
if we saw other uses like for the Department of Defense or for the SCRO.
I mean, we could change it in that way.

It was fairly broad in so much as it allowed the administrator to
determine when the situation warranted. I agree with you that is pretty
broad. I guess we were trying to have our cake and eat it, too.

We are trying to keep our options open so that if something comes up,
the administrator can make the \-- the administrator would make that
decision. I agree that might not make our respondents feel very
comfortable.

We don\'t want this to become an ongoing paper exercise so we were
trying to just \-- I guess we were going to do it once and get agreement
by the company and then basically until they rescind it, we would have
that in place so we weren\'t going to specify time periods with regard
to the data.

We were just going to cover all the data on the form. We were trying to
keep it broad simply not knowing what emergencies are gong to come up or
how the data would be used or what specific data would be needed. That
may not be the best way to handle it.

I\'ll look at the 524 criteria for this hearing between BLS, BA, and
Census. I\'ll look at that some more. Currently what we do when we share
information outside of EIA on any of our current confidential, we do
data sharing agreements.

We do specify how the information is going to be used, that it will
remain confidential, what the organization will do when it puts out
information products, what we expect of them, what they will do if they
get requests for the individually identifiable information.

We do have those kind of agreements in place for our existing \-- when
we are doing sharing work with the Statistic Petroleum Reserve or the
Department of Defense. They are basically signed off by Mr. Caruso and
someone very high up at the receiving organization to try and make sure
there is a commitment on both sides recognizing the importance of
confidentiality, the importance of proper uses.

We do have the law which unfortunately is out there which says we are
going to share for official purposes which it is very broad and it does
obligate us to share for nonstatistical purposes that might be
regulatory or legal or some other type of administrative action against
a company.

We can\'t really promise that we won\'t do that because the law says we
must. We are very resistant to it. We try and handle it in a number of
different ways but we can\'t make those kind of promises.

Yet, I guess we try and avoid telling our respondents that because we
don\'t want them to know that. We as an agency have been very
successful, as you know, in fighting those kind of requests but,
unfortunately, we can\'t make promises that we aren\'t sure we can keep.

MR. WOOD: You know, we\'ve just sent forward the EIA-23 reserves forms
for clearance. We dealt with some of this issue by saying that the
production data supply would not be covered but reserves data would.

In that agreement we have clearly stated for a number of years that for
statistical purposes we share that information with the Interior
Department and the Mineral Management Service to compare reserve
estimates that they run compared to the ones that we run.

We have shared for official purposes with another Government agency I
think one piece of data on one field out of approximately 50,000 to
100,000 levels of data like that and that was done under our direct
supervision in our office under extremely high-level request.

Now, we are involved very directly in critical infrastructure planning,
part of Homeland Security national defense issues, and in a ongoing
basis preparation planning for vulnerability analysis, emergency
response, recovery operations.

Now, I have no problem at all telling any respondent of EIA-23 data that
we are doing that. But what I would like you to clarify that very
specific response. Is that a acceptable set of things to release data
for as opposed to keeping it confidential?

To share it with those committees that we are actively participating
with for critical infrastructure planning, recovery from theft, or
homeland security?

Should we just say we\'re doing that or are we actually prevented from
doing that under CIPSEA? And if we are prevented from doing that under
CIPSEA, then we would pull all reserve data out tomorrow.

It\'s actually kind of important. A\-- Is it obviously the type of thing
that we would be expected to share the data? We should just tell people
that we will share the data for critical infrastructure protection plan.

MR. CASSELBERRY: If you are going to be sharing it, I guess we would
have to look at how they are going to use it. I mean, if they\'re going
to use it to be focusing on the individual oil and gas reserve fields
and things like that, that probably might not be statistical and you
would really then have to get \-- if you had collected under CIPSEA you
would have to get some type of informed consent from those respondents
for that use.

I think we would have to look at how Homeland Security or anyone else is
going to use it. If they are going to use it to look at the broader
picture of reserves where reserves are located, how the Federal
Government would respond to some threat or some other problem, then that
would be one thing.

But if they are going to be focusing on individual fields and making
decisions or taking some actions based on that, that would probably be a
different issue. I think we would have to talk about it.

MR. KENT: Have you asked for CIPSEA protection for any of that EIA-23?

MR. WOOD: All of the prereserved data that comes in was \-- well, we
just sent it out last Friday.

MR. CASSELBERRY: It\'s at OMB. We\'ve asked them to approve it so we are
proposing for the 2003 data that we will collect in 2004 that it would
be under CIPSEA, reserves data.

MR. WOOD: And I would assume that for the pipeline infrastructure, major
refinery complexes would also necessarily need to be in that
vulnerability planning process. I can\'t remember of the 10 up there if
any of them dealt with that.

You can say it\'s for statistical purposes but there are very skewed
distributions of where the gas and oil flow and where it comes from and
that means you really need to talk about individual fields and
individual pipelines, individual nodes of the collection systems period.
What does everyone think?

MS. KHANNA: I have a very basic question. Some of the services that are
now going to be under CIPSEA historically were available in public
domain without this kind of security on them. No?

MR. CASSELBERRY: No. All the ones that are going to be were
confidential.

MS. KHANNA: Were already confidential?

MR. CASSELBERRY: Yes.

MS. KIRKENDALL: Under the limited ability we have to protect it, they
were protected as best we could.

MS. KHANNA: They will be additionally protected. That\'s all.

MS. KIRKENDALL: Right.

MR. BREIDT: Sorry to crack the whip here but we\'re out of time for this
session and we\'ll head directly to the breakouts.

Oh, do you have something quick, Polly?

MS. PHIPPS: I just want to strongly agree. I mean, the first thing you
do in a form consent is tell people how their information is going to be
used and this doesn\'t do it so it doesn\'t meet the needs of this.

The other thing I would suggest is if you\'re going to make an argument
about emergency situation, I would try to reduce the kind of
bureaucratic language and make stronger arguments in your third
paragraph and your fourth paragraph should be really what you\'re going
to protect the person on. It has to be understandable to the respondent
and I think there\'s just lots of vague bureaucratic language that needs
to be reduced.

MR. BREIDT: Thank you. So we\'ll now head to our breakout sessions. The
data edits for the EIA-906 will be held in this room and the relative
standard errors discussion will be in 5E069.

(Whereupon, at 9:12 a.m. off the record until 9:19 a.m.)

B-R-E-A-K-O-U-T S-E-S-S-I-O-N

9:19 a.m.

MR. RUTCHIK: Good morning. I\'m Bob Rutchik of Statistics and Methods
Group EIA. This breakout session is on the editing for the form EIA-920,
previously form EIA-906. Let me bring a little confusion to clarity and
explain that.

Last spring we had another breakout session. We discussed the daily
requirements and data collection models for the EIA-920 combined heat
and power plant report. In its previous incarnation the EIA-906 power
plant report. Then there were two populations for the EIA-906, regulated
power producers and complied heat and power facilities.

Now the EIA-906 will just go to regulated power producers and we now
have the EIA-920 that will go to combined heat and power facilities. In
short, we\'re going to discuss a different aspect of the same survey
that we discussed last spring except that the surveys number and its
name have been changed.

What is the EIA-920, combined heat and power plant report? There are two
survey populations for it. There\'s a monthly schedule and an annual
schedule for the CHP facilities. The monthly goes to a sample of
respondents and the remainder of the respondents will provide annual
data.

Statistically we collect data on four variables, electric generation,
fuel use, fuel characteristics of fossil fuel stocks. In fuel use
there\'s two aspects related to that. One is total fuel use at the
combined heat and power plant facility and this is the important
variable of the survey, fuel use to generate electricity at the CHP
facilities.

Now, for the editing, and I also have for those who have the handout the
draft screens for the EIA-920 online survey because this is what the
editing will apply to. This is, as I said, just a draft. That\'s not its
final version by any stretch of the imagination, I think.

We have two types of edits that will be on the online data collection
system. One is consistency checks and these are just, for example, if
you use natural gas and residual fuel oil last month did you use it this
month or this year.

Then there is statistical edits based on previous submissions. If your
generation last month was X and this month is Y and that Y is a certain
percentage, for example, greater than last month, then the respondent
would get an edit flag and have to make adjustments.

Then the second type of edit will be in the EIA-920 editing system, a
summary level edit. These are for EIA staff to use in data cleaning.
Again, there would be consistency checks, statistical edits based on
previous submissions and edits based on other company submissions.

The summary level edits are basically an analytical tool for EIA to see
where there may be problems in the data or even trends in the data. They
would be mainly at an aggregate level aggregating up all the
respondents.

There is going to be a great deal of overlaps between the types of edits
between both the summary level and respondent level edits. For example,
the respondent level could use contact information, energy source if the
respondent put an energy source that was different or not on a list that
EIA has to get an edit flag.

Also, by a prime mover type and a prime mover to give you a complex
engineering explanation is a boiler turbine configuration out of
combined heat and power plant. Then also the respondent level edit would
be at total generation.

The summary level edits, examples of where they would be applied would
be the total amount of fuel use in the combined heat and power system,
total amount of fuel used to generate electricity, total electricity
generation and stocks.

Just as a reminder again connected to what we have been doing on
EIA-906, now EIA-920, is that the substitution of the total amount of
fuel used to generate electricity is a substitute or a replacement for
useful thermal output. This is, as I mentioned before, the major
substantive change from the EIA-906 that is going to combine heat and
power facilities.

Now we want to evaluate our edits. In our evaluation of edits before
they are implemented we are going to propose a number of edit evaluation
criteria. Pardon me for no space between the D and the E. One, proposed
criteria for seeing what edits would work best would be a standard
deviation of month to month changes, calculated standard deviation and
kind of shout it along month to month, year to year.

Comparison to an exponentially smooth mean, percent change from previous
months or same month in the previous year. In other words, April of 2002
to April 2003, and the use maybe of heating and cooling degree days.
Heating and cooling degree days are a measure of how hot or how cold the
location is over a period of time relative to a base temperature. Most
commonly six to five degrees fahrenheit.

If you have a lot of cooling degree days, those would be, let\'s say, in
the summer and your prediction would be that there could be a lot more
fuel use for the air conditioning and things like that. Some of the
problems we\'ve been having was evaluating the data that we do have for
the EIA-906 \-- pardon me again if I use 906 and 920 interchangeably \--
is that we only have two years of data.

The second, data quality is not optimal. This applies particularly to
the variable useful thermal output that I have talked about that will
not be on the new survey, the EIA-920.

Finally, these are the questions we would like to pose to the Committee
members. Other suggested edits appropriate and you\'ve got a couple of
reports that we\'ve had done for us. Should additional edits be
considered.

What should the edit output look like, both the online user and the
survey, both at the respondent level and the summary level. Four, how
should edits be evaluated prior to implementation. Finally, what should
the \-- how should the edits be evaluated once the survey is actually
underway and going to respondents.

I want to thank you and turn it over to the Committee for their
discussion.

MR. BLAIR: That was quick. Anyone have any \-- I guess before maybe we
go exactly to those questions, are there any questions about the
information that was just presented to us that anyone wants to raise?
Okay, I\'ll raise one or two.

This is your third slide on respondent level edits. You mentioned
something came up that I think essentially seemed implausible, the older
range or some such, that the respondent would have to make an
adjustment. I guess one question I have is that there are true
implausible values.

I don\'t know about these particular data but, in general, there can be
legitimate outliers and legitimate implausible values. I\'m curious as
to what\'s the process or how is the edit conceived to work. If someone
comes up with a value the program says, \"Gee, this doesn\'t look
right,\" or whatever, gives them an error message, and they say, \"Yeah,
it is, in fact, right,\" can they override?

MR. RUTCHIK: I think there may be two things going on here. One,
there\'s a comment section in the survey where they can explain the EIA.
Two, I also think this is one of the purposes of the summary level edits
so EIA itself can examine these things and determine is this a real true
outlier or is this really something wrong.

MS. KIRKENDALL: But I think that the answer is the respondent in the
edits he would see he would have to be able to somehow validate, \"Yeah,
I think that\'s the right data,\" and then put in a comment. We might
ask him to describe why or if there is something unusual that\'s going
on. That kind of thing has to be worked out. We aren\'t quite there with
how the edits will work.

MR. BLAIR: I guess two things occurred to me. One is that \-- I\'m not
taking these quite in order but hopefully we\'ll cover them \-- is that
this could be one way that edits would be evaluated. That is, if you
find after a first phase, first survey that you are getting a lot of
these needs for comments, a lot of implausible values that have to be
explained, obviously respondents aren\'t going to like that but also
from the perspective of how well the edits are working is going to
suggest that shouldn\'t happen.

That is, if the edits are working are working properly, the respondent
shouldn\'t have to, it seems to me, make a lot of notes and a lot of
explanation for putting in values that they have checked and they think
are correct.

I would think that might be one criteria for evaluating once this survey
is \-- well, I guess this is at the summary level because you wouldn\'t
have all these comments until after the data were collected.

MR. HENGARTNER: I think I just want to emphasize what you were saying,
John. Numbers might have taken standard deviations away from the mean.
That doesn\'t mean that they\'re wrong to start with. My concern is even
if you identify suspicious looking numbers, you haven\'t told me how you
are going to replace them. I mean, are you going to impute?

MS. KIRKENDALL: We\'re not going to replace them. This is the respondent
who is going to be seeing them. This is for an online survey.

MR. HENGARTNER: Oh, online. Okay.

MS. KIRKENDALL: It would be a flag to the guy filling in the data form
and he can either say, \"Yeah, I believe that\'s the right thing.\" He
might be asked for an explanation.

MR. EDMONDS: Is it self-referencing? That is, you\'re looking at
outliers not from the mean of all the data but from past entries that
this site \--

MS. KIRKENDALL: Yeah, we haven\'t seen the results yet but it would be
based on his own past submissions.

MR. HENGARTNER: But how can you do that if you aren\'t allowed to use
cookies?

MS. KIRKENDALL: We have his past data and we will embed the summary
statistics of everything that we sent them.

MR. NEERCHAL: In that regard, when you say something is an outlier, we
are actually giving them some information about the rest of the data you
have collected.

MS. KIRKENDALL: I\'m talking about him.

MR. NEERCHAL: Maybe sometime you may end up saying that looks like your
way out there only related in not some \-- you know, that is somewhat
limited.

MS. KIRKENDALL: I don\'t understand.

MR. NEERCHAL: In the process of telling him or her that you are an
outlier, you might end up giving some information that you really don\'t
mean to give out.

MR. EDMONDS: No, it\'s an outlier from his own past record. In other
words, if he\'s buying gas or goal or whatever it is and he reports a
number that has one extra zero in it and it\'s like 10 times larger than
anything he\'s ever put in in the past, this is going to pop up but
it\'s not going to get a flag simply every time simply because he\'s a
small user and he\'s always two standard deviations off the mean. He\'s
not going to get a flag for that.

MR. NEERCHAL: On that point I want to mention in this regard is that
there could be another kind of inlier problem because I did this kind of
with the TRIS data, Toxic Release Inventory System data. It turns out
that some facilities are essentially reporting the same number year
after year after year and none of your outlier tests I know of can
really detect that kind of a problem.

In the case of TRIS it turns out that even if by doing that they are
actually complying with the law because they are only supposed to do an
estimate of what their release is and say, \"Hey, our production
function has not changed. Our release has not changed so this is the
same number they are plugging in.\" I think you may want to consider
that portal inliers problem. I don\'t know what they are going to call
it.

AUDIENCE MEMBER: One of the things Bob mentioned, though, was that there
were really two sets of data. One of them are the individual edits that
the company sees when they fill in the form. The other, the edits that
EIA looks at once the data have come back and been aggregated. And I
think and I\'m not sure what variables you would use but you could
address the problem that you suggested.

If the company\'s reports are incredibly consistent over time and don\'t
match what\'s going on in the industry in their geographic location or
the industry overall or looking at some economic situations, that could
raise a flag. Now, you wouldn\'t find that out each month when the
company submits it. It might take a while for that to come up but keep
in mind there were those two sets of things.

MR. NEERCHAL: Immediately you are comparing it to other records.

MR. RUTCHIK: At the summary level we had one set of data from just one
company but let\'s say other companies. An example would be, and this
gets into the type of survey this is, one of the major respondents for
buying heat power plant surveys are paper mills.

You have one paper mill that is, as you said, it\'s just constant,
let\'s say, for fuel use, but all your other paper mills have shown a
steady increase or a certain patent that\'s different from that one
inlier. I think that\'s the thing you\'re talking about, the type of
example that Stan mentioned.

MR. BLAIR: I want to come back. It may be that I\'m just not
understanding the process but if the implausible value or the legal
range of value is based on that particular respondent\'s past reports,
it would seem to me that from one variable to another that it\'s going
to take some time before you know what\'s a sensible range.

I\'m not sure sort of when you make the decision that, yeah, you\'ve
reached a point that you\'ve got a sensible range for this variable and
it may take more time to reach a point where you\'ve got enough
information to say what\'s the proper range for another value.

I just wonder, and someone who knows more about these kinds of data
would have to speak to this but that whether this is something that is
really going to be workable at a respondent level for a whole range of
variables to come up on this micro level what makes sense for that
particular respondent.

In other kinds of data, at least I\'m familiar with, in survey data not
typically how ranges are done but more sort of what\'s reasonable at
some higher aggregate level for group of similar companies, for example.
Maybe you could say a little bit about if I\'m misunderstanding or how
that would work.

MR. RUTCHIK: I think you raise a point in one of the things that we
raised about the problems with the data. We really don\'t know at this
point what is sensible for a respondent or for an aggregate because we
only have two years of data for EIA-906 to combine heat and power
plants.

MS. KIRKENDALL: What they did, they did something similar in the
petroleum surveys years ago. They had more past data so that was an
advantage they had. They would come up with what could be a different
one. They use an exponentially smooth mean at the company level.

They came up with a cutoff value at the company level. An exponentially
smooth mean absolute deviation at the company level. Then the cutoff
value was a percent of that. The cutoff value would vary based on the
kind of data field. Each kind of field might have a different cut off.

I don\'t remember but sometimes you need to worry about size of company,
too, because there\'s more variation in little companies than there is
in big companies and I don\'t remember how that was implemented. We have
a lot to look at yet with this survey.

MR. BLAIR: I guess \-- I mean, being under the situation where you can
say you don\'t have sufficient data at this point, you only have the two
years of data or whatever, but do you have a sense of sort of at what
point should you have. I mean, how much data do you need? Do you need
three years, four years, five?

MS. KIRKENDALL: Three years of data ought to do a pretty good job if the
data are reasonable. That would be one of the things we have to look at.

AUDIENCE MEMBER: We want to ask the Committee how much data \-- how much
monthly data is a sensible amount?

MS. KIRKENDALL: Except that we have no more and no matter what you say,
we\'re still not going to have anymore so we\'re going to have to do
something.

MR. BURTON: But we could make you feel bad, though.

MS. KIRKENDALL: You can\'t do this until you have five years of data.

MR. NEERCHAL: These is the kind of data where the respondents would be
\-- the inadequacies in the data are like typos and stuff like that,
misprints, or is it something that the respondent is kind of trying to
wriggle through?

I don\'t know the nature of the data that is being collected. Like TRIS,
you know, it\'s self-reported data and they don\'t really want to tell
\-- you know, they just need to comply and that\'s it.

There is a requirement that they have filed that form and they don\'t
really care that much about the quality. The person who is collecting
the data is much more worried about the quality of the data than the one
who is filling in.

MS. KIRKENDALL: I think that will always be true. The respondent
doesn\'t care nearly as much about the accuracy of what he tells us as
we do. I think most of them probably want to follow the rules and give
us the number that we ask for.

I don\'t think they have \-- most of them, at least, don\'t have an
incentive to lie to us. I think they want something that\'s easy that
doesn\'t take them much time so they can get done with it and send it
in.

MR. BLAIR: I guess on this question of should additional edits be
considered and are these adjustments appropriate and so forth, I wonder
if this is going to have to be answered over time by kind of shift and
balance between what can be done at the respondent level and what\'s
done later at this summary level.

Hopefully over time as you get better at this more can be done at the
respondent level but that early on without creating a lot of frustration
for respondents it may be that you can\'t do as much at the respondent
level as you would like.

After you have more data and have looked at more things in aggregate,
maybe you can make better judgments about what those, at least in terms
of legal right, should be at the respondent level so that shift between
those two stages of the editing may change over time.

MS. KIRKENDALL: The other issue, of course, we want something that the
respondents can use in January 2004 so we are on a really tight time
frame. That may limit what we can do right away. I think the thought
that we have to look at this as an evolving process is probably right.

MR. EDMONDS: I think at least from the respondent what you\'re doing is
you\'re giving them some self feedback because, as you\'re saying, it
ties into what you have entered in the past.

Therefore, if you only have one data point, you could probably say if
you have a factor of 10, you probably will want to take another look.
But as you get more data, you are going to get a better estimate of what
this operation looks like and the variance within that operating
procedure.

The other thing you have to think about is if they are running a gas
turbine or something like that and they swap out the old turbine and
they bring in a new combined cycle turbine and all of a sudden, boom,
you know, you take a big step down in the amount of gas that is being
purchased.

You actually need some way of once they go in there they have made a
change of state that it doesn\'t flag them for the next ten years
because they went outside the old range. That\'s one thing you might
want to think about because it\'s getting real pesky.

MR. HENGARTNER: That\'s what the exponentially moving average would do
if you lower the trend.

MR. EDMONDS: That might take it a while. What you might want to do is
once they enter a note that says, \"The reason that I\'ve only got a
third of the natural gas is that I just brought in a combined cycle
turbine.\" It\'s almost like you need to start over because your state
has changed.

MR. NEERCHAL: And I think the related issue because of this kind of
thing is how much data you need is going to kind of depend on once the
state changes, then you are starting the process over so you are asking
yourself how much data \-- how many more data points do I need on that
particular thing. So how many data points I need, I think, is obviously
a complicated question.

MR. BURTON: Since this is going to be an online survey, you could almost
incorporate a switch into the survey that if you have made substantial
changes in the technology check here and then from that point it knows
to start over. It doesn\'t have to be done manually at all. It could be
incorporated in that.

MR. HENGARTNER: One of my fears is that all these automated things is
like when you do your taxes. Then at the end of it you have run audit
check. Essentially that\'s the idea here. It flags that this number
doesn\'t jive or this one is relatively high and you might have to raise
suspicion.

And people, because that\'s the feeling or the experience they have with
self-checks, at least that\'s my experience, I wouldn\'t want to say
instead of entering an explanation and drawing attention to me, I\'m
going to put in the number they want to see.

MR. RUTCHIK: That sounds somewhat similar to he mentioned about the
inliers.

MR. HENGARTNER: Yeah, but inliers wouldn\'t care. I mean, the inliers in
his case are things that are more regular than they ought to be here.
You are actually encouraging them to do that.

MR. NEERCHAL: And that\'s what I had in mind when I asked if they have
incentive because no matter how many rules you make, once somebody knows
all the rules, then they can always make a number that fits all the
rules and still be not the truth.

MS. KIRKENDALL: You have to be a little careful. You don\'t want to have
something that flags an item every time he puts it in there. He\'s going
to try to figure out what you\'re doing so he doesn\'t have to look at
all the edits.

MR. NEERCHAL: Because the first time they are going to be really
surprised. You entered some number and immediately say, \"No, last year
you entered this much.\"

MR. BLAIR: That raises, I guess, a question that isn\'t on the list and
that is what is it that you are planning to tell respondents prior to
the implementation of this process. Again, people don\'t like surprises
and if they have been used to providing these data in one way and now
they are going to go online and suddenly, you know, they are going to
get all of these error messages and so forth, is there some thought
about sort of \--

MS. KIRKENDALL: No, I don\'t think we have thought about it much. Maybe
Bob could say.

AUDIENCE MEMBER: It exist already. We have been collecting data since
April so these people that could be filing, we already have a report of
all the forms coming in over the Internet.

One quarter of all of the monthly forms are coming in over the Internet
so we are going to be making a very large push this year for next year
to get as many people to submit over the Internet as possible. They have
been aware of it for quite a while now.

MS. KIRKENDALL: It sounds like, though, when we implement new edits,
when you go out with the new edits we might need to let them know that
we are putting in a new system with some new edits and there may be some
start-up problems and ask them to help us and try to get their
cooperation ahead of time.

AUDIENCE MEMBER: Introducing a slightly different form here we would
have to inform them anyway.

MS. KIRKENDALL: Right.

AUDIENCE MEMBER: There will be something going on ahead of time.

MR. BLAIR: I think the other part of a system like this, and this is I
guess maybe related to the evaluation once the survey is underway, is
the need to recontact some respondents.

That is, someone enters something and they put in the comment and maybe
you have to talk to them about the comment before you know since that is
typically a part of edit.

Again, this may be something and I don\'t know how much of this has
happened in the past but that certainly if people are getting a lot of
error flags and have to put in a lot of comments, there are going to be
places where you can\'t come to a conclusion of what should go into the
data set without getting back to the respondent. They, again, need to
kind of know about that.

MS. KIRKENDALL: EIA has frequent contact with respondents.

MR. BLAIR: Time is zipping by here. I\'m not sure and I hope someone
else can say something about this third point of what should the edit
output look like. I don\'t know how to begin to respond to that. I\'m
not quite sure what it is that you are looking for. That may be part of
my problem.

MS. KIRKENDALL: One thing I guess on the first one is any thoughts on
what would be the most informative way for the user if you have an
online system and you enter in something that fails, do you change the
color of the box and put a little note that pops up? That might be one
thing. Alert them that there is a problem and this is hopefully an
informative description of what it means.

MR. BLAIR: Oh, okay. I misunderstood what output was. Okay.

MS. KIRKENDALL: I think the second one is for the survey manager, what
kind of edit reports would be useful. That\'s harder, I think.

MR. HENGARTNER: That raises too much suspicion. Maybe after the form is
complete you gather all the suspicious numbers, you know, therefore
these numbers are correct. Then if he sees a gross mistake, he will
catch it.

MR. NEERCHAL: How much information do you really \-- I think it\'s very
frustrating for me if a machine comes back and says, \"What you put in
is wrong,\" but it doesn\'t give me any idea \--

MR. HENGARTNER: No, don\'t say it\'s wrong. Just say, \"Please verify.\"

MR. EDMONDS: \"Did you really mean to do this?\"

MR. HENGARTNER: Yes.

MR. EDMONDS: \"Is this the number you intended?\"

MR. HENGARTNER: Just plain, \"Is that what you intend?\" If he clicks on
return \--

AUDIENCE MEMBER: Have a really loud buzzer.

MR. BLAIR: You are suggesting that they get all of this at the end?

MR. HENGARTNER: You get this at the end.

MS. KIRKENDALL: The model is Turbo Tax.

MR. EDMONDS: I think it\'s a lot easier if you\'re going through and it
just says, \"Did you mean to put this in?\"

AUDIENCE MEMBER: There are two schools of though even inside our office.
One school is what Mr. Edmonds was just talking about which is enter a
number and right then and there it says, \"This number is not good.\"
Say you enter the content of a ton of coal, say 50. It comes back and
says, \"No.\" It does not allow you to go on until you fix that or you
write a comment.

Another school of thought is you let them enter all their data which is
actually the way the electric forms are made right now. Allow them to
enter all their data and then they can voluntarily press the error/edit
button. Or they might say, \"I\'m going to submit this.\"

They press the submit button. When they press the submit button it
automatically runs the edits for them as if they had pressed the button
just like with Turbo Tax, puts them in the back there, and then it says
very kindly, \"There are some errors. Would you like to look at them or
do you still want to submit?\"

Our philosophy was, particularly on some of the larger forms \-- this is
not one of the larger forms but we have some forms that are 12 to 16
pages long. We didn\'t want them as they were going through it say,
\"Okay. Stop here. Fix this one. Stop here. Fix this one.\"

I figure they are never going to submit it. We said let\'s get them
hooked on submitting the data on the Internet and then we can get a
little bit more script. The fact is that they can go \-- after the edits
are run they have the choice of submitting it with those errors not
being fixed at all, or they can go to the error report and click on what
the problem is.

It gives them the range, the typical range of what it would be. For coal
it might not be in particular what their coal range was but what
bituminous coal range would be. Then it tells them what line \-- I think
it tells them what the range is and what they put in there so they can
think about it all in one spot. You can go back, fix it. When they run
the error report again that line goes away. There are two schools of
thought here which are very divergent.

MR. HENGARTNER: Let me just tell you one advantage of the second school
of thought is that not only are we looking for gross outflow but
somewhere down the line there was a notion of consistency. Consistency
means that somehow the entry in one box 3 and the entry in box 7 ought
to be related loosely.

I don\'t know which one of those boxes is wrong. You start at each box
and wait, wait, wait, wait. You might actually miss the self-consistency
within the form. A good form should have some redundance. Maybe Johnny
will disagree but self-consistency is a good idea because you can catch,
you know, errors.

MS. KIRKENDALL: Don\'t you think there is some relationship between
total fuel use and the amount of electricity generated?

MR. HENGARTNER: Number of money earned and all kinds of things.

MR. EDMONDS: I think the one thing you might want to be careful of is
your vocabulary. I don\'t think you necessarily want to call it an
error. I think I want it to be a flag and you want to say it\'s not an
error.

MR. HENGARTNER: It\'s not an error. It\'s not an error. It\'s just \--

MR. BLAIR: I think that some \-- coming back to the point about
consistency is a good one and is one of the things you commonly want to
do in editing is see that not just individually variables and ranges
make sense, but that in relation to others they make sense.

I think a decision that you need to give some thought to is how much of
that you want to do at the respondent level and how much at the summary
level. It\'s fairly complicated as you\'re going along to do that
because depending on what you want to check against you may have to back
check against something that they have already entered versus checking
against something that they have. It becomes a simpler process to do
consistency checks, I think, once you have the full data.

The one thing I wanted to say on this notion of sort of as you go along
giving the flags. If you have the requirement that you have to enter
something before you can proceed, I think that goes back to some of the
things I said before, kind of encouraging the respondent to put in
something because they are stuck at that point and they can\'t go
forward if that is the way the system is set up.

It may be a number that is hard to get or they think it\'s right. If you
stop them sort of as they go along, I think that has some problems. I
think it\'s one thing to flag and say, \"Do you want to change it or
not?\" But to give them the option if they don\'t want to change it to
still be able to proceed.

Depending on how much of that there is, that can be really frustrating
for the respondent. On the other hand and, again, I\'m just kind of on
the one hand and then on another, I\'m not sure what the answer to this
is but getting all of this at the end, if you\'ve got a lot of flags and
you go through and you think, \"Ah, I\'ve just finished this.\" Submit
and it comes back, \"No, you really haven\'t. Here\'s your score and
here are the things you need to fix.\"

MR. HENGARTNER: You still can submit what you have put in there. You
look at the numbers and the numbers are the ones you wanted to enter.

MR. BLAIR: Well, it\'s unlikely to be an all or nothing. I mean, if I
get a list of 10 things, gee, the third and the sixth I want to go back
and now I\'ve got to \-- well, maybe the system makes it easy to do that
but I\'ve got to kind of look at them one at a time.

Maybe some I want to submit and some I don\'t. I would err in the
direction \-- I don\'t think there is a good solution \-- of two things.
One is in the early phase to put less burden on the respondent. Again,
this is this balance between what is done at the respondent level and
what is done at the summary level at the beginning to not overly
frustrate the respondents and try to do maybe more at the summary level.

Hopefully over time as you get more information you can do a better job
at the respondent level. As far as some of these issues, there probably
isn\'t an answer to. This won\'t give you an answer either but it will
give you more information, and that is to talk to respondents about it.
I mean, what do they prefer? It may be that there is more agreement out
there than you might expect as to how respondents want to do this.

MR. NEERCHAL: Some of this ID types of things like address and things
like that, I think you can immediately tell them that, \"Last year you
said you live in Virginia and now you are in Maryland.\" People
wouldn\'t mind getting that feedback right away and get it out of the
way.

But, on the other hand, something sensitive kind of information maybe at
the end laid on them slowly saying that, \"Based on the last year\'s
data your data seems out of range,\" or something like that. Some very
nonthreatening language.

MR. BLAIR: Kind of in line with that, in other types of data there is
this notion of sort of fatal errors versus others. You don\'t want to
let the person go forward with the wrong state or something. There may
be certain variables that you really do have to stop and say, \"Wait a
minute. This doesn\'t wash.\"

Hopefully not a lot of those but there may be some. The criterion is
would this respondent\'s data be usable if this error or if this
remains. That is, you know, would it render that data element unusable
for analysis or very problematic for analysis. This is a common kind of
thing that is done in other sorts of edits, particularly with business
surveys to have these critical or fatal errors that have to be dealt
with.

MR. NEERCHAL: Another question, I think, does it slow down the process
since you have been doing it for a few months now? Does it slow down?
They might be doing it on the Internet, you know, Internet connection
speed that they are using and things like that may be factors as a slow
down.

AUDIENCE MEMBER: We have about \-- I\'m not just including this
particular form but all the monthlies and the annuals I figured the
other day that our division processes 39,000 forms a year, 15,000 of
them coming in over the Internet. That says to me that people are
anxious to get it done that way and get it out of the way.

The edits actually help them because then we don\'t call them back if
all is fixed. Now, it might induce them to make up data. That\'s a
possibility. I hadn\'t really thought about that. If they drop with
errors and they don\'t correct them, we will call them.

Our goal and their goal is for us to never speak with them. We want them
to give us good data the first time through and we\'ll just run with it
then. In fact, I was just asking yesterday my staff of the ones that
have submitted it over the Internet, this was across the forms, have
they fixed the edits. Have they come back with no error or problem. They
said certainly over 90 percent, if not more than that, come back clean.

That\'s very encouraging to us. In fact, next year we are hoping to have
a whole lot more come in over the Internet and then the idea is to save
us time and save them time and call-backs and improve the data, get it
out fast and all the good stuff that goes with it.

MR. BLAIR: Okay. I think, if I\'m reading the schedule correctly, we
don\'t have very much more time. Are there either on these questions or
things, you know, that you want us to make any additional comments on?
We have kind of jumped around.

MR. RUTCHIK: The fourth bullet I think we haven\'t really discussed all
that much. I would like to get your ideas on that. No. 4, the system,
the edits are implemented. Do you have any other ideas for evaluation
criteria?

MR. EDMONDS: Are you asking about before you put this system into effect
how do you decide go or no go?

MR. RUTCHIK: Yes.

MR. EDMONDS: I think Johnny\'s comment earlier was a really good one
which was one of the things you want to do is go out and ask people. Get
some data testers and have people try it and get some feedback from
them.

MR. BLAIR: Other comments, anyone on the Committee? Okay. I\'m not sure
how we\'ll summarize this. I will ask when we get to that point if
people on the Committee if I neglect to repeat one of your points, it\'s
because I wasn\'t so good at writing and listening at the same time.
It\'s not that I intended to. Feel free when we do the summary to jump
in.

MR. NEERCHAL: This is probably obvious but for the last bullet what you
mentioned just now is an evaluation of how the edits are helping. I
think it is too obvious a point.

MS. KIRKENDALL: What was that?

MR. NEERCHAL: I think what he said about 90 percent coming error free, I
think that is already an evaluation of how the edits are working. You
keep track of that number over the life.

MS. KIRKENDALL: That\'s a good statistic.

AUDIENCE MEMBER: I made up that number.

MS. KIRKENDALL: It\'s a good statistic. That\'s one that the system
could probably produce pretty easily once we decide that the system
should produce it.

(Whereupon, at 10:05 a.m. the breakout session was adjourned.)

MR. BREIDT: First we will summarize the relative standard error breakout
session. Bill.

MR. WEINIG: We already did the summary.

AUDIENCE MEMBER: No, we didn\'t.

MR. WEINIG: No, we did not. I\'m sorry.

MR. FEDERER: First of all, what is the relative standard error. It is
the ratio of the standard error divided by the estimate when the
estimate is positive. Indeed, that particular statistic is used mainly
when the estimate in question is one that tends to be positive such as
total.

The questions to the Committee were, if I can get my slides here, what
guidance of users of RSCs is needed for serving managers. What guidance
on RSCs and how to use them are needed for internal and external data
users. And what experience have members had with RSCs and other
techniques.

RSCs are also commonly known as CVs and they are widely used to judge
the quality of estimates. It\'s a bit problematic when you have certain
types of estimates. For instance, if they tend to be close to zero
because then the estimate might be very large. Useful, though, for
instance, for monitoring quality of data to detect outliers. Actually
this is one of the users that had been in mind by the presenters.

We talked about some alternatives to using that. We noted that some
users of data take estimates as the true value. Then they look for even
some statistics that you can get from the table which are not actually
published. They take differences of estimates and look at it as the
estimate of change and so on.

The Committee suggested to use RSCs for editing data to monitor for
outliers. Not just high RSCs but also too low RSCs. One way to detect
such outliers would be perhaps by using some modeling such as
generalized variance functions and then automatic detection of an
outlier if the observed value of the RSC is very different from the
predicted one from a model.

There was one mentioned by a Committee member that in the case of time
series data variances are not the only thing that one should look for
but also autocorrelations in the series. That has to be also looked at.
Some problems related to RSCs, and I\'m sorry for not going in the order
the discussion was. The proportion that is being estimated is very rare.
The RSC can be very high.

Then there was a discussion what to publish and how to flag those
estimates that are poor quality. Many committee members suggested that
instead of publishing point estimates and either standard errors or RSCs
publish confidence intervals. That has a few advantages.

One of the advantages is that if there is estimate by itself doesn\'t
provide enough information about the quality if the sample size is small
and, hence, the degree is very low. We know the confidence interval is
not just two times the center on either side.

Moreover, this is based on normality assumptions and certain estimates
such as proportions are highly normal. We suggested to the EIA members
to perhaps publish confidence intervals.

There was a certain concern about it by EIA members and others that
it\'s maybe going to double the information and people are going to take
the average of the 10 points anyway. In addition, some people will not
appreciate what it means and in some ways have to deal with that was
also discussed.

MR. BREIDT: Thank you.

(Whereupon, at 10:14 a.m. off the record until 10:45 a.m.)

MR. BREIDT: Okay. I guess we\'re set here. We have managed to reassemble
so I\'ll turn it over to Doug Hale.

MR. HALE: Okay. I\'m not quite sure whether I can close this thing up.
Just a couple of things before we get started. One, as some of you know,
I\'m an EIA employee but I\'m spending a year at Lawrence Berkeley Lab.

The idea was that I would finish up this little project in a couple of
weeks and then go on to other things, things that were a little
different than what I was doing here and the hope being maybe I bring
some different contacts, different kinds of methodologies back to EIA.

Sometimes you get into a project and you think it\'s a little tough and
you get in a little farther and you think, \"Gee, this is getting really
tough.\" I guess the long and short of it is I\'ve been on several
projects but of all of them this one has been the most difficult. There
is a combination of technology, regulation, history, limited data
collection, and public policy questions that all seem to be coming
together to make sorting through this very difficult.

We have a first draft of the report from which I\'m talking has been
sent around. We were very fortunate that there were many people who are
deep into tough love and have pointed out mistakes, misunderstandings,
omissions, that sort of thing. We are busily working on a second draft.

In the past I have been rash and made forecasts as to when the next
version of things were coming out. I have learned through experience
that I\'m not going to do that. We are hopeful soon, but don\'t pin me
down on that.

What I would like to do today is talk about this project and share with
you kind of our approach and the sorts of things we\'re finding, how we
are kind of breaking at it. I am also looking for your suggestions,
especially in one area, and that has to do with the huge amount of data
that is out there that isn\'t controlled by the Federal Government
directly but is produced under Government auspices.

That\'s the stuff put out by the independent systems operators. Which
we\'ll get to in a minute what they are. Primarily they have Internet
websites. And also some information from the reliability folks, NERC.

The purpose of this project was to assess the Federal Government\'s
needs for transmission of information. For information basically we are
needing data, models, analytical tools. We are trying to suggest changes
to the data collections that might better meet the needs of the
Government.

This talk describes, first, basically we\'re trying to see if the data
can help us answer questions of federal interest, to try to link the
kinds of data needed to the questions that are being asked, identify
specific unmet needs. I think they are ending up with one suggestion for
an evolutionary approach for meeting many of the needs, the unmet needs
we\'ve identified.

There are a lot of reasons to collect data. For the Federal Government
the bottom line often times enforced by OMB is that we collect the data
because of the federal interest in it. If the data collection doesn\'t
have a clear federal interest, it\'s going to be very difficult to get
approval for the collection.

In the area of transmission, the main concerns have long been \-- have
been around for a long time reliability and national security, economic
regulation. Economic regulation is extremely important because the
transmission system which is interstate is primarily under a FERC
regulation, Federal Energy Regulatory Commission.

FERC has for a long time used a cost of service approach to regulation
and has had voluminous data collections to assure itself that costs are
reasonable, that expansions are occurring at the right times, and also
to assure ourself that the systems are doing adequate planning for
reliability.

The third area, and probably the one that the Government first got
started in in a big way is economic growth. In particular, regional
economic growth. That\'s what TVA was all about. That\'s what Bonneville
was all about. That\'s less of a concern certainly on that scale now.
There is a lot of concern will the grid be able to support the growth in
the southwest, for example, industrial centers, that sort of thing.

Now, talking about these needs is just fine but you have to do it in the
context of realizing there\'s just a huge amount of numbers out there
right now. We\'re not starting with a clean slate. We\'re not saying,
\"Here is this new industry. What do we really need to know about it?\"

EIA and FERC are the two dominant, or predominant, federal data
collection agencies that have information that is transmission relevant.
There is also a great deal of information held by these independent
systems operators that were created by FERC\'s regulatory folks. I\'ll
tell you what they are in a second. And the North American Electrical
Reliability Council, known as NERC.

Nevertheless, even though there is a lot of data and it\'s in a lot of
different hands, there\'s still some kinds of data which are important
which just don\'t seem to exist anywhere. We\'ll get to that.

Now, the basic factual questions or the kinds of questions you\'re
thinking should be answered by the data that are responsive to the
federal needs are such things as how reliable is a grid? Is it improving
or deteriorating? What does transmission really cost? What are the
revenues, prices, and returns associated with transmission of investment
and transmission? The regions face similar costs. There are some regions
that are very high cost or low return, low cost, high return, whatever,
and what is the reason for that.

In the area of economic regulation, one of the big concerns with FERC is
whether the grid is shielding firms with market power from competition
and is the grid accommodating economic trade. Another area of great
interest now, especially the Department of Energy, is whether the
appropriate investments are being made to expand and modernize the grid.

There is a lot of concern that there are no incentives now to invest and
that the American grid is obsolete. I don\'t have an opinion on that.
I\'m just saying those were the kinds of questions. You\'ve got to be
able to go to the data in this context to get some handle on it.

Now, having said that, I want to say that many of these areas of
interest the Government has had for, like I said, 70 years in the case
of economic growth. Conservative reliability has been around for a long
time. Especially pronounced after the blackout of \'65.

We have been regulating this industry economically for another 70 years
as well. Why are we looking at it again? I just want to spend a second
on that because it\'s important.

What has happened is FERC is trying to bring competition into electric
power and when you change the nature of the business, you change the
nature of the data you need, you change the nature of the whole thing of
where can you get the data. There is data now that we want such as
market prices but no individual firm can actually tally. We need
something like a market to do that.

One of the big things they did was to unbundle transmission and energy.
They are separately priced. They also want to have transmission treated
as a stand-alone unit so the transmission now stands on its own from
generation to distribution.

They are trying to use markets to price energy so that\'s the
competitive markets to try to price energy instead of using cost of
production. The transmission tariff which FERC controls will now under
their standard market design and other proposals be based on regional
capital recovery and cost and not just the capital recovery of each
little individual firm.

One thing that we\'re going to talk about a bit, too, is that now
congestion is being priced using market prices. Congestion is when you
can\'t get power through the lines to a customer so there is a price, a
social price, not being able to do that, or cost of doing it. They\'re
actually trying to price it. They\'re trying to make it explicit and
we\'re using market prices to do it.

They are now looking more and more at regional planning for grid
expansion. The expansion wouldn\'t just be something that companies did
merely by themselves. I can\'t emphasize enough how important this
congestion pricing is to the data and also to the kind of FERC\'s
approach actually pricing congestion, you know, the lines is new for
transmission. It has not been done before.

These congestion values are kind of the cost of not being able to get
power through are very important for figuring out where you should site
your generators and for guiding investments. You would guide
investments, presumably, to try to relieve or reduce congestion cost.

The ticker in all this from just a pure theory point of view is that
this approach makes a huge amount of sense so long as the prices that
you have coming out are competitive. If they\'re competitive, economists
are not going to go crazy about the way you are trying to use this
information. If they are noncompetitive, then the argument for doing
this is not overwhelmingly good. It\'s not overwhelming is the way to
put it.

Where structuring over impacts have been that we have had major changes
and world powers flowing. Lines, big lines, that used to be underloaded
are now loaded to the gills. Lines that were loaded to the gills are
not. Moreover, who is overloaded and when changes in ways that have
never been seen before. It changed quickly, presumably in response to
market forces. Presumably.

The other thing we have discovered is market power and congestion are
really common. Going into the restructuring of the New England area,
which is where they now have an independent operator, systems operator,
people that have performed construction said, \"We don\'t have a problem
with congestion.

We have overbuilt the system so much that you can get through as much as
you want.\" That was fine until they started looking at the numbers and
they actually put in a system so they can see if there was congestion
and how severe it was.

They discovered it was always there or maybe it\'s just because of a
change in patterns the systems are much more congested than previously
appreciated. In the case of market power, I don\'t think I have to
remind you much about what happened in California. That one is still
reeling from that one.

As I mentioned before, it\'s very difficult when you have an industry in
transition like this where the rules for proper recovery and investment
are so fluid to encourage a whole lot of investment. That seems to be
going on. There seems to be some slow down.

The other thing is that we now have at least three regulatory regimes
for electricity which means you\'ve got three different approaches to
information as well as everything else. First, in New England, New York,
Pennsylvania, Maryland, New Jersey, you have three independent system
operators.

They are trying to \-- they have established formal explicit markets for
electricity related goods such as reserves. They have prices,
quantities, trade data that is available to anyone on their Internet
sites.

They are the farthest along by general agreement in the direction of the
competitor restructuring that the FERC, they\'ve been looking for.
California is still recovering from the disaster and they have about
given up on the idea of an independent systems operator but their design
is still evolving, I think.

Then you have the great state of Texas public power cooperatives and
municipals which seem to be primarily under the federal state
regulation. ERCOT which, again, is mainly Texas. It\'s not under FERC
regulation. It\'s not interstate commerce set for very small pieces so
they are not regulated.

The remainder of the country which includes the southeast, northwest,
most of the midwest, is essentially still under something similar to the
old cost of service regime. In terms of the actual data they operate,
clearly there\'s competition. There are also some generators popping up,
merchant generators. The regulatory structure is still pretty much of
that store.

What we did was we said, okay, the main areas that we\'re concerned
about are reliability and associated deliverability of transmission. We
are concerned about the cost revenues, investment that are associated
with transmission. We are concerned with whether the grid is actually
being expanded and developed and how that\'s done.

Finally, we wanted to know something about how competitive the markets
are for energy. Again, the reason that\'s important for transmission is
because if those energy markets are not competitive, then the way you
price congestion is hard to defend and so that is why particulars put so
much stress on competition and market power.

What we did is go through each area and I think some of the Committee
members have seen a rough draft or a draft of the reliability section.
We\'ll try to lay out the area, work through what\'s important to know
about it, work through the kinds of data you need and the data that\'s
out there to answer these kinds of questions and then come up with some
conclusions about the availability of data and the gaps.

When we did that, we found that the data needs seem to fall into certain
classes. First, you need electrical models of regions and interconnects.
Interconnects are just big, huge regions. There are three of them in
this country. You need that in order to confirm reliability of
deliverability. You need it to quantify the effects of investment. You
need it to establish market boundaries.

Now, if you are looking at diesel fuel or something like that, you
don\'t need a model of the transportation system to kind of figure out
what\'s going on. Electricity you really do because the capacities do
not add up. The fact that you have trends that seem to be strong
doesn\'t mean they will continue. Very small changes far removed from
the areas you are interested in. It can make it impossible to either
send or receive electricity. You actually need a model in order to
establish some of these things.

In particular, you can\'t say how much more power can be delivered from
Southern California to Northern California if this congested corridor is
upgraded. You can\'t do that just by looking at the causes and the money
involved. You need these kinds of models.

The other thing you need is data clearly on transmission cost, revenue
investment. FERC needs it for rate determination. You need it if we are
going to confirm that these regulated transmission entities are
financially sound. And you need it to monitor replacement modernization
and expansion of the grid.

There are two other general areas where you need data. You need
information on the energy markets and how well the system is operating.
From first point of view the main thing is to detect market power, but
you also want to see what the values of congestion are. You want to
identify low pockets where certain generators exercise a lot of market
power. And you would like to be able to monitor reliability. For that
you need your market or operational data.

Finally, there\'s public policy interest in how are the actual
participants, the NGE\'s participants, affected by transmission. To do
something on that you need information on generators, how many there
are, where they are, what they are like, that sort of thing, to assess
competition and also to monitor transmission access and the quality and
cost of transmission.

I think what we generally found when we went through all of this is
that, first, there is just a huge amount of data or data elements out
there, but often times to use the data that is currently available to
answer questions in the context of a new kind of regulation that FERC is
doing.

In the case of reliability investment and market boundaries, we really
have very limited \-- the Government and its contractors have limited
ability to analyze these things independent of the power companies
themselves. Part of the reason for that we\'ll get to.

Financial analysis of the grid as a stand-alone entity, and transmission
is a stand alone entity, just wasn\'t possible. The data simply is not
there. This is traditional finance analysis. It\'s also extremely
difficult to make regional cost comparisons to save that it\'s a lot
cheaper in the southeast than it is in the midwest for something like
that.

A huge problem is that market data prices, quantities, imports, exports
simply don\'t exist outside of the northeast and, to a lesser extent,
California and Texas. Although there is a lot of suggested data, it\'s
hard to get tight estimates of what users are paying in terms of the
cost or what quality of service they are actually getting and what risk
they are taking.

Now, that\'s not such great news but it doesn\'t mean that there isn\'t
a whole lot that might be done. I think what can be done tends to fall
in three areas. One, simply extensions of the framework we now have. We
will the establish the framework we have now for collecting information.
I want to talk about that first.

FERC, if we are going to be able to address these kind of questions with
policy information or data that\'s available for Government policy
analysis, FERC is going to have to do some things that are hard but they
are still consistent with what they\'ve been doing for years. I\'m not
trying to minimize that.

If FERC fails to act, then I don\'t see how you make great improvements
in our analytical ability to analyze and describe these industries. The
first area is model data and what we suggest there is to vastly improve
the recording of the FERC 715 to add things like contingency list to
identify generators and planned facilities in such a way that they can
be linked with other sources.

In particular, EIA. The 714 has potentially very important data on the
power flows, some of them being trade, some of them just being pure
electrical flows in various regions within the country.

That data could be extremely important for assessing competition within
the various regions. The data has not been very reliable in the past \--
I don\'t want to say that. There are a lot of reporting problems with
it. It would be useful to have those cleaned up to see what the flows do
look like.

MR. EDMONDS: Just a quick question. What is a contingency list?

MR. HALE: The transmission system is not run at its physical limits.
Instead what is done is they do a series \-- regions do a series of
analyses, what-if analyses. If this line went out, would the system
continue to work. If this big generator went out, would it continue to
work. Then they come up with upper limits on how the systems would
operate.

You always have \-- always. Saying always in this industry is nuts.
It\'s very common that you can look at lines and say, \"Holy cat, you
can get a lot more power through there.\" In electrical models, yeah,
you can get a lot more power through there but it doesn\'t happen
because of the fear that if you did that, an alarm somewhere or a
generate went down, you would burn up the entire system or it would
collapse on you.

One example of that is when we\'ve looked at the power lines going into
Florida and done electrical analysis of that, it looks like they could
in principle physically move a lot more power than is actually being
moved. But, again, it\'s clearly some sort of contingency or concern of
that sort.

FERC does not maintain those lists. It does not let you know what they
are so it\'s very difficult to go from the model to, say, to use the
model to make realistic assessments in what the grid can do. That\'s
all.

The final one is that if we are going to get a handle on what the
financial situation of the transmission system is as a stand-alone,
we\'re going to have to know what the grid\'s actual cost, actual
revenues and investments are. We have a fair amount of information about
the grid cost on a stand-alone basis.

We have a fair amount of information about the grid\'s investments. The
revenue is very murky. It is just incredibly murky. That has been kind
of a bugaboo. In order to get a better handle on that, I would think
that FERC would have to unbundle the FERC 1.

The FERC 1, if you have never seen it, is like this. I think it\'s 200
pages. That is an impossible thing to do. It\'s not an impossible thing
to do but it\'s asking them to relook at the standard system of accounts
so that is a very, very difficult thing to get an organization to think
about.

EIA by itself, however, can make some contributions, in our opinion. In
the area of investment and grid expansion, we have fabulous data on
generators and we know when new generators come on line and connect.

It would be very useful to know the direct cost, how much it cost these
guys to actually connect to the grid and how much time it takes. We know
that there\'s a lot of entry in the industry. This gives us some handle
as to the cost of entering in the industry in various locations.

Another area where we can make a contribution is merchant transmission.
Right now merchant transmission has to go to FERC for rate approval,
upper level to approve their approach to rates. Merchant transmission
does not report investments or the projects, electrical care, that sort
of thing, and that would be useful to complete the balance there.

I want to say one other thing. On merchant transmission, this is very
small. We\'re talking about right now half a dozen projects. It\'s
small, it\'s potentially important, and it\'s more to completeness.

In the market data it\'s happening very slowly and is very restrictive
right now is that we are finally getting some limited demand response to
increased prices. People are getting \-- not people but industries, big
companies, are now being metered in such a way that when the price is
high they can cut back and when it\'s lower they can use more. There are
a considerable amount of experiments including New York, Niagara Mohawk,
which has been doing it for 15, 20 years.

There is actually some of this being used in the country according to
the ISO websites. It would be useful to get a handle on how much demand
really is needed to allow that to potentially happen and get some idea
of how much of this is also potentially actually time of consumption
charge of some sort.

Again, this isn\'t something that is going to make a big difference now
but overtime if these things develop the way economists dream that they
will, and who knows, it could be very, very important for understanding
the operation of the grid.

The other thing that I think would be very useful would be to have some
information on generator transmission quality and cost and also the same
sorts of things from the low-serving entities. It would be very useful
to know what percentage of generation is under long-term contract. There
are a couple of reasons. That tells you something about the availability
of service in a way.

The percent of firm transmission also helps. The particular thing about
long-term contracts is there is a lot \-- there is a theoretical and
empirical research now that seems to indicate that if you have a system
where you have a lot of long-term contracts, a lot of your output is
under long-term contract, the incentives for the generators to go ahead
and jack up prices is much, much reduced than what it is now so that
these long-term contracts in a funny sort of way, a subtle sort of way,
have a way of inducing more competitive behavior than we have without
them.

That isn\'t to say that long-term contracts are going to lower your
cost, that you could always do better by buying long-term or short-term
or something like that. It simply says once a generator has itself a
number of these long-term contracts, their incentives are different.
Finally, it would be useful to know what the transmission charges really
are by the kinds of generators and similar things for the load serving
entities.

There is a huge mine of gold out there at the ISOs and in the NERC
regions and FERC controls the keys to the mine, if you will. FERC has to
approve ISOs. The ISOs report to FERC, if you will. NERC is an
independent organization that at last count but there is a lot of \--
there has been legislation proposed to give NERC absolute authority over
making requirements of the industry and putting them under FERC
oversight, whatever those words mean.

In the case of the ISOs, I think FERC would do us all a favor by
standardizing some of the ISO data reporting requirements and archiving
the data so that as special needs arise, including FERC\'s needs for
marked power studies, that data could be accessed.

I also think that if they would simply consolidate and archive files
they already have such as OASIS on things like the availability and
approval of contract requests or transmission requests, that can also be
a very important source of information for analysis.

In the case of NERC, NERC is about the only one that really has, I
think, the power flows, line loadings, curtailment data and information
dealing with reliability metrics and tools. Typically what happens now
is every time there is a blackout, and I know because I\'ve been working
with the Transmission Office, the Government pulls in a bunch of its
experts and everybody goes scurrying up to Princeton, New Jersey, and
spends months with NERC\'s staff trying to figure out what in the heck
happened and also interviewing cases all over.

It\'s a very reactive sort of approach. What I\'m suggesting here is
that perhaps FERC and NERC where they could get together a little more
and think about what we need to give us a better handle where we are and
also to allow us to analyze these things better more efficiently after
the fact. That would be a great help. We don\'t have very specific
recommendations for that at the moment.

We go through all this but in terms of areas outside of the ISOs you
still would be missing the big pieces of information. They mainly have
to do with the markets and trade, less to do with reliability, cost,
user impacts. The fact is that if you don\'t have markets it\'s hard to
have wholesale market prices. The markets they have are informal. I
mean, they are markets but they are not the formal markets where you can
easily look it up.

Outside of the ISOs there is congestion that is getting recorded more
and more. How to value it since you don\'t have market prices, again, is
a real puzzle. Avercha has tried to do it. I actually helped them. Tom
actually helped them but these are very rough, rough estimates of what
the cost congestions are.

Similarly, as you get closer and closer to markets, I believe you are
having \-- we are having trouble finding trade flow data that we think
is \-- well, it appears to be quite accurate and timely, that sort of
thing.

So I think overall what we are finding is that there is a huge amount of
data out there. The data that is available is useful but that I think we
could improve the utility of the data quite a bit in three ways. One, we
would have to extend some of our existing data collections. I say that
and it sounds easy but it\'s not. FERC would have to do an awful lot of
work in particular.

Second, I think FERC could do us all a big service by organizing or
coming up with standard displays of the ISO data that would then allow
analysis of the ISO and its neighbors.

Finally, there are these areas that even if you do all that, there are
some areas outside of the ISOs where the market data just isn\'t there
and if you\'re going to do something about it, that takes some real
research into how you come up with good proxies for market prices in
these informal markets. That is something statisticians I\'m sure can do
but it\'s not trivial and it\'s not something to solve over night.

That\'s kind of where we are in looking at the industry, looking at the
kind of questions that we think you would want the data to ask that
respond to federal interest. And looking at the data collections that
are out there and how they might be developed to better meet your needs
for information.

That\'s it if there are any questions, complaints.

MR. BREIDT: Thanks. So I\'ll turn it over now to our formal discussants
and we are going to depart from the agenda a little bit and Mark Burton
will discuss first.

MR. BURTON: Cal and I were talking this morning. I\'m getting old and I
can\'t remember how many years I\'ve been coming to these meetings but I
do remember the first assignment I ever had with the Committee that Doug
was working on at the time. I remember feeling wholly inadequate. I\'ve
been around here enough now and I feel comfortable with the group.
However, I still feel wholly inadequate.

For those of you who have the opportunity to read the material, there is
a huge amount of information there and it is very compelling work. I
want to do sort of three things with my discussion. I\'ll try and
summarize as well as I can my take on what Doug has uncovered.

I want to make a comparison between the electric utility industry and
restructuring there, the comparison between that and what happened in
telecommunications because I think from an economic standpoint a lot of
what has motivated restructuring in the utility industry has been the
happy outcomes attained in telecommunications. I think there are some
important differences and maybe some lessons, too. Then lastly sort of
try and suggest in my opinion what all this may mean to us as we move
into the future.

The fundamental point Doug makes very early in one of his documents is
that reliability is still the obligation of the providers. Restructuring
has not changed that. What restructuring has done, though, is
fundamentally alter both loads and flows. Electricity is moving places
in volumes and in directions that nobody anticipated as the system was
designed or as it has evolved.

Which means that we need to be able to evaluate a lot of things that
we\'ve not seen before and, as he aptly described, the data either are
not consistent or not compatible or don\'t exist and, frankly, my
suspicion is because the data don\'t exist that the overall
comprehensive models probably don\'t exist either so that both from a
modeling side and a data collection side there is a great need.

What Doug doesn\'t say is that in my view, at least, the great changes
that we are observing in association with restructuring mean that events
like last August may be more frequent than they have been historically.
Again, a lot, I believe, of what the motivation or momentum behind
restructuring is attributable to what we have seen in
telecommunications.

Now, in 2003 finding an economist who will admit that he or she thought
divestiture in telecom is a bad idea is about as hard as finding an old
Hippie that will admit they weren\'t at Woodstock. Everybody says, \"We
said all along this is going to be a great thing.\"

The fact is if you go back and you read the literature from the early
1980s, there was a lot of dissent and a lot of naysaying. One of the
principal complaints was the impact that restructuring and
telecommunications was likely to have on reliability.

The argument was a competition wouldn\'t provide enough revenue or
sufficient investment long distance and that without the cross-subsidies
that were occurring, there wouldn\'t be enough money for investment in
local exchange service. There was a great deal of concern. It didn\'t
happen, however. In fact, there is compelling evidence that service
quality and telecommunications has improved.

However, telecom and the electric utility industry are very different.
At the time of divestiture \-- well, let me back up. Restructuring in
telecommunications has come primarily through one order from a federal
judge in 1982 and one federal statute, the 1996 Telecom Act and is
applied universally coast to coast.

You haven\'t had sort of this hodge podge of varying legislation from
one state to another. At the time of divestiture in 1984 there were
eight firms that controlled well over 90 percent of all
telecommunication services. In 2003 you still essentially have eight
firms, albeit a few different ones now, that still control the vast
majority of telecommunication services.

Doug, what would you say if you looked at generation and distribution
and transmission, how many important \--

MR. HALE: It would have been six, I\'m sure.

AUDIENCE MEMBER: Traditionally we still have about 3,300.

MR. BURTON: So if you make the comparison of having eight providers
versus 3,300 relavent participants, you can see you have a much
different situation. The asset lifes in telecommunications, the
dollar-weighted asset life on average for most telecom providers is six
years.

I don\'t know what it is in the electric utility industry but I have a
feeling it is considerably longer than six years. Issues regarding past
investments and the ability to recover necessary revenues for those old
investments don\'t really matter in telecommunications. They do really
matter in the electric utility industry.

In telecommunications new market participants traditionally have entered
simply by reselling the excess capacity of incumbent firms so that you
didn\'t have new firms coming in with a lot of new capacity. Eventually
a lot of that emerged but they came in reselling incumbent capacity as
opposed to developing their own networks through build down.

I think from our standpoint maybe the most important thing is that while
the public utility commissions and telecommunications certainly have an
important role to play, there is one single federal entity that still
rules the roost without much question and that is the Federal
Communications Commission.

Particularly from a data standpoint that\'s where the data live and as
an analyst it\'s wonderful because it\'s one-stop shopping. When you go
to the FCC website if it exist in the world, it\'s going to be there.
The challenge that we face in trying to assess what\'s going to happen
under various scenarios within the electric utility industry is much
greater than it was with telecommunications. I think the expectations in
terms of outcomes are probably much different.

The last few comments I have I have under the heading of what can,
should, will happen which is kind of the world according to Mark. My
guess is that the experience in California and the experience with the
blackout in August probably will slow the process of restructuring to
some degree.

I think we\'ve seen that since California and then what happened in
August will probably reinforce that. That may not be a bad thing. We may
need some time to take a breath \-- catch our breath and take account of
what\'s happening.

There has been a considerable amount of consolidation and I expect that
is likely to continue. As an economist, consolidation troubles me
obviously because of its implications on competition. However, I think
from a managerial standpoint in terms of managing the system and
collecting and analyzing the necessary data, having fewer rather than
more firms might be a helpful thing.

Finally, and, again, Cal and I had discussed this a little bit this
morning, I feel like that consolidating the authority for data
collection and dissemination into a single entity may be necessary. In
any case, the sort of additional data means that Doug\'s work points
very compellingly toward is going to require \-- probably require at
least statutory changes in reporting requirements.

I hope I didn\'t do your work too much of a disservice. If you all will
excuse me, I\'m going to go catch a train.

MR. HALE: Mark, thanks very much. I think those are useful comments. I
would really have liked to have been able to come here and say here\'s a
data model for this industry. You fill in these little things and, boy,
not only do you describe the industry, you know, which is a typical sort
of thing to do, but you can also really get right at many of the \-- you
know, the heart of the policy questions.

I\'m just finding \-- we\'re finding maybe there is such a model but we
haven\'t found it. It looks to me like the data is about as fragmented
as the industry and what you really have to do is just try to work with
what you\'ve got and make a few, well \--

MR. BURTON: Big challenge.

MR. BREIDT: Thanks, Mark.

Now we have our other ASA discussant, Jim Hammitt.

MR. HAMMITT: Thanks. I was going to echo some of what Mark said and I
don\'t think you should feel inadequate at all. That was masterful.
Electricity transmission obviously is a complicated situation. I don\'t
know much about it. When I was asked to do this I said sure because I
know Doug is always involved in some interesting and intriguing problem.
I think that presentation just illustrates as usual how clearly you
think about this and can lay it out for us.

As I was saying here, this is the NERC map of the regions in the
transmission grid, I guess, in this very room so we can see right here
some of the complexity in the fact that these interconnects aren\'t
really separate. There are connections between everything everywhere
that matters as we know.

Let me just try and say a few things different from what\'s come up
already. In terms of determining what kinds of information are needed, I
was wondering one way to go about that is you\'ve got a bunch of
questions and we don\'t know exactly all the policy questions of
interest but we know some of them.

We know that to address any of those it\'s going to require modeling
because of the interconnections matter. One idea would be either some
Government agency constructs the model that is used for answering all
these questions or that can be used.

And by doing that you figure out what data are needed to drive that
model. Of course, a related idea is not one Government model but sort of
promote the establishment of these kinds of models in several
organizations.

I\'m sure Doug is aware of the energy modeling forum at Stanford which
is a venue through which people with models who do similar things can
come together and compare their results and learn from each other. EMF
may already be in this area for all I know.

The down side to that is sort of driven very much by the small or the
set of models. If you fail to think of some relevant questions, you may
not collect data useful for those questions you didn\'t think of yet.

Another approach would be a much broader thing like look at this as a
general industry or set of industries and try and list all the kinds of
data you could imagine collecting and just be more about trying to be
comprehensive grab bag with less up-front thinking about exactly why you
would want different elements. That is sort of in the nature of two
general ways to approach the general problem of what kinds of
information to collect.

Another issues that comes up here would be the data sharing which we
talked about earlier this morning. It sounds as if at a minimum there
will be needs to share data between EIA, FERC, and then NERC which is
not part of the Government, I guess, but maybe will be. Presumably some
attention needs to be given to how that can be done and confidentiality
and all this other stuff.

In terms of reliability, one idea I had which, I guess, it sounds like
some is already done, is since complete system failures are fortunately
rare, we want to be aware of sort of precursors to failures.

I guess there is already in the EIA form on energy incident or emergency
incident and disturbance report. I\'m not sure if the things that are
reported there are a good set of things to be looking at. People with
knowledge of liability issues might have other thoughts. I\'m going to
stop there.

MR. BREIDT: Thanks. Any comments?

MR. HALE: I\'m grateful for you all looking at this material. I think
one of the things that is really different about this industry is that
for many kinds of questions at some level or another, you have to have
an electrical model. You simply cannot tell where power goes and how it
distributes and who pays and who is affected by some apriority
principle.

You can\'t tell if guys are in the same markets or not very well without
that sort of thing. The problem with electrical models is they tend to
be extremely complicated. There are the same few equations repeated over
and over and over and they require an awful lot of data.

I think there is a general recognition that if there were some sort of
reference models that were out there that weren\'t meant to be totally
accurate pictures of the grid but to kind of get across how the grid
would react under certain kinds of investments, for example, under
certain kinds of loadings and do it close enough to use that as a basis
for more detailed knowledge.

I think that would be a very useful thing to have. When we did the grid
study at the Department we had to use models that couldn\'t represent
transmission in any reasonable way at all. The engineers screamed bloody
murder. This thing gave us billions of dollars of cost. It may be right
but when you have the engineers so angry at you, it\'s tough to feel
real confident you\'re on the right path.

The NERC, in fact, has for its own reasons developed interconnect wide
models. I think that it would be really useful and this is certainly
beyond the paper. If somehow NERC and FERC can find a way to work
together so that there is a model that we can all kind of refer to and
not have to fight forever about what\'s the right way to look at these
things.

That\'s an idealistic sort of view of the world. It certainly has
advantages over having the Federal Government rely on models that some
people don\'t trust and then using that as a basis for making some
policy suggestions. That is the alternative, the one we\'re dealing with
right now.

MR. BREIDT: Cal.

MR. KENT: Just to make a couple of comments. The first is just from a
homeland security aspect, the need for much better data and much more
reliable data in the sort of modeling I think may not become apparent to
the general public until we have some sort of a crisis and this relates
back to previous work that you have done.

I think that it is extremely urgent that we push forward in this area.
Mark and I have talked a lot about this back in 1992 when we did the
EPAct which basically allowed this all to happen. Several of us were
critical because legislation in essence created this problem that we
have with transmission and did nothing to solve the problem and
certainly did nothing to provide any method of getting the information
that we need.

Here we are 11 almost going in to 12 years later and the problem is just
compounding itself with what I think many of us believe is gross
under-investment in certain parts of the country in reliable
transmission facilities.

I think it is going to require federal legislation not just from the
regulation standpoint but legislation in terms of data collection
because you\'re not going to get private investment with the lack of
reliable information because people won\'t have any way of computing the
most simple ROIs right now.

They\'ve got some sort of a vague idea that they ought to but the data
just is simply not there so I guess this is a plea for correcting a
decade old problem that we\'ve had. It\'s not to say that we shouldn\'t
have done it. It\'s simply to say that this was an error in omission
that is past due for correction. The second thing I want to know is how
does all of this feed back into NEMS?

MR. HALE: That is actually a very good question. Years ago I got started
in this stuff for basically two reasons. They were restructuring an
industry for economic reasons which an economist \-- I mean, if you are
going to wallow in anything, that\'s the thing to wallow in. That\'s
good stuff. Just like a statistician with a tremendous new data set, I
guess.

The other thing is that EIA\'s NEMS model has these various regions.
They sort of solve each one independently but then allow a certain
amount of flow between the regions. As I learned about congestion and
network effects a little bit more, I talked to Marian and said, \"Look,
you know, where do these things come from? How sure are you of these
things?\"

She, at least, you know, supported it in saying, \"Why don\'t you guys
try to do some electrical modeling and see if we can relate what you
find to the flows in NEMS. That is basically how I got started.

I think that in a case like NEMS you need to assure yourself \-- you\'re
not going to put the grid in there as a real time thing but you want to
assure yourself that the power flows you\'re bringing in from other
areas are reasonable, are defensible. The other thing you would like to
know some information about is what generators and load-serving entities
are really paying for this.

Right now it\'s kind of in a wash. It seems to me that can make a
difference for generation. You would want to know something about things
like cost of generation to better anticipate entry, that sort of thing.
The NEMS relation, I think, the biggest thing is right through the power
flow, the inner-regional trades.

MR. EDMONDS: Actually, I was very impressed with how far you have been
able to get and, in particular, one of your early asides almost was that
little tutorial on the regulatory regimes that are in effect. You went
down this long list of different institutions that currently exist in
the field.

I think that one of the observations I would make is how you collect
data, where you gather the data is contingent on exactly how this
industry is structured. This industry is in the middle of a
restructuring and maybe only at the beginning of restructuring so it\'s
extremely difficult to say definitively how you\'re going to do this.

It also brings me back to one of your other comments about the modeling
which I think was important, and that is one of the things that this
modeling can do is help inform the evolution of this system. What you
have now is an unstable situation.

You cannot possibly go on with the market arrangements as they currently
exist. You can predict with virtual certainty that 10 years from now
it\'s not going to look anything like it currently looks and it\'s going
to be a very different environment.

Actually, one of the things you need to do \-- one of the very useful
things you can do is structure the model in a way that allows you to
say, \"Listen, there are better ways of configuring this and poorer ways
of configuring this and here are the data requirements under those
different worlds.\"

MR. BREIDT: Any other discussion? Okay. Thanks, Doug.

MR. HALE: Thank you all very much.

MR. BREIDT: According to the agenda we now have some time, not much
time, for suggested topics for the spring 2004 meeting. This is
something we can continue discussion over lunch. Maybe I\'ll just call
for any public comment or questions.

MR. HAMMITT: I was wondering about the model of this advisory committee
and contrasting it with one I know pretty well and another one which is
the EPA Science Advisory Board which is really a conglomeration of
several different boards for different topics and so forth.

The way the SAB committees, whatever they are, function is in a much
more kind of formalized process. They address a smaller number of things
in much greater detail. Typically the agency will charge one of its
boards with providing advice on some topic like guidelines for doing
economic analysis or one of the ones I\'m on is the EPA has required
under the Clean Air Act amendments to do these benefit/cost analyses of
the Clean Air Act.

One of the pieces of the SAB comments on that process so the agency
prepares a 200-page document describing how they are going to do the
analysis and then we prepare a 100-page response to that so it\'s much
more the SABs provide reports documenting their recommendations to the
agency. Whereas the way this group has worked in my experience is sort
of relatively short presentations and a wide range of things to which we
respond only orally usually.

Then to the wide range of things, some of it is quite technical and
specific where some members of the committee have a lot of expertise
but, of course, only some people on the committee have expertise in each
of these topics. Some are sampling stuff, survey design, some are
modeling.

Then we have spent a fair amount of time on various points here on
general management stuff and issues having to do with a large number of
EIA staff likely to retire in the near term and stuff like that where I
think most of the committee members don\'t have much formal expertise
except to the extent that some of the committee members are managers of
groups themselves except for Cal probably much smaller groups than EIA.

I just wanted to throw out that as something that may be worth talking
about. Big changes in the model of how this works and whether that would
be good or not good.

MS. KIRKENDALL: It would be tough for us to prepare a 200-page document
for you guys then for you to prepare a 100-page document in response.
That would be a significant change. It\'s certainly worth discussing
moving more towards that direction. I think the Committee before has
sometimes talked about spending more time on a smaller number of things,
a more focused number of things. That\'s not a huge change to the model.

MR. HAMMITT: I\'m not saying I even think this is the good direction to
be going in. I just thought it was useful to put a different model out
there.

MR. BREIDT: We have taken a step in that direction. A baby step compared
to a huge step which was the breakout sessions to spend more time on a
smaller number of topics so the breakouts were specifically designed for
even more presentations with even less time for discussion. I think
breakouts have been beneficial.

MS. KIRKENDALL: And breakouts were also intended to get committee
involvement earlier in the process.

MR. FEDERER: We appreciated getting shown the slides and discussion
papers in advance.

MR. NEERCHAL: What is the reaction from your staff after reading this?
Do they think what they referred to is prepared presentation because it
does take time to make a good presentation.

MS. KIRKENDALL: And, in fact, right now it\'s mostly my staff and the
reason why it\'s hitting my staff is because I have a little control
over it. I think they get a kick out of it. One of the things we do is
everybody writes their papers. They do dry runs. They have to listen to
each other\'s talks and comment and help them improve them so it\'s kind
of a good collaborative effort. Kara has been with me for not even quite
six months.

MS. NORMAN: I\'ve only been here since May but I presented \-- I did a
presentation yesterday and then I helped Shawna and I presented today. I
fairly enjoyed it. Both presentations were kind of different. One was a
group project that we had been working on for a while and I just sort of
summarized and presented.

The other one was one that just started as a discussion amongst a group
of staff members that kind of turned into, \"Maybe we should ask some
people about this.\" It was interesting seeing both sides of that. The
presentations themselves \-- maybe I\'m just speaking personally \--
didn\'t take any significant amount of time to get together.

In the case of the natural gas production I actually had the final
product. I have enjoyed it. I have enjoyed the feedback and I\'m anxious
to kind of see now what the next step is for me personally, how to
implement it and how to use it.

MS. KIRKENDALL: How do we take what we\'ve heard and change what we\'re
doing.

MR. EDMONDS: I was just thinking about the comparison between the EPA,
SAB, and the ASA boards. I was thinking that really the function of the
SAB is really more one of reviewing certification which is a function
that you really haven\'t utilized this. You haven\'t said take a look at
something and we want you to pass judgment on whether it is ready for
prime time.

Sometimes you have informally but not asking the committee to put it
down on paper and sign your name which is really what\'s going on. That
is actually a function that is worth keeping in the back of your mind.
There may be things that it would be useful to have a group that has
been convened as experts to, in fact, take on that function from time to
time.

I don\'t know that is necessarily the way you want to change this group
to do that as its primary function but it is something that it may prove
useful in the way that the SAB is structured in the sense they will
bring on specialist.

They get a problem that is out there. Say Doug is ready to release that
report and is fearful of his trip up to the Hill. He would like to have
some peer review certification that this has gone through a rigorous
process and there are prominent people who have signed up to say this
has been done to the highest standards.

MS. PHIPPS: How do those committees work because you don\'t have a huge
amount of time to put together a 100-page document. How does that work?

MS. KIRKENDALL: That is kind of a limitation of this committee. There is
only so much they can ask of you.

MR. EDMONDS: It\'s a little bit like an academy panel.

MR. KENT: I was going to say it\'s very much like it. If you go back to
that rather laborious history of this committee that I did, you find out
that that was not really how this committee was constituted and that was
not the view of this committee when it was originally established to
serve as an SAB.

It is really performing its function quite well but there may be a need
to go in the direction that Jim does on specific topics in which EIA
itself or even DOE says this is so hot we need to start bringing in a
group of experts which may include people off of this committee to
woodshed a particular problem or a particular issue and do that.

It would require a rethinking of the responsibilities of the committee
if it were to start assuming some sort of certification or validation or
something like this which I think most of us are perfectly willing to
give our advice as long as we\'re not held accountable.

MS. KIRKENDALL: There is one thing that this has reminded me of is that
there is a new reporting requirement. The Government is pushing
initiatives to tie our budget to measures a little bit better. There is
a reporting requirement called PART. I don\'t know very much about it
yet and I think we are probably going to be among the last of the
federal agencies to have to do it.

The Census Bureau and the BEA have had to go through the process. BEA
has said that they have done pretty well in the PART process. This is
where you get the red, yellow, green light.

MR. KENT: I thought that the PART team has disappeared?

MS. KIRKENDALL: Those PART teams disappeared. This is something new with
the same name.

MR. KENT: Sounds like the same bad idea.

MS. KIRKENDALL: This is a reporting requirement. We would fill out this
form or something and report to OMB. As we evolve into that, we may
learn that there is a role for this committee that might be a little bit
different.

BEA has said that they did well in this by getting green lights and
things by demonstrating a linkage of measures, getting advice from
outside committees and how they act on the advice.

It may be that kind of role for you \-- I mean, you are already doing
that to some extent but maybe there\'s some modifications to what we are
doing that would help us with that. As we learn more about it, and I
mean learn more about it over the next year.

I think Kara had something.

MS. NORMAN: I was just going to say two things. I agree with the
statement that was made earlier about the breakout sessions being
helpful and easier. I think we should be free to be able to get into the
information and understand what\'s going on and offer feedback.

The other one, and I\'m not sure what the restrictions are on this, but
I know that you mentioned that you are basically \-- you feel like you
are only able to respond orally. I don\'t know if maybe \-- I mean, I
wouldn\'t be opposed to voluntary written responses or e-mails or
anything like that. I mean, once people have a chance to kind of let the
information sink in.

MS. KIRKENDALL: Yeah, we do that after the meetings. Send e-mails and
we\'ll share them around.

MR. HAMMITT: Not many people avail themselves to that opportunity I
think.

MR. HENGARTNER: My feeling is the proof is in the pudding and it\'s both
a question of what are the needs. I think Cal mentioned that this
committee historically has a different function than the Science
Advisory Board.

But I do feel that the breakout sessions are useful. The other thing I
found over the few years I\'ve been here is there is much more
participation from the EIA than there was initially which proves
something to me that somehow we are able to engage them and that\'s
important.

I mean, if this format helps the EIA to have discussion among themself
and be engaged, they will accomplish something. I\'m not sure if a
formal report would be more useful. It would definitely have more
prestige than an e-mail, let\'s face it.

MS. KIRKENDALL: We\'re doing a lot of work, my office. We are doing a
lot of work with the other program offices and one of the reasons we
take the responsibility for presentations is that way we can get it on
the agenda. They are willing to come and listen and participate in the
discussion. They are welcome to provide input on what we do for our
presentations, too. That\'s just a way of trying to make it so that more
people can be involved.

MR. HAMMITT: Isn\'t there more staff involved?

MS. KIRKENDALL: I think so because I think more of the people from other
offices are coming. I think there are some of the breakout sessions that
they are interested in and people come and listen and that\'s good.

MS. BLESSING: I think more people are coming to the meetings or the
presentations.

MR. KENT: That\'s a bit of a bother because back a thousand years ago
the only people who were doing the presentations were the old
Statistical Standards group and there was resistance because those are
the cops over there and they are going to this committee and they are
saying how bad we\'re doing in collecting our data or reliability of our
data.

MS. KIRKENDALL: People didn\'t like being cops.

MR. KENT: No. It didn\'t work very well either. One of the things we
actually started, and I don\'t know how you could do it, to have the
people from the other offices that are involved or affected by these
sorts of things. Why couldn\'t the discussion that we had this morning
which related so directly to Max. I think it would have been good if we
had somebody from Max who would have been there and talked about how
these problems were actually impacted.

MS. NORMAN: Well, we did a little bit yesterday with Campbell who had
background information for natural gas.

MR. KENT: That was very good to have that.

MR. HAMMITT: On another sort of different tangent, I wonder if \-- I
think it would be a good idea if more members of the board could come
from industries that either supply data or are major users of it. I
would imagine many of these firms have some people of professional and
statistical expertise.

MS. KIRKENDALL: In the past, and this is going back a long time, we did
have statisticians from oil companies and goal companies and they were
great. They were wonderful participants. They really do know the
industry. I think we have worked through most of them but we should look
again.

MR. HAMMITT: Recycle them.

MR. KENT: One of the funny things is that in the deregulation and the
competitive thing you have lost a lot of the in-house expertise in this
area. They are not keeping up because it\'s one way you can cut cost and
rely on people like EIA and other organizations to do that sort of thing
with you. Now, in the finance area, yeah, they are still pretty strong
there but they are not doing the same level as statistical analysis.

MR. HAMMITT: They are huge firms. You only need two people.

MR. KENT: Yeah. And those people ought to be sought out.

MR. EDMONDS: Going back to your original question, let me throw out on
the table the notion of an encompassing theme as opposed to a particular
session which would be the EIA and energy security. What the data needs,
the data that\'s being collected, and what are the issues that surround
that part of the deal, service to that part of the deal.

MR. HENGARTNER: That would require actually on-site speakers.

MS. KIRKENDALL: Maybe not. I didn\'t know John Wood was involved with
some group that was doing the energy security but it turns out he put in
his form 23 to go under CIPSEA only he wants to share that data now with
a group that he works with on natural security.

MR. KENT: I would ask him the question.

MS. KIRKENDALL: Well, I\'m glad somebody brought it up because it turns
out we don\'t want another CIPSEA if we\'re going to do that.

MR. EDMONDS: Why would you do that if this was going to be so important
and then you make the statement like, \"If we can\'t use it for that
purpose, why do we even do it?\" It was the statement that he made.

MS. KIRKENDALL: Anyhow, he has some experience working with those
groups. He would probably be one of the speakers and there are probably
others, too. I would have to search them out because I don\'t know who
is working on that but there are inside people working on that.

MR. HAMMITT: Energy security is many things. Homeland security is
obviously a new thing and that seems like one to pay attention to.
Energy security is also one and foreign policy and North Korea.

MR. EDMONDS: I am not proposing a workshop on energy security. That
isn\'t what I was proposing. What I was proposing was using that as an
organizational metaphor and using it as sort of a theme that goes
through the issues the EIA has to deal with that are associated with
energy security. I\'m not proposing that we \--

MR. KENT: It goes all the way from what Doug talked about,
confidentiality to all of these things.

MS. KIRKENDALL: It would be an interesting way or could be. We would
have to think about it. It might be an interesting way to organize it.
Yeah, you\'re right.

MR. EDMONDS: In fact, one of the things I have observed is that energy
security is very difficult to get your arms around. In fact, one of the
things the EIA does \-- I mean, if EIA is the place where the data is
that you need to understand energy security.

In fact, there ought to be some thought about what this critter is. In
fact, in organizing a session about what are the issues the EIA faces
under that topic area. You might actually find it useful in just
thinking through what do I put in a session like that.

MS. KIRKENDALL: The other thing is if we could figure out all those
relationships. That\'s not bad marketing to put in our budget submission
why we\'re important.

MR. EDMONDS: No, it\'s not a bad idea.

MS. KIRKENDALL: It\'s certainly worth thinking about.

MR. BREIDT: What is the lunch status?

MR. WEINIG: Lunch is available for committee and guests down the hall.
When you break I\'ll just take you right down there. It\'s all set up.
You need to bring a couple of your chairs.

MR. BREIDT: Okay.

MR. WEINIG: I might segue that a couple of committee members had asked
about the spring meeting date. I will send you an e-mail the first of
the week to suggest the Thursday, Friday sessions. Next spring with
Passover and Easter I might suggest moving the spring meeting into March
but we\'ll look at it. I\'ll put the array out there and just ask
committee members to say when they are available. You probably hear the
clamor of calls for public comment.

MR. BREIDT: I do indeed. We had one public comment. Is there any further
public comment? I think we will go ahead and adjourn the meeting and we
can continue after lunch.

(Whereupon, at 12:12 p.m. the meeting was adjourned.)