AMERICAN STATISTICAL ASSOCIATION

\+ + + + +

COMMITTEE ON ENERGY STATISTICS

\+ + + + +

SPRING MEETING

\+ + + + +

FRIDAY

APRIL 14, 2000

\+ + + + +

The Committee met at 8:55 a.m., in Conference Room 8E-089 in the
Forrestal Building at 10th Street and Independence Avenue, S.W.,
Washington, D.C., Carol Gotway Crawford, Chair, presiding.

MEMBERS PRESENT:

CAROL A. GOTWAY CRAWFORD, Ph.D., Chair

JAY BREIDT, Ph.D., Member

THOMAS G. COWING, Ph.D., Member

CALVIN A. KENT, Ph.D., Member

W. DAVID MONTGOMERY, Ph.D., Member

WILLIAM G. MOSS, Ph.D., Member

POLLY A. PHIPPS, Ph.D., Member

RANDY R. SITTER, Ph.D., Member

ROY WHITMORE, ASA, Ph.D., Member

PERRY M. LINDSTROM, EIA, Member

GUESTS PRESENT:

JOHNNY BLAIR, Ph.D.

JOAN HEINKEL, Natural Gas Division

MARY CARLSON, Natural Gas Division

ANGIE KENT, Natural Gas Division

KAREN FREEDMAN, National Energy

Information Center

GUESTS PRESENT:

ANN DUCCA

ROGER SACKLEY, Electric Power Division

ROGER SACQUETY

JAY BREIDT

STAN FREEDMAN

JAY HAKES

MS. BENROHBHANE

NANCY KIRKENDALL

STEVE WADE

DWIGHT FRENCH

ANTOINETTE MARTIN

C-O-N-T-E-N-T-S

[PAGE]{.underline}

Open the Meeting 4

Presentation by Ms. Heinkel 6

Presentation by Ms. Phipps 28

Presentation by Stan Freeman 35

Panel questions and discussion 46

Announcement of Winners of 7th Annual EIA

Graphics Contest by Jay Hakes 62

Recess at 10:07 a.m. until 10:34 a.m. 68

Presentation by Perry Lindstrom 69

Presentation by Roy Whitmore 19

Panel questions and discussion 102

Panel discussion: Challenges in Measuring Data Quality 116

Presentation by Jay Hakes 116

Presentation by Johnny Blair 130

Closing of Meeting 154

P-R-O-C-E-E-D-I-N-G-S

(8:55 a.m.)

CHAIRPERSON CRAWFORD: Good morning. I\'d like to go ahead and we\'ll
reconvene. We actually did get started at 8:30 and then postponed the
meeting until now. First, I\'d like to announce that anyone who did not
sign in yesterday is asked to sign in this morning, either at the break
or before you leave.

Lunch for the Committee will be held at the conclusion of this meeting,
downstairs again in 1E226. The business from Thursday\'s meeting was,
again, to appoint Seymour to be Vice-Chair, and I think we\'ll just
table that and work on it outside of the Committee, or revisit it at the
next meeting, depending on Seymour\'s health at that time.

I am still open to other nominations and would be interested to hear
your thoughts on that in case for some reason Seymour changes his mind
and decides he doesn\'t want that position anymore.

Finally, I think any EIA staff or any Committee member or member of the
public who was not present yesterday should introduce themselves. And
I\'m not sure, maybe we\'ll start with Joan. Were you here yesterday?

MS. HEINKEL: Yes.

CHAIRPERSON CRAWFORD: Okay.

MS. HEINKEL: I\'m Joan Heinkel, with the Natural Gas Division.

MS. CARLSON: Mary Carlson, of the Natural Gas Division. I was here
yesterday afternoon, but not in the morning.

CHAIRPERSON CRAWFORD: Okay.

MS. KENT: I\'m Angie Kent with the Natural Gas Division.

MS. FREEDMAN: I\'m Karen Freedman, with the National Energy Information
Center.

MR. SACKLEY: Good morning. My name is Roger Sackley. I\'m at the
Electric Power Division.

CHAIRPERSON CRAWFORD: Thank you. We\'ll move on to our first
presentation this morning: EIA Responses to Market Changes in Natural
Gas, and Polly will be our discussant.

MR. HAKES: Polly, would you like to come up and join Stan?

MS. PHIPPS: That\'d be fine.

MR. HAKES: Joan, are you using the light ‑‑ 

MS. HEINKEL: Yes, I\'m working on that now.

(Pause.)

MS. HEINKEL: Just while that\'s warming up I would like to say, like
many of us, I was sorry to hear about Seymour. He had been working with
us on this project and we had hoped he would be able to hear the
progress of the project, and like everyone, wish him well and a speedy
recovery.

I want to thank Stan who\'s sitting here. I guess I\'m giving the
presentation because I probably talk more, and I found when I walked in
this morning that I have a twin. So obviously, if you don\'t like
something about this project, John will be happy to answer your
questions.

(Laughter.)

MS. HEINKEL: Anyway, seriously, today I\'d like to bring you up-to-date
on some of the progress we\'ve been making on our natural gas ‑‑ Next
Generation \* Natural Gas Information Program, how we\'re modifying the
data, as well as the systems, to address some of the market changes and
some of the changing needs of our customers.

I mentioned that we have been working with Seymour and Polly and Jay on
some of these aspects. They\'ve given us a lot of good ideas and we\'ve
tried to incorporate them in the project as we worked along. And so
hopefully, we\'ll find out how well we\'re doing on some of this.

Before I get into that, the overall discussion, I\'d like to make a
couple of points. Like the electric power people, we\'re looking at this
not just as data collection, but a whole information system, not just
how we collect ‑‑ not just what we collect, how we collect it, but also,
how we present it to our customers and provide access to them.

Unlike electric power, we\'re doing the implementation a little bit
differently. The gas industry has basically been going under
restructuring a little bit longer than the electric power. We have some
pressing needs that we feel we need to address immediately, and so we
are looking at a phased implementation of this program over time.

Now, just to give you a brief background on how this has all transpired,
some of the market changes, in the late \'70s we had some very
significant shortages of natural gas.

This prompted Congress to take action, which started a long period of
deregulation of the commodity price of natural gas, and over time we saw
many changes in the transportation structure, as well. We now have a
very competitive market.

The price of natural gas is very volatile and in terms of the futures
price of gas it\'s one of the most volatile energy futures prices.
We\'ve seen new players in the market, as well as new institutional
arrangements.

I won\'t go into them in detail because there are certain things I\'d
like to focus on today, but I did want to bring them to the forefront
and mention them. Okay. This figure does a couple of things.

It\'s basically a very simple schematic of the industry, and the top
part shows the physical flow of natural gas. It\'s a little more
complicated than that, but fundamentally, this is the general flow. It
goes from producers, pipelines, LDCs to the end-users.

Prior to the unbundling of the commodity and the transportation
arrangements, basically, the commodity, the gas sales paralleled the
transportation path. The pipelines took ownership of the gas and bought
it from the producers and sold it to the LDCs.

From a data point of view if you\'re looking for volume information and
pricing information it\'s very simple. You could go to the pipelines and
the LDCs and you could get the basic information you needed on end-use
consumption, and our data program was fundamentally based on that.

Now, as you can see on the bottom part, this is trying to capture more
of the purchase and sale of the gas. It\'s much more complicated. And at
this point in time what we\'re really able to get is what we call,
prices paid by our on-system end-users, basically those who are still
purchasing their gas from the local distribution companies.

We are losing price coverage of this group, those who have the
flexibility to purchase their gas from other sources. The one thing I
will say is that we still get our information on electric generation and
electric power from the electric power people.

So that\'s kind of outside this whole arrangement. Now, this has had
some major changes or some very significant impacts on our gas data
collection. One, as I had mentioned, we\'re losing coverage of some
important price information.

Also, we are having more and more problem with timeliness of the data,
actually getting it from some of these companies as they\'re being
merged and sold and basically restructured themselves. And there are a
number of areas of the industry that we simply are not capturing any
information on.

In the past, for example, storage was just part of the local
distribution company or the pipeline\'s operating system. It became part
of their rate base, and there wasn\'t a lot of interest in terms of
costs or much of what was going on there, aside from just fundamental
stocks.

Now, that capacity is being bought and sold, pipelines, storage capacity
and there\'s a great deal more interest in what\'s happening in those
areas in terms of looking at a competitive market and necessity of
competitive market.

I wanted to point out, this just highlights the extent of the problem
that we\'re having with this price coverage issue. The bottom row is the
industrial prices. This has been a problem for quite awhile.

Right now, we are getting approximately 20 percent of the industrial
prices. We\'re having a similar problem. It\'s not quite as severe with
commercial, and an area of great concern is right up here with some of
the retail choice programs.

We\'re seeing a little bit of degradation of our coverage of residential
prices. This just kind of highlights what\'s going on with some of these
retail choice programs. This is taken from a web site.

We have a web site on there on some of these retail choice programs, if
you\'re interested. Without going into all the specifics of it, these
retail choice programs are in varying states in terms of implementation,
pilot testing.

The ones in the white here are ones where there is simply no action
happening right now. But I wanted to put this up to highlight the fact
that we have 50 different states and we could potentially be faced with
50 different programs.

That\'s highly unlikely, but as we go through this transition I think
what we\'re going to see is some melding of data collection systems,
some based on our current system, which will work quite well in some of
these states, but transitioning into a slightly different data
collection system.

I think over time it\'ll merge into one simple data system, but right
now, as I think Seymour coined the phrase of \"messy combinations.\" I
think we\'re going to see that over the next few years as this will
close out.

Here I\'d just like to highlight, we\'ve taken some heroic assumptions,
I guess, as to how some of these programs might be implemented over
time. And I think what it points out, this is ‑‑ we\'ve got some regents
here.

What it points out, I think, is that right now we\'re not in too bad a
shape and if we move fairly quickly and really tackle this problem, we
can head off something which could be quite significant down in a few
years or so.

And one of the real concerns here is that we could just lose whole
states. So you might have, for example, some of the western states where
there\'s not very much gas, and you might lose New York, Georgia,
California, New Jersey, and that\'s very significant in terms of the
kind of information we\'d have.

Okay. What are we doing about this? I wanted to talk first
organizationally what we\'re doing. We\'ve put together a team using
staff from the Statistical Methods Group. They bring a great deal of
expertise in terms of survey design, survey methodology.

Simple things are not so simple things from our perspective, like the
clearance procedure, communications, and as well as survey instrument
design. And then from the natural gas side we\'ve put together a group
who was actually involved in the day-to-day data collection,
dissemination processes, as well as some of our analysts who are
bringing a great deal of expertise in terms of the kinds of questions
that they deal with on a day in and day out basis.

Fundamentally, now, in terms of a process you\'ve got to figure out what
it is you need to collect. And we started ‑‑ we did a couple things
here. One, we went and talked to our users.

We did, I think about 10 focus group meetings, we had three executive
interviews with Congressional people, and we got some interesting
feedback. One of the things is they liked us, they like our work and
they\'d like us a lot better if we did a few more things, one of which
is to work on timeliness.

Generally, I think people were supportive of the baseline information
that we were collecting. They\'d like to see us improve some of the
coverage problems we\'ve got, as well as address some of these new
market arrangements that we do not have coverage of right now.

And we ourselves took a lot of time and talked, what are the issues that
we have to deal with on a day in and day out basis? What are the issues
that we see developing?

And I won\'t spend a lot of time on these, but I think they\'re
fundamental in terms of how we approach this, what our priorities are
and fundamentally what we collect and how we collect it. I mentioned
price volatility.

That\'s not just here, but the impact, spill-over impact of some of the
things that happened in the gas industry, like interruptable gas
contracts on ‑‑ like heating on our price volatility. That\'s been an
issue that\'s been ‑‑ we\'ve been paying a lot of attention to lately.

Competition in retail markets. One of the Public Utility Commissioners
mentioned in one of our focus groups, it\'s like people are going to
want to know, why am I paying what I\'m paying and is it fair. I think
we have to be prepared to address that, and that\'s one of the reasons
why we\'re concerned about the residential price coverages.

There is significant market growth, approximately 50 percent expected in
the next 15 years. There are infrastructure issues that we need to be
working on and dealing with, and issues like that.

Market convergence. Much of that growth is expected to be seen in the
electric power area. It behooves us as a group to be aware of those
interactions and make sure that we stay on top of them.

And then additionally, the environmental concerns, which is tied back in
to some of the electric generation work. Okay. We took this work and we
combined it with what we called our industry conceptual design.

And there, we sat down as a group and just talked through what we knew
about the industry, new relationships and new interactions,
fundamentally, so we could make sure as we went through this that we had
a good handle on what we should be capturing as well as the places to
capture.

This we pulled together in a data requirements report, and we call it an
Information Requirements Report. It\'s out on our web site. I\'ll give
you the cite to this a little bit later.

But we\'ve put together what I think is a really interesting matrix,
which lists the data elements along one side and it shows what we think
the frequency of it should be, the geographic coverage, where we think
we might be able to get it, as well as some comments about some of the
difficulties that we see right now.

I have a slide of that but I don\'t actually ‑‑ I don\'t have it in
here. But then again, it\'s on our web site. We took that information,
then, and started our Information Collection Redesign, and we are now
going through pilot testing of a new survey, as well as we\'re getting
close to pilot testing a second.

And we\'re looking at implementation, basically over the next couple of
years. Hopefully, this program is going to be mostly completed, pretty
much completed, the major elements of it, by the beginning of 2003.

Unfortunately, you probably can\'t see this very well, can you. This is
another schematic of the industry flows. I wanted to use this to just
explain what some of our priorities are and where we\'re focusing right
now.

This is the end-user sector, and as I mentioned before, it\'s no
surprise with all the talk of the decline in coverage. This is an area
where we are focusing very heavily right now. It\'s basically our first
priority.

At the same time, in the production area we\'re working with the Natural
Gas Supply Association on some initiatives to improve the timeliness of
production data, as well as the quality of that.

And in between, there\'s a lot of work that needs to be done looking at
some of these distribution transportation issues, storage, some of the
new information about marketers.

That\'s not really captured here, but right now we are working on just
looking at secondary sources, trying to get a handle on what\'s out
there, what we can use, where we might be able to make a unique
contribution as we go through this process.

We started ‑‑ well, this is our conceptual design for the commodity
transactions. Basically, in our new, flexible world, almost anybody
except some residential and core customers can buy and sell gas from
anybody and sell it to anybody.

And we refer to this as our spaghetti graph. And at some point we may be
addressing some of these transactions, but right now, really, all we
want to focus on again are the consumers. So we took this and just
highlighted the transactions with the consumers.

And we devised four different strategies, and I\'ll start at the bottom.
Strategy 4: we can go with end-user surveys. You\'ve got two
‑‑ basically, two parties to a contract, the people who purchase it and
the people who sell it.

If you go to the people who purchase it, hopefully they know what they
purchased and how much they paid for it. We\'ll talk a little bit more.
There are frame issues. The size of the frame makes a difference in
terms of what you can and can\'t do much of, physically through the
process.

Another strategy, strategy three, is to go to state agencies. We have
done some of that in the past and there ‑‑ we\'ve had some limited
success with that. One of the reasons we\'ve had limited success is that
typically we\'re tapping into a data system that\'s being used for
another purpose.

It\'s not being used for timely information. It\'s being used for tax
purposes or something like that. So we\'re at cross-purposes. I will say
right now, there\'s some interest in Congress in terms of getting some
more timely residential prices and we\'re working on scoping out some
cooperative arrangements with the states in terms of trying to improve
that.

Strategy two is basically the strategy of going to the suppliers, the
people who sell it. And with the diversity in the market, the split
between transportation, the unbundling, you have to go to more than one
in terms of getting the delivered prices.

And then the strategy 1 is, as we go into, you know, more of this retail
choice program, there are billing companies that are coming up, popping
up, that are, as I understand it, combining some of the ‑‑ basically,
they\'re combining the commodity and distribution charges into one bill,
just sending one bill to a consumer.

It\'s like getting, you know, your long-distance telephone user pooling
in all of your long-distance calls into one so that you don\'t get four
or five different phone bills.

I\'d like to start on the investor side and talk about what our approach
is here. I\'d kind of outlined this in the paper, some of the
complexities of trying to get this information from the suppliers.

Several years ago we did try ‑‑ we went out and talked to some marketers
about this, and they did point out some concerns. If you are looking for
state ‑‑ for information on where the gas is consumed, the flexibility
of some of these arrangements may make it difficult to capture that.

For example, a large industrial customer who has plants in several
states could take delivery of gas, for example, in Chicago, arrange
their delivery to Michigan, Wisconsin, Indiana.

The information that we would collect from a supplier would be what gas
was sold at a certain place, not what was consumed in different states,
and we would lose the transportation component of that. We effectively
would not be getting a final delivered price.

The frame for industrial users is about 200 and some thousand industrial
users. The census does this all the time. They maintain the frame. They
have great confidentiality provisions. So our approach on this was to
try a pilot test of an industrial survey with using censuses as the
agent.

And that\'s actually being pilot-tested right now with 150 companies. We
are hoping for some results within the next couple of months, and this
year we\'re hoping to evaluate this and potentially have a new survey in
place next year, which will improve our industrial coverage greatly.

MR. MONTGOMERY: What was the frequency you said that was? I missed that.

MS. HEINKEL: I\'m sorry?

MR. MONTGOMERY: What was the frequency of that survey going to be?

MS. HEINKEL: We are working on a monthly one right now. Residential and
commercial users\' approach had to be a little bit different. If you
look at the end-user, we\'re talking over 50 million residential
consumers alone on a monthly survey.

I don\'t even want to think about what that would be. And it\'s
unnecessary. Right now, for most of them they\'re continuing to buy from
their local distribution company. They will always get some kind of
distribution services from those people.

So it makes sense to look at the suppliers\' side of it here. And we are
‑‑ we\'ve got a little bit of time with this. So right now we are
looking at strategies, going to the billing companies where we can, or
alternatively, a combination of pulling together information on
transportation charges from the distribution companies and getting the
pricing information from the marketers.

We\'re planning ‑‑ we\'re developing a pilot test for five states right
now. I can\'t remember them all. It\'s Ohio, Georgia ‑‑ thank you, Stan
‑‑ Michigan, Maryland and New York. And we\'re hoping to have that
fielded later this year.

One of the things that we\'ve also been working real hard with in this
process is communications. We had the focus group meetings. We have a
web site up, and this is the information that we have on our web site.

This talks about the purposes. This is a link to our retail choice web
site. This is the Information Requirements Report that I mentioned
before. We have a section there which tells people how they can get
involved, as well as an e-mail notification.

We haven\'t just done this electronically. We\'ve also taken a summary
of the information requirements and we\'ve put that into our monthly
data pack. And we\'ve sent out a letter to almost anybody we can think
of who might be interested in this, telling them about the project and
inviting their involvement.

And we are periodically meeting with some of the trade groups. That\'s
another area where I think we\'ve been pretty successful in terms of
interacting with them and getting good information and cooperation.

Now, in summary, personally, I have been very pleased with the results
of this group, the work they\'ve been doing, and I think we\'ve made a
lot of progress. There\'s been a lot of thought that\'s gone into
fundamentally the conceptual design of this, and I think we\'ve got a
lot of ideas that we\'re just beginning to ‑‑ that are just beginning to
come to fruition.

Some of the challenges, though, that we continue to ‑‑ that we will see
throughout this project is there will be continuing changes in the
industry. We have to be flexible in terms of the way we approach it.

But as I mentioned earlier, we\'re seeing major changes in the electric
generation area, and they have very significant impacts on the gas
industry, and we have to keep that in mind as we move through this
process.

Always, maximizing timeliness, utility, accessibility, I think these are
just basic tenets that we have to keep reminding ourselves, you know,
every time we look at some facet of this project.

And then finally, something which we\'re terming right now, \"response
encouragement,\" convincing people that it\'s in their best interest to
provide us this information.

And you know, in thinking about this, I was thinking, you know, in a
regulatory structure it was definitely in people\'s best interest to
provide information.

They may have provided it grudgingly, but they knew that if they didn\'t
provide this information they weren\'t going to get their rates
approved. So there was kind of a common purpose here in terms of
providing some of this information.

It\'s quite different, obviously, in a competitive interest. People
trade on information and there\'s a great reluctance to provide anything
that they don\'t have to provide.

We\'ve found in working with some of the affiliates of the local
distribution companies, those who are used to providing information were
not as reluctant. You go out to some of these new companies and
they\'re, like, \"Who are you?\"

So I think there\'s a lot of activities. There\'s a lot of groundwork,
homework that has to be done to encourage people to respond. One area
that I think is going to be potentially very helpful to us is working
with the National Association of Regulatory Utility Commissioners,
NARUC.

We have been working with them on an extended basis. There was a meeting
in March and I think they\'re becoming more and more aware of some of
the developments in this area, and we\'re hoping that they can assist us
in this in terms of encouraging marketers to submit some of this
information.

And finally, there\'s just an ongoing need for better methods to improve
timeliness. In one of our meetings, and I know the issue came up about
working, for example, with software companies to try to get some of our
reporting requirements incorporated in that, I think that\'s an area
we\'d like to move in over time.

But fundamentally, as these companies become more competitive and you
have people moving in and out of positions, it\'s a real tracking game
trying to stay on top of that, remind them of their commitments and find
the people who will actually respond.

That concludes my presentation. Are we ‑‑ 

CHAIRPERSON CRAWFORD: Okay. Thank you Joan.

Our discussant is Polly Phipps.

MS. HEINKEL: Okay. I\'ll turn this off.

MS. PHIPPS: I\'m not going to ‑‑ don\'t have any overheads here. So I\'m
going to just ‑‑ I\'ve been following the work on the natural gas now
for about two years as a member of the Committee since I\'ve started,
and I just want to note that I\'m very impressed with the scope of work,
the planning, the testing and all the detail of the EIA effort.

This is, I think, an extraordinarily large and complicated project and I
think everyone on the EIA staff deserves a great deal of praise on their
efforts. They\'ve set out the issues well, and in collaboration with the
major stakeholders they\'ve set out focus objectives, detailed
information requirements, and now they\'re to the part of the final kind
of data requirement, I\'d say, and data collection stages.

One thing I\'ve been quite impressed with is the well-focused industry
effort, and with this kind of scope of change in restructuring, a real
detailed assessment of this problem, the respondent loss and the
development of multiple new players as new merchant functions, state
variation, there\'s a lot of factors to face in what\'s going on in
coverage, and loss of respondents.

I think the phased approach is really sensible and I applaud them on
this. You\'ve just kind of got to move into this phase by phase. The
focus on price data now, as they\'ve pointed out in the paper as the
major kind of priority, is one of the most challenging objectives.

I\'m going to speak just on ‑‑ mostly on the survey methods and others
on the Committee may have discussion and question on other issues.

One of the steps that I was interested in the findings of, and I wasn\'t
at the last meeting so I may be embarrassing myself here a little bit,
but I was interested in knowing a little more about the findings of the
pre-test site visits that were done in Maryland, Virginia, Georgia and
the District of Columbia, and it would refresh my memory and I think the
new people here.

I\'d be particularly interested in these pre-design site visits in the
reaction of the marketers. They were the ‑‑ the visits involved
marketers, local distribution companies and one public service
commission.

Also, I\'m interested in these visits with an eye towards earlier
meetings and difficulties that were coming up in the natural gas
industry issues; such things as records-keeping and compatibility. There
were differences in billing cycles between calendar and billing months.

Also, in the sectors there were problems with the EIA\'s use of
residential and commercial, while the companies tended to use rate
class. So I\'d be interested in the record-keeping issues that you ‑‑ I
don\'t know if you uncovered there.

Also, the EIA and industry concepts and definitions, there\'s a notation
in an earlier meeting of differences and mismatches between those. And
if you were able to get at any of these issues in addition to this
larger issue of just industry change, were these problems apparent in
your visits and do you have any solutions for them?

The focus on the residential and commercial prices, EIA noted they had
found a number of data models, the unified bills, the kind of multiple
data sources, or I think you might have called them something different,
and this time the state agencies and the end-user surveys deciding, as
they discussed in their paper, focusing now kind of on the unified
billing way to approach the residential and commercial prices.

I\'d like a little bit to know about how this decision was made, and as
I looked at the information requirements, it looked like the multiple
data sources may be the waves of the future. And so I\'d be interested
in knowing if you\'re going to move forward in that direction.

The other thing, you know, I would always urge caution and flexibility,
kind of. You\'re basing your movement at this point I think on these
pre-visit design ‑‑ pre-design visits on three states and District of
Columbia.

And I was going to encourage you to do your cognitive testing in states
outside of those three states, which it sounds like for a number of
cases you\'ve already gone ahead and did that. They ‑‑ all those do it;
anticipated me there.

But also, as you do these things I think some of the other alternatives
may merge, and I think you noted that there\'s going to have to be a
great deal of flexibility. In the cognitive testing and operations and
pilot testing, which EIA is moving into now, there was a discussion of
doing cognitive and limited field testing.

And I\'ve had some conversations with EIA\'s staff on this. I would
encourage them to take advantage of every opportunity for field testing,
both kind of in an operations sense and a cognitive testing.

There\'s a lot of different issues going on here. It\'s not just the
understanding of the respondents, which is the main focus of the
cognitive testing, but there\'s also operations and timeliness issues,
such as the type of data collection you\'re going to use, movement from
mail to Internet and telephone.

And so there\'s kind of a number of different focuses going on as you
conduct ‑‑ go into this cognitive and field testing. One thing that
I\'ve always found useful in these types of situations is doing
structured respondent debriefings over the telephone, what were often
called the response analysis surveys.

Basically, they give you a chance to both assess the respondent\'s
understanding and their ability to provide data in a timely fashion. So
you can go over certain issues, you know, did they understand concepts
and definitions, and you can also understand why they were able and
unable to provide the data.

I think this would also be useful ‑‑ I don\'t know if you\'re planning
on doing these for the Census Bureau in the industrial pre-testing
you\'re doing there, but that might be a useful thing to do.

Finally, I\'m just interested in the forms design issues. And as I said,
I wasn\'t at the last meeting on the strategic design, but there was a
discussion, I think, of the standard kind of template, and I\'m
interested in your ability to incorporate this into your cognitive and
field testing now.

I don\'t know if that\'s moved forward at all. I just want to say I
really applaud this effort. I think it\'s a great group of people kind
of interdisciplinarian; sounds like cross-groups in the agency.

And I think they\'re doing a great job in being flexible, which is not
an innovate in a big project like this, which I think should be
applauded. And I look forward to hearing about the results in the
future.

CHAIRPERSON CRAWFORD: Thank you.

Joan or Stan, do you have some comments?

MS. HEINKEL: Actually, Stan was on all of these site visits we did. You
went to Georgia ‑‑ 

MR. FREEDMAN: Maryland.

MS. HEINKEL: ‑‑ Maryland.

MR. FREEDMAN: And Virginia.

MS. HEINKEL: Right.

MR. FREEDMAN: Actually, I wasn\'t on every single one of them. I missed
a couple in Virginia.

Polly, first of all, thank you for your kind words about the efforts of
the group, because they ‑‑ many of them are in the audience here and I
know that they very much appreciate those comments.

You raised lots of issues and let me talk about a couple of them early
on, and that is the record-keeping and things we found out in the site
visits that we\'ve done so far. All of the problems that existed for
years at EIA with record-keeping not matching EIA concepts still exist
now.

Nothing in the change in the industry has really done anything to change
that. The different cycles or the billing months, for example, local
distribution companies still use about 25 or 26 different cycles every
month, and it didn\'t appear that there was any reason that they were
going to change their entire way of doing operations just so they could
have a calendar month to match EIA\'s.

What they told us ‑‑ nor did we suggest that they do that.

(Laughter.)

What they told us was that they have their own estimation techniques to
give EIA about what they think we\'re asking for, and quite frankly,
given some of the other issues that we have to deal with in forms
design, I think for now we\'re probably going to allow that to continue.

The same issue was true on sectors. It may actually be getting a little
bit worse. What\'s happening ‑‑ and Ann, you were on many of these
visits. So jump in if I either mis-speak or forget something.

What\'s happening in most cases is that the marketers who were selling
gas are picking up the sector classifications from the classifications
that the utilities used in the past. They\'re not coming up with their
own because they\'re not really interested in the sector.

They\'re interested in selling residential and commercial gas. They tend
to make distinctions, and this is why it\'s a little bit worse. They
take residential and small commercial and lump them together for their
marketing efforts, and then look at large commercial and small
industrial, sometimes in another organizational unit, sometimes in the
same unit.

One marketer we visited early on said, we\'re not going to be dealing
with industrial customers and ‑‑ but he gave us the name of a contact
who was not answering our phone calls.

And when we called him back he said, You know, I really didn\'t mean to
mislead you, but last week we reorganized and we\'re now doing
industrial also. So that\'s going to be in a state of flux.

For the most part, they\'re not dealing with the really, really large
industrial customers like steel plants and things like that who have
their own purchasing staffs, as Joan said, who go off and buy gas
directly from market hubs or from wellheads.

So the sector issues are probably going to get a little bit worse before
they get better.

MS. DUCCA: Let me add a couple of things there.

MR. FREEDMAN: Please.

MS. DUCCA: What seems to be important to them is how much gas a customer
is using. They don\'t really care if they\'re residential, commercial,
industrial. They\'re either using a little bit or a lot, and that\'s how
they classify them.

And for the industrials, not only, as Stan said, do they have their own
purchasing department, but they have a very increasingly sophisticated
purchasing department.

Economists, people that are watching the market, that this could be
‑‑ the amount that they pay for energy could affect, you know, their
profit margin. That sometimes may be the only thing that they have where
they can make an adjustment.

So the industrial, if anything, is getting more difficult to pin down
because they\'re getting increasingly sophisticated in how they make
their deals for gas.

MR. FREEDMAN: Thanks, Ann. I think what I described so far is the bad
news from the site visits. The good news, and Joan mentioned this
earlier, is that those new actors in the system who have come from a
regulated environment appear to be willing to supply the data.

They are used to supplying the data. They have a general understanding
of what EIA wants, and indicated to us that their records are kept
pretty much in a way that they could supply us with that information.

And these are people who come from utility backgrounds. The new actors
are much more reluctant, although it appears that they do keep their
records in sufficient detail that they could provide us ‑‑ there may be
some problems with the sectors, as Ann talked about, the enlarged
customers versus small, as opposed to residential versus commercial.

But they do have basic volume data and they do have revenue information.
Their willingness to supply it, we\'re just going to have to see. There
were real concerns about confidentiality and market share and
competition.

They were very interested in seeing information from and about their
competitors, and not very interested in their competitors seeing their
own information. We always spoke to them in terms of aggregation at the
state level and at the sector level.

And they generally allowed as that probably wouldn\'t be a problem,
unless they happened to be one of the big guys who they were concerned
that their ‑‑ that people could figure out what their market share was.

And we\'re going to have to deal with that in terms of confidentiality
and the release of the data. I do think, though, that there is a general
reluctance on these new actors to report to the government, simply
because they\'re not used to doing that, and I\'m not sure they see the
benefit in it.

And so we\'re going to have to work on that. Speaking to why we looked
at the billing approach as sort of our first line for collecting the
data, I think conceptually going after what an end-user is billed is as
close to the burner tip as we can actually get, and that\'s important
for the reasons that Joan talked about.

Ideally, if you could go to the end-user and they would give you 12
months worth of bills or a bill every month, that would be ideal. The
other thing is that at least initially when we did our pre-design
visits, it appeared that in all states there was an option to do a
unified bill.

And what we mean by that is a consumer would receive one bill and there
would be a cost for the commodity, the molecules of the gas, a cost for
the distribution or transportation from what we call the city gate.

The point where the pipeline drops the gas off to the local distribution
company, there is a portion of the bill that accounts for that. And then
there ‑‑ also on the bill are some taxes and other charges.

There was an interest in the data requirements section of our work that
we look at the components of the bill, rather than the total ‑‑ in
addition to the total bill, as EIA has done in the past.

And the reason for that was so that issues of large swings ‑‑ that we
would have more information in looking at the data in terms of large
swings in price. Was it a jump up in the commodity price or was it a
change in a regulation that related to the distribution price?

MS. HEINKEL: If I could just jump in here.

MR. FREEDMAN: Go ahead.

MS. HEINKEL: One of the things that has been very difficult to deal with
in terms of analysis is if you look at the per BT ‑‑ per MCF price of
gas during the year, it\'s counter-cyclical. I mean, you get your lowest
prices in the winter, your highest prices in the summer.

And you know, trying to do any kind of analysis of that is really
challenging. And it speaks to this commodity versus the distribution
charges, where you know, one overwhelms the other during certain parts
of the ‑‑ volume in the winter overwhelms the commodity charge.

Whereas, the commodity charge becomes just huge in the summer, and so
you see the prices going up in the summer. So from an analytical
perspective that would be really helpful, although I guess one would
have to say that\'s not necessarily a compelling reason for doing that.

But secondly, as states go through this retail choice program I think in
terms of doing any assessment of how these programs are playing out, you
have to have that information. You have to be able to know what people
are paying for that.

And so I think there really is a compelling reason for doing that at
this point in time with the retail choice programs.

MR. FREEDMAN: The commodity is actually the thing that the people get to
choose. You don\'t get to choose your distribution rate. That is still
regulated.

So we thought at least conceptually that looking at the bill, and these
bills in the choice states are separated out that way, we also hoped
that going to the billing, using the billing approach in those states
that had unified billing ‑‑ and some states now require that there be a
unified bill.

You can get it either from the distribution company or the marketer, but
you have to have a unified bill. And in some states the customer gets to
pick. We were hoping, and what we had felt initially from the site
visits, is that most people were going with the unified bill, so that we
could get that information all in one place.

It would help to eliminate double-counting and mismatches on billing
cycles and things like that, because we\'d have all the components we
were interested in from one bill.

And we\'re going to have to see how that works out in terms of the
testing, because the industry is very volatile, and the rules in the
states are changing all the time. Initially, when we started this there
were not very many independent billing companies that were out there.

It was mostly the local distribution companies that were doing the
billing because they had the infrastructure set up to do it. In the past
few months we\'ve seen the rise in billing companies in states that are
unbundled.

Very often it\'s in the same states where electricity has been
deregulated and these companies go to electric marketers and to gas
marketers and provide a whole series of computer services for that, one
of which happens to be the billing of end-use customers, but may not be
their most important electronic service that they provide.

So we want to try to get to these companies, find out what they\'re
doing, and as Joan said, perhaps use those ‑‑ work with them to set up
software that would allow them to report automatically. And so we\'re
going to have to see how that works out.

We are, Polly, going to do what you suggested in terms of the
field-testing and the debriefing over the phones. We\'re going to take
two approaches. One will be the on-site, more traditional kind of
cognitive interviews.

The others will be where we send out the form and then debrief them over
the phone. And we\'re doing this not only because it\'s a good way to do
it, but also because of time and cost constraints. Does that pretty much
answer your questions?

MS. HEINKEL: It\'s great.

MR. FREEDMAN: Okay.

CHAIRPERSON CRAWFORD: Any other comments, Polly?

MS. PHIPPS: No, I don\'t.

CHAIRPERSON CRAWFORD: Any questions or comments from the Committee?

Bill?

MR. MOSS: I have a question. When you said you were going to go to NARUC
and see if they can help you with the marketers, what kind of leverage
did they have over the marketers? I mean, if this is retail, choice,
there\'s free entry into the industry, they\'re not part of the
regulated piece of the supply chain.

MS. HEINKEL: In some cases they have to be certified. So ‑‑ and they
have to ‑‑ let me see. It\'s my ‑‑ they have to have some type of credit
worthiness, and I\'m not exactly sure of the specifics of that.

MR. FREEDMAN: I can ‑‑ 

MS. HEINKEL: Yes.

MR. FREEDMAN: ‑‑ I can jump in.

MS. HEINKEL: Take it away, Stan.

MR. FREEDMAN: Okay.

(Laughter.)

MR. FREEDMAN: What we found in our visits, and this was not the same
thing that we found when we were working with the electricity folks; in
electricity, generally, the Public Utility Commissions regulated
people\'s entry into the market.

They had to be approved by the Public Utility Commission. In gas that
wasn\'t so much the case initially. They had to be approved by the
utility because of the financial arrangements with buying capacity ahead
of time and things like that.

The utilities wanted to make sure that they had the bucks to deliver the
gas to the city gate every day. But the utilities keep ‑‑ have to
approve these marketers. So we know who they are from the utilities.

But what we did find out is that all the gas marketers are real
concerned that they\'re going to go to state regulation for approval
because of what\'s happening in electricity. So as they move towards
that or if they do, they may be able to play a larger role simply
because of that requirement for regulation.

And also, at least in Georgia, the Public Utility Commission ‑‑ and I
know that Carol can probably speak to this because she was telling me
about her \$800 gas bill ‑‑ they are getting involved when marketers are
engaging in unfair marketing practices, or they\'re going out of
business and leaving people high and dry.

So I think that there will be some influence there, the extent of which
I don\'t know, but I think it\'s a good avenue to pursue. And Polly, to
get to the question that you asked that I didn\'t answer, we actually
have a mockup of an Internet-based form right now that the gas folks on
the team haven\'t actually seen yet.

We really have just finished it. But what we have tried to do with that
is use English language words to ask the marketers and the LDCs what we
want, and then using some computer techniques to guide them through the
interview process, providing them with help with more technical
definitions.

But the one thing we did find on our site visits, and in fact, when we
did some cognitive interviewer training for EIA staff, we practiced on
natural gas service and we didn\'t even know we were going to be working
with gas at the time.

It was very fortuitous. And those people in the industry that we always
thought understood some of the terms, well, it turns out maybe they
didn\'t always. So we learned a lot from that in terms of redoing our
questionnaire wording.

We\'ll just have to see how it goes, and we\'ll be happy to share that
form with the Committee as soon as we have one that we\'re ready to go
out and test with, and we\'ll send it out and can get comments back, and
I\'m sure we\'re going to be making revisions to that.

MS. DUCCA: At the moment we\'re planning on an Internet filing. We
don\'t know how that\'s going to play out, but we think that that may
fit well with the culture of the people that are going to report to us.
They work, you know, in real time and immediate situations.

So that\'s our plan and Stan has been working very hard on a series of
English-like questions that you would be stepped through. But we are
hoping that we can do this through the Internet, but we still have to
check that out.

MR. FREEDMAN: Actually, not just Stan, but Antoinette Martin, who is
sitting in the back, and Bob Rutchek, who isn\'t here right now, are
constantly hollering at me when I introduce something that sounds like
I\'m a bureaucrat if I make a suggestion in how to word something.

So they\'re the ones who are really responsible for making it readable
and understandable.

MS. DUCCA: This is a bit of a cultural shift for EIA, too. I mean,
we\'re used to seeing this matrix-like form and many people in EIA when
they think of Internet filing, they think of taking that form and just
putting it up on the Internet, as it\'s always been click on little
boxes.

So we have something different in mind, but we\'re not sure how that\'s
going to play out.

CHAIRPERSON CRAWFORD: David.

MR. MONTGOMERY: Yes. I was intrigued by your discussion of the fact that
residential gas prices are high in summer and low in winter, since I
just stumbled across that myself recently.

I was trying to actually correlate heating oil prices with natural gas
prices and discovered you get a perfect correlation. The problem
assigned was the opposite of what I thought it was going to be.

(Laughter.)

MR. MONTGOMERY: So I think that\'s a really good idea to kind of get
those components broken out. I think it would also be important to find
out how they\'re broken out on the actual bills. That is, whether people
receive bills where they can distinguish the commodity and the capacity
charges.

MR. FREEDMAN: They do. In the ‑‑ I have not seen an exception to this;
in the unbundled states they can.

MR. MONTGOMERY: Okay.

MR. FREEDMAN: One of the problems with that, David, that I really
didn\'t mention is how those distinctions are made, and this is where we
get into the messy combinations.

MR. MONTGOMERY: Okay.

MR. FREEDMAN: It\'s not uniform from state to state.

MR. MONTGOMERY: Where do you pay the fare and transportation charge for
getting gas to the distribution company, for example? Is that in the
commodity charge or in the capacity charge?

MR. FREEDMAN: Generally speaking, yes, it is.

MR. MONTGOMERY: Yes.

MR. FREEDMAN: But there are things that I would consider to be a
distribution charge; that is, the balancing of the gas on the system and
the gas that needs to be held in the above-ground tanks to make sure
there\'s sufficient flow through the pipes.

And in Maryland that actually gets put onto the commodity charge. So if
you live in Maryland and you get an advertisement from a marketer, which
I have, that says you\'re going to pay 31 cents a therm, read the fine
print very carefully because sometimes it\'s 31 cents plus 5 cents
balancing.

And sometimes, that advertisement includes the balancing charge, and
that\'s going to be a difficult data collection issue for us because the
states are doing it differently, and they have stranded the electricity
‑‑ the gas analogy to the stranded assets charges.

MR. MONTGOMERY: Yes.

MR. FREEDMAN: And they\'re not uniformly applied across states. And so
the state comparisons may be very, very difficult. And I think it\'s
going to be, I don\'t want to say impossiblem, nothing is impossible,
but it\'s going to be difficult to try to get the respondents to put
things in the categories where EIA thinks they should, because they\'re
not going to keep their records that way because the bills don\'t work
out that way.

One solution to that may be just to do a commodity charge and then a
charge that\'s \"other,\" which is everything else, in the ‑‑ with the
idea that the commodity charge is really the market competition issue
and all the others are still fixed by the Public Utility Commissions.

But we\'re going to see what happens when we do the testing and look at
the data when it comes back before we make a decision whether or not
that\'s what we should do. But we have thrown that idea around in the
group in terms of being able to compare the data.

MR. MONTGOMERY: Yes. That makes my second suggestion even probably more
questionable, but that it would also be really nice if you could get a
benchmark quickly so that you could look over time at how these billing
practices change and what that implies for kind of residential
volatility.

MR. FREEDMAN: Okay.

MR. MONTGOMERY: If I could ask a third question which ‑‑ of Jay
‑‑ rather than Joan.

It\'s been occurring to me that the reporting of end-use prices and
quantities for ‑‑ on the oil side has always been very ‑‑ is very
different from natural gas. And historically I think it\'s been because
you always survey the suppliers, and the suppliers know more about their
customers on the natural gas side than on the fuel oil side.

But in the case of natural gas, if you\'re not clearly starting to think
about different ways of collecting that information, have you given any
thought on the fuel oil side to doing something that would let it
‑‑ make it possible to get higher frequency data on prices and
quantities by sector, rather than just the kind of annual data that we
see now?

MR. BREIDT: Our focus has been on natural gas and electricity, because
we thought those were the biggest challenges. But fuel oil, of course,
is a very sensitive issue to a lot of people. And I think this year
we\'re trying to support similar efforts more on the oil side.

We\'ve sort of started. We\'ve been at this, what, now, for two years in
electricity and natural gas. We felt that was about as much support for
this we could get at the time. And in this year\'s appropriation we have
more money for oil and the intent is to follow-up on some of these.

I think actually in the fiscal 2001 budget we\'ll get even better
support for these programs, because high oil prices raise people\'s
sensitivity to data quality.

CHAIRPERSON CRAWFORD: Cal.

MR. KENT: As I was listening to your presentation I was thinking of my
own situation, and that is, first of all, I get a level bill from my gas
company and they adjust it every six months. And I\'ve never had an
\$800 gas bill yet.

But I\'ve had some that have been up in the three- and \$400 level
before I went to, you know, the level billing. And I just wonder how
that\'s going to be able to work in if you\'re surveying the residential
customer and you ask me, you know, what am I paying.

Right now, I\'m paying \$87.50 a month, you know, for gas. I get one
bill. That\'s all they tell me. It doesn\'t have anything else broken
down or anything of that nature. I also, though, do happen to know that
Mountaineer Gas, which is our gas company, is now buying gas from a
whole bunch of different places.

It\'s purely a distribution company, and I have no idea how much
they\'re paying for gas. All I know is how much I\'m getting charged.
And I just begin to wonder how in the world ‑‑ and I think of
telecommunications now where I get four telephone bills instead of, you
know, the one that I used to get ‑‑ how in the world you\'re all ever
going to untie all of those knots, and then come up with anything that
represents 50 states, all of which are not moving in the same directions
at all.

And it just blows my mind that I have the same problem with, when I was
listening to Bob Schnapp yesterday, you know, how you\'re going to be
able to come up with retail, residential figures that mean anything.

And I don\'t know if that was just a comment, a complaint or what, or
just a lament is perhaps the best word for it, because if you were to go
out in the telecommunications business today and ask how much people are
paying for telephone service, I don\'t know if you can get an answer,
you know, that means anything.

MS. HEINKEL: Well, I recall an interesting discussion we had ‑‑ maybe
Stan will remember this, too ‑‑ where we were talking precisely about
this issue. You have very ‑‑ you know, ideally, from an economist\'s
point of view what you really want to see is the reaction, the price
reaction, over time.

When you look at some of the flexibility and the opportunities that
people have you can basically, you know, buy gas and have that price
adjusted over time. You can have a fixed price.

You can have levelized billing with an adjustment at some point in time,
or you can have a fixed price contract. And one part of me said I\'d
really like to know what kinds of programs that people are actually
buying their gas under.

And we discussed that and I think we came to the conclusion that that
was a lot of information and we probably weren\'t going to be able to
get it, and that I think right now we are just simply looking at trying
to get a handle of what people are actually paying.

You know, some of the underlying aspects of some of these programs\'
affect, they would be nice to know, but I don\'t think we\'re going to
be able to know that. In terms of the number of different states, the
variety of the programs, my guess is that over time we\'re going to end
up with very similar models that make sense.

And ‑‑ but the extent to which, you know, there\'s this unified billing
versus getting billing from a number of companies, I think you\'re
basically, probably from a residential user you\'re likely to just have
a distribution charge and probably just be dealing with one marketer.

So the telephone issue isn\'t ‑‑ I don\'t think we\'re going to have
that same problem. But I don\'t know. That\'s just my sense of how this
will play out over time.

CHAIRPERSON CRAWFORD: I think we have some time for just one more brief
comment.

MR. MONTGOMERY: Let me make it a little briefer, then, which is despite
my curiosity about all these arrangements, too, it has occurred to me in
trying to figure out how to deal with the data we have now that it might
be possible to ‑‑ it might make sense to focus on total expenditures and
BTUs, which are two things you probably can measure, and then do the
rest of the work ourselves.

For example, if you have total expenditures and then do some kind of
‑‑ and volumes ‑‑ and do some kind of weather adjustment on the volumes
or some other kind of, you know, removing either cycles or removing
trends, it might be possible to find the things that we want to find.

For example, I\'m thinking about, you know, correlating oil and gas
prices. If we remove the normal cycle in gas prices we actually don\'t
need to worry so much about this fixed charge, because you can probably
eliminate it statistically ‑‑ 

MR. FREEDMAN: Okay.

MR. MONTGOMERY: ‑‑ when you\'re trying to analyze things. Now, I guess
that would be my suggestion. Think about the simplest things you can
collect, like expenditures and volumes, and then think about what
working statistically on the things that we know vary in a predictable
way might let you do so that you can get to about the same place in
doing the analysis that you would if you had lots of ‑‑ 

MR. FREEDMAN: Okay.

MR. MONTGOMERY: ‑‑ if you had a different kind of data.

MR. FREEDMAN: Okay. That\'s certainly a different way of thinking about
it than we have been, but one that we should think about.

CHAIRPERSON CRAWFORD: Thank you very much. I\'d like to move on now, and
turn things over to Jay, who\'s going to present the winners of the
graphics contest.

And before I do that I\'d like to say that the members of the Committee
that went to dinner last night had the pleasure of looking at the
graphs, and I think it really is sort of like the Oscars. It is a
privilege just to be nominated.

We had fun looking at them and we thought that they were all very well
done, and we had some discussion about which ones we should choose. So
congratulations to all of the finalists, and in particular to the
winners.

MR. HAKES: Okay. This is a project that I\'ve had a personal interest
in. It\'s no accident that I\'ve been here for seven years and this is
the 7th Annual Graphics Contest.

(Laughter.)

MR. HAKES: We try to get ‑‑ there are some very simple rules that
anything graphic that\'s publicly used, whether it was in a printed
publication or some other media in 1999, is eligible for consideration.

And we have, what, 36 entries this year, compared to 20. And we had a
winnowing process to go down to the semi-finalists. We had some EIA
judges that deserve to be thanked: Ann Whitfield, Jim Disbrough, Marie
Terri Barley, Mark Rodekohr, Larry Stroud and Ed Flynn.

And then last night, as was mentioned, the Committee went through a
rather, I\'d say, protracted discussion. There was not unanimity I think
on any of the graphs, but they were well liked, and so I get to
recognize the winners.

Now, before I do that there\'s a person who doesn\'t have her own
submission, but in fact is involved in almost all of our graphs. That\'s
Ann Whitfield. She\'s been supporting EIA graphics for 20 years, and I
would just say is someone who presents our graphics before Congressional
committees and in briefings to various government officials; that
graphics often set the tone for how people perceive an agency.

An agency that writes well, that has good graphics, is perceived as
being better than an agency that doesn\'t do those things. So she\'s
been right at the heart of EIA and how it\'s perceived, and there\'s
hardly a publication or a brochure that she\'s not involved in.

She knows printing and reproduction, so she is able to carry these
projects from start to finish. She\'s sought after by virtually everyone
that\'s involved in this. There are often big lines outside of her door,
and she\'s been a mentor and a teacher for many people.

And she also doesn\'t get recognized as much as she should. So we\'re
trying to partially correct that today by recognizing Ann. So if Ann
would come up, we have a certificate for her.

(Applause.)

MS. WHITFIELD: I knew there was a reason John wanted me with him, beside
him coming up here. Oh, Geez. Well, I am a little taken aback by this.

MR. HAKES: Well, it\'s well deserved.

MS. WHITFIELD: Thank you so much, and I\'m really going to miss you.

MR. HAKES: Well, keep up the good work, Ann.

MS. WHITFIELD: Yes, and I thank all of you for being so supportive. I
really do. Pardon me? Oh.

CHAIRPERSON CRAWFORD: That\'s good.

MS. WHITFIELD: Thank you. Thank you.

(Applause.)

MR. HAKES: Now, I\'m going to recognize the winners. One is Figure 16,
the Use of Natural Gas to Generate Electricity ‑‑ \"The Use of Electric
Gas to Generate Electricity is Expected to Grow,\" from [Natural Gas
1998 Issues and Trends]{.underline}, the winner is Mary Carlson.

(Applause.)

MR. HAKES: Information on types of fuels used to generate electricity,
both currently and projected for the future. The story is in all the
headlines included in the figure.

Mary, would you tell us how many times you have won a graphics award?

MS. CARLSON: It might be number five. I\'m not sure.

MR. HAKES: Seven years and she has won five graphics awards.

(Applause.)

MS. CARLSON: Thank you.

MR. HAKES: I haven\'t really mentioned what the prizes are, but the
prize, of course, is a certificate. The winning entries are hung outside
my office for a year, because I figure that people that come to visit me
at least ought to receive some education as they come in.

And then I take the winners to lunch, usually at a pretty good place,
and it\'s been a lot of fun. I enjoy that part of it.

The next winner is the \"Electricity Info Card 1998,\" which appeared at
HTTP Cola\\\\www.com. This is an electronic product and it has its ‑‑ I
don\'t have one with me, but it\'s a card that\'s sort of like this.
Okay. Polly has one.

MS. PHIPPS: Everybody has one.

MR. HAKES: These are very, very popular around this town, I can tell
you. And this info card provides the users with quick facts about
electricity that can easily be carried in a pocket.

The card includes four graphs and two tables, and we\'ve received many
compliments on this card, and it was submitted by Sandy Smith and Roger
Sacquety will pick up the award. Thanks, Roger.

MR. SACQUETY: Thank you.

(Applause.)

MR. HAKES: The next one is one that the press is always eagerly waiting
for so they can write a story about it. It\'s \"OPEC Oil Revenues 1972
to 2000,\" which is an OPEC fact sheet that gets a lot of circulation on
the web.

The purpose of the graph is to show OPEC oil export revenues, nominal
and real, from 1972 through 2000, submitted by Lowell Feld.

(Applause.)

MR. HAKES: I\'ve been looking for an excuse to take Lowell to lunch.
Now, I have one.

MR. FELD: Now, I get my chance.

MR. HAKES: Congratulations.

MR. FELD: Thank you.

MR. HAKES: Congratulations. And the final winner, the title is \"OECD,
Oil Stocks Versus World Oil Prices, 1990 to 2000.\" This appeared in a
short-term energy outlook of October 1999.

It shows the inverse relationship between world oil prices and oil
inventories, an issue that\'s been much discussed by oil analysts and
oil ministers around the world in the last few months. This was
submitted by Doug McIntyre.

(Applause.)

MR. HAKES: Congratulations, Doug.

MR. McINTYRE: Thank you.

MR. HAKES: You missed by one vote last year.

(Laughter.)

MR. HAKES: Congratulations. And I think that\'s it.

CHAIRPERSON CRAWFORD: So we\'re roughly a little ahead of schedule for
the break, but we\'ll go ahead and take one and reconvene at 10:35.

(Whereupon, a recess was taken at 10:07 a.m. until 10:34 a.m.)

CHAIRPERSON CRAWFORD: I see Perry\'s excited and we need to get going,
so I\'m not going to stand in his way. So I\'ll turn the floor over to
him so that he can talk to you about measuring uncertainty in the energy
CO^2^ emissions.

MR. LINDSTROM: Thank you much, Carol.

This is my third appearance before this Committee, so I kind of feel
right at home here. This is a new area for me. I don\'t have a
background, really, in statistics. So when I heard that we were going to
be embarking on this I said, well, the first thing I should do is get
all the free advice I can get.

And so the first thing that came to mind was the ASA Committee. So
that\'s when I asked to speak to you all. What I\'m going to do today is
not a technical presentation, but more of a qualitative presentation and
to try to motivate what we\'re doing in a way that makes sense, and to
really get feedback from you all so that the work we do over the summer
will go in the right direction.

And then hopefully in the fall I\'ll come back and present our results
and see where we stand from there. An interesting thing happened
yesterday, I got a call from ABC News, and the person was very excited
about the issue and seemed to be much more attentive than usual.

And I wasn\'t aware of this, but she said, Have you heard about Leo Gate
and I was, No, but evidently, I guess, they were going to have DeCaprio
interview the President on global warming and this caused a big uproar.

So I guess we\'re now in the ‑‑ become a topic that\'s really in the
mainstream, and so we\'re excited. Just briefly, I was just going to
first give an overview of our program and, you know, what we do, and
then we\'re just going to go into the types of uncertainty as far as my
understanding, which is very simplistic, and then we\'re going to
discuss the Monte Carlo approach and also some alternatives, mainly
Latin Hypercube.

And then we\'ll go into what we hope to accomplish. And then I do have
some questions for the Committee that I don\'t think made it into your
packet, but we\'ll be able to discuss them. And hopefully, it\'s
something, if you don\'t have answers right now, you can get to me over
the summer as we\'re working on this, and this should be a big help.

The overview is going to consist of the Legislative mandate, the
inter-agency cooperation, just a little look at our data organization.
These are probably things you already know, so we\'ll go quickly through
them.

A quick look at the method of estimation and then a discussion of the
IPCC meeting and the guidelines that kind of brought this work about. As
you may or may not know, we do this work under the Energy Policy Act of
1992, the so-called Section 1605.

There\'s an (a) and (b) component to that section. Under (a), we do an
inventory of the major greenhouse gases, and under Section (b) my
cohorts are involved in collecting voluntary data on people\'s efforts
to cut back on emissions of these gases.

So those are the two components that come out of our office. So I think
yesterday somebody mentioned expertise in energy data versus, you know,
these other gases.

Because of this Legislative mandate we are forced to know about nitrous
oxide and methane and these other more complex gases, at least on a
basic level. So it\'s something that Congress has given us to do, and
until they change their minds we\'ll be doing work in that area.

But because of our comparative advantages in the energy area we do have
a memorandum of understanding with EPA. In fact, I invited some of my
EPA cohorts to come today. They\'re probably downstairs trying to get
into the building at this very moment because of the enhanced security.
Hopefully, they\'ll make it.

What we do under this memorandum of understanding is provide the energy
consumption-related data to EPA, and that they put into the official
inventory, which comes out ‑‑ in fact, it\'s just now coming out.

But because of the EPACT legislation we also do estimate the other
gases. So what our report becomes is the preliminary U.S. estimate,
since our report comes out in October and the official inventory,
because of EPA processes and public comments, comes out in April, after
our report.

So there\'s about a six-month lag in the two reports. And what you\'ll
find if you look at the two reports is that we agree very closely on the
energy data. The only difference might be if there had been any updates
to our data since when we put the report to press and when they put the
report to press.

And most of the differences will be in the other gases, because nitrous
oxide, for example, the techniques and the ideas on how this emission
occurs, every year we\'re getting new information on it.

So over the course of six months, I believe the new estimates that are
coming out in their report now are 20 percent higher than what we have
in our report last October. So that gives you an idea of the
fluctuations that can occur in the estimation methods for these gases.

A quick look at the EIA\'s data collecting structure, most of you know
this, we get data ‑‑ we get some data directly from the oil and gas,
mainly the petroleum supply annual, and we use some of the monthly data
also before the annual is available.

We get a lot of data from the ‑‑ that comes out of EMEU, energy markets
and end-use. We use the state data reports through ninth ‑‑ for example,
for this year through 1997, because we like ‑‑ some of the
disaggregations are better for our purposes.

And then for 1998 and the current year that we\'re working on, 1999, we
use the monthly energy review data, which also is tallied up on an
annual basis. So for the two most recent years we use a different data
source than for before that.

And then once we get the new state data we update the next year that
that\'s available. We also get U.S. territory data from EMEU and we\'re
letting them do the estimates for us, because that\'s a ‑‑ it\'s a
difficult area to figure out because we don\'t have the kind of data
collection mechanism that we do in the other data.

We use Federal Highway Administration data for some of our reports. So
that, you know, comes through the system and we use that, also. Oh, the
other thing of importance that we get from energy markets and end-use is
when they do, for example, the four-year surveys in detail,
manufacturing survey.

They did one in \'94 and there was one done in \'98. We use that data to
kind of true up some of the assumptions that we have to make between
those years. So it\'s very important when we get that data into the
system.

Often our projections or our estimates change when we get the new,
better data from the surveys. The method is just to put all the data
into a very large spreadsheet. There\'s nothing conceptually difficult
about it.

The difficulties have to do with just all the details that you have to
keep track of. We apply our carbon coefficients to the consumption data
that\'s provided to us. We also calculate non-fuel use in the making of
plastics and that sort of thing, which sequesters the carbon in the
product itself.

And then we make adjustments. As I said, we get the data for the U.S.
territories that we add and we also subtract international bunker fuels,
which is a requirement under the international protocols on this.

This is mainly, certain countries, if the bunker fuels are not
subtracted. For example, the Netherlands, 25 percent of their petroleum
emissions are international bunker fuels related to the large airport
there.

So this kind of adjustment is very important to a country like that.
It\'s a little less important for the United States. Just to give you an
idea of the picture, this is the pie chart relating to \'98 emissions.

You see that the big chunk of it is energy-related CO^2^. There\'s a
smaller amount that\'s other CO^2^, and then methane and nitrous oxide,
about 15 percent combined. But as I mentioned, every year these gases
seem to fluctuate more in the estimation method, and they tend to go up.

Now, I don\'t know if EPA\'s doing that on purpose so they get more
control over the numbers, but we could ask my EPA counterparts, if they
were here.

THE AUDIENCE: We\'re over here.

MR. LINDSTROM: Oh, there they are. Great. Just in time for that
question.

THE AUDIENCE: Yes.

(Laughter.)

MR. LINDSTROM: But as you see, still the lion\'s share is energy-related
CO^2^, and that\'s what we\'re concentrating on. This work came out of a
meeting in last October in England that Arthur Rapinsky attended.

He\'s since moved on to the policy office, but one of the last things he
did was to initiate this work. It\'s recommended that all Annex I
countries do this formal uncertainty analysis in their inventories. And
so there\'s a contract in place to conduct this over the summer.

And we\'ll be coordinating this work with EPA, and we can maybe talk
later about the level of coordination with ‑‑ since the EPA folks are
here. And there is a formal report that will go along with the ‑‑ that
will come out of the IPCC.

That\'s not available yet. I believe it\'s going to be available in May
or sometime in the summer. And so we\'ll be able to ‑‑ hopefully, if we
need to adjust things at that time we\'ll be able to. But I think the
important thing is we\'re out, you know, we\'re in front of the curve
here in getting the work done early.

So let\'s talk a little now about types of uncertainty. Certainly,
there\'s a range of uncertainty, as I mentioned, between the various
greenhouse gases. There\'s this concept of formal uncertainty versus
natural variability in the data, something that I didn\'t quite have a
feel for until I started doing this work.

There\'s obviously random versus bias error, which I think is an
important distinction to make and we\'ll see in some of the data how
that can affect things.

There\'s uncertainty across the sectors, and the earlier discussion I
think was very appropriate to this issue that often, people believe our
data is, you know, reported by sector; whereas, often it\'s reported by
rate class, which is a very different thing.

And then we\'ll get into some specific examples. The good news is that
energy CO^2^ is the most certain of the estimates. And that\'s good
because it is the lion\'s share. However, because it is so large, even a
small fluctuation in that number can mean a big difference in the total
picture.

Nitrous oxide is probably the least certain of the gases. I haven\'t
talked at all about sinks and sequestration. That\'s a whole other
bailiwick and that\'s probably the least certain of any of it.

But in terms of emissions of gases, nitrous oxide might have an
uncertainty of 100 percent associated with it. And even though it\'s a
smaller percentage relatively, because that uncertainty is so high it
can contribute quite a bit to the overall uncertainty in the national
inventory.

One thing we don\'t examine in this study is the uncertainties inherent
in the global warming potentials. That\'s because that\'s essentially
handled by the IPCC itself. They give us the GWPs and so we ‑‑ every
country uses the same GWP.

And if they change those potentials we do it en masse. Therefore, you
know, that\'s kind of taken care of by them and we don\'t have to deal
with that issue. But certainly, there is a large amount of uncertainty
within those numbers.

Now, this is the issue here of uncertainty as a true uncertainty versus
the natural variability that will be in the data. Things that would be
more uncertainty versus variability would be the measurement of the fuel
consumption, although I would imagine in the actual ‑‑ for example,
measuring gas flow, there is a certain amount of natural variability in
the measurement technique.

And I\'m thinking more of the institutional framework that would be in
the overall consumption number that I\'ll talk about a little later.
Obviously, unknown sources or sinks can cause a bias one way or the
other.

And the natural variability would be in the energy values of the fuels
consumed and in the emissions coefficients that we use. I should also
mention that there\'s a certain amount of uncertainty in the work we do
ourselves.

I mean, we can make mistakes in the spreadsheet. We can ‑‑ we have to
use quasi data sometimes to estimate certain splits between fuels. So
there is a certain amount of that inherent in any sort of calculation
like this.

If we think about random versus a bias error, the random fluctuations
can be a problem when you\'re comparing obviously to a base year, the
way we are to 1990. If you\'re consistently off with a bias error where
you\'re, you know, consistently lower or higher, that\'s not going to
affect your delta between 1990 and 1998.

But if we\'ve got these random fluctuations in the data, then that can
really show up in there. And so that\'s an area of concern to us,
obviously, and something that we want to, you know, grapple with when we
do this analysis.

We touched earlier on this issue, and as I mentioned before, the issue
of going from a total number to a number by sectors, and I would say the
uncertainty is multiple fold when you go from presenting a total
emissions to, let\'s say, emissions from the industrial sector.

There are many things that come into play, and I\'m going to ‑‑ when we
have the panel discussion after this I\'ve got a couple overheads that
I\'m going to show in that area. So I\'ll keep that discussion for then.

But another thing that plays into the data, of course, largely is the
weather effect. And we see that across sectors and that\'s an important
thing to think about when we\'re looking at year to year fluctuations.

Other things that could affect that would be, for example, I don\'t know
what the Y2K effect was in the last quarter of 1999, but there may have
been some, you know, hoarding of petroleum that would show up as
increased consumption; whereas, in fact, it really isn\'t consumed in
that time period.

And that\'s, of course, a function of the amount of private storage that
might exist out there that we\'re not really aware of that we wouldn\'t
capture in our regular stock data. Then there\'s the issue of, you know,
when can you pull a trend out of the data?

We\'ve heard recently a lot about this Internet effect and how emissions
are flattening, even though the economy\'s growing. So I want to talk a
little about that and how the uncertainty in the data might influence
our analysis of that sort of thing.

The sort of thing that might have ‑‑ by the way, I didn\'t coordinate my
presentation with Joan Heinkel, for those of you who were here before.
It just happened to work out well this way that I\'m going to talk a lot
about natural gas, I think mainly because it\'s the fuel of interest
right now.

It\'s the fuel that\'s on the margin and there\'s a lot going on in that
area. The sort of things that are going on are likely, actually, to
increase the random error, because you just ‑‑ you don\'t know the
affect and you ‑‑ but you know there\'s something going on there.

You\'ve got the combination of natural gas deregulation and electricity
deregulation. Natural gas is the fuel of choice for the nonutility
generators, and so the interaction between those two is very important.

Then recently, you\'ve had mergers in the petroleum industry. You\'ve
got changes in how they are organized internally. They\'re cutting back
on their corporate planning offices, and that\'s often where we get the
data from.

So there\'s a lot of things that go on, on an institutional level that
you have to be aware of when you\'re trying to construct these
uncertainty bands. Recently, we\'re aware of an uncounted source in the
coal area, which we will be adding this year.

It\'s coal waste that has previously not really been counted, either on
the production nor consumption side, since it\'s a waste. And but it is
consumed, in fact, and the number will probably ‑‑ and it\'s gone up
since 1990.

That\'s why it\'s somewhat important. Had it been consistent through
time going back before 1990, it wouldn\'t be that important because
you\'d adjust all the numbers by roughly the same amount. But it\'s
about ‑‑ it was close to zero in 1990, and now it\'s I believe about 9
or 10 million metric tons, making it about a one percent increase in the
coal number.

As I said, I want to use natural gas as an example of the complexity.
You know, people often have this idea that we\'ve got one number that we
can use for consumption, but we\'ve got a lot of numbers that interact.

You know, we start with gross withdrawals and have to subtract things
for repressuring, nonhydrocarbon gases, vented and flared. Now, vented
and flared is an interesting issue because it moves between one source
and the other.

If you flare all your natural gas, you\'ve got a CO^2^ emission. The
amount that you vent is methane emission. So either way, you\'re going
to have an emission. It\'s just trying to figure out what that split is.

And the states just reported as one category. So we\'ve started just
putting everything in there as a flare, as a CO^2^ emission. But that\'s
just a decision we made to do ‑‑ we made several years ago, but it may
not be totally correct.

So you end up with your total marketed production, which is wet gas.
From that you take your natural gas liquids, and there\'s an extraction
loss associated with that and that gives you your total dry gas
production.

So that\'s just to get you to that level, and then you have to add in
these other things, supplemental gases, net imports, which is an
increasingly large amount. I think it\'s up to about 14 percent now.

You have to keep track of stock withdrawals. And this last item is an
item I want to focus on because it\'s an important item in recent years,
and this is the balancing item. And so after you go through all that you
get your total consumption of natural gas.

And as you can see, there are a lot of opportunities for things, for
interacting numbers to yield, you know, various outcomes that does
affect the sensitivity of the number. And so ‑‑ 

CHAIRPERSON CRAWFORD: You need to watch your time, so.

MR. LINDSTROM: Are we running here?

CHAIRPERSON CRAWFORD: Yeah, we\'re five minutes over.

MR. LINDSTROM: Five minutes over?

CHAIRPERSON CRAWFORD: Yes.

MR. LINDSTROM: Oh, sorry.

CHAIRPERSON CRAWFORD: That\'s okay. So summarize ‑‑ 

MR. LINDSTROM: Okay.

CHAIRPERSON CRAWFORD: ‑‑ maybe in two minutes or less.

MR. LINDSTROM: Okay. Let me summarize quickly, and actually, this is a
good place to summarize, just looking at this particular example. That
is, this is the balancing item for the natural gas, and I\'ve been told
by my natural gas friends that this number will ‑‑ by the time we do our
inventory we\'ll be up by here.

So it would be more ‑‑ closer to how it\'s been in the past. But what
you see is you can clearly see a random signal going through there,
because in the past it was ‑‑ almost always stayed at that level, which
it looks like a bias error.

But when you look at this you can see this signal, and of course, that
coincides with both the deregulation of the natural gas industry,
deregulation of the electricity industry, and it\'s the sort of thing
that we, you know, have to keep an eye on when it comes to coming up
with these emissions estimates.

It probably represents about a ‑‑ even if you knock this back to 400
billion cubic feet, it\'s probably about a four percent, four or five
percent swing between the positive and negative number that we have in
there.

So because you\'re experts on the Monte Carlo simulations, most of you
anyhow, I probably don\'t need to spend much time on that. Let\'s see if
there\'s anything here we really need to ‑‑ you\'re probably aware of
what it does.

The main thing is to aggregate PDFs up to a level that gives us an
overall uncertainty band, and allows us to consider variability and
uncertainty, allows us to look at the uncertainty inherent in the fuel
versus the activity data.

And we\'re going to check this against the Latin Hypercube and which
allows us to do a representative sampling, rather than the method that
the Monte Carlo estimate does. These are the general procedures that
probably most of you are familiar with.

The most difficult I think that I\'ve found already is step one, trying
to figure out what are the shapes of these things. And so I think once
we get that, the other stuff is just a matter of plugging things into
the computer.

But until we get that we\'re going to have to ‑‑ it\'s going to take
quite an effort to get those PDFs to begin with. This is just a schedule
which is over the summer. And I think what we hope to accomplish is, you
know, a better understanding of the nature of uncertainty in a
quantifiable way.

And hopefully, it will be a tool to help policy-makers understand the
kind of uncertainty that\'s inherent in our data so that they don\'t
make assumptions or that we don\'t mislead them as to the level of
accuracy that we have in the data.

So this is a good way to, I guess, end, just to say, here are some
questions that I\'m interested in. It seems that the PDFs do shift over
time, and I\'m trying to think about how best to capture that.

Ideas that people might have on instrumental variables that would help
us bracket uncertainty, where if we don\'t have the direct data there\'s
a way to look at something related to it that would give us a good idea.

And I\'m very interested in any work that\'s going on with graduate
students or anything that we can leverage out there that you might be
aware of, because we need all the help we can get. The reason I wear
cowboy boots is my shoestrings went into our budget.

And so we\'re trying to do a lot with not that much. Luckily, I do have
a Ph.D. statistician from SAIC, Suet bin Ramadan. Did I get that right?
And she\'s doing ‑‑ she\'ll be doing the statistical work.

Why don\'t you hold your hand up for me, which is good. And so from
there I just welcome your comments.

CHAIRPERSON CRAWFORD: Okay. Thank you. Our discussant is Roy Whitmore.

Roy, do you want to come up here, please?

MR. WHITMORE: I don\'t have any slides.

CHAIRPERSON CRAWFORD: That\'s okay.

MR. WHITMORE: You\'d rather me come up there?

CHAIRPERSON CRAWFORD: Yes.

MR. WHITMORE: Okay. I\'d be glad to do it.

CHAIRPERSON CRAWFORD: Well, I just ‑‑ I take my marching orders from
this guy, so.

(Laughter.)

MR. WHITMORE: He can be tough.

CHAIRPERSON CRAWFORD: I know.

MR. WHITMORE: Not being real familiar with the modeling of energy and
the ins and outs of the various sources of uncertainty, I\'m not going
to say a lot about the model, per se, but more about the statistical
methods that can be used.

So I\'m going to focus my comments primarily on the last portion that
Perry went through very quickly regarding the Monte Carlo methods and
such. And maybe some of the other Committee members can more directly
address some of the modeling, and maybe sources and sinks and things
that may have been overlooked and things like that.

First of all, the paper talked about, you know, calibrating the Monte
Carlo against a Latin Hypercube, basically, in each case looking at, you
know, taking all of these input distributions that represent variability
and uncertainty in the inputs to the model and using a Monte Carlo
simulation approach to estimating the final distribution of CO^2^ or
emissions from CO^2^, or the greenhouse emissions.

And I didn\'t quite follow the concept of the Monte Carlo model versus
calibrating it against a Latin Hypercube, because the Latin Hypercube
really is a way of sampling from the input distribution. So I just
wanted to kind of briefly go through some of the ideas there and the way
I think that they should be approached.

Maybe this is actually exactly what you\'re talking about doing, but it
wasn\'t clear from what I read or what I heard. So let me just start
with that a bit. First of all, the kind of the general idea of the Monte
Carlo simulation is to take ‑‑ you\'ve got a model with lots of
different inputs.

They\'re not fixed values. Each input has a distribution. You\'re
sampling from those distributions. You may sample from marginal
distributions or joint distributions, the input variables, if they are
not statistically independent of each other.

Sampling from those distributions, you give the various inputs to the
model and by simulating that over and over, replicating the samples from
the input distributions, you build up a distribution of the output.

And that\'s kind of the basic idea of the Monte Carlo approach to
modeling, and basically generating, instead of a deterministic model, a
stochastic model of a distribution of values of the output that you\'re
trying to model.

The idea of the Latin Hypercube sampling was developed by McKay, Beckman
and Connover in \'79. In fact, I think they coined the term Latin
Hypercube sampling, and it\'s simply an efficient way of sampling from
the input distributions that result in the Monte Carlo simulation
converging more quickly than if you\'re simply taking random samples
from all of the input distributions.

Or if we want to look at that a little bit more precisely, what it
really does is it produces ‑‑ for any fixed number of iterations of the
Monte Carlo simulation ‑‑ produces an estimate with smaller variance
than if you were just doing random samples.

And in \'87 there was a paper by Stein that generalizes what McKay had
done before, because McKay was looking at a situation where all of the
input variables were assumed to be independent of each other.

And I\'m not sure in terms of these models if, you know, the
independence assumption is close enough to reality that that\'s good
enough, but if in fact you need to account for the fact that some input
variables are correlated with each other and use really joint
distributions of the inputs rather than independent marginal
distributions for all the inputs.

In \'87 Stein did a generalization of the Latin Hypercube method that
allows you to use joint distributions for input variables where that\'s
necessary. In a case where the input variables are all statistically
independent it does simplify to the Latin Hypercube method that McKay
had developed.

And in Stein\'s paper he also ‑‑ it\'s more of a mathematical paper than
McKay\'s. McKay\'s was a very applied paper and Stein was kind of
developing some of the mathematics.

And one of the mathematical results that he came up with is that the
Latin Hypercube sampling produces an estimator with smaller variance
than a simple random sampling so long as the number of iterations of the
Monte Carlo simulation is much greater than the number of input
variables.

There are potential problems if there are not sufficient number of
iterations. So that\'s just, you know, maybe the things that were
already known by the folks working on this Monte Carlo modeling, but if
not, I have copies of those papers.

With regard to several of the problems that were mentioned in the paper
that you\'d like to get a handle on, probably the first and most obvious
being differentiating ‑‑ the difference between natural variability in
the population and uncertainty regarding what may be the distributions
of input factors or other sources of just uncertain information, lack of
information.

One way of handling that in the Monte Carlo simulation is using compound
distributions for the input variables. For example, if a particular
input variable in the model you were fairly sure was ‑‑ you know
‑‑ could reasonably be approximated as a normal probability
distribution, but you\'re not real sure of the mean and variance of that
normal distribution, a way of handling that in the Monte Carlo
simulation is to use a normal distribution for that input variable, but
also specify distributions for the mean and the variance, maybe a
uniform or a normal for the mean, some kind of a gamma distribution for
the variance.

That represents the uncertainty so that you feel fairly confident it\'s
normal, but you\'re not quite sure about exactly where it\'s located and
what the spread or scale of the distribution is.

And a way of kind of separating out, then, in the simulation the effect
of natural variability versus uncertainty is to run the Monte Carlo
simulations first where you just use kind of the point estimates of the
mean and the variance of the input distributions, and you get a
simulation that reflects the distribution only accounting for a natural
variability.

And then re-run the Monte Carlo simulation where you\'re also sampling
from the input distributions that specify the mean and the variance of
the individual input factor. And that adds additional variability into
the simulation, will spread out the distribution and you\'ll see the
effect of the uncertainty on the overall estimate.

Another way that those compound distributions for the input factors can
help would be, you mentioned that the ‑‑ some of the distributions that
you\'re looking at are time dependent, that they change from year to
year, you know, and that there may be in this case something like ‑‑ you
know ‑‑ you might have ‑‑ a normal distribution of an input variable
makes sense, but the mean should be time dependent.

Maybe even the variance should be time dependent, but at least the mean,
so that you could easily build that into the model by having time
dependent distributions for the parameters for the input distributions.

Another effect that you were considering was the weather affect, and
that might also be something that you could simulate in the same way,
that you might have a distribution for an input variable that\'s kind of
a standard or an average, and have a distribution for the ‑‑ that
specifies what the mean and the variance of the input variable looks
like that accounts for weather changes.

So additional variability associated with weather pattern affects or
uncertainty regarding what the weather next year might be. So those,
using compound distributions might be a way of handling a number of
those problems and being able to separate out different factors and
different effects.

Another thing I would suggest is doing some sensitivity analysis to
determine which factor is given a final model to determine which factors
were having the greatest influence on the uncertainty and variability in
the final predicted distribution.

The simplest way to do the sensitivity analysis is to simply hold all
the factors except one at a central value, a mean or median value, and
only sample from the distribution of the one factor for which you\'re
evaluating the sensitivity and see what kind of a distribution of
outputs result, taking each factor one at a time.

But if you\'ve got a big, complex model, that may be impractical. And
we\'ve done ‑‑ I\'ve been involved in some things lately where we were
looking at Monte Carlo simulations for exposure, human exposure and risk
where we have taken a database from the Monte Carlo simulation which had
maybe 1,000 runs.

You have 1,000 values of the input variables that were sampled and the
1,000 or maybe 10,000 outputs from the model, and doing a regression
modeling, kind of a response surface type of model to see how the values
of the input variables relate to the predicted output and looking at
what particular inputs have the largest coefficients or most significant
coefficients, and that way identify which inputs have the greatest
impact on the final model value.

And that way, again, identify those variables for which the model is
most sensitive and thereby, you know, if what we have ‑‑ what we\'re
looking at in that variable is uncertainty instead of natural
variability, then that suggests, well, if we can go out and get more
information regarding this particular variable, you know, tie down the
range or distribution of that variable better, then we\'re going to be
able to reduce the variability in the final model.

So you can get ‑‑ through the sensitivity analysis you can identify
those particular variables for which you need additional information or
for which additional information would be most valuable in reducing the
variability in the final analysis.

And I guess the last thing in terms of characterizing the uncertainty is
you\'re going to build up this complex model and run these Monte Carlo
simulations, and that provides kind of a picture of the uncertainty and
variability based on that particular model.

But certainly, another source of uncertainty is whether or not you\'ve
really got the right model. Did you include all the sinks and sources or
are there other things, or maybe things ‑‑ could things interact
differently.

If there\'s some potential of developing alternative models, another way
of looking at uncertainty is, well, you know, we could have used model A
instead of model B. Or maybe you\'ve got a set of five models that are
potential, alternative models.

You can run the simulations with alternative models and get another
dimension of uncertainty with regard to which model really is the most
appropriate model. I think those are my comments.

CHAIRPERSON CRAWFORD: Okay. Thank you, Roy.

Perry, do you have any quick comments?

MR. LINDSTROM: The area that I think was the most important that you
touched upon that I, because I lost track of time, didn\'t really talk
about, what is the sensitivity analysis, I think especially in the idea
that we want to isolate those things, when I go back and finish this and
talk to the data people and they\'re looking for areas to enhance their
work and, you know, put budget money, I think what we want to do is
coordinate these sort of things.

And so that if we see an area that really could benefit from increased
knowledge, we should really think about that in terms of allocating
resources to that area, especially, you know, the importance of this
topic right now, and decision-makers possibly making decisions which
could mean, you know, a lot to our economy and that sort of thing.

So I think we want to pay particular attention to that aspect of this
analysis, and they will.

CHAIRPERSON CRAWFORD: Are there any comments or questions from the
Committee?

Jay?

MR. BREIDT: Just a quick question and comment. What kind of software are
you planning to use? Because I know there exists add-ins to Excel which
allow you to do exactly this kind of thing.

MR. LINDSTROM: Yes, At Risk is the one that we\'re going to be using.

CHAIRPERSON CRAWFORD: Okay.

MR. WHITMORE: Yeah.

MR. BREIDT: Okay.

MR. MONTGOMERY: One thing you passed over real quickly that I\'ve
worried about in the uncertainties, and I wondered if you\'re including
it in the source of the uncertainties, how do you handle petrochemical
feedstocks?

I mean, I would have thought the amount ‑‑ in calculating non-fuel use
of energy, petrochemical ‑‑ you mentioned petrochemical feedstocks. I
would think there are a lot of issues in terms of whether or not those
really do count as sequestration activities.

And how are you ‑‑ are you planning on looking at that?

MR. LINDSTROM: Yes. Well, there\'s kind of two ‑‑ there\'s two parts to
that question. One is the calculation of it, to begin with.

MR. MONTGOMERY: What are the petrochemical segments?

MR. LINDSTROM: And it\'s the most complex calculation in our spreadsheet
by far, because it\'s the area that we have to use the most ‑‑ kind of
the largest fudge factor is, is in some of that data. And so there is
probably a much higher degree of uncertainty in that data than the other
things.

The other part to that question is, then, what happens to these things?
Does it really sequester the carbon forever or does it end up in a
landfill and get ‑‑ or does it get burned or what happens to it?

And I think we\'re going to probably be adding some component. I know I
think EPA already has, right? Is that right, in the current submittal
now.

MR. WHITMORE: Yes.

MR. LINDSTROM: There is a component now added for the burning of
plastics and that sort of thing. So we will probably then turn around
and add that also in, so that we\'re in sync with them. But you\'re
right, figuring out how much of that really stays sequestered is another
issue.

It is probably the most difficult area, other than the area of sinks
and, you know, trees and all that stuff, which is a whole other
bailiwick which I don\'t want to get into.

MR. WHITMORE: I was just going to add to that, we\'re ‑‑ this will be a
surprise to Perry ‑‑ but we\'re starting to talk about now at EPA trying
to fund a little more work to do more bottom-up estimates, like the
plastics and landfills and how much goes to waste incinerators, looking
at, you know, some of the other non-fuel feedstocks, coke and things
like that, what goes into metals and things like that, trying to, you
know, do a bottom-up to try and match to the top-down data that Perry
and some of the people at the EIA think about that.

MR. MONTGOMERY: Do we have time for one real naive question?

CHAIRPERSON CRAWFORD: Okay.

MR. MONTGOMERY: Why are you doing a Monte Carlo analysis? It looks to me
as if you have basically an accounting framework, which is essentially a
set of linear transformations on the energy flows.

And I don\'t see why you can\'t just sit down and calculate straight off
what the uncertainties are from your original PDFs.

MR. LINDSTROM: Well, you\'re asking a question that probably goes beyond
my expertise. However, what I\'ve heard is this is what\'s going to come
out of the IPCC best practices guidelines, that the Monte Carlo is the
way to go.

And I think it has to do with the idea that a lot of these distributions
are non-normal, and so combining them, we have to do it this way. And
that\'s about the extent of my knowledge right there. Suet might have
something to add on that, you know.

But essentially, we\'ve been told that this is the direction that the
international community is going in and so we\'re trying to get ahead of
the power curve. Now, if we come back in May or June and say, Oh, we
just really have to do this, we\'ll rethink things and I think we\'ll
still have to time to.

But I think what I found is just in the process of thinking about this
I\'ve learned so much about our data and what goes into it that it
probably doesn\'t even matter what happens to it later on, you know.

It\'s just such an educational process, figuring out what these curves
look like and ‑‑ 

CHAIRPERSON CRAWFORD: Lost your ten.

MR. WHITMORE: Lost my ten if that\'s over. Okay. I\'m not sure in terms
of this particular application, but I think in general the idea is that
a Monte Carlo simulation allows you to produce an interval estimate, as
opposed to just a point estimate.

And that interval estimate can then reflect both, you know, the
variability in the population and if there is very little variability,
there might be a very narrow interval, if you\'re really able to just
focus on natural variability.

And then it also lies with characterized uncertainty, which may, you
know, widen the interval. You know, if you had ‑‑ knew all of these
things, maybe there really is a deterministic value or a very narrow
distribution.

But if in fact there\'s a lot of uncertainty about variable ‑‑ various
things, then you wind up with a fairly broad interval estimate. This
just gives you a lot more information than a point estimate.

CHAIRPERSON CRAWFORD: I just had a few quick comments about your input
variables not being independent, and I wanted to say something about
what Roy said about extending the Latin Hypercube approach.

Ron Iman did something on that with the use of rank correlations, and I
have a few technical reports of his and a paper on that, if you\'re
interested. The other thing is, is that if you have input variables that
do vary over time, chances are you might want to incorporate temporal
auto-correlation in your Monte Carlo simulations.

I\'m not sure if At Risk does that. But again, you have to really extend
the Latin Hypercube approach to be able to account for that, and also
any Monte Carlo approach. And there is a paper. It\'s on spatial
statistics.

It\'s in [Technometrics]{.underline}. It\'s pretty recent. I think it
appeared last year late, but anything that you could do in space, you
could do in time. You just reduce the dimension by one. So that might be
helpful if you want to incorporate those correlations. Any other
questions?

MS. BENROHBHANE: I would like just to make few comments. There was like
a question about why we are introducing now the Latin Hypercube
sampling. And I think personally, and based on my experience, when I was
doing a risk assessment, actually, I was using Monte Carlo to go to a
probabilistic approach to the whole procedure in doing risk assessment.

And in my work I was finding all the time that the Latin Hypercube gives
me the opportunity to go through the stratification that you talked
about, avoiding the clustering problem that happens mostly with Monte
Carlo.

This is just to say why I wanted to propose maybe we should benchmark
our results with Latin Hypercube sampling in the same time, if it could
reduce the time of running our simulations. Or also, as I said, getting
away from those possibilities of the clustering in our results.

The other issue about At Risk, I think it chooses both. So we don\'t
have, really, a major problem. It\'s ‑‑ choosing Monte Carlo simulations
or Latin Hypercube simulation would be as easy as just clicking on a
different button and doing that.

The correlation between different factors, I think it\'s also a
possibility that is integrated in At Risk, and could be used by both of
them. So we could account for the dependence between different variables
in our model.

The only problem is we need to really identify how these parameters
depend on each other, and that\'s very critical, I mean, to introducing
those factors in our simulation. I\'m sorry. Maybe I didn\'t get most of
what you were asking about.

Are you proposing there are some probably references that we need to
talk to ‑‑ 

CHAIRPERSON CRAWFORD: No. I was just saying that there\'s a different
type of dependence that you might need to consider for any particular
input variable if it varies over time, and that would be temporal auto
correlation, and simulating auto correlated variables is slightly more
complicated, particularly in the Latin Hypercube approach. So that was
all I was saying.

MS. BENROHBHANE: That\'s correct, you\'re right. And probably this time
issue, maybe we need to run the different scenarios, like have different
scenarios and then combine them manually. I mean, I\'m not sure if the
computer is really able to incorporate all those kind of variabilities
to be studied in the same time.

We could just take them factor by factor and do our analyses in the end
on the results that we get.

MR. WHITMORE: I think the reason that I was thrown a little bit is just
the terminology of benchmarking the Monte Carlo against Latin Hypercube,
because the Latin Hypercube is simply a way of sampling from the input
distributions.

And when you speak Monte Carlo I think you\'re just talking about
independently sampling each input distribution. In fact, McKay\'s paper
talked about the random sampling, stratified sampling and Latin
Hypercube. They\'re all Monte Carlo approaches.

CHAIRPERSON CRAWFORD: Right.

MR. WHITMORE: They\'re just different ways of sampling from the input
distributions. And in the McKay\'s then Stein\'s papers, if they would
look ‑‑ generalize that again to if some of the variables are dependent
instead of independent.

And I don\'t know if At Risk would include ‑‑ it probably can handle the
dependent when you\'re sampling from each individual variable or maybe
each correlated. I don\'t know if it would include the generalized
method as a Latin Hypercube for dependent method.

CHAIRPERSON CRAWFORD: I doubt it.

MR. WHITMORE: Do you think it does?

CHAIRPERSON CRAWFORD: No.

MR. WHITMORE: Yes. I would be surprised. But generally, it\'s fairly
easy to write SAS programs or whatever to do the simulations. 

CHAIRPERSON CRAWFORD: Actually, SAS does have a new procedure that\'ll
do it in spatial statistics it generates, at least if you want a sample
from a Gausian distribution. So you could reduce the dimension on that.

But it probably won\'t help you, you know, if you think you have some
auto correlated uniforms or something, then you might try transforming
that if you need to. But I think you probably have a lot bigger things
to worry about then the auto correlated variables.

MS. BENROHBHANE: Yes. I think I would say, I mean, I have been in other
conferences where people talk about Monte Carlo simulations and the
uncertainty that is already inherent to the whole procedure, which is
pretty high.

I mean, it\'s between 20 and 30 percent; sometimes your results are in
that range. I think what I want to say is at this stage we will be happy
to have at least numbers and ranges that we can work with, rather than
not having anything and saying, okay, we\'re not really ‑‑ we\'re just
uncertain of how accurate those numbers are.

And having a feel of really what kind of, you know, ranges we have
around our estimate will be quite a good start, because this is probably
the first time you are doing this kind of systematic uncertainty
analyses.

And I\'m very actually pleased to be a part of it because I would like
to rectify something, because I am the environmental engineer and I am
an applied statistician because of the use of these kind of simulations
in my research and the risk assessment that I used to do. So I think
it\'s a good start.

CHAIRPERSON CRAWFORD: Nancy.

MS. KIRKENDALL: Yes. Ramesh Dandekar has written software that does take
your Latin Hypercube data and give it the correlation structure you
want. So there\'s software available in-house that you might want to
take a look at.

CHAIRPERSON CRAWFORD: Thank you.

MS. KIRKENDALL: And it was based on the papers that you just mentioned,
the ones by Iman. So he collected all the papers and implemented the
algorithms. So that\'ll be available.

CHAIRPERSON CRAWFORD: Well, thank you. Now, we\'re going to ‑‑ I\'m
going to turn things over again to Jay for our panel discussion. He\'s
going to lead that. Some of the Committee members ‑‑ or I guess it\'s
just me and Tom; Roy, you need to go back to your seat.

(Laughter.)

CHAIRPERSON CRAWFORD: And Tom and I have to move to someplace because
the panel members are going to sit up front here.

MR. WHITMORE: Jay, Andy, Nancy Kirkendall, Steve, John. Maybe over there
where Perry has been. Perry, come on over here, Perry.

(Pause; people changing seats.)

MR. HAKES: As long as you\'re up here at the front, that\'s really what
we care about.

Okay. Time is tight, so let me get started. As you know, I have a
continuing and growing interest in the field of history, and so I
thought it might be good to point out that interest in data quality
problems is not new.

It goes back far into history, and I have an excerpt from the April
11th, [Oil Daily]{.underline} from the year 1960. And it noted:

\"A 73 percent discrepancy in the number of service stations as reported
by two of its agencies is still unexplained by the Commerce Department
nearly six months after the difference was reported. The Census Bureau
counts 194,000 service stations. The Office of Business Economics comes
up with 355,000. The Internal Revenue Service also came up with its own
number, 218,000.\"

So maybe we\'ve made some progress since then. I was just going to make
six very brief points about data quality. One is, as you become more
visible as an agency data quality becomes more important. Because
you\'re in the middle of the fray, people are going to shoot at you.

Second, as issues become more controversial data quality becomes more
important. Example, volatile gasoline prices or whenever we monetize
carbon it will be the same thing.

Three, it\'s more important to get data quality when you have changing
industries. We\'ve just hit that pretty hard. But the fourth thing that
I don\'t think we\'ve discussed yet is that EIA\'s visibility helps data
quality.

And the reason that it does is we\'ve got more clout in the
policy-making process. I think there has been a tendency because of
experience we\'ve had in the past to feel we have to beg for data or
that if some respondent goes and complains to the Congress that
immediately the Congress will erupt in a protest and we\'ll get thrown
off kilter.

I think the fact that our data is being so widely used in the Congress
right now is very, very helpful. In fact, during the break I got a call
from the chairman of one of the major committees in Congress, his senior
staff.

He was about to leave on a trip and needed information on that
particular country. I quickly lent them the country analysis briefs and
boom, they had exactly what they needed. So that helped.

Fifth, performance measures help a lot. We should be tracking major
quality trends with performance measures. That helps the people who are
working directly on the problem, but it also makes it easy to interpret
the significance of the issue throughout the organization.

And my final point is that if you can explain the data quality problems
in plain English there is a very good chance that the administration and
the Congress will actually deem this something worthy of additional
support.

So it\'s up to us to explain it, and then it\'s up to them to help us
solve the problem. That\'s all I have to do and I\'ll pass it over to
our chief statistician, Nancy Kirkendall.

MS. KIRKENDALL: Okay. This is a little bit different session. I mean, we
snared Jay to say a few words and I was going to say a few words, and
we\'re going to have Steve say a few words. And then we invited people
from the various offices to come here and sit around the table.

And we got them to come by saying they didn\'t have to say anything if
they didn\'t want to. But what we\'re trying to do is to have people
here from each office participate in a discussion and just throw ideas
around about how we should be measuring data quality in EIA.

And some of it\'s easy and some of it\'s not easy. So we did send you a
one-page document with our strategic plan, and you can see that the data
quality issues in the strategic plan are two. We\'re concerned about
response rates and I think Jay showed you a chart about response rates
yesterday.

And we\'re concerned about frames coverage. And the way it looks in the
strategic plan, if we\'ve got those two things targeted for improvement
and we make some improvements, then we\'ll call our data good.

Well, it\'s probably not that simple, but at least those are certainly
the areas we\'re concerned with now. And for response rates, at least
measuring them is easy and we do plan to include measures in our data
processing system.

We already have measures. We can track them. At least we can keep
‑‑ that one\'s an easy one to measure. The other one, on the other hand,
frames coverage is a little bit harder to measure in some cases.

Other data quality measures, survey-related measures can easily be
incorporated into our processing system, and our Performance Measures
Committee will be leading an effort to do that. Tom Groan over here is
going to lead the Committee.

Some of the ones you can expect to see are certainly response rates,
revision errors, percent imputed and whatever else makes sense. So I
think that we\'ll have a good handle on the easy ones, but the frames
coverage is another question.

We saw another chart this morning. Joan showed us a chart, this one.
This shows the declining coverage of our revenue information. This is
our industrial coverage.

These are volumes that we do have, but we are not getting revenues for
those same volumes and it\'s because the people we asked don\'t have the
revenue information. So that\'s one kind of frame deterioration.

This is the one kind that we can measure because we don\'t think we lost
the volumes, or at least this doesn\'t show that we lost the volumes,
because it\'s just the ratio of the revenue we collect to the volume we
collect.

So our harder problem is, well, how do we know in the natural gas and
the electric power areas when we do have better frame coverage? Have we
lost volumes with natural gas? We hear from modelers or from outside
people that maybe our natural gas coverage isn\'t quite as good as we
think it is in terms of volumes.

How do we find that out? How do we come up with the measure? How will we
know when we\'re done? That\'s true in electric power, also. We haven\'t
seen the deterioration in frames coverage, but then we don\'t have the
data.

One way of doing that has been to compare to other data sources, and if
they start diverging then you know something\'s wrong. If there\'s a gap
you know there\'s something wrong, and sometimes, all you have to do is
explain the gap.

That may or may not show you a frames coverage. Another interesting idea
was in Perry\'s talk he showed us the balancing item for natural gas.
And the balancing item is a difference ‑‑ it\'s basically a difference
between two data collections.

You have the supply side and the consumption side and they\'re supposed
to be equal. The balancing item is how far apart they are. So that
helps. That\'s another graphic you can use, another measure you could
look at to monitor data quality.

So those are just some ideas. Now, Jay\'s talked about in his
introduction ‑‑ in fact, your introduction yesterday was what I thought
your introduction to the session would be. So it was kind of
interesting.

But he\'s always been a big user of our data quality measures. And he\'s
talked about the measures that he needs to have to talk about our
products and the quality of our products to outsiders. The people who
run the surveys need to have data quality measures so they can see what
they need to target for improvement, how well are they doing.

And of course, the users of the data, and we have a nice big modeling
section in EIA of users of our data, are also concerned with data
quality. So the other issue that we haven\'t talked about much is how to
get input from our modelers on the data quality.

So with that I\'m going to throw it over to one modeler.

MR. WADE: I don\'t expect you to actually be able to see this overhead
that well, but EIA has a number of different models. They use a variety
of techniques, all the way from econometric, statistical modeling to
structural simulation-type models.

And I want to just give you a perspective of some of the data quality
issues that we have in the NEMS building sector modeling system. I\'ve
color-coded this chart because it\'s too detailed to really view with
one ‑‑ one ceiling from the back of the room, I\'m sure, and I\'ll just
go through the different panels real quickly here.

Basically, we\'re at very high level data here in the blue area, and
then we move down the chart and get into more and more detailed data,
and then issues that are involved, change as you go through different
levels.

Okay. For NEMS, NEMS accounting of energy starts with EIA\'s data system
sets. This is called a supplier survey data system and the type of data
are ‑‑ it provides a direct measure of total consumption in the United
States.

You can get sectoral, state level consumption estimates by fuel type
from it with a note that the sector definitions are based on supplier
records or rate schedules from the utilities. And the data example that
would come from NEMS would be commercial sector consumption of natural
gas in Florida.

That\'s the type of data element that you can get out of that. The
quality of the data is not a statistical sample, so we\'re not, you
know, measuring variances or anything of the data at this level. And we
basically take this as our given in our modeling efforts.

Now, there are a couple issues with this type of data. First of all, the
supplier definitions may not coincide with what the different sectors
are modeling. For example, public transportation, electricity, might
show up under a commercial or industrial rate schedule where it really
belongs in the transportation model, because that\'s the use of the
fuel.

And also, estimation of some types of fuel consumption can be somewhat
tricky, for example, with fuel oil that probably could be stored by the
consumers. So you have to estimate actually how much you think is
consumption for a particular year.

So that\'s the top level of NEMS accounting. And we go to another type
of a survey here. It\'s called the end-user surveys, and examples of
them for the billing sector are residential, RECS, and commercial, CBECS
surveys.

Now, I bring in some external data here in terms of characterizing the
type of data in that some of the frame information comes from, you know,
third-party sources. But basically, it\'s a survey of buildings.

Each one has, I guess we\'ll say, about 5,000 observations and you get
‑‑ the type of information you can now get from this would be, say,
office building consumption of natural gas from the South Atlantic
Census Division.

So we\'ve added a dimension. From SEDS you don\'t know anything about
what particular building type consumption may be occurring in. One of
the ideas of this survey is that given that it\'s a stratified
statistical sample type of data, you actually get estimated variances
for your concepts.

Some of the other data issues are that, you know, it requires possibly
frame design, which is a large component of manual labor and fieldwork.
And I know there\'s some efforts currently underway in EIA to reduce
that.

And for commercial buildings you don\'t really have a good frame for
selecting your stratified sample from. So that complicates the
commercial survey, too.

Then taking the data down another level, the type of data we\'d be
talking about here is not just office building consumption of natural
gas in the South Atlantic Census Division, but it would also now be by
end-use.

That\'s also provided by the surveys, but generally they bring in more
external data to provide that. For example, there\'s statistical
analysis of the survey data, simulation modeling of the survey data. You
might bring in engineering estimates or other reports.

At this point the data quality, either the variances have increased or
they\'re actually unknown because of the external data that you apply to
the ‑‑ to break the data down into the next level.

Now, you could do something different. You could bring in more direct
observations of the equipment efficiency, but I think that was tried in
RECS and that\'s been difficult to collect.

Direct estimates of energy consumption by metering appliances is
prohibitively costly, at least compared to the current cost of the
surveys, and metering every appliance is not feasible.

Now, there\'s an interesting article in the [Energy Journal]{.underline}
recently that kind of discusses optimal selection of appliances to
meter. In these conditional demand analyses there are certain types of
end-uses that are virtually unidentified, and so you can kind of decide
to put your meters where you can get the most benefit from them.

I\'m going to skip the next layer because, really, all the issues are
the same as the ultimate level of detail that we use. So our buildings
model basically starts from here and builds up, and we bring in a lot of
external data at this point in conjunction with the survey data.

What we want to know at this level, not only office building consumption
of natural gas in South Atlantic Census Division for space cooling, but
we want to know, for example, by type of equipment. In this case, in the
example, it\'s just to say standard efficiency centrifical chiller
systems with a COP at say 5.9.

The type of data we bring in to basically take the survey data below the
level of detail that we did, we\'ll get reports from contractors.
Shipment data are sometimes available from some of the manufacturers and
we have, you know, used a large amount of judgment at this point.

And at this stage of the analysis we don\'t even to think about
variance, really, because the ability to quantify it at this stage would
be truly for the road. But the interesting thing is, is that if you want
to address the current policy questions you\'ve got to start down at
this level of detail.

If you want to know the effects of the standard on commercial cooling
equipment, or the effects of a tax credit for particularly efficient
equipment you\'ve got to have some sense of what\'s being consumed all
the way down at that level.

So from a modeling perspective that\'s just some of the issues that we
would have.

CHAIRPERSON CRAWFORD: Johnny, were you asked ‑‑ 

MR. BLAIR: Yes.

CHAIRPERSON CRAWFORD: You\'re part of the panel ‑‑ 

MR. BLAIR: Yes.

CHAIRPERSON CRAWFORD: All right.

MR. BLAIR: Yes, that\'s my understanding.

MR. HAKES: Johnny Blair is our guest member of the Committee and is
going to be the ASA respondent on this issue.

MR. BLAIR: Okay. Well, I guess the area that I\'d like to focus on that
was mentioned earlier is the response rates. And as was mentioned, in a
general sense response rates are easy to measure, but what we\'re really
concerned about, of course, with the potential or the reality of
declining response rates, is potential nonresponse bias.

And when one starts to look at nonresponse, things are not necessarily
so easy to measure or so clear. I think it\'s important that one start
to look at not just what total nonresponse is, but what are the
components of nonresponse.

Quality measured by and looked at from this point of view can vary over
time. So you might see the same level of nonresponse, but the components
of it, who are the nonrespondents, the reasons for nonresponse, the
various inputs into nonresponse may be different over time.

Both in looking at ways to reduce nonresponse, which is obviously the
best solution, is to reduce it, requires a careful look at what those
components are.

Secondly, by knowing more about what those components are, you can do
more careful and more useful nonresponse weighting adjustments, rather
than the kind of gross weighting adjustments that you do if you just
kind of look at nonresponse as a whole.

The issue was raised of resources in maintaining data quality. And I
think that, again, this area of nonresponse, potential nonresponse bias
is something that is often overlooked in the allocation of resources and
that it has to really be an ongoing effort; that the measures of
nonresponse and the kinds of things that I\'ve mentioned need to be
something that are part of every survey.

And that\'s an ongoing effort rather than something that\'s sort of
looked at now and then, which typically happens. Say there\'s a dip in
response rate, so everyone kind of jumps up and pays a lot of attention.

And often, at that point you\'re well behind the curve and it\'s a lot
harder to start to do things about it than if you\'ve been tracking it
and looking at these components over time, and doing things which are
not inexpensive, like samples of nonrespondents, which often are
obviously far more costly per case, but give you good information in
terms of estimates of nonresponse bias.

The other area of nonresponse that\'s often overlooked, but that I think
is important to consider in this kind of a program is item nonresponse.
The focus is typically on unit nonresponse.

It is when a ‑‑ whatever the unit happens to be ‑‑ a business or a
consumer, end-user, whatever, but often it is ‑‑ well, let me just say
it\'s not unusual to see that as response rates go down that you start
to see also a degradation in the data that you are getting, and item
nonresponse can creep up.

Item nonresponse I think can be especially damaging because it can
contribute in a couple of ways to total survey error, in that when you
get item nonresponse you have the increase in sampling variance, because
for those items you\'ve got smaller sample sizes.

At the same time you have a potential increase in bias because that item
nonresponse is generally not randomly distributed. So that source of
nonresponse when you start looking at item nonresponse can really have
an important effect on total means wherever you\'re measuring total
survey error, because of it contributing both to sampling variance and
potentially to sample bias.

And I guess I\'ll stop at that point in terms of that area. I think that
that\'s something that everyone, lots of agencies are looking at. And I
guess the last thing I would say is to take advantage of the fact that
other agencies and people outside of government, pretty much everyone in
the survey world, is focusing a lot on nonresponse because people are
finding that either response rates are declining or they\'re having to
spend a lot more to maintain the response rates.

And as I mentioned, those components of nonresponse are shifting over
time.

CHAIRPERSON CRAWFORD: Let\'s just see if anybody has anything they\'d
like to add, either from the Panel or from the Committee.

MR. FRENCH: A couple of philosophical comments. Number one, Jay, was the
second thing that you said: the more visible you are, the more important
data quality is. That was a statement you made.

Therein lies an enormous problem for this organization, because our
visibility is widely variable over time. Now, what data quality are we
supposed to insure over time?

The data quality that corresponds to the very precise information that
people want at the time that we are in the headlines, such as in the
last three months, or are we supposed to get along, as we have been over
most of the last decade, struggling along under constrained budgets and
other types of resources because energy was out of the news and nobody
really cared.

One can preach about quality as much as one wants. I happen to be a
little bit concerned about sampling error myself. I operate surveys that
are not operated in anywhere near an optimum way, because the sample
sizes are not anywhere near what the development activities, the cost of
the development activities should be supporting.

But there\'s a limit to what we can do under the resource levels that we
have. And it\'s an ongoing fight for us to try to convince people that
in times when people don\'t care about energy that it\'s still important
to keep the quality of the information up, and I don\'t know how we do
that.

The other thing I would say about quality measurement is when I came
into the government 29 years ago the National Center for Health
Statistics was trying to develop an overall measure of health status in
the United States population.

They gave up because health status is such an anomalous and multifaceted
concept that while you can measure pieces of it and look at the parts of
the elephant, you have a very great deal of difficulty holding onto all
of the elephant at once.

The same is true of data quality. While performance measures can tell us
a lot of things, there are so many aspects to data quality that we will
probably never get our hands on the elephant in total. And indeed, we do
emphasize many of the things that have been discussed here.

But sometimes, when I think about the idea of, we need performance
measures and we just need a couple of performance measures because we
don\'t want to overload people with measures of our performance, we\'re
probably shortchanging something of the area of quality.

And I don\'t have any answer for that. You just ‑‑ you can\'t get hold
of the entire elephant at one time.

MR. HAKES: Can I address this, because this sort of is a question of
where the statistical world interacts with the political world.

MR. FRENCH: With the real world, yes.

MR. HAKES: And we\'re sort of seeing in the census now one way in which
that interaction occurs. During the 1990s we had significant cuts in our
budget. We went from 85 million a year, I think we got as low as 66 or
65, something like that.

That\'s a pretty significant cut, a very significant cut. And I think
collectively as an organization ‑‑ I had a lot of influence on the
decision, but I think it was collaborative in many respects ‑‑ we
decided to keep almost all of our data series going and sort of defer
maintenance.

Whereas, another approach we could have taken was we could have
discontinued more data series and said, we\'re only going to maintain
data series that we can maintain the highest level of quality.

But I think most people in the organization were reluctant to give up
data series. Now, if we\'d done that it would have been an interesting
game of chicken with the Congress, because then we would have gotten
some reaction, Hey, this data series is gone; I need it for my work, and
then perhaps maybe the cuts wouldn\'t have been as great.

But the deferring maintenance is more invisible to the outside world. I
believe that you need sort of to be able to be multi-lingual when
talking about these data quality problems. If Nancy\'s talking about it
with Dwight, she may have, you know, 20 different measures of quality to
fully get the richness of the issues.

But if I\'m going over to one of the Congressional committees, having 20
measures is about as useful as having none, because at some point you
have to communicate with an audience and the Appropriations Committee is
dealing with more than the Department of Energy.

And the Department of Energy budget is \$18 billion and our is 72. You
can do the math and what the percentage is. There\'s a limited amount of
time. So I think another term I use for performance measures is summary
statistics.

And there are times that if you cannot summarize, you cannot be
effective. This is not to say the summary should be mistaken for the
most careful and intellectual work that we\'re capable of doing. That
needs to be done, also.

So that\'s why I have urged the approach of this in having very
sophisticated measurement tools that allow you to get down into a lot of
depth and have common-ground measures, but also have another set of
measures that can be part of broader strategic thinking and spread out
to a larger audience.

I think if we can do both of those we can be extremely successful as an
organization. Now, what I think we have been able to do with strategic
planning is to elevate the quality issue into the political world where
it\'s sort of understood.

And now, if we go to the Seventh Floor or the Congress and say, we got a
data quality issue here; here\'s our assessment of it, over the last two
years we have gotten budget increases, albeit small, based on that
argument.

This year we will get a budget increase which I think will be
substantially larger. And I think if we can go back next year and say,
here\'s what you gave us, here\'s what we delivered, explain it in
English so that they understand what it was we delivered and why the
data is better and what difference that makes, I think there\'s a chance
for another sizable increase next year.

I think it takes awhile to establish this communication between the
statistical world and the political world, but I think that we\'re
moving rather strongly ahead. But I think we cannot treat it solely as a
statistical.

Now, there are a number of things that can be done with very limited
resources. You know, for instance, redesigning the cost of air may
increase the quality of your data. That has some cost to it, but it\'s
not as big a resource issue as redoing a frame.

Calling up the CEO of a delinquent company and telling them to get on
board doesn\'t cost very much money, but it might improve the data
quality a lot. So there\'s whole sets of things that we can do within
limited resources.

But ultimately, if you\'re talking about readjusting ‑‑ or adjusting to
the 2000 census, as Dwight is doing, and Dwight was never able to adjust
to the 1990 census, which I think is shame on all of us that that was
allowed to happen, but that takes money.

And ultimately, we have to do frames. That takes money. But so far,
we\'ve been able to get the money for Dwight to adjust to the 2000
census, and we have been able to get money for these projects in natural
gas and electricity.

And we ‑‑ you know ‑‑ is it as much as we would like? No. But will we
get more in the future? I think we will.

MR. BLAIR: If I could comment. I\'d like to comment on a couple of
things that Dwight raised. One, this sort of philosophical question of
data quality, and all of the different components of it.

And I think one of the things that we often do is to measure those
things that are easy to measure, which is one of the reasons, you know,
we know a lot more about sampling variance than we do about sample bias,
because we have the tools to measure that.

Sometimes, that\'s the right thing to do; sometimes, not. I think that
some fair amount of effort needs to be devoted to looking at, of those
various measures of data quality what are the ones, whether easy or
difficult to measure, that really are most important, given the kinds of
estimates that you\'re trying to produce.

And whether you have limited or, in some perfect world, unlimited
resources, still the focus should be on those things that really
contribute most to the total effect on data quality. And that often
means that sometimes you are going to have to make efforts to measure
things that aren\'t so easy to measure.

And then just a small point on the question of sample size, which of
course is always an issue. Everyone would like to have a larger sample
size. It\'s important to keep in mind that there is always a trade-off
between how many units you collect data from and how much data you
collect from each of those units, and that there are a couple of ways to
look at sample size.

One is that if you want a larger sample it may mean that you\'ve got to
collect somewhat less data from the units in order to get a larger
sample size with either the same cost or some acceptable increase in
cost.

And secondly or finally, it\'s not necessary in terms of increasing
sample size to collect the same amount of data from everyone. It may be
useful in some instances to increase sample size by adding units to your
sample.

But collecting only a small number of key variables from them that can
still perhaps be useful, both in your overall estimates and perhaps in
making adjustments for that sample that you\'ve collected the full set
of data from.

MR. HAKES: Other EIA officers want to make comments?

THE AUDIENCE: I have a question for Johnny Blair. You mentioned a while
back that you felt that response rates, people were taking response
rates more seriously. And I did a small survey about six months ago
about 20 firms, you know, I once called at random, but we tried to get
large and small and different parts of the country.

And I didn\'t really find that. I found that there were two major
things, you know. Those who were ‑‑ who had government customers, you
know, had a higher rate because the government was sort of paying more
money, and those that were dealing with private customers we found, you
know, 60, 50, sometimes even 40 percent.

And a lot of them felt that they were able to use those smaller response
rates because they sort of knew what the customer wanted, a little here
and there. And it was just so hard. But you know, my impression was that
it was sort of just going in the opposite direction.

MR. BLAIR: Well, certainly there is a difference between the government
and nongovernment surveys. And I think that there is ‑‑ certainly, I\'ve
talked with a lot of people, for example, at Westat, who do a lot of
government surveys, and there is a concern about response rates.

And though the decline isn\'t as marked with the government surveys, the
standards are higher for those surveys. And so a smaller decline is as
much a concern sometimes as a larger decline in some other surveys.

So I think a lot of attention is being given to that. And again, the
point I would make is when there starts to be signs of these types of
problems is the time to start paying attention and devoting resource to
them, before they start to go over some threshold where you then really
have concerns about the validity of your data.

And oh, I guess just one other thing I would say along that line is that
because even when you are only seeing small declines in response rates,
I think you also have to look at the cost factor, because what certainly
some ‑‑ and I don\'t know if there is any published, careful analysis of
this ‑‑ but certainly what some organizations are finding, Michigan, for
example, I know, is to maintain response rates, which they\'ve been able
to do in a lot of their surveys, is costing them a lot more per case
than, you know, if you look at the change over time.

And then finally, on this issue of nonresponse, a recent book by Groze
and Cooper or maybe it\'s Cooper and Groze, I\'m not sure which, on
nonresponse is that sometimes there isn\'t as much difference between
the nonrespondents and respondents as you think.

And when they\'ve done analysis of sort of what would the data set look
like if response rate had been cut off at this point, versus what would
it look like if, you know, it achieved the response rate it did, the
results are quite surprising sometimes in that you don\'t find the
differences that you expect, which is why everyone is always going to be
concerned about response rate, because it\'s visible.

It goes back I think to the point I made that you really have to look at
these things more carefully to see what\'s really affecting your data,
because sometimes it\'s not what at first blush you might think.

MR. HAKES: Jim.

JIM: One of the things that\'s important with statistics is to make sure
that you\'re measuring what\'s there and not ignoring what\'s also
there. Perry very politely referred to the non-petrochemical ‑‑ or the
non-fuel uses of petrochemical feedstocks.

Somewhere over half the propane that comes out of the ground and over 60
percent of the ethane that comes out of the ground goes into plastics.
You know, and then there\'s the butanes, and we don\'t really measure
those at all.

They\'re sort of derived indirectly for the modelers. And yet on the
survey side because it\'s complex and difficult, it has been
historically ignored since the very beginning. And it\'s not the only
area in which there are physical changes in the use of fuel that affect
the way that data is collected.

And it\'s important to look at the current flow of the physical, you
know, commodities to see where they\'re going and how they\'re winding
up. You know, an example is in the reformulated gasolines. We really
don\'t have a good handle on those in terms of what exotics are going in
to replace the benzenes that came out.

And you know, these are statistics, but they aren\'t really addressed
well, and they\'re difficult.

MR. HAKES: Roy, it\'s your ‑‑ 

MR. WHITMORE: Yes. I just wanted to know what it looked like, kind of an
interesting trend the way this was presented here in the overhead. You
know, it started on the very aggregate statistics. You know, there\'s
certainly measurement errors and such things as that.

But essentially, you\'re dealing with census level data there and point
estimates without any real characterization of uncertainty do a pretty
good job. And then you go to the next level and you\'re dealing with
survey data and you\'ve got sampling weights and known probabilities of
selection and strata and whatever and can calculate sampling variances,
and therefore, very directly get interval estimates, constant and then
centerable estimates.

And then as you go to the additional layers you have some additional
modeling superimposed on the process. And then it becomes much more
uncertain as to how you should quantify the uncertainty.

And it just kind of reminded me of the Monte Carlo sorts of things that
we were looking at in this last paper as being something that, you know,
you probably wouldn\'t want to do for everyone one of the kind of
model-based estimates.

But certainly a way of doing that is maybe you\'re looking at certain
percentage changes or whatever, or a percentage of this product may be
deviated to that use, particular end-use technology class or whatever,
and you might go with the uniform distribution of what that percentage
looks like.

You think, well, there\'s probably about 30 percent that goes here, but
you know, it could well be somewhere between 28 and 33. You can use the
uniform distribution on that range to do a Monte Carlo simulation, and
generate interval estimates when it\'s necessary or important to
quantify uncertainty on some of these more model-based estimates.

MR. HAKES: We were just talking about that before I came up here, the
idea that you\'d really have to judgmentally just estimate what the
variances were at the bottom level for your distributions to sample
from, because you really don\'t know.

But I guess it\'d be more akin to a sensitivity analysis at that point.

MR. MONTGOMERY: I was going to comment on that same point, and maybe I
could make a suggestion. I think this is something that we ought to take
up at one of these meetings. I think it\'s really important and very
difficult.

I would kind of raise it maybe coming out the other end, whether
‑‑ there are some threshold issues. And I think the threshold issue here
might be whether EIA should be using models that are based fundamentally
on data that don\'t exist, and how close we are to that situation here
and whether it\'s feasible to do the kinds of simulation that it would
take to tell how inaccurate these models are, because there are
alternative approaches to these technology based efforts.

And you know, I actually found that very enlightening because I think
you seem to be ‑‑ you really were suggesting the data that are required
in order to make these models literally don\'t exist. There are some
sources of information, but they\'re not the information that the model
is using directly.

It\'s kind of using it to act as if there was a great deal more data,
and we should know more about that.

MR. WADE: It\'s not that the data don\'t exist. It\'s that the national
samples that EIA supports don\'t go all the way down.

MR. MONTGOMERY: Yes.

MR. WADE: You might find a utility, for example, that\'s done some
metering and bring in a metering study that maybe doesn\'t have the
national representative list that you\'d want of your sample, but it\'s
actually data.

It\'s just data of limited geographical exposure and you don\'t know how
its being combined.

MR. MONTGOMERY: Exactly. And I think ‑‑ that\'s why I think it would be
worthwhile kind of addressing it in some more depth.

MR. SACQUETY: I\'d like to comment just on the electric power side.
That\'s one of the challenges we\'ve been having in restructuring. I\'ll
just bring out a simple area of new plants, especially for nonutilities.

We basically add them to our data or to our frame when they\'re getting
sited, because we do have the agreement with the Environmental
Protection Agency that we will give them a plant number or a code so
that they can do an environmental impact statement.

But then you just hear about Duke Power Company saying that they\'re
purchasing all these generators from G.E. Well, you know, in the next
three years they\'re putting up a lot of plants, but they\'re not even
sited yet.

They have just ‑‑ you know ‑‑ they\'ve announced to the world that
we\'re going to be putting this much generation on, but we don\'t even
know what state they\'re going to be in. We don\'t even know what
location.

You know, we work with NARUC and they have projections ahead; it\'s
supposed to be five years. But we also know that some are sited and some
are just paper plants that they need to try to balance their equations
for what their demand is.

And that\'s something that I think we have to work together. I think
staff has to try to be better prepared to look at that. It\'s a real
challenge to me on the type of data that we\'re getting and how much we
can rely on it.

MR. HAKES: Well, studies have shown that the performance of humans is of
higher quality when they have the time and resources to eat lunch.

(Laughter.)

MR. HAKES: So before David made his comment, Nancy and I were just
whispering that I think this panel has the seeds of several future
panels for ASA meetings. I mean, I sort of apologize for going over some
of these items almost superficially, that I think each of these
presentations could be almost a whole panel.

And I would hope that this would be a strong, strong focus, because I
think the bar is going to continue to get raised for a whole variety of
reasons, and I think this is an area where EIA has a lot of work to do.

And I think the Committee is just performing a totally valuable service
to interact with us on these and advise us. I\'ll turn it back to Carol.

CHAIRPERSON CRAWFORD: Okay. Thank you, Jay.

I guess at this point I\'d like to invite public comments on this
discussion. And if there are no further comments, I\'ll adjourn the
meeting.

Thank you very much ASA Committee members and also EIA members for your
nice presentation and all the work you did. Thank you very much.

(Whereupon, the Spring Meeting of the American Statistical Association
was adjourned at 12:14 p.m.)