AMERICAN STATISTICAL ASSOCIATION

\- - - - -

COMMITTEE ON ENERGY STATISTICS

\- - - - -

MEETING

\- - - - -

FRIDAY, APRIL 26, 1996

The Committee convened in the Clark Room of the Holiday Inn Capitol, 550
C Street, S.W., Washington, D.C., at 9:00 a.m., DR. TIMOTHY D. MOUNT,
Chair, presiding.

PRESENT: TIMOTHY D. MOUNT, Chair

SAMPRIT CHATTERJEE

BRENDA G. COX

JOHN D. GRACE

CALVIN KENT

GRETA M. LJUNG

RICHARD A. LOCKHART

DANIEL A. RELLES

BRADLEY O. SKARPNESS

G. CAMPBELL WATKINS

ALSO PRESENT: RENEE MILLER

YVONNE M. BISHOP

DIANE LIQUE

L.A. PETTIS

JAY HAKES

JOHN WOOD

GORDON M. KAUFMAN

ROY KASS

NANCY LEACH

I-N-D-E-X

Introductory Remarks:

Announcement of Winners of the Contest on

Graphs and Visuals Displays 3

*Restructuring the Oil and Gas Crude*

*Reserves Program* (Agenda Item 5)

Presenter: John Wood, Office of Oil and Gas 8

Discussants: Gordon Kaufman, MIT 19

Brenda Cox, ASA 29

*An Update on Issues Pertaining to the*

*Restructuring of the Natural Gas Industry*

(Agenda Item 6)

Presenter: Roy Kass, Office of Oil and Gas 54

Statistical Issues Pertaining to

Re-engineering at EIA (Agenda Item 7) 77

*Measurement Model for Information Management*

*Process* (Agenda Item 7a)

Presenter: Nancy Leach, Energy Markets

and End Use 77

*Is the a Summary Measure of Data Quality*

(Agenda Item 7b)

Presenter: Renee Miller, Office of

Statistical Standards 84

Discussant: Dan Relles, ASA Committee 93

Public Comment 117

P-R-O-C-E-E-D-I-N-G-S

MR. MOUNT: I\'d like to get started, please. If people would get seated.
I\'d like to welcome everybody to the meeting of the ASA Committee. A
couple of introductory remarks: I\'d like to welcome Larry Pettis and
Gordon Kaufman from MIT, a former committee member; and I would also
like to congratu­late Jay Hakes for his TV appearance last night and a
promotion of the Web site. I imagine we\'ll see a great surge of
interest as a result of that.

We\'re now turning over to a very important event, and that is to
announce the winners of the contest on graphs and visual displays. Jay
Hakes.

MR. HAKES: Good morning. I don\'t think we need with this group to
discuss the importance of graphic portrayal of information. We live in
an age where at least some people need to absorb information very
quickly. They don\'t have time to go over a lot of numbers, and graphics
often tell a good story. I think you all have had some good discussions
in the Committee here that have helped us see the role of graphs. I
would report to you that we\'ve had consid­erable success over the last
year or two getting a number of our graphs into the popular press.

[USA Today]{.underline}, from time to time, will carry EIA graphs. There
have been ‑‑ I think there was one a day or two ago in the [New York
Times]{.underline}. They\'ve had several recently. The [Wall Street
Journal]{.underline} has had some and the [Washington Post]{.underline}.
They now do a better job of attributing EIA as the source. So I think
the emphasis on graphics has helped educate the public on energy issues
and brought attention to the even richer data that is available for
people who want to drill down and find it.

A couple of years ago, we collectively decided that it might be a good
idea to highlight those who did good graphical work and give them some
special recognition. One of the roles of a manager is to keep track of
different ideas and make sure they\'re well coordinated, and I failed a
little bit in that regard because I offered at that time to personally
take out to eat the members who won the graphic contest and neglected to
remember that when I, several weeks later, emphasized the importance of
teams. So now there are more and more people to take out to dinner every
time we have this contest. So there will be less lobster and more pizza.

I\'d like to thank the nine judges who worked on this. We had three from
the Committee: Samprit Chatterjee, John Grace and Bradley Skarpness. We
appreciate your work, going over and making these hard decisions; and
then within EIA we had Theresa Hellqui­st, Bob Rutchik, Susan Shaw,
Sandra Smith, Alan Swenson and Ann Whitfield.

We\'re about to announce the winners. Winner No. 1 is the team of Mary
Carlson and Phil Shambaugh; and accepting for this group is Jim Todaro
from the Office of Oil and Gas.

Come on up.

Mary is a repeat winner, incidentally. I remember she won either last
year or the year before.

Jim?

This is the graph that won this particular award, and I actually had
seen this before now, but if you can see how the graphic does give you a
better sense of the impacts of temperature.

Then our second winner is John Herbert, also of the Office of Oil and
Gas; and he developed this graphic on the premium value of gas,
reflecting supply uncertainty and weather conditions. That is another
one. I know John fairly well. I play tennis with him occasionally, and I
suspect that\'s what he\'s doing today. So deliver that back to John. We
would appreci­ate it.

Then our next award winners ‑‑ actually, one of the winners I see is
here: Michael Lawrence and Hattie Ramseur from Energy Markets and End
Use. This is their graphic. This office has done an excellent job of
developing color brochures that summarize a big, large document.

Come on up, Mike.

These brochures have been widely used, and they\'ve also helped bring
press attention to the bigger document. We find a lot of policy makers
don\'t have time to read a big document, but they will read a brochure.
In fact, the only problem I\'ve had, I handed them out at some meetings
‑‑ secretary\'s senior staff meetings, these brochures ‑‑ and the
problem was as we moved on to the other agenda items, people kept
reading the EIA brochures. So I guess that\'s a sign you did a good job.

Then the final set of winners would be Joelle Davis and Imelda Rivers
from Energy Markets and End Use, and accepting on behalf of the team
would be Nancy Leach.

MS. LEACH: They\'re also off.

MR. HAKES: Yes. Okay, this is another pamphlet. If you haven\'t seen
this pamphlet, I think you would want to get a hold of it. Okay, it\'s
being distributed, and I think you do see it is a nice way. In a sense,
this is part of our vision, I think, that in the future we may be having
smaller printed publications that may be viewed somewhat like these
pamphlets that are then backed up by a rich array of resources
electronically. That mix seems to make some sense.

I don\'t know if any of the winners want to thank their mothers or
anybody else; but other than that, I will pass it back to Tim.

I always hate to miss a presentation by John Wood, but I do have to go
give a briefing on energy to some of the staffers in the Senate this
morning. We will be discussing EIA\'s \'97 budget with them, so I think
I need to be there, John. But I\'ll talk with you later.

MR. WOOD: Good luck.

MR. HAKES: I appreciate it. Thank you.

MR. MOUNT: So we move on now to the first presentation on Restructuring
the Oil and Gas Crude Reserves Program. The presenter is John Wood,
Office of Oil and Gas.

MR. WOOD: Good morning. Due to budget reductions, our resources in the
Reserves Program must be reduced. The annual reserves report is in the
EIA Flagship publication. It contains the highest pro-rated data series
in the Office of Oil and Gas because of the uniqueness of the data.
There\'s a 50-year history of the annual reserves data. The last 18
years of this data has been published by EIA, and we would like to
retain as high a quality annual program as resources permit.

And just some of the ‑‑ we have a 99.8 percent response rate from our
respondents. Just there are certain things that we do that we\'re very
proud of.

Next slide.

But we do have a problem. Due to budget reductions, the resources
devoted to the Reserves Program must be reduced. At this time, we do not
plan to conduct a full reserve survey. That\'s a 3800 sample out of
23,000 operators for the 1996 data year. The 1995 data year\'s survey is
in progress.

Next slide.

We\'ve asked the ASA Committee to comment on three possible options.
Option one: a complete reserve survey every other year, with no reserves
report or survey made for intervening years. Two: a complete reserve
survey every other year, with published reserve estimates based upon
model reserve\'s behavior made for the intervening years. Three: a
complete reserve survey every other year, with published reserve
estimates based upon data from a sample of the largest 150 operators.
The remaining reserves would be modeled and estimated.

We have two questions: (A) In light of the reduced resources, what
option would the Committee recommend; and (B) Does the Committee have
any other ideas or proposals on how to implement restructuring for the
Oil and Gas Crude Reserve Program?

Next slide.

EIA created the Reserves Program to estab­lish a unified, verifiable,
comprehensive and continu­ing statistical series for crude reserves and
crude oil and natural gas. The annual reserve report provides the most
accurate yearly estimates of U.S. pre-reserves of crude oil, natural gas
and natural gas liquids.

These estimates were considered essential in the development,
implementation and evaluation of natural energy policy and legislation.
It is based upon data filed by operators of oil and gas wells and the
operators of natural gas processing plants, customers, our Congress,
federal and state agencies, the oil and gas industry, the financial
community and the public.

The reserves data is used to inform Congress and resource assessment,
strategic planning and modeling. In particular, it is the annual field
reserve data resulting from the survey ‑‑ this is field-level data ‑‑
that have allowed EIA and other groups, like United States Geologic
Survey, to reassess and enlarge the oil and gas resource-base estimates.
That data, then, fields the file ‑‑ the oil and gas integrated field
file which EIA has built and maintains.

A substantial loss of detail in these databases would hamper future
resource assessment and modeling efforts. The higher resource
assessments that have been made lately have allowed, for example, U.S.
production estimates for gas to grow through the year 2015. The latest
USGS resource assessment reflects future additions to crude gas reserves
from known fields of 322 trillion cubic feet. That\'s 247 percent higher
than the previous estimate. Similarly, for crude oil, an increase of 60
billion barrels, or 184 percent higher as assessed. This is just the
growth of reserves in known fields.

Next slide.

Now what are crude reserves? EIA defines crude reserves as those volumes
of oil and gas that geologic and engineering data demonstrate, with
reasonable certainty, to be recoverable in future years from known
reservoirs under existing economic and operating conditions. Now over
time, increasing knowledge of an individual field, changes in price,
changes in technology all make a difference in reserve estimates over
time; and the net effect has been generally for increases.

Next slide, please.

This slide shows the types of reserve data collected at the field level:
the data on new field discoveries, new reservoirs and old fields,
extensions to the reserves resulting from extending the actual known
boundaries of a given field, and revisions and adjustments to reserves.
The sum is reserve addi­tions.

Note that the rather small ‑‑ the first positive stack there is the new
field discoveries. The volumetric contribution of the new field
discover­ies is usually quite small in the individual year. The sum of
these various components is the reserve additions.

It is the reserve additions in old fields, as more of the oil in places
recovered and estimated to be recovered, that dominates the oil reserve
additions. That\'s the largest component.

Production has been larger than reserves in most years. The red line
shows the net reserve change, and it\'s generally negative.

In 1994, there was a 2.2 percent decline in the crude oil reserves, and
this is the smallest decline in four years. The current oil reserves are
22.5 billion barrels.

Next slide, please.

Crude oil reserves change annually. It\'s changes vary by region. You
have plotted up here, the blue line, the U.S. total. Texas, the largest
oil reserves state, and Wyoming, which is a state with significant
reserves plotted in green.

In 1986, as oil prices and drilling cratere­d, all three dropped. But the
U.S. decline was 5 percent; Texas, 8 percent; and Wyoming, 11 percent.

In 1990, Texas was up 2 percent. The U.S. total was down 1 percent, and
Wyoming was down 4 percent.

And the last set of points plotted, in 1994, the U.S. total was down 2
percent; Texas was down 5 percent; and Wyoming was down 9 percent.

I just put this up to show their significant annual changes, and these
changes are not necessarily the same in any individual region.

Next slide.

The reserves data are presented regionally. The reserves are regionally
concentrated, with five areas containing 64 percent of the natural gas
reserves. Thirty-three states in the federal off-short have oil and/or
gas reserves.

In 1994, gas reserves were up for the first time in four years. It was a
1 percent increase and it\'s a good thing. There is a relationship
between crude reserves and production, and you can\'t have increasing
production for a long period of time without stable or increasing crude
reserves. The U.S. is counting on increased gas production for the year
2015. In 1994, the gas reserves were 163.8 trillion cubic feet.

Go to the next slide, please.

EIA\'s Reserve Program depends fundamentally on survey data gathered on
Form EIA 23. We receive company confidential reserves data that are
provided to us on the survey each year. This is a sample survey of
23,000 operators. This represents opera­tions in 45,000 fields. The
larger operators, the 650 or so companies, provide field-level data. The
smaller operators provide state-level estimates for their reserves. In
1994, the original sample was 4100 operators.

Next slide, please.

Our current sampling strategy depends of stratified sampling. There\'s a
certain E-sample stratum of Category One: large operators; Category Two:
intermediate operators ‑‑ They report at the field level ‑‑ and Category
Three: small operators above production cutoff were selected with
certainty. There\'s a random-sample stratum where there\'s an 8 percent
sample collected. State by state, there\'s a process run which minimizes
the total number of respondents and still meets our target sampling area
for that state.

Base-started sampling areas are: 1 percent for national estimates; 1
percent for each of the five states having subdivisions: Alaska,
California, Louisiana, New Mexico and Texas; 2.5 percent for each state
having 1 percent or more of estimated U.S. reserves for production; 4
percent for each state having less than 1 percent of estimated U.S.
produc­tion; and 8 percent for states not published separate­ly.

Could I have the next slide?

Now the size of operators is very skewed. In the 1994 frame, there were
161 Category One operators, and they were sampled with certainty, and
those operators represented 87.5 percent of the oil reserves and 83.6
percent of the crude gas reserves in 1994.

The Category Two operators, 482 of them, represented 5.1 percent of the
crude oil reserves and 10.5 percent on the gas reserves.

In the Category Three operators, again, the frame had 22,211. Of those,
we pulled a sample of 3,431, about half and half with certainty and
random-sampled operators; and they represented 7.4 percent of the crude
oil reserves and 5.9 percent of the crude gas reserves.

Now it is this distribution that leaves us to believe that a certain new
sample of Category One operators would provide a solid set of data every
other year that reserve estimates could be based on.

Could I have the next slide?

Now I\'d like to review the options we asked the Committee to consider.
Option one is basically no report, no survey, every other year. The
pros: there\'s no modeling required. There\'s no expenditures for
publishing data. The cons: there\'s no data collected. The data series
is interrupted. The field-level data is lost. EIA would be unable to
provide annual national or regional reserve informa­tion.

Could I have the next slide?

Option two would be a report of reserves made during non-survey years
based on model reserve behavior. The pros: it would provide an estimate
of national reserves annually for customers. The cons: there\'s no data
collected. The data series is interrupted. The field-level data would be
lost. The accuracy of the national reserves would be question­able, and
the regional reserve estimates even more questionable.

Okay, the next slide.

Option three is a report of reserves based on a smaller survey every
other year. The pros: the data series is less than complete, but is not
inter­rupted. The field-level data for data editing and resource studies
would basically be available. It would provide estimates of national
regional reserves annually that is based on data. It\'s more reliable
than option two in intervening years. The cons: a sample of roughly 150
operators would be surveyed instead of 3800. More estimation would be
required to expand the limited data sample, and the accuracy of regional
reserves would be diminished.

Could I have the next slide?

I\'d like to review the questions we asked the Committee to consider,
and they were: (A) In light of reduced resources, what option would the
Committee recommend; and (B) Does the Committee have any other ideas or
proposals on how to implement restructuring for the Oil and Gas Crude
Reserve Program?

Thank you.

MR. MOUNT: Thanks a lot.

First discussing it is Gordon Kaufman for MIT.

MR. KAUFMAN: Thank you. It\'s a pleasure to be here.

MR. MOUNT: Do you want to come up here or do you want to be there?

MR. KAUFMAN: I\'m happy to sit here. I have a drill sergeant voice.
Everybody can here me.

I\'m delighted to be here and, in particular, to share with you some
thoughts on this set of issues. I have followed EIA since its inception,
and my heart has always been principally in what John Wood has been
talking about. Indeed, EIA 23 is a flagship program, and it\'s
universally recognized to be so.

One of the questions that John asked is a technical question conditional
upon the budget being recast, and we have to do this. What particular
steps can we take to do imputation, or whatever it may be. You\'ve got
to cover the holes that would be left by a substantive reduction in EIA
23. And as I thought about this, I said, \"Well, we could spend a lot of
time, instead of fishing, just talking about details and frames and
structures and so on.\" But I would rather not do that at this juncture,
because I don\'t believe that there\'s a clear-cut technical solution to
the degrada­tion of quality of data that would follow a drastic axing of
pieces of EIA 23.

In particular, it\'s my personal belief that the loss of field-level
data will ultimately be a disaster. And why do I believe that? It\'s
probably not true if you wish to compile, as you are required to do, an
historical record of the resource base for purposes of statistical
preview by those who like to look at the historical record. But the real
power of this exercise comes in its ability to provide data that is
useful for detailed policy analysis both within Government and outside
of it.

John\'s presentation accents what I would call Morry Adelman\'s dictum.
What\'s most important to measure are not stocks. They are flows, and
flows are changes. When you talk about inferred reserves and proved
reserves, if you back away from a tradi­tional view, you\'re really
talking about flows; and the proportional variations in flows ‑‑
proportional to the mean or medium size of these flows ‑‑ tends to be
much larger than it is in stocks.

The old saying that \"the devil in the details is in the details\" is
present here in this activity, and I\'d like to return to that thought
after asking some ‑‑ well, not so rhetorical questions, although you
started answering some of them for us, John. What are the costs of
ensuring quality in this particular domain? It is always bad politics to
talk about trade-offs, but I\'m neither a politician nor a government
administrator. But as somebody\'s who\'s a concerned citizen, I feel
it\'s reasonable to talk about trade-offs.

There are alternatives to allocation of funds, just as there are
alternatives to deciding how much \"f\" work to put into a sampling
strata, a sampling design with a fixed budget. And it\'s hard for me, at
least personally, to make a recommendation as to which course of action,
among the ones that you have put down, is, in my personal judgment, the
best without having some kind of referent.

What I mean by that is that you said, \"Let\'s cut EIA 23,\" and
presumably other things are being cut; and I would hope that as you
consider the strategy, the kind of EIA corporate strategy, if you like,
that you might give some consideration to reallocation. So what are the
costs of ensuring quality? I don\'t know the answer to that.

Renee is going to talk to us about measuring data quality, which I think
is quite a\` propos and relevant to this discussion, and she\'s going to
talk about, as I gather in your nice summary of this, consistency and
continuity. Can policy analysis be done with the same degree of rigor in
precision as time goes on if the magnitude of the cuts that you propose
in EIA 23 take place?

Ultimately, if you do not maintain the quality of the field-level data,
it\'s my personal belief that you will substantively damage your ability
to do rigorous policy analysis as the current degree of precision the
EIA has. It has taken years to build up this particular program and
system, and you have an enormous amount of in-house expertise and
knowledge ability. It gets back to \"the devil is in the details.\"

What\'s happened to the resource base ‑‑ the oil and gas resource base?
It\'s self-evident that the size distribution of fields has shifted and
that much more of the activity and contribution to the flows here is
coming from fields of smaller magnitudes. And you\'ve got to keep track
of that if you want precision in this domain.

So at this juncture, I will be happy, if pushed, if you force me into
the mold of saying, \"Well, we have Option One, Option Two and Option
Three, Gordon. Which do you choose?\" But what I\'m arguing for here is
a review of this activity to see if there is not a way of absolutely
ensuring the quality of the field-level data; and we can move later to
talk about the technicalities about how you might do that, but I\'d like
to just stop there.

MR. MOUNT: We have an extra discussant from the Committee, John Grace.

MR. GRACE: Unfortunately, I don\'t have the eloquence or diplomacy of
Gordon. I\'ll just say straight out it\'s a stupid idea.

The database that\'s collected by EIA in this activity is not only
absolutely critical to current policy studies, but its degree of detail
and ability as a database to be responsive to what will be changing
needs of analysts and policy makers is going to grow dramatically.

As John mentioned, the USGS, when they came out with the national
assessment last year, which I worked on for the last three or four
years, estimated that around 300 trillion cubic feet of natural gas and
60 billion barrels of oil can be expected to come as additions to
reserves from discovered fields. That\'s not new discoveries. Compared
to the proved resource base, it\'s over double the amount of gas and
triple the amount of oil.

The thing that we need to know in order to make those assessments is the
detailed level of data that we find in the OGIFF file, the oil and gas
field information file; and without that level of detail, we can\'t do
even the volumetric estimates of how much we can depend on in the future
volumetrically. And these quantities are the backbone of the future
production of oil and gas over the next 15 or 20 years. They\'re not the
undiscovered resources that, in the survey, we spend so much time and
effort thinking about.

As John showed, the contribution of undis­covered fields to the annual
increase through discov­eries is very tiny compared to these, and this is
what we learn about in the annual survey done by EIA.

Could I see the next slide?

Even though the OGIFF file gives us the ability to understand now what
the volumetric contri­bution might be, the great gap that exists in our
understanding of the supply function of oil and gas in the United States
is now large and is getting larger because we haven\'t mined the OGIFF
file to the degree we need to to understand the relationship between the
marginal costs of those inferred resources and the volumes that will be
brought to market as prices go up and down.

We have an idea of what the economic\'s proved resources are by the
definition of proved resources. Necessarily, they\'re the resources that
are capable of being brought to market under existing price and cost
relationships.

We have a good idea now, as a result of the 1995 assessment, of what the
economic relationships of undiscovered resources are ‑‑ that tiny sliver
which is added every year. But that big whopping part of the bar graph
in John\'s graph, which is additions through revisions, extensions and
adjustments, is coming from that section of the supply function in the
middle, about which we now have some estimates of volume, but have
really no understanding of the marginal cost structure. And if we lose
the resolu­tion by reduction in sampling size or other strategies ‑‑
which I understand the budget problem, but then again, that\'s not my
job ‑‑ if we lose the resolution that we have in that middle segment, we
will have lost the ability to analyze and, therefore, understand the
nature of supplies that are going to come from the most important
segment of the resource base.

Could you put the last slide up?

The role of revisions is clear. It will undergird supply in the future,
and it will undergird the vast majority of supply ‑‑ of additions to
supply in the future.

As Gordon\'s already mentioned, the nature of depletion in a fixed
resource base is the movement away from large fields to smaller fields;
and that also requires that not only smaller fields be more carefully
sampled, but smaller operators who are the operators who operate smaller
fields must be sampled.

The changes in price and technology, especially in an industry which is
becoming increas­ingly dominated in terms of the number of wells and the
number of fields by smaller operators, is some­thing that\'s going to be
much harder to understand in the future just because of the diffusion of
technology over a large number of small operators as opposed to calling
Exxon and Chevron up on the phone and asking them how they feel about
horizontal drilling. I understand that\'s not quite the role. The effect
is the same.

Finally, we see a major level of divestiture by the large oil companies
and gas companies as they sell off their smaller and marginal assets
which are, from their cost structure, smaller and marginal assets, to
smaller, more nimble and adept operators who could profitably produce
them.

So in conclusion, we definitely need the ability to see the fine texture
in the resource base in order to understand the most important component
of volumes that support future supplies. We need to be able to be
responsive to the dynamics of changes in the resource base towards
smaller fields and changes in the industrial organization of its
production by the increasing role of smaller operators and the ability
to see, in this new rather different world, what the role of prices and
technology will be.

I know that the questions that were posed were technical questions as to
how to respond to a bad situation. The message that I\'m giving ‑‑ and I
don\'t need to speak for him, but I believe the message that Gordon\'s
given is that the issue needs to be raised to a different plane and
addressed as a matter of policy rather than optimization under a
constraint. Let\'s look at the constraint.

MR. MOUNT: Thank you, John.

The third discussant is Brenda Cox.

MS. COX: Well, in many ways I can repeat what the two speakers said,
knowing nothing about oil and gas. But just looking at the statistics
coming out of this program, you see a situation in which change can be
rapid; and the difference of a year or two could be important.

However, I approach this topic as ‑‑ I\'m a statistician. What\'s the
best thing to do? What\'s it mean? What\'s it going to cost? What are
the quality aspects? So that\'s how I looked at this.

The first thing I did is look at the basic design of the survey,
regardless of how often it\'s done; and I found ‑‑ I think there are
some improve­ments needed in the basic design. In the sampling plan
that\'s being used, I think ‑‑ there appears to me that there may be
more certainties than actually need to be. In business surveys, you have
a very skewed distribution. So a classic business design is to have
certainty selections and then differing probabilities of selection based
upon size.

In this design, you\'re either certainty or you\'re 8 percent, and I
think there might be something that could be done there that would
improve the efficiency.

Another thing I notice is that a rotating design is not being used here.
It\'s very common when you\'re measuring change and when change is
highly important over time to use a rotating panel approach in which
some people are deliberately kept in the sample from year to year and
rotated out so that a certain percentage of your sample is common from
one year to the next. And I think that kind of approach should also be
considered. So with that respect, I think there are approaches that
could be used to improve the basic design.

Then I wanted to comment on what\'s being done in estimation from a
statistical sense. This design has a phenomenal response rate; just
absolutely phenomenal. Phenomenal is 100 percent for the certainty and
99.something for the non-certainty. That\'s just incredible. It says
something about the value of this data system that people will respond
at such a rate.

Now presently it doesn\'t appear to me that any adjustment at all,
though, is being done for what non-response actually exists, and I would
recommend there are some fairly simple weighting class adjustments that
could be made. You don\'t need to do an imputation the way it was
mentioned in the documenta­tion here. But I would suggest a weighting
class adjustment there.

Then I was a little concerned that there\'s a lot of imputation
occurring. There\'s a lot of imputation occurring because operators
don\'t have information that they can report to you. Some operators,
particularly the very small ones, are reporting production. They don\'t
report reserves at all, so reserves are estimated for them, as well as
adjustments, revisions, new discoveries, et cetera. And that\'s a little
scary, in fact, for me.

How those imputations, shall we say, are being done is by developing a
model for what their reserves would be based upon their production. That
ratio, which is used to estimate that, is based upon the people who
actually report reserves. Well, the people reporting tend to be the
larger operators. So you kind of have ‑‑ you\'re estimating what your
smaller operators are doing ‑‑ your very small operators are doing ‑‑
based upon people who are bigger than they are, and it just makes me
feel very uncomfortable. I don\'t know there\'s anything that can be
done about it, but it\'s something really that needs to be recognized.

In reading the results of this study, you don\'t see in actual survey
estimates that they make enough ‑‑ I don\'t think you make enough point
about the fact that, in fact, some of these estimates, even for the
survey itself, are not actual estimates. They\'re approximations or
guesses. And that\'s not to say that, in the aggregate, they\'re not
good. I couldn\'t respond to that. It\'s just that a lot is being
estimated.

Now to get to the basic questions we were asked, I suspect there may be
no choice but to do something. A lot of programs are being cut now and,
frankly, the cuts in agency budgets don\'t always flow from what agency
has a fat budget, unfortunately. So assuming something has to be done, I
think that we should look to the subject of cost versus quality and, in
particular, to say that EIA will not produce estimates, approximations,
between-year projections if they cannot create something of satisfactory
quality.

Now EIA has a perfect opportunity to see what it can do. It\'s got a
past data series that you can use to model, to say, \"Well, what would
have happened if we had modeled for the intervening years.\" In fact,
you can develop an estimation approach, use a certain portion of the
data to develop your estima­tion approach, and then use another portion
to test it out and to say, \"Okay, if we had used this first ten years
here to develop our estimation approach, how well would we have done for
the next few years,\" just to see, you know, the circumstances in which
your ‑‑ you probably shouldn\'t call it \"estimation,\" by the way. We
probably should call it \"projection.\"

Under what circumstances would your projec­tion approach work? What kind
of circumstances would lead to large variations in your projections
versus reality?

So that would be the first thing that I would suggest: that you do a
little pen and pencil. I think you\'ve got all the tools available to
you now to do that and determine that; and that might answer one
question, and that is, how well can you do with a sample of 150? How
much does 150 large operators give you in terms of making those
projections versus none at all?

Now the other question I have to bring up is the subject of your frames.
Frames can go only so long without being updated, and I don\'t know what
the ideal update period would be for this frame. I noticed in the
documentation that it said that some units were updated every two years.
I don\'t know what the normal range would be; but definitely, if you
skip a year, there\'s going to have to be a full-frame updating before
the next survey.

In other words, some of the cost will be greater for this
every-other-year survey. You\'ll be doing more updating. The updating
may not be as efficient of your frame. You may have to trace more
operators to find out what happened to them.

There\'s also a potential for coverage. This survey is doing something
that I would do, but you have to recognize that it can be a problem ‑‑
or I\'ve done in the past for business surveys, and that is, when you go
out with an existing frame and you start looking for people to find out
they\'re there, after you search and search and search, and you can\'t
find them, you conclude that they\'re out of business and gone. But that
may not be the case. It just may be that you can\'t find them.

Now I\'ve made that same assumption myself, but it\'s a potential source
of under-coverage that you have to worry about; and it could lead to a
potential continuing deterioration in quality. There really wasn\'t a
discussion of coverage for this frame, so I don\'t know how well the
coverage is of the existing frame.

Then we get to the issue of how well can you estimate; and then given,
and then next, how much better could you estimate with large units here;
and then what would be the cost of all of this, because I think it\'s a
trade-off between all of these. Then, in addition, I\'d say, \"What
would EIA do personally were its own uses?\" Forget about users. For its
own uses, what would EIA do if they didn\'t have these estimates in the
off-years? How does it affect EIA\'s own internal programs, procedures,
your models, your decision making.

Thank you.

MR. MOUNT: Thank you, Brenda.

So we\'re open to the Committee, Campbell.

MR. WATKINS: Let me make three comments. When I saw this item five on
the agenda that John had provided, as a euphemism I was going to say,
\"Well, it\'s sad.\" But I\'m actually shocked with the situa­tion in
which you\'re placed.

I say that because, if anything, the case can be made ‑‑ and I think it
can be made strongly ‑‑ that the reserve analysis and reserve data, the
United States, should be expanded rather than contracted. The reason for
that is that there are still gaps, significant gaps, in knowledge about
what is happening to the United States\' oil and gas reserves.

You might argue that a country, for example, like Saudi Arabia, where
they really don\'t know and don\'t care too much about what their
reserves are ‑‑ if you were in that position, maybe this doesn\'t matter
that much. But this is a situation at least with respect to crude oil
reserves where the reserves are seemingly declining, and it\'s that much
more important that detailed reserve data be available.

I think I\'m right ‑‑ and Gordon, you can comment on this ‑‑ that
perhaps the best and most detailed data available are reserves for any
country is the information available in Canada.

MR. KAUFMAN: A western ‑‑

MR. WATKINS: A western sedimentary base. Hopefully, the United States\'
information would be moving in the direction of that sort of detail, and
the types of areas or the types of information that would be useful to
see further extensions of U.S. data would be in the critical areas of
distinguishing affirmly between enhance recovery reserve additions; that
is, a gauge of proportion of the recovery of the oil in place for a
given area ‑‑ distinguish between that and, say, what I call aerial
extensions in the reservoir by developing drilling ‑‑ reserves added in
that way ‑‑ distinguishing more between types of technology, and John
has mentioned horizontal drill­ing, and providing finer detail on
reserves by play, and thereby voiding in terms of reserves analysis of
what the term or what I can term \"aggregation prob­lems.\"

So the fact that you seem to be a place with the situation, where
you\'re going the opposite direction, is, as I said at the start of my
column, really quite shocking.

The second comment is that the uses of these data seem to be expanding
rather than contracting. For example, there are proposals to augment the
national accounts to reflect resource depletion.

John, that\'s not on your list of functions here, but I\'ll add to it.

Gordon is aware of the work that Morry Adelman and myself have been
doing on valuing in situ reserves and the way that might be utilized to
address the problem of adjusting the national accounts if, in fact, that
was pursued. Any application of that kind of analysis relies on accurate
reserve information and, in particular, accurate information on the
changes in the reserves.

So that the situation, in terms of the importance or the underlying
importance of these data to my mind, is increasing, and it\'s increasing
from desires to more properly accommodate and reflect the question of
the valuation of reserves in a context of accounting for depletion or
appreciation, as the case may be.

My last comment is that everybody thus far seems to have jointly avoided
the questions that John ‑‑ the options that John has put to the
Committee. So that may at least give you my preference, which is clearly
for option three. You do not want to avoid publishing the reserves
report, make it every other year; and at least if you were to have the
sum of 150 as the distribution that clearly shows in your own data, you
are going to pick up a lot of information by doing that. So my vote, if
you are forcing the situation ‑‑ I hope that would not be the case ‑‑
would be for option three.

MR. MOUNT: Cal?

MR. KENT: Can I begin with a very mundane question; that is, how much
money are we talking about? What\'s being spent every year ‑‑ I have
kind of a vague recollection ‑‑ and what are the costs of the three
options?

MR. WOOD: Maybe a three-year time we\'ve gone from about \$1.7 million
to \$1.5 to \$1.3 to what we are targeting under out latest budget
scenario is more like \$870,000 a year.

MR. KENT: That\'s to do the full ‑‑

MR. WOOD: Right. And then that is the full cost of running the surveys,
maintaining the system, maintaining the frame activities, the quality
assur­ance activities.

MR. KENT: And what are the costs of your options, then?

MR. WOOD: I\'m sorry?

MR. KENT: What are the costs? If we went with option one, option two and
option three, what would those ‑‑ obviously, option one costs us almost
nothing.

MR. WOOD: They\'re not, we hope, just tremendously different. If you
have a cost of running a survey one year and you decide that you will
run it every other year, but in fact try to maintain the same qualities,
the same frame quality ‑‑ and the frame quality is almost the key to
getting a good set of reserve estimates. Over the years, we\'ve
investigated lots and lots, more and more sophisticated sampling to
lower the sampling rate and keep the same target rate, et cetera. But it
always comes back to: the frame has to be good or you\'re not going to
get anywhere.

So coming back to the question, you can\'t simply take the cost of doing
the survey every year on a smooth ongoing basis and divide by two. That
just isn\'t the way it happens. We haven\'t really gotten down to what
we would call precise estimates on, you know, dollar-by-dollar kind of
savings; but it is in the several hundred thousand dollar range, in kind
of marginal costs, to go get the smaller operators.

MR. KENT: Well, I think it makes a whale of a lot of difference as to
which option one recommends if one knows what the cost savings actually
are on these; and I don\'t know how you could make a decision, quite
frankly, unless you\'ve got a good estimate of: if we go with No. 1, our
current budget is \$800-plus thousand. It will only be \$200,000; or if
we go with option three, we\'re only going to save \$100,000 if we went
with option three. Then for that \$100,000, it may be an entirely
different conclusion than if you say \$500,000 by going with option
three. Even making an imprecise mental cost-benefit calculation, you
know, I\'d need to know what the price tags, or at least the approximate
price tags, are here so that one could make the evaluation, at least
mentally, how much are we going to lose under each one of these compared
to the cost savings that we would experience under each one of these.

MR. WOOD: Okay. I would comment on the option two. The first year or two
costs may well be higher if we executed it well in just the additional
modeling study.

MR. KENT: That\'s not costless.

MR. WOOD: What?

MR. KENT: That\'s not costless to do that.

MR. WOOD: So in fact, you know, on several grounds option two doesn\'t
look particularly good.

On the option one, there are things that we know are somehow qualitative
in concept, but we know are costly to avoid that we just haven\'t tried
to make assessments of, and that is that we know we have to do some very
serious frame maintenance to make sure that, on an every-other-year
cycle, we really approach the same type of quality that we\'re used to.

And two, the operators tend to have to store the data in a slightly
different way for us on an operated basis than they do ‑‑ have to for
their own personal use. So if you skip a year, there\'s enough turnover
in the staffs of the companies that we just anticipate a great deal more
problems in executing the survey.

So obviously I\'m hesitating to try to give a precise quantification of
the difference between, for example, option one and option three. It
would, you know, probably be in the 10 percent of the budget kind of
options.

MR. KENT: Okay. And the second comment that I would want to make,
although I do think you should generate more precise cost estimates of
these three options, would be to call attention to what Brenda had
previously said; and that is, with 50 years of good data, 18 of them are
your own, if you\'re going to use some sort of a modeling technique,
you\'ve got enough data to be able to back-cast or something like that
and come up with ‑‑ to see really what you are going to lose of this
fine texture by just going back and saying, \"If we had done this.\"

Now, again, in the first year, that\'s not going to be an inexpensive
process. But you could very easily establish whether or not this
approach for the intervening years is going to lead to a signifi­cant
diminution in quality just of your data just by going back and testing
this option in the way that Brenda has suggested.

MR. MOUNT: Richard?

MR. LOCKHART: I have a comment and a question, I guess. The comment is
to probably one you\'ve already thought about, but when you come to
1998, and it\'s the second time you haven\'t done a survey or a full
survey, or you\'re looking only at a fragmentary survey, you won\'t have
\'96 data. So when you go back and when you\'re developing modeling
methods for filling in the intervening years, you\'ll be able to look at
modeling methods which utilized data from two years ago, as well as
simple one-step-ahead-type forecasts. And those modeling methods will
not be nearly as useful when the two-data ‑‑ two-step-behind data is
missing. And so you\'ll want to think about how well you\'ll be able to
modify your models when you don\'t have this two-year-old data come,
say, 2,000 or 1998.

The second ‑‑ that\'s just a remark.

The question I wanted to raise was that you showed us two graphs: one
which showed that large operators were roughly responsible for 85
percent of reserves; and then you focused on a graph which talked about
things in reserve, and Gordon talked about the importance of flows.
Large operators are responsible for what fraction of change of reserves?
In other words, this certainty sample, 150, you have 85 percent total
reserves, but we\'re really trying to measure changes. So how important
are they for that?

MR. WOOD: Well, actually I have an overhead that would probably make
that reasonably clear. This is for the changes in percent of the annual
crude oil year by year, so we have 18 surveys in here or 17 changes. The
percent change is the solid line for the total U.S. crude oil reserves,
and the dotted line is a category one reported crude oil reserves.

So the flippant answer is that, you know, they represent 85 percent of
the change because they represent 85 percent of the reserves, and the
changes tend to match.

It is kind of interesting, the biggest mismatch ‑‑ for example, one of
them is in 1986, when the price of crude oil went from \$25 a barrel in
December of 1985 to \$9 a barrel in July of 1996, and the drilling
dropped several thousand rigs in a six- or seven-month period and
there\'s tremendous turmoil. So the idea of ‑‑ you know, there are
things that happen which may explain when there are bigger changes.

Now we actually, in fact, looked at how you might do simple projections
of this at the national level.

Bob, would you put that next overhead on?

This is about the simplest thing you could do for the annual
national-level crude oil reserves. What we did there was use the first
12 changes to calibrate that model and then projected the next file; and
the average absolute error in those five years is about .4 percent, and
the largest one was about .6 percent.

Now obviously bigger changes than that do happen, as it happened in
1986; but then again there are other things that might help explain the
changes in certain years. Again, you would expect to be a little better
in oil because the percentage represent­ed by the large operators is a
little higher. As you go to the regional level, the larger states would
probably not be too much different than this. As you move into the
smaller states, you would expect certainty because often the larger
operators don\'t represent as big a piece of it.

I might comment on two or three threes things as kind of a ready answer.
We do have some imputation processes designed that account for response
rate in the size and the expected behavior of the very small operator.
In the last two years, we simply haven\'t implemented it because, you
know, there was no need to do it for the small certainty opera­tors, and
there\'s a very small problem for the category three.

Some of the imputation processes for the small operator reserves, they
are significantly different than for the large operators. On a national
level, the reserve-to-production ratio might be 9 or 10 to 1. In New
Mexico for gas, it\'s like 15 to 1. For the smaller operators, it\'s
more like 6 to 1. So that sample of more comparably sized operators who
make their estimates in kind of similar ways, you know, there\'s a lot
of difference in the way the smaller operators are actually estimating
compared to the reported data for the larger ones.

MR. MOUNT: Why don\'t you leave that up there, because I think it will
stimulate some more discussion.

John?

MR. GRACE: I would just suggest, though that graph itself does provide
some confidence in the national-level aggregate, the greatest concern
that I have is not in the ability to peg the national-level aggregates
or even state-level. It\'s in the field data. And it\'s by the analysis
of the field data that we\'re going to get to an understanding of how
much of these volumes of adjustments, revisions and extensions are going
to be contributed to supply; and that ultimately is what we care about.

I mean, even more to Morry Adelman\'s point of the flows are what
matter, changes in the stock can be conceived as flows, but really what
we all care about is the flows that come out of the ground not moving
categories between reserves. And it\'s the flows that come out of the
ground that are affected by the structure of marginal cost, and our
ability to understand marginal cost is dependent on that fine texture
that\'s available in the data that never shows up in the book.

MR. MOUNT: Does anybody else from the Committee want to make comments?

MS. COX: Just the emphasis on field-level data. Field-level data isn\'t
being obtained for the small operators; is that correct?

MR. WOOD: That\'s correct. Probably 5 percent of the data is at the
state level, and that data is very carefully matched painstakingly by
Bill Monroe, Office of Oil and Gas, back to the fields where we appear
to not have the coverage; but we do, from supplementary data sources,
get to dimension. The reserve data, then, is distributed mostly over the
small fields ‑‑ very small percent back over those. So it\'s accounted
for in the system. There is always at the margin, there\'s data you
don\'t know at the field level.

MR. MOUNT: Is there anybody from the public who would like to make
comments?

I have a few. Basically, I would like to endorse comments that have
already been made: that I think that the importance of the costs and the
trade-offs that Cal talked about for the different options is clearly
important; the value of a rolling sample that Brenda talked about for
understanding a very dynamic situation, and the ability to do the sort
of modeling that you have just described with data that have already
been gathered ‑‑ that a number of Committee members have mentioned. But
I want to add one more thing, and that is that these data really do
matter, and the cost of acquiring them is really relatively small
compared to the cost of many other components of government.

I think it would be very unfortunate if we got into the situation where
we\'re looking at one data set versus another data set within EIA. I
think the important thing is to recognize that in a situation where
things are changing, and here the ability to extract oil has improved
dramatically over the last few years, sort of understanding these things
better is absolutely essential really for the benefit of the country.

This is a highly strategic commodity, and the same thing can be said for
the issues that we discussed yesterday about the changing structure of
the electric utility industry. This is a very dynamic situation and that
we cannot understand it without having good data. The role of EIA is
absolutely essential in this sort of endeavor.

So if you want to have anymore remarks ‑‑ have you covered all of your
responses?

MR. WOOD: Well, other than that we\'ll certainly look into the panel
approaches and, then again, would also allow for a certain amount of
certainty sampling in various panels and potentially the sampling
according to size. But I think the driver of what we have arrived at
over the years as the sampling procedures is, you know, we have kept
refining the sample size for a given target.

Two, there is such a turnover in the companies \-- you know, where a 15
percent change in the frame or more is not uncommon ‑‑ that, you know,
we spend a lot of time keeping track of that; and two, any given company
can acquire a very large proportion of the reserve base, I mean, half
percent and things, even though they\'re not a particularly large
company, through acquisition or something else and the company owner of
it.

There\'s been a tremendous movement of the fields operatorship amongst
the companies during the last seven years. In fact, we\'ve documented
and tracked that. So there\'s been at least a 50 percent turnover inside
the companies on their field operatorship. What that always makes me
worry about is that if they\'ll be such potential for erratic change in
a given year, that if you missed any of the sample size and if you
don\'t sample, you know, the larger groups, basically we\'ll sample. And
I thank you for your comment.

MR. MOUNT: So it looks as though we\'re a little bit ahead of schedule,
which is very good. I propose that we take a break now and meet back in
15 minutes, because our last discussant, Dan Relles, wants to rush off
at the end here, and that will give him a better chance of getting to
his next meeting.

So I thank all of the contributors this morning. I think this has been a
very important discussion.

(Whereupon, a short recess was taken.)

MR. MOUNT: So the first presentation, we\'re revisiting a topic that the
Committee has been concerned about for a number of past sessions. This
is an update on issues pertaining to the restructuring of the natural
gas industry. Roy Kass, Office of Oil and Gas.

MR. KASS: Hi. Good morning. I\'m Roy Kass, from Reserves in Natural Gas
Division of the Office of Oil and Gas; and I\'m going to talk about a
sub-part of what has been distributed to you in the past, and I
understand came along in a most recent package of material: the results
of an EIA Process Improvement Team that was looking at gaps in our data.
Specifi­cally, they focused on gaps in the gas data. We\'ve got a lot of
them.

The two that are most important are, first, there were missing
information that would enable separate identification of what it costs
to transport gas. There is no way that any reasonable analyst can parse
apart the stuff in our data series and get an estimate that they would
be confident in; that this is a transportation cost specifically for a
given leg or to get gas from one place to another.

The second problem that they focused on was coverage in the measurement
of prices. This has to do with the changes in the industry that have
happened over the past 10 or 15 years. I remember coming and speaking to
the ASA Committee two years ago when we had noticed the problem.

The problem really comes from the fact that all of our surveys go to
companies that physically move the gas. Our monthly survey and our
annual survey go to those companies, and we asked them essentially two
questions.

The first question is: How much gas do you sell to customers that you
deliver gas to, and what did you get for it? What\'s your revenue
stream?

The second question is: How much gas do you ‑‑ as we pointedly put it ‑‑
transport for the account of others?

In the second case, our respondents obvious­ly have absolutely no idea
what the cost was. All they can tell us is what they generated as the
revenues for their transportation function. Because of that, we do what
we call \"shade in\" the revenue thing. We don\'t ask anything about
that. Keep that thought in the back of your mind. It\'s going to come up
later.

As the industry has changed over the past several years, more and more
gas has been transferred to what we call transportation gas. Customers,
especially large customers, can cut deals from all sorts of sources to
buy gas for less cost than if they went to their friendly local
distribution company. As a result, we are getting, as I said, less and
less of the gas.

We know that. We try to be up-front about it. We publish a
representation of how much gas, by state, is covered in our price
series. But people tend to not look at footnotes. They tend to interpret
table titles the way they want, and a lot of folks say that the number
we\'re publishing ‑‑ for instance, for industrial gas ‑‑ is the cost of
gas for industrial customers in a state.

This shows the representation of the past several years. The earlier
data series is industrial. We\'ve been monitoring industrial
transportation versus sales for longer than we\'ve been monitoring
commer­cial. And you can see that the industrial is in pretty poor shape.

Nationally, in 1994, we caught about 25 percent of the gas; \'95 trended
down even more; and we figure our preliminary estimates are something
like 20 to 22 percent of the gas nationally. There are some states where
we capture 100 percent; there were some states we capture substantially
less than 5 percent. If anybody is working for information in the state
of New Mexico, for instance, and wants to do price, beware. It is
glitchy as all get out; okay? And that\'s because we\'re capturing
approximately 3 percent of New Mexico gas. I got a call on that a couple
of weeks ago.

Well, in terms of the coverage issue and the misinterpretation issue,
there was an article in [NG]{.underline} magazine in the last session
that mentioned me and it came to my attention, and I\'ll quote from it:

\"It seems that the U.S. Government has no idea what industriales really
pay for natural gas. The prices that me and my group painstakingly
release every month in EIA\'s natural gas monthly publication under the
rubric average price for natural gas sold to industrial consumers by
state are nothing of the kind, and the Government readily admits it. The
question is, what do we do about it.\"

What I\'m doing here today is really a progress report. It\'s a heads-up
on what we are doing about it, what we intend to do about it, and it\'s
something to keep in mind. I have some concerns that I\'ll voice at the
end about this.

Okay. So as I said, we monitor natural gas delivered to end-users. By
doing the measurement on the molecules, we\'re missing more and more of
the gas. We have a basic approach that we came up with in terms of how
could we fill in this gap, and this really has two-sided parts ‑‑ two
parts.

One is: go to the people who pay the bills. They know what the bills
are. The other is: go to the people who send the bills. They know what
the bills are. On the one hand, we\'ve got the consumers. There are, in
the industrial sector, about 20,000 transportation consumers. That\'s
off-system sales. Their average consumption a year is something on the
order of 300 million cubic feet of gas. That\'s a lot of gas. They are
big users, comparing them to the 185,000 on-system users that have an
average of about 11 million a year, okay? So we\'re talking a big
difference.

That proportional difference is approximate­ly the same as commercial
sector as well. We\'re talking really big commercial users that cut
their own deals; relatively small commercial users who don\'t.

Well, if we\'ve got a frame of 20,000 and we wanted estimates by state,
we\'re talking about a daunting sampling effort. If all things were
equal, I would opt for doing something other than that on a monthly
survey.

There is the other than that. Those are the guys who pay the bills. Go
to the guys who send the bills; okay? There is a marketing industry that
has developed and consolidated over the last few years. About three
years ago, what it took to be a marketer was a business card and a
phone. But each of them have bought each other out. It\'s gotten bigger
and bigger. They have gotten bigger an bigger, rather; and now we have a
pretty stable group of marketers.

There\'s an organization up in Bethesda called Ben Schlesinger and
Associates that each year puts together a directory. It is the only
source that we have located that represents itself as a comprehen­sive
listing of marketers. It has approximately 450 entries. The top 65 to 70
of them account for about 90 percent of the gas. So that represents, you
know, a target of opportunity that we\'re trying to take advantage of.

We\'re going to go to the folks who send the bills, the marketers.
We\'re going to enumerate the top 65, a cut-off sample; all right?

Among the advantages of going to the marketers are that they represent
the set of actors who sell the gas. They could potentially capture sales
to commercial and residential consumers. The residential off-system sale
component is small now, but there have recently been a lot of changes in
state regulation and state rules; and looming on the horizon over the
next five years is an increase. There are a lot of experiments going on
in several states. There are a lot of marketing affiliates set up by
OBC\'s to cap that market ‑‑ and, as I said, it\'s a small target.

Disadvantages are that they\'re not used to EIA. They are not used to
regulation. They are not used to filling in our silly forms. And we
frankly do not know how, you know, the impact of that is going to play
out.

Conversely, EIA is not used to them. We really don\'t know what they
have in their data system; okay? We really don\'t know, on a day-to-day
basis, what they need to do their jobs. One thing I learned when I came
into the statistics business is that if you ask people something, if you
ask companies something, that something better be in their routine
business records, or you are not going to get anything worthwhile.

So the proposed solution is: go to the folks who send the bills. We have
a draft survey form.

Could I have No. 3? Oh, let me have No. 4.

This is really drafty. It\'s last week\'s version and it\'s changing day
to day. It\'s the operational part.

What we\'re going to ask them, by state, is how much the gas they
deliver, f.o.b. burner tip, and how much gas they deliver ‑‑ they sell,
rather, f.o.b. burner tip, and how much gas they sell f.o.b. citygate;
and for the citygate price, that just gets it into the local
distribution system. We\'re not going to ask them this thing called
\"distribution charges,\" because that\'s what the local distribution
system charges to get it from the pipeline to the customer\'s company.

There are anecdotes that say that in some cases that last ten miles has
a confiscatory rate. Why it\'s important is that in the case where we
got the burner tip sales, we want to unduplicate. In the case where we
have the f.o.b. citygate sales, we want to add in order to get a
representation of what the true retail price might be; okay?

Now may we have the old three?

In terms of the whole flow of gas, that\'s a real simplified schematic.
Every place that a line hits the box or a line hits a circle is a place
where a sale can take place. A marketer can sell at the well head. A
marketer can sell at a market hub. A marketer can sell at a storage
facility, at an interconnect, at a place where one pipeline meets
another. We do not know how much of a marketer\'s sale they\'re going to
be able to tell us took place to a consumer in a given state. This is
something that we\'re going to go out and find out. So we have that as a
caveat.

Another thing we have as a caveat is that we\'re interested in states to
consumers ‑‑ I\'m sorry, sales to consumers by state, and we have
anecdotal stories from a couple of the marketers already that say they
sell to a big customer. The customer takes it up in, let\'s say, the
state of Ohio and moves it to his plants that are located in Ohio,
Kentucky and Michigan. They know this, but they don\'t know how much gas
goes to Ohio, to Kentucky and to Michigan. So again, a problem that
might end up really being problematic.

Okay. We\'re in a pretest mode now. We\'ve sent a draft to OMB for
clearance in our pretest; we intend to mail out to the companies next
week. We\'re going to ask them to fill in a form for one state, get it
back; and mostly we\'re doing this for two reasons: first, to find out
whether they can fill it in from their company records. I imagine if
they can\'t, we will hear very quickly. Second, to get entree to them.
We\'ve tried telephone contacts, and frankly we ran into a lot of stone
walls. They won\'t return our calls. We are not business for them. As I
said earlier, they\'re not used to doing business with EIA; EIA is not
used to doing business with them.

Okay. Additional questions which we can\'t do in a pretest because it
wasn\'t in the original OMB clearance.

Jerry ‑‑ is Jerry still here?

MR. MOUNT: Just gone.

MR. KASS: Just gone; okay. He, I was hoping, could make me straight
about what we can and cannot do at the OMB connection.

But with the OMB clearance process now, there are two iterations of
federal register notices. In the first federal register notice, we
intend to find out what states the marketers do business in and the
stage of the gas flow, in which the transaction takes place. If we find
that we\'re going to be able to capture a lot of gas at those places
with two \"x\'s,\" that\'s different from where the transactions take
place in other of those potential places.

Okay. We see some upcoming problems, and this is where, oh, maybe next
year we\'ll come to the Committee and ask for some advice. But this is
really a heads-up.

What if we determine that we can expect to fill in some of the gap in
coverage? Right now, we\'re missing 80 percent; okay? Let\'s say that we
can fill in, in a best-case analysis, 80 percent of that; okay? That\'s
different from if we can fill in 10 or 15 percent of that. This could
happen because trades that are out of scope. It could happen because of
insufficient data in corporate records. It could happen for a lot of
reasons. But we\'ve got, on the one hand, we could fill in a lot of the
gap. On the other hand, we can\'t fill in a whole lot.

Then we\'ve got to make some decisions. The first is, at what point is
it worthwhile to continue with the survey? We call it the \"901\"
because we gave it a name. Now with this survey will come correspond­ing
modifications to the monthly 857. We\'re going to unshade what our 400
respondents, through our monthly form, report. They\'re going to have to
report the transportation revenue for their transportation gas. We\'re
going to unshade our annual report transporta­tion thing. So on an annual
basis, all companies are going to report a transportation revenue.

At what point is it worthwhile to go to that effort? Basically, the
question, I think, comes to what\'s going to be the most beneficial to
our custom­ers: if we go forward with the 901 or if we do something else.

And the something elses that we can do, just to give you three examples:
one, we can continue as we do. We publish all reported prices. We give a
caveat table saying this is the coverage. We let our customers do with
it as they will. A lot of customers misinterpret, misrepresent what our
numbers are. This is especially problematic when you get part of the
industry press picking up the price table, not mentioning at all what
the coverage is, and going forward saying the price in state X is \$10
this month. Well, it is for a very little bit of gas.

The second thing we could do is report a price for state only if the
coverage exceeds some threshold. For instance, I said that there are
states with less than 5 percent. Are we doing anybody any real benefit
by reporting out that state? Well, if the 5 percent is too little,
what\'s enough; okay? What\'s the threshold going to be?

Finally, we could drop the price series from publication altogether. Are
we doing a service for publishing a 20 percent coverage price? The
belief has been that the prices we publish, the prices we gather, are a
biased estimate of what a true price for that category of customer is,
but it tracks well with what the true price is. This is really an
article of faith, and it\'s been the religion that I\'ve been working
with for a while. But we don\'t know; and if people share that article
of faith, that\'s one thing. If they think that it\'s a true data point,
they\'re clearly wrong. We know it\'s a bias. We\'re assuming that it\'s
a constant bias. We have no idea what the magnitude to the bias is.

Thank you.

MR. MOUNT: We have no formal discussant for this presentation, but
hopefully there are some comments from the Committee.

MR. KENT: Well, this is really off the cuff, but will the end of the
world really come if you quit publishing the price data? And I ask that
really, I guess, as a serious question, because years ago, every time
everybody said they wanted us to publish prices, so we published prices
so they could beat us up about the prices we published. And, I mean,
this almost seems like it\'s self-flagellation or something on our part.
You know, why do we need this grief? I mean, unlike what the
presentation we heard earlier on reserves, you know, where I think we
all understand the absolute essential nature of that, what would we lose
if we quit doing it?

MR. KASS: Okay. I can give you anecdotes; okay? I get calls from time to
time from people who are in contracts that were written years ago, where
a price they paid is geared to a published price. That\'s one use of our
data. I understand with the advent of the spot market and with the
advent of the future\'s market, more and more contracts are not relying
on EIA, and that\'s probably as it should be. But we don\'t know how our
customers are using our data; okay? The one way to find out is to not do
it and see who squeals loudest.

MR. MOUNT: Campbell?

MR. WATKINS: Let me make a couple of comments and suggestions. First,
have you looked at the possibility of collecting information,
particularly on the transportation rates from the EBB\'s or electric
bulletin boards and the lifelines?

MR. KASS: Extensively. It does not fill the entire transportation
classification. The EBB\'s are at the margin of the rate.

MS. BISHOP: Use the mic.

MR. KASS: I\'m sorry. The EBB\'s, to the extent that they cover a price,
cover the price at the margin; not the long-term commitments. That\'s
one thing that we looked at early on.

MR. WATKINS: But an increasing proportion of the gas is transported
so-called at the margin.

MR. KENT: Yeah.

MR. WATKINS: So that is not insignificant information by any means.

The second idea is ‑‑ you touched on the contracts ‑‑ whether you could,
by a sampling process, acquire information from contracts on the pricing
provisions, transportation provisions. There are some data that are in
the public domain in any case. I think I\'m right in saying that any gas
that crosses international borders, that those contracts have to be
filed in the public domain in the United States, and there is some
information or source of information on prices and transportation.

MR. KASS: That\'s interesting.

MS. BISHOP: People are saying they cannot hear.

MR. WATKINS: Sorry?

MS. BISHOP: Talk louder.

MR. WATKINS: Sorry. Let me repeat what I was saying about use of the
contract\'s information, where it can be in the public domain, or
obtained by satellite. A lot of the contract ‑‑ or some contract
information for international flows is in the public domain in any case.
They offer ‑‑ I mean, I was really surprised, and so must you be very
uneasy about the amount of information you have on the industrial
sector. I mean, if your average is 25 percent, as you pointed out, with
respect to New Mexico, you\'re very, very low.

Now since all that international gas does reach a lot of market areas ‑‑
I mean, it\'s not as if you\'re talking about a rim prospect. I suggest
you could at least look at that.

Could you put up the flow of natural gas graph, again, because I had a
couple of comments on that.

It goes to the question, a point of clarifi­cation: I take it where you
had the \"X\" with the LDC, that indicates a citygate process.

MR. KASS: Right.

MR. WATKINS: The omission, or what seemed to be an omission, is why you
wouldn\'t want to have an \"X\" either in the well head or the gas plant
as well.

MR. KASS: The way the survey was designed, we\'re really aiming at
getting a retail price. The further away from the site of consumption
the sale takes place, the more transportation costs we\'re missing,
okay, which flows back to the fact that we can\'t monitor transportation
costs.

MR. WATKINS: But perhaps related to my earlier comment, if you did
collect some of the, let\'s say, point of \-- whether the sale at the ‑‑

MR. KENT: Let\'s say the point of \"X\" is the gas pump.

MR. WATKINS: ‑‑ which is the base for a lot of it, and again going back
to the facts that you may have some information in the public domain,
the transporta­tion to the point of the price, that could give you, say,
a citygate price somewhere down the line ‑‑

MR. KENT: That\'s what I thought they did.

MR. WATKINS: ‑‑ if you had the well head or gas plant exit price as
well.

I\'m just trying and groping around for ways you could try and fill some
of these gaps. That\'s all I have.

MR. MOUNT: Anybody else on the Committee?

Well, I suppose I would like to say the obvious: the price does matter
for analysis, particu­larly of the industrial sector, and I find the idea
of trying to get information from the brokers on the supply side very
interesting. I\'m sorry that the Committee hasn\'t commented on that.
Maybe I could encourage somebody to say something.

I feel that, in addition, this is the sort of thing that may have to be
considered for the electric industry. It seems inevitable that there are
going to be brokers who both represent groups of suppliers and brokers
who represent groups of consum­ers who were involved in competitive
markets; and I think that if there is one group that I would like to
know the price of gas about, it is independent power producers, and
presumably they\'re one of the ones that you\'re missing.

Are there any comments from the public?

Oh, sorry, Campbell.

MR. WATKINS: Let me just make another comment on what you just raised. I
should say there is a lot of information in the trade press, at least on
spot prices, hub centers and various points along the system. So all is
not completely bleak on the gas pricing outlook, and the trade press
publications do provide some key information.

MR. MOUNT: Cal?

MR. KENT: Well, let me just raise an issue about who should be
collecting this price data, since you brought up the electric people.
What role should FERC be playing in this? And I throw it out as a
theoretical question because somebody\'s going to have to have the
clout.

I\'m not that much familiar with other gas markets, other than one in
West Virginia anyway, but I do think that what Roy said, if you go to
these brokers, they\'ve got no incentive to report to you. They may not
really have the data in the way that you want it. The way they keep data
is basically the way they\'re accountant wants it for their tax
purposes, and I just really begin to wonder if we\'ve got a problem
here.

Certainly what you pointed out about the IPP\'s and their consumer of
gas is that if there\'s a real regulatory issue here, that we\'re just
going to spend a lot of time beating our heads against the wall, trying
to get information that if somebody doesn\'t have a bit of a stick,
we\'re not going to be able to get.

That was all I was saying, because I think you\'re absolutely right. As
the electricity goes the same path that natural gas went, we\'re going
to have more and more difficulty; and this even comes back to the
confidentiality issue that we\'ve had before, because brokers are
certainly not going to want to have their data put out there where it
might become public record for their competitors.

MR. MOUNT: Well, to offer a carrot in this process, certainly a lot of
the things that are done in land grant universities gather information
that individuals might be reluctant to release, but that they\'re
actually very interested in looking at reports that contain all the
data, obviously without the individual entities identified. So this is
an advantage to the people who are providing the data on this, you know,
as a long-standing tradition with agriculture commodities.

Have you any final comments?

MR. KASS: No. I think I would like to follow up with Campbell about the
reporting of important places.

MR. MOUNT: So thank you all for presenta­tion.

We can move on to the next topic and pick up some time for Dan. So
we\'re moving now on to an issue that I think many of us are interested
in, statistical issues pertaining to re-engineering at EIA. All I can
say as an introduction is I hope that re-engineering is going better
here than it is at Cornell.

The first presentation is Measurement Model for Information Management
Processes by Nancy Leach, Energy Markets and End Use.

MS. LEACH: As Tim said, I\'m Nancy Leach. I\'m reporting today as a
member of EIA\'s business re-engineering or BR team. For those of you
who have handouts ‑‑ it looks like this ‑‑ the last three pages are
details on each measure. Oh, I see Renee is giving them out right now.
It provides the measure, the target, the collection frequency and the
data collection method. As I talk, I may not give every single piece of
those, but you do have that in the back.

What I\'m going to do this morning is provide an overview of the
performance measures for only the information management.

Okay, the next slide \-- or the IM sections of the BR.

Want to do the next slide?

Essentially, these are the survey-related processes: how to design the
survey, collect the data, process the survey, and then put the data out.
As you see, develop the design, create and update the frame, request
respondent data, receive respondent data.

Next.

Then we import information. This is when we use non-EIA data sources. We
call it imported data. We clean up the data, and we protect the
confidential data, and then we create what we call the information
components, or these are your graphs, your charts, your tables, your
explanatory text.

The first set of measures are for survey information. The first two
refer to the availability time for the survey estimates and for the
survey data. The idea is to get the data out faster than we currently
do.

The third measure is to track the change in respondent burden. This is
to keep up with our legislative mandate of ‑‑ I think it\'s a 10 percent
reduction in burden each year.

Our fourth is for edit performance statis­tics.

You want to bring up the next chart?

The idea here is to identify good edits; that is, edits that track truly
erroneous data with a minimum of respondent burden and processing
burden. A good edit changes bad data and then provides better data,
better quality.

Our performance statistics we defined were frequency of errors detected,
the number of erroneous records identified, the number of true errors
identi­fied ‑‑ again, trying to get at was this valid data? Impact of
identified errors. Did it make a differ­ence?

Sure, you\'ve identified all of these errors. You went back to
respondent. Respondent said, \"It\'s fine with me. That\'s exactly what
I told you, and I still agree that that\'s the correct answer.\" Well,
we don\'t change the record. We have wasted our time by calling back and
irritate a respondent. So maybe that\'s an error that we would want to
drop. And then the number of changes made as a result of validation.

Again, sure, identified all these errors, but did I change anything
because of it?

The emphasis ‑‑ let me just say ‑‑ can we go back to the one with the
yellow tag on it?

The emphasis from the BR Team was to make these automatic; to try to put
all of these flags, these counts, within the edit and imputation
programs themselves. Right now a lot of this work is done manually. You
go to your error, your error list, and you say, \"Oh, yeah, I got five
errors flagged.\" It\'s more likely 500 usually ‑‑ and then checking it
all.

So the idea is to get all of this created automatically so we don\'t
have to sit up there and keep track of all of these edits.

The fifth survey information performance measure is imputation
performance statistics. Now I was told ‑‑ now we can go to the
imputation slide ‑‑ that some of you all might not be used to dealing
with the real world where we have recalcitrant and imper­fect
respondents. So we do have to do some imputa­tion, and that\'s
essentially the filling in of the missing data.

We have unit non-response, where the respondent has refused to complete
the form at all, or item response where somewhere in the midst of the
form, someone just doesn\'t answer a question. It\'s skipped or
whatever.

Do you have the next one?

So the imputation statistics that the group came up with: the number of
total data items imputed, the percent of one particular data item
imputed, percent of the total data set imputed, and the effect on the
file data. Does imputation make a difference? Is it worth the time and
effort?

We identified two measures associated with data requests. Essentially,
number one is your postal returns. In the old world, where we mailed
forms as would be your postal return, how many bad addresses did you
have?

Number two is the cost. How much did you have to spend to get the report
out? And that\'s the cost fer form. Again, the idea is to minimize
these. Let\'s get fewer returns and for less money.

Data receipt measures. We had four of those. It\'s your percent data
entry error by method of entry. Your percent response is received by the
due date. This is a critical one for many of our surveys. Our
respondents don\'t respond on time. Therefore, it throws everything off.
So our target here is to try to minimize that.

The cost per data item sell. How much does it cost to get a particular
form keyed? Is OCR more expensive, as direct as key, whatever.

Then the fourth is time from deadline to continue the next process. How
long does it take to get from one step to the next. In general, again,
the idea is to kind of cut down on the amount of time spent between all
of these steps.

Next one.

We also looked at a performance measure for imported ‑‑ again, this is
your non-EIA data. We often bring non-EIA data in to benchmark our own
data or to check sets of our data sources. Here, the measure is the
amount of time spent cleaning the imported data. You know, do you spend
all of your time cleaning the data, so it\'s really not worth bringing
it in?

We also defined a frame or sample measure. Again, trying to use our
resources most efficiently, we\'re looking at the number of births,
deaths, and updates catego­rized by source and volume. Is this particular
source worth the amount of time or effort and money that we put in? Do
we update our form? Is it really worth our frame updates?

And our last measure is on the information component. Again, this is
where you\'re creating your graphs, your tables, your explanatory text,
reports. Here, the measure was designed to be number of accesses or hits
in the electronic data basis and transmittal of virtual and/or actual
hard copies. Again, the idea is to maximize those.

And Renee will continue on.

MR. MOUNT: Thank you.

The second presentation: Is There a Summary Measure of Data Quality,
Renee Miller, Office of Statistical Standards.

MS. MILLER: Good morning. Can you all hear me okay? Okay. Nancy told you
about all kinds of measures that we developed during our business
re-engineering process. But the one thing that we don\'t have is a
summary measure of data quality.

Now this issue came up several times, and several times we concluded
that it was very important, particularly if we make our data more
timely. But we also concluded that we didn\'t know how to do it. So
we\'re now turning it over to you and have the follow­ing questions.

Is it worthwhile having a summary measure? Do you have ideas on what it
should be? Or maybe we should look at this from a different point of
view and concentrate on the explanatory notes in our publica­tions.
Should we do more to standardize the descrip­tive material?

Not to worry. We\'re not asking you to look at this from scratch. I\'m
going to describe available measures, give background on previous
efforts, and give you an outline of a proposal that Dwight French and I
worked on during the business re-engineering process, but that wasn\'t
adopted.

Okay. To start with what\'s available, we compute and publish revision
error, the difference between our preliminary and final estimates. Now
this measure in particular has been criticized as not being a good
measure of data quality because it doesn\'t tell us about the final
estimates. How do we know that the final estimates are any good?

There\'s an additional problem. Suppose we don\'t revise the data? Does
that mean there\'s no error?

We also compute and publish sampling error and response rates; and while
these are considered important measures, the general feeling was they
don\'t tell the whole story. We also discuss non-sampling error in our
publications, but it\'s a discussion. It\'s not a particular measure.

Going back in time, EIA used to conduct what we called validation
studies. They were cradle-to-grave examinations of the data; and as I
describe in the paper, they proved to be both extensive and very
expensive and were eventually discontinued when our budget was reduced
in the 1980\'s. But they did provide information on the major sources on
non-sampling error.

For example, they included an audit of company records which gave us
information on measure­ment error. They also included a search for
deficien­cies in the frame which gave us information on coverage, and
they also included a comparison of hard copy with the automated file
which gave us information on processing error. And sometimes the
information we got was actually quantifiable. But what we didn\'t have
was a way of adding it all up to get total survey error because
sometimes the errors were offsetting.

We also, in the early days, did a lot of comparisons with other series;
and in the early days, the comparative series were plentiful. We
considered the series that we were interested in the \"reference\"
series, and we computed the comparative series as a percentage of the
reference series.

So let\'s take crude oil imports, for example. We had three comparative
series and we had data for three years. So we considered it nine
independent estimates. In other words, we had nine independent estimates
of the ratio described in the first bullet. Well, with nine estimates,
you can compute a mean, a standard deviation of the mean, at a 95
percent confidence interval. And that\'s what we did. So for crude oil
imports, the 95 percent confidence interval was 99.2 to 100.8, and we
therefore concluded that our data on imports were accurate to within 1
percent.

Well, as you might imagine, these conclu­sions about accuracy were not
well received, because we didn\'t have all that much information about
the comparative series. In some cases, we didn\'t really even have a
clear idea of how the comparative series were obtained.

So although we continued to perform compari­sons, we stopped coming to
conclusions about data quality based on them, and we shifted our focus
to explaining and resolving discrepancies.

Okay. Moving on to more recent times, we\'ve looked at elements of data
quality, and this may look familiar to Dr. Kent. It\'s from his 1991
presentation at the annual ASA meeting.

We identified four elements: timeliness, customer satisfaction,
consistency and continuity. This approach differs from the work that was
performed in the validation studies where we were trying to measure
total survey error. Here we were looking at the fitness of use of our
statistical products. And since 1991, we\'ve made a lot of progress in
measuring timeliness and customer satisfaction, but we don\'t have
measures for consistency or continuity. By consistency, we meant how
well our data compare with other series. Were they consistent over time
‑‑ or rather, were they internally consistent; and for continuity, we
meant: are we measuring the same thing over time?

Okay. Moving on to even more recent developments, some of you may recall
at a few meetings ago, Paul Biemer suggested that we prepare quality
profiles where we describe what\'s known about each source of
non-sampling error. And this came up when we were discussing data on
imports and exports.

Well, we didn\'t prepare a quality profile for imports and exports, but
we recently completed one for the Residential Energy Consumption Survey.
It\'s this purple publication. This was a joint effort between the
Office of Energy Markets and End Use and Office of Statistical
Standards, and Tom Jabine, a former committee member, was the principal
author.

Another development also in the energy consumption area ‑‑ which, by the
way, is Nancy\'s area here ‑‑ is a succinct set of notes for the
Commercial Building Energy Consumption Survey, or CBECS. And that\'s
what you have in attachment one of the paper. These notes describe the
survey methodology, sampling and non-sampling error, in a very concise
way.

So now that I\'ve given you an idea of what we\'ve looked at in the
past, let me move on to the business re-engineering proposal. As I said,
this issue came up several times, and several times we thought it was
important, but that it wasn\'t doable. But Dwight French and I went
ahead and worked on it anyway.

We realized the reason that it seems undoable was that there were
several dimensions of data quality, and we listed some of them. There\'s
sampling error, measurement, coverage, non-response, what we called
\"methodological consistency, which is really the same thing that Dr.
Kent called \"continu­ity.\" And these dimensions differ in the ease in
which they can be quantified. Sampling error, for instance, can be
computed directly from the data. Now methodological consistency, we
didn\'t know how to quantify that.

Ideally, you would like a series to be stable over time ‑‑ not to have
any breaks ‑‑ but sometimes, due to changes in the industry, it\'s
inevitable that you revise your data collection; and the issue is: Does
the series get penalized for having breaks?

Now some of these other dimensions ‑‑ it sounds like they should be
easily quantified, you know, such as measurement error. But we don\'t
have information for each survey on an ongoing basis. This is the type
of information that we obtained from the validation studies which have
been discontinued. So the question is: what to do?

Well, we thought that since we did have some information from each
survey, what we should do is gather the information together and then
rate each survey on each of the dimensions. And we would have two
categories: the level of knowledge we have and the quality level. We
thought we would use a 1-to-5 scale since that seems to work in our
customer satisfaction surveys, where 5 meant things were just wonderful.
On the other hand, 1 meant they were not quite so wonderful.

So to give an example of how this would work, let\'s take non-response.
A survey might get a score of 5 on the level of knowledge for a
non-response if, say, there was documentation available on the response
rate, on our follow-up and imputation procedures, and the key
information was presented in the publications. A survey may get a 5 on
the quality level for a non-response if, say, the response rate was 98
percent in terms of both number of respondents and volumes reported. And
in attachment two of the paper, there are lots of other examples of how
this scale might work.

Well, as you can see, attachment two, you know, was getting kind of
complicated. As a result of these complications and other unresolved
issues, this Business Re-engineering Team decided not to pursue this
procedure.

Some of the issues were: who would do the rating? Now the Committee
might be thinking, \"Well, you know, this is kind of an interesting
abstract issue.\" But one of the possibilities was we were thinking that
maybe the Committee would help us do the rating.

One of the other issues was the time involved. The perception was that
this would be very time consuming.

The third issue was: Could we really ensure consistency? A general
feeling was if we could be precise enough that we could ensure
consistency, would we really be giving any more information than you
could obtain from the explanatory notes?

So that brings us back to the original questions. Is it worthwhile
having a summary measure? Are there ideas on what it should be? Or
should we take a different approach and try to do more to standardize
the descriptive material.

Thank you.

MR. MOUNT: Discussant Dan Relles.

MR. RELLES: Regarding the question: \"Should you have a summary
measure,\" yesterday my answer was no, but today it\'s yes. So I\'ll say
why.

The \"no\" was a knee-jerk reaction based on two observations. One is
the summary quality measure that has come out of statistics related to
modeling is one of the most misused objects I know of; namely,
R-squared. R-squared is interpreted by most people as being sort of a
measure of the quality of a regression model. It\'s equal to 1 minus the
sum of square\'s error over the sum of square\'s total; and the sum of
square\'s error indeed is a measure of quality in the sense that the
smaller that gets, the better you are. But the denominator is a measure
of diversity of the original population, and that\'s something you
couldn\'t do anything about.

So if, for example, I took the amount of heating oil consumed during the
winter from RECS, and I kind of took the homogeneous population of
people who lived in, I don\'t know, Tennessee, and regressed that on
income, I\'d probably get an R-square that was around .15 maybe. But if
I took the entire country, I think the regression would rediscover that
the south and the west are a little warmer than the northeast, and I\'d
probably get a regression in the neighborhood of .90.

Nevertheless, R-squared is highly touted by people as a measure of
quality; and if you talk to social scientists, the first question
they\'ll ask you is: Was it over 20 percent? If the answer\'s no, they
don\'t want to talk to you anymore even though you can have R-squares of
.05 that are tremendously important for policy purposes, because they
can measure fairly steep relationships.

So, okay, the potential for misuse is one thing to recognize. The other
thing to recognize is that any measure of quality is even more
complicated than what you described because it depends on the uses to
which it will be put.

So let\'s take a number like 1.0, which many of you will recognize. But
I don\'t think that many of you know that it\'s the sine of 89 degrees
to one significant digit. Now is that a quality number? Well, I
calculated it last night on my calculator, and I assure you ‑‑ and I did
it three times and I got the same answer each time. So I assure you
it\'s a quality number.

But now ask me, \"Okay, well, what am I going to use that number for?\"
Well, if I were to try to use it to plot a sin curve, it would correctly
get me up to close to one. So that would be good. If I wanted to compute
a cosine, it\'s 1 minus the sin-squared, square rooted, I think, and
that would get me close to zero. But if I wanted to compute a tangent,
it would point me to infinity. So it\'s pretty lousy for that. So what
was apparently a quality number is no good for that particular
application.

By the way, that quality number has one attribute that I think you all
wish the reserve numbers would have; namely, the value of the number
doesn\'t degrade over time. I mean, the sin of 1 is going to be the same
thing tomorrow. So you can ask the question, \"Well, okay, that\'s a
quality number. But how can I get an even higher quality number?\" The
answer is, \"Well, let\'s compute it to two significant digits.\" And
the answer there is .98. That\'s still probably not good enough for
computing tangents, but it\'s better than 1.0 for many other things.

I could go on and perhaps produce ‑‑ do the best job I can and produce
it out to 100 significant digits but the main point that I\'m trying to
get across is that you really have to ask what are you going to use that
number for?

I guess another number that comes up a lot in statistics is the
cumulative normal probabilities. Let\'s say at 3 or 4 the value is
1.0000, which is fine for a lot of things, like how significant is this
regression coefficient. But it\'s pretty lousy for things like computing
hazard functions or likelihood functions or other kinds of things that
depend critically on, not the cumulative CDF itself, but sort of 1 minus
the cumulative.

So with that preamble, I was really sure that the idea of a single
summary measure was really a bad idea.

But on the other hand, you look at how many smart people have wrestled
with this, and you have to say, \"Well, there must be something there.\"

It\'s kind of like ravioli\'s a really good idea, because if you look at
how many cultures have adopted something similar to ravioli, like the
Chinese have dim sum and the Jews have kreplach, it must be a good idea.

So what I tried to do is look for kind of a thread that would unite a
lot of the stuff that I saw today. The idea, I believe, is that quality
really is a number that you ought to attach to a process that data have
gone through; and I want to propose a process where the numbers are 1,
2, 3 and 4. I guess I would assert that the higher the number, the
higher the quality.

The quality, what it\'s really trying to measure, is the degree to which
you\'ve empowered the user of the number to deal intelligently with the
problems of the numbers. So by telling you that the sign of 89 degrees
is 1.0 to two significant digits ‑‑ or to one significant digit, I\'ve
given you all the information you need to deal intelligently with that
number.

And that\'s what I want to try to do with regard to the things you
collect in surveys. My intention is to sort of find a place to put each
of the things we saw today, as well as Calvin\'s earlier remarks. So
I\'m going to propose a 4 point scale. The higher you go on the scale,
the higher the quality of the data or, equivalently, sort of the more of
the data are understood so you can deal with its problems.

On the first level, I think timeliness, consistency ‑‑ oh, and these
things also tend to correspond to the order in which you do these things
and the order in which you might actually chop off doing them if your
budget comes into play. I think everything that EIA tries to do, sort of
achieves at least level one, namely, it\'s timely, consistent and
continuous. I think over the presentations I\'ve seen over the last
couple of years, I mean, there\'s every effort given to getting the data
in faster, higher quality, pay attention to whether or not it\'s
consis­tently coded over time. If not, alert the user of that. And I
believe these are the natural initial steps that one takes when
compiling any data set. So I\'ll call those sort of the prima facie
validity observations. I believe they\'re satisfied with virtually all
data that we see here.

The second level I would call \"metadata,\" which means data about data,
and that\'s where I\'d put a lot of the stuff that Nancy and Renee
showed us this morning. First, you have to know what values the data can
take; and what I think of there is building a database that describes
the data you have. I think of things like value the data can take on.
What are the values of null values? What is the frequency of imputing
something? How often are the data missing? What\'s the time rate of
decay of the information?

It\'s sort of all the stuff about a variable or a data set you\'ll need
to kind of make initial judgments about it. I think of that as perhaps
something that could exist on the Web. Wherever you see a number on ‑‑
if you had a database that de­scribed all of your variables, a consistent
metadata base ‑‑ when I was on the Web before I came here, I thought how
nice it might have been to be able to click on a data value and kind of
open up something that explained more about that than I could see. As a
matter of fact, I couldn\'t see any metadata on the Web when I went
looking for it.

But again, if you had a database organized like that, it would be easy
for you, I believe, to click on any number and get sort of the
standardized description out of that. I believe your lists today that
include errors of various kinds and everything else would be valuable
things to pop up when I clicked on that number.

The other thing might be as a management tool, because when EIA is
having to face information, face its cuts, it would be really nice to be
able to have kind of a common format for descriptions of databases and
variables in it so that if, for example, you decided it was really
important to look at, in eliminating a survey, whether timeliness was
really that critical in it, that you could just kind of grab the time
dimension descriptors of the data and at least be reminded now that this
reserves data set is really time important and perhaps another data set
is not.

So I think metadata building is really an important activity and that
all of the things that you and Nancy ‑‑ Renee and Nancy discussed this
morning really ought to go into metadata descriptions. So anything that
had full metadata on it, I would give it two.

Then there\'s another level, which I\'ll call level three, which is ‑‑ I
think what the user would need in addition to be able to deal
intelligently with the errors, and that\'s a document like this, which I
think is a marvelous document. It describes a survey in a way that if
you gave me the data and gave me this book, I think I wouldn\'t shoot
myself in the foot. It talks about the design, the coverage,
non-response, measurement error, data processing and imputation, how to
do estimation and sampling error ‑‑ which includes a discussion of
weights ‑‑ comparing it with other estimators, and some suggestions for
data users.

I really think this is a terrific model of how to document your surveys,
and I believe that ‑‑ oh, yeah, it\'s the thing that Renee waved too.
It\'s the residential energy consumption survey quality profile. There
was a similar one in a much earlier draft stage for the commercial ‑‑
what was it? CBEC, Commercial Buildings Energy Consumption. But RECS is
definitely at least a three in quality given that, and the other one
isn\'t quite done yet.

Again, that\'s likely to come after you assert your timeliness and you
build your metadata. It\'s a lot of effort to produce one of these
things, and I fear that in the budget cutbacks, you\'re not going to be
able to do too many of these. But that\'s okay. You just advertise your
data as a two instead of a three and blame it on the budget.

How do you make it a four? Well, I think making it a four is building
still more metadata, and that\'s sort of user-oriented metadata. It\'s
about the users\' use of your data; and again I\'d like to see databases
developed that have that information, although I recognize that\'s even
more expensive.

But a good example would be yesterday we heard about NEMS, and there\'s
certain data that goes into NEMS, like RECS, and is RECS quality data?
Well, I guess the question I\'d want to ask is: Suppose the RECS data ‑‑
what sort of overall contribution of RECS to the total error produced at
the end-user result by NEMS ‑‑ I guess I\'d want to know the degree to
which RECS, even though it\'s already a three, whether it\'s
contributing 1 percent of the error, 10 percent of the error, 50 percent
of the error; and I\'d like to bring that information up front. You
know, the user that I presume needs the document, that would be the NEMS
people. It\'s another plug to get them to look at random error and error
in general. I guess I\'d like to ‑‑ since RECS is a survey, I\'d kind of
like to see them boot-strap RECS and, for various boot-strap samples,
look at the variations in outcomes from NEMS of key quantities and, from
that, get a sense of what the contribution of RECS is to their
uncertainty.

I think in discussing reserves this morning, I must apologize for
listening with one ear because I was trying to think about what I was
going to say, but it was pretty clear that one didn\'t need a fancy
model like NEMS to argue that the time decay rate of reserve estimates
was so sensitive that there\'s all these incredible decisions that are
being made that are going to incur tremendous error if RECS doesn\'t ‑‑
if the reserves go to once every two years. Again, that\'s the kind of
thing that I think needs to find its way into metadata bases that
describe things like the reserve series so that it doesn\'t escape
manage­ment attention when budgets are being looked at, if for no other
reason.

But again, as a user, not only are you giving me a three, but you\'re
also giving me a database of what other users are doing, and I think
that would be, you know, really valuable for me to make even more sense
of things.

So I guess at level four, I would say if a data set has been augmented
by a metadata set that describes user requirements ‑‑ not user
satisfaction, because I don\'t care about user satisfaction as much as I
care about how much ‑‑ well, because users are never ‑‑ it doesn\'t cost
them anything, so they\'re never going to be satisfied. But if I
understood how much the data was impacting the precision in what the
user cares about, then I\'d be able to make an intelli­gent decision
about whether it\'s worth spending more money to increase the quality of
the data. And I just don\'t think that information is known at this
point.

It says here I should pontificate, so I guess I\'ll do that at this
point. It seems like we\'re always coming back to the same issue.
Quality gets mentioned everywhere, and quality issues are just always to
nest. It\'s kind of like if you mention quality enough times, you\'ve
shown your awareness of it, so you don\'t have to deal with it. That\'s
a little extreme, but I even saw it in the AEO which I got sent a couple
of days ago. Right in the front, there was all this preamble about how
important quality is and how uncertain we were about certain things, and
then, you know, there\'s all this discus­sion out to the year 2015 that
doesn\'t mention any uncertainty the rest of the time.

So I guess I want to pontificate more on the need to deal with
uncertainty and to deal with error and to try to be more imaginative on
doing that. I know that in the past ‑‑ and I think statistics have come
a long way on that dimension, too. It used to be the way to deal with
that was develop more complicated models. I remember earlier, you know,
we were hearing about uncertainty in NEMS, and there was a whole team
trying to deal with those models on an analytical level; and those are
hard problems ‑‑ especially when you\'ve got a black-box computer
program that you\'re not going to reopen no matter what because it runs
to completion without error.

But I think statistics has developed techniques over the years for
looking at uncertainty ‑‑ things like the boot strap, things like
multiple imputation, things like just seat-of-the-pants-perturbing
parameters that basically argue there\'s enough compute power out there.
We can afford to re-run our models many times and see how much variation
there is there. And I believe that, you know, quality ultimately has to
be answered in terms of how much does it impact the predictions that we
really care about, and my feeling is that we haven\'t seen enough of
that here.

So Tim\'s been asking me, before I run off, what do I think I would like
to see? Obviously, a continuation of my discussion for next time. I
think that\'s all I really want to say.

Thank you.

MR. MOUNT: So, thank you. Samprit?

MR. CHATTERJEE: I have no wisdom, or special comments to add. No special
wisdom, but just some comments to add and maybe a few questions.

The first thing is I think to liven up the proceeding, I disagree with
Dan that quality can be summed up in one number. You know, it\'s much
too complex a process to be summed up in a number, although I think we
should, in your publication, indicate the quality of the data which
you\'re presenting; and from that point of view, one of the points which
is mentioned, which I like very much, is for each data series or data
surveys to present what you call a quality profile, which is a
multi-attrib­ute, multi-variate characteristic which would give the
person, the user, roughly the kind of things which they need to know in
using the data analysis. And some of the points which you also showed in
your transparency No. 7, sampling error, measurement error, coverage,
non-response, proportion imputed and so forth, I think we\'ll give the
person using it as much information as they need to know in order to use
the data for the analysis.

The second point which I had is among the elements of data quality, I
think Dan said about that, too, is customer satisfaction. I think the
customer satisfaction is a function of the data quality. You know, what
else the customer should be satisfied with if the quality of data which
he or she is getting is not very good. So I think elements of customer
satisfaction is a function of elements of data quality, and I know in
that I might be stepping on toes, but anyway.

The other point which I had was in thinking about quality, if you look
at a quality profile, it\'s not just a time ‑‑ you know, a one-time
thing. I think quality can be understood if, for successive rounds,
successive years, see how the numbers are changing. It\'s the trend in
these dimensions which will indicate improvement of quality or
stagnation or decrease in quality. So I think it\'s the quality measured
when one point of time is very meaningless, because it is ‑‑ what do you
call it ‑‑ a standard; and how it does is shown by, I think, the trend
over time.

MR. MOUNT: Bradley?

MR. SKARPNESS: You know, Dan brought up many very good points about data
quality and what\'s really important, and I just wanted to reemphasize
that, that the biggest thing that I get in trouble with with a data set
is that I do an analysis and then I find out later that I didn\'t have
some critical piece of information, and I really am making a statement
and I didn\'t understand the data as well as I should have in the first
place. And the reason for that is that I\'m using other people\'s data
and that I have to read their documentation and depend on that; and as I
delve into it more, they either omitted a critical step or a critical
part of how they collected the data, or I wasn\'t informed of that, and
then until, you know, it\'s too late sometimes. And so the documenta­tion
of how the data was collected and what were the circumstances in those
types of pieces of information are real critical and should be carried
on here.

So there was one of these ‑‑ is the descrip­tive material and the
quality, but I would like to say, you know, not only the summary stuff,
but actually some of the effort in things that occurred while you were
collecting the data is also very important to try to understand what its
usefulness is and its consistency, too.

On another note, the customer satisfaction part I would break out a
little bit. Because you do have this data out on the Web or on the
Internet or in a Web site, the transportability of the data becomes a
real pain sometimes. It costs a lot of effort to get the data to us, do
something with it and get it into SAS, basically.

I would like people, you know, to think about that part of it, and then
the utility and usefulness ultimately ‑‑ sort of the two extremes of
what you were saying. You\'ve got this timeliness. Get it in, but then
how useful is it in trying to quantify that. That\'s all.

MR. MOUNT: Brenda?

MS. COX: I also wanted to comment on Nancy\'s paper, which, by the way,
I thought was excellent. Actually, I thought both papers were excellent,
but Nancy\'s I\'m going to send to someone at MPR because we\'ve been
trying to develop the same methods. Do you have all these in there?

I think at a minimum we should at least try to describe the aspect of
quality. You\'re coming at it more the quality of what we\'re doing,
physically producing ‑‑ not looking at timeliness so much as this
product here. Once you finally get it, what\'s it like? I would say that
at a minimum you want to describe and quantify your quality so that at
least your sophisticated users, which would include your­self, can
understand the quality of what you\'ve got. And you\'ve got measures
here that are going in that direction. Like the percent of data that\'s
imputed, I think at least descriptions of the surveys should include
things like this.

Now I would say ‑‑ I\'ve noticed this in the EIA publications, too, by
the way. I would suggest that you think about a standard outline for
your methodology discussions and a standard outline where you actually
tell people, \"Here\'s the kind of things you should be discussing.\"
You should give an explicit definition for your target population. You
should give the objectives of the survey. You should tell what is the
unit from which you\'re collecting data and who are they reporting for.

In the Reserves Program, it wasn\'t totally clear to me. Some of those
items weren\'t totally clear to me from the discussion in the back.

And then things like what percent of the data are imputed and items like
this. So a general ‑‑ so that people actually know what they should be
going for would be helpful, I think.

At least, then, with measures like this and a description that would let
me get a feel for the whole thing, then I could say, \"Well, gee, I felt
pretty confident here. Ninety-nine percent response rate. Great.\" But
then when you go in and you look and you go, \"Wait a minute. A lot of
this data is being estimated.\" I don\'t know how much, by the way. The
report didn\'t say ‑‑ or maybe I didn\'t look closely enough. I will say
that. I might not have looked closely enough.

There\'s some other areas where you might want to think about quality
measures, like sampling and weighting. Just some measures like design
effects or the design effect running away. Maybe some clustering effects
and things like that, so that you get a sense of what the data are like.
They\'re more useful for us as designers and doers in the sense that you
look at them and we say, \"Gee, look at that design effect. Maybe you\'d
better look a little further.\"

I was looking at a design where they were over sampling intentionally,
but I got down in the cell that was defining the ultimate level of over
sampling ‑‑ calculated design effects running away, and I was seeing
design effects of nine and I went \"Wait a minute.\" You know exactly
what that means. Bad design. Bad design. But those kind of measures
would be very helpful, I think.

So that would be what I suggest is a simpler technique.

MR. MOUNT: Anybody else with comments from the Committee?

I wanted to say that I was entirely persuad­ed by Dan, and I also started
off as a major skeptic about being able to come up with a measure. I
think that really what you\'ve come up with is a sort of categorization
and that the value of this, particular­ly for people trying to use data
electronically, of having sort of a compact summary of what you can
expect to be able to find about a data set, I think is very valuable.

However, it obviously isn\'t a substitute for metadata itself; and I
think that there, as I\'ve argued before, there really are opportunities
to standardize the way that data are documented electrically, and there
ought to be much more consistency not just across different data sets,
data files, but also across federal agencies.

I think that it\'s still early enough to really do something to avoid
getting a very fragmented way of doing this. So I think these are very
impor­tant things that are being discussed today. I don\'t know if
that\'s going to stimulate any more comments.

Gordon?

MR. KAUFMAN: Well, I can\'t resist the comment. In Conan Doyle\'s \"The
Hound of Baskervilles,\" Watson remarks that the hound did not bay that
night, and Holmes replies to the effect, \"Exactly, my dear Watson.
That\'s what\'s really important.\"

On this issue of data quality, a key dimension of data quality would
seem to be to me identification of what the data does not provide and
that\'s necessary to meet the objectives as to why it was gathered in
the first place. And when you begin to think along those lines, away
from numerical measures of quality, you begin to think of scenarios with
which many of us are familiar.

If you go back to the oil and gas upstream data, it\'s possible to
provide an aggregated projec­tion time series for oil and gas at various
levels that is extremely timely. It\'s extremely accurate. It\'s
consistent. It\'s got a lot of continuity to it. The problem with it is
when a Senator from the Hill calls and says, \"I would like in two
months an answer to the following policy question,\" the data isn\'t
there to meet that customer need.

What\'s the bottom line with respect to measuring customer satisfaction?
Some animals are equals, but some are more equal than others; okay? From
EIA\'s point of view, as a legislative service as well as servicing a
wide domain of users, this notion of accurate measurement of consumer
satisfaction for the purpose of providing the quality of the data that
it provides is going to be an awfully elusive thing, and I would hope
that doing quantitative performance measure wouldn\'t detract from
careful attention to that dimension to data quality.

MR. MOUNT: So any other final comments? Any comments from the public?

Dan?

MR. RELLES: Since I got hit three times on short shrift of customer
satisfaction, let me say that I agree with everybody on that issue. I
simply was suggesting that it would be a byproduct of minimizing or at
least understanding the amount of error that they\'re incurring, rather
than a direct goal that you can pursue, because I frankly don\'t know
how to measure it. I think I have a chance of measuring the amount of
error that a user is incurring by the problems that are in the data.

MR. MOUNT: So I\'d like to thank Nancy, Renee and Dan for the
presentations and the other people who have contributed; thank EIA for
being a good host; and I think that we can call the meeting to an end
early.

I would like to ask the Committee, could we meet at 12:30 p.m. for
lunch? That gives us a half an hour to check out for those who want to
do it and meet at 12:30 rather than 12:45 p.m. Is that okay?

(Whereupon, at 12:05 p.m., the meeting was concluded.)