AMERICAN STATISTICAL ASSOCIATION

\- - -

COMMITTEE ON ENERGY STATISTICS

\- - -

Friday, April 21, 1995

\- - -

The Committee met in the Columbia Room at the Holiday Inn Capitol, 550 C
Street S.W., Washington, D.C., at 9:00 a.m., Timothy D. Mount, Chairman,
presiding.

[PRESENT:]{.underline}

TIMOTHY D. MOUNT, Chair

DAVID R. BELLHOUSE

CHARLES W. BISCHOFF

BRENDA G. COX

FAYE DUCHIN

JOHN D. GRACE

PHILIP HANSWER

CALVIN KENT

GRETA M. LJUNG

JAMES L. O\'BRIEN

DANIEL A. RELLES

BRADLEY O. SKARPNESS

G. CAMPBELL WATKINS

A-G-E-N-D-A

[Page No.]{.underline}

Introductory Remarks, **TIMOTHY MOUNT**, Chairman 3

Announcement of Winners on Contest On 3

Statistical Graphs, **LARRY PETTIS**

Review of Survey Design for Residential Energy

Consumption Survey

**BRENDA COX**, Discussant 7

**DAVID BELLHOUSE**, Discussant 29

Dates for Future Meetings, **TIMOTHY MOUNT** 42

Estimates of State Level Natural Gas Consumption

from RECS, **LINDA OWENS**, Presenter 43

**JOHN D. GRACE**, Discussant 51

Results and Overall Plans for Customer Surveys,

**NANCY KIRKENDALL**, Presenter 70

**MIKE LAURENCE**, Presenter 76

**JIM O\'BRIEN**, Discussant 85

**DAN RELLES,** Discussant 104

P-R-O-C-E-E-D-I-N-G-S

9:11 a.m.

CHAIRMAN MOUNT: This is the meeting of the American Statistical
Association Committee on Energy Statistics, and I have a couple of
announcements. Any EIA staff or ASA members who were not here yesterday,
maybe they could identify themselves. We\'ve got new people in the back
there.

MR. O\'BRIEN: My name is Jim O\'Brien. I was here yesterday, but I
arrived late. I\'m a former member of the Census Bureau, and I\'ve been
on the committee since, I believe, 1991.

CHAIRMAN MOUNT: Anybody in the public new? And, if anybody in the public
would like to receive a copy of the meeting highlights they should sign
the register.

So, the first event of the day is to announce the winners of the contest
on statistical graphs, and I\'m very pleased to hand the lectern over to
Larry Pettis.

MR. PETTIS: Actually, I\'m one of those people Yvonne was talking about.
I\'m Larry Pettis, I wasn\'t here yesterday. But, I\'m really glad to be
here today, because this beats the heck out of sitting through strategic
alignment meetings, where I spent my day yesterday.

The graph contest, I\'m not sure I can really tell the history of this,
but I think it was really Jay\'s idea, and it was \-- he wanted to sort
of acknowledge some of the best work in EIA as a way of promoting good
work all across the organization.

And so, I think Renee has organized this thing for us, and there were
six judges in EIA and three judges from the ASA Committee who judged the
graphs that were submitted. The EIA judges were Eugene Burns, Barry
Cohen, Bob Rutchik, Robin Reichenbach, Ann Whitfield and Mary Zitomer.
And, the committee members, which we appreciate your assistance in this
effort, were Faye Duchin, Dan Relles and Bradley Skarpness.

Now, the winners of winning graphs, I think everybody knows, gets to
have lunch with Jay, and since Jay is not here I\'m sure he\'ll take you
some place like Lion D\'Or for lunch.

And, the other thing is that these graphs will hang outside on the wall
in front of the Administrator\'s office for the next year. So, all of
the people that walk down the halls can see some of our best work.

So, the winners are, Kitty Seiferlein is the first winner. Do you want
to come up, Kitty? Congratulations.

MS. SEIFERLEIN: Thank you very much.

(Applause.)

MR. PETTIS: There\'s a lot of people that like this graph. Somebody had
an idea about putting this up on Internet, so you could click on to the
various areas and go to different information sources within EIA.

The second winner is Irv Chamberlain. Thanks, Irv.

MR. CHAMBERLAIN: Thank you.

(Applause.)

MR. PETTIS: And, the third winner is a joint entry by Mary Carlson and
John Herbert, and you\'ll both get half a lunch. Congratulations.

(Applause.)

MR. PETTIS: And, the fourth winner is Barbara Fichman.

MS. FICHMAN: I did not do this. I just admired it, okay?

MR. PETTIS: Oh, you did? So, you get an award for admiring it.

(Applause.)

MS. FICHMAN: It\'s interesting, it\'s a good example of team work,
because I think it started in natural gas issues and trends, and we
published it in the highlights in the Monthly Energy Review, and later
on in the Annual Energy Review, the whole, all three together, first
appeared in the Annual Energy Review, and Kitty Seiferlein designed that
page. So, but she didn\'t want to come up twice, so I\'m here.

(Applause.)

MR. PETTIS: And, the last winner is Jim Disbroy, who I don\'t think is
here today. Is somebody going to accept this for him? We\'ll make sure
he gets it anyway. I think this ended up in our annual report to
Congress, as I recall.

So, anyway, thanks to everybody who participated in submitting the
graphs, preparing the graphs, judging the graphs, and we\'ll do this
next year, and admiring the graphs.

CHAIRMAN MOUNT: We\'ll start the first session now, which is a review of
survey design for the Residential Energy Consumption Survey, and this is
going to be a discussion by the committee, and the first person is
Brenda Cox.

MS. COX: I have to begin by making reference to what the Administrator
said yesterday about the Consumer Satisfaction Survey, asking whether
EIA\'s customers were satisfied with the products, and the answer was
\-- with the quality of the products, and the answer was, well, we
assumed it would be high quality or you wouldn\'t show it to us.

Now, I think RECS is a good example of a high-quality survey, and I
would hate, in cost saving days, to have this changed. I\'m going to
give you several suggestions for improvements, but I\'d like to start by
saying, this is a good design. It\'s attacking very hard topics to get
data on, and it\'s doing it in a really nice way.

But, I would feel that I wasn\'t worth the salary I was paid if I
couldn\'t tell you something you could do differently. This is a
recurring design, every three years. Some parts are actual longitudinal,
they are followed over time, other parts are just a cross-sectional
survey.

The unit that is of interest is interesting here. It is both the housing
unit and the household, I would say. I saw some appreciation for that in
the literature, but I would like to emphasize in energy use it is the
characteristics of the two that are of interest, not just the housing
unit. And that, in particular, has an impact for the longitudinal
survey, because it does follow people every three years and about 20
percent of the Nation\'s population moves every year.

This is a multi-stage area cluster design, where every three years half
rotates in and out, so that you have some units staying in and reporting
that reported last time so you have some continuity. That\'s often done
for change estimates, to try to get more precision. It leads to a more
expensive design, in particular, when you change censuses.

The first question that we were asked by the committee was, do we really
have to redo the design with the next census? Oh, yeah. In fact, when
I\'ve been at the end of a decennial census doing a survey, the quality
of the data, you can tell the way the quality starts declining. And, one
way you can tell is when you start doing your weights, and all of a
sudden you are noticing under coverage problems and other difficulties,
that really says that the size measures you were using for selection,
which were, in my case I was using 1980 size measures in 1988, and it
wasn\'t a good thing. The very distribution of the way households were
arrayed changed under my feet. For a survey doing new construction,
well, new construction occurs where construction wasn\'t before, or
tends to occur where nothing was before. So, new construction becomes
another problem.

During the last \-- there was a lot of changes in the sample design from
the decade of the \'80s to today. I liked many of those changes. I
thought they made sense. There was one change that I wanted to say I
kind of liked the way they did it in the \'80s better, and that is,
during the last decade they would \-- they are doing a probability
proportional to size selection. For the size, in the last decade the
size was the population count, the number of people, and that was a bad
size measure. This decade they are using housing units or households,
which is a good size measure. But, they were selecting, they were
creating zones and making a random selection within the zone, with the
zone being the equivalent of the skip interval for a systematic sample.

Now, this time they are just doing a systematic sample, and I wanted to
point out that there\'s a technique that is equivalent to systematic
sampling but avoids the problem they had in the last decade. In the last
decade, if there was a unit that crossed the boundary of the zone
because of its size measure, it would have \-- it could have more than
one opportunity of selection when it didn\'t deserve to. This technique
has Chromy sequential PMR selection technique. It\'s easy to use. I
think he would make the software available to you, and what it does is
the equivalent of the random selection within zones, but it controls for
that cross-zone selection probability.

At RTI, I used this design. You can come up with exactly equal weights
for an area household design with this approach, without all of the
tailoring and adjustment that I observed in your current design. In
other words, it takes less professional time and I think creates a
better end product.

Now, generally, it appeared that this design did give or was trying for
equal weights, generally, within domains. In other words, new
construction, the poor were over sampled, but, generally, within
whatever the domains were, and it wasn\'t totally clear to me what the
domains were where over sampling was desired, but it was clear that
within those domains equal weighting was the intent.

It wasn\'t clear that it was exactly being achieved, and that\'s the
reason I wanted to mention, with this technique you can exactly achieve
it. It was something that RTI actually learned from Marty Frankel in the
late \'70s, and I would recommend it, and I\'d be happy to tell you more
about it.

I would say that having done it both ways, with the rounding cause by
trying to \-- by needing an integer number of selections, by defining
self-representing PSUs in advance, et cetera, the difference between
that and this other technique, just the rounding effect can be as much
as a five percent increase in variance. So, it\'s not a trivial thing.

To go on to another subject, the poor or poor income housing, there\'s
an attempt in this survey to over sample. I don\'t like how it\'s done.
This is one part where I really didn\'t feel comfortable with how the
poor and new construction were identified, Interviewers, in listing the
segment, travel around the segment and classify the housing unit. This
is a very subjective measurement. For new construction, it means that
you are trying to identify a house that may be as much as six years old,
and I just wasn\'t sure how efficiently it could be done.

I have a much better technique for identifying the poor that is
objective. I\'m sorry I can\'t give you a better technique for new
construction. For the poor \-- I know, this is what you want most, but
for the poor what you can do is a technique RTI has used in the past,
I\'ve used it myself while I was there, it works very effectively, you
take, within a PSU you take your segments, which are like city blocks,
you order them from smallest to largest in terms of the median rent, and
you assign just a numerical order, where they come in the list.

Then, you reorder those segments. You accord that value, call it a rent
index. Then, you reorder your segments by median housing unit value and
assign a number from smallest to largest and just assign a number where
they fall in the list. Call that housing index.

Then, you create a composite of those two, where you composite them
based upon the proportion who rent and the proportion who own. And, that
becomes your housing unit index.

Then, sort by that index and use, for low income, a certain percentage
down in that index. I usually use the lower third. It works very
effectively in identifying neighborhoods of very poor housing. You
wouldn\'t do so well if you had a hovel amount millionaires, because
that would kind of get lost, but since these tend to be more grouped
it\'s very effective.

In the past, other things have been used by RTI, like presence of
bathrooms and things like that, but it doesn\'t work as well anymore
because people tend to have indoor plumbing these days.

For new construction, I thought you really should try to find an
objective index, but I\'ve never tried to do it, so I don\'t know how
you could do it. I thought, well, if census geography would stay the
same between censuses, then you could, perhaps, look at just the growth
that\'s observed there. But, unfortunately, at least from what I hear
from the Census Bureau, they feel no obligation to keep their geography
in any way similar, so the geography stays the same so that you can\'t
make comparisons across decades.

Then, not only that, there is a variable that says, did you live in this
housing unit at the time of the last census, and that\'s about all that
I can suggest that might help a little to get at the new construction.

One of the things that\'s done in updating for new construction is the
visual inspection by the interviewers. It appears to me that you are
giving a lot of weight to the half-open interval technique, in
identifying both new construction and unlisted units. I wanted to give
you a warning, and, again, this is based upon experience with some old
surveys I did at RTI, and that is, we were using general purpose samples
to do two surveys, one in 1977 and one in 1980. The \'77 survey was a
combination of RTI\'s general purpose sample, which was freshly listed
immediately before the survey, and NORC\'s sample, which had been listed
three years previously. Now, the sample was selected, then with the use
of the half-open interval technique for both surveys, but it had been
more important for NORC\'s, which were using three-year old lines.

The problem, there was a problem in that survey. It wasn\'t detected at
the time, because in the course of post-stratification RTI\'s sample
actually got counts higher than census projections, NORC\'s were lower,
and the two kind of canceled out. So, the fact that there was a problem
there wasn\'t recognized.

By the time the survey was repeated in 1980, RTI was using lines that
were listings that were three years old. NORC was using listings that
now were more than six years old to select their sample. In doing that
survey, RTI found that it had ten percent under coverage, and the person
doing the weight discovered they had ten percent under coverage, and I
was asked to look into it to see what was wrong.

Well, I went back to the original 1977 survey, looked at the two surveys
independently, and what I saw was that RTI\'s counts for major
demographic domains appeared to be very much on target with what the
census projections should have been. Some of you may not be aware that
the census projections in the \'70s were too low, and they discovered it
at the time of the 1980 census. Well, RTI\'s counts that we were getting
were actually more than the census projections, particularly for domains
like young Black males, which is a chronic problem for the census
anyhow.

NORC\'s, however, were about, I would say about five percent low. The
two samples were independent, so, basically, RTI is above the census
projections, NORC is below.

At the time of the next survey, we looked again, now RTI\'s counts are
five percent below the census\', and NORC\'s were 15 percent below, and
when combined we got the ten percent under coverage.

We looked into it and could find no good reason. It appears that
although in practice the half-open interval technique should help us get
rid of this problem, it doesn\'t appear to. In other words, there seems
to be some consistent under coverage there, and it can be serious
depending on how long it\'s been since you did the listings.

I\'m not sure of the extent to which having your interviewers go back
with the same listings and try to validate it, how much that helps.
That, I think, was already being done by RTI and NORC.

So, I wanted to warn you about that. I\'ll come back to this point in a
minute when I get to your weighting approach.

I would say that in general this interview appears to be unsuitable for
telephone implementation. In other words, a telephone implementation
would be another survey. You have your mail survey. I could see how \--
there\'s a mail survey that is actually sent around to non-respondents.
That might be able to be \-- those kind of questions could be used in a
telephone interview, but I want to warn you that a lot of the categories
in your personal interview, you have lots of categories in the answer,
and those are very difficult for a telephone interview. Categories are a
problem listening to, you can\'t make the context, it\'s a concentration
factor when you are not looking at someone, it\'s a lot harder to
concentrate and keep the context.

So, the questions look difficult by themselves, before you bring in the
problem of the actual measurement that you are doing. And, I think the
dimensions of rooms are being measured by the interviewer, I consider
that rather crucial, because I think many people would not know the
square footage of their home, and heated space, unheated space and
things like that, so it seems to be crucial.

Before I actually went with a totally telephone approach, I would try to
see what could be done in terms of cutting back but preserving the
quality of what you have. In other words, I\'d rather have a good
quality estimate that doesn\'t allow me to do divisional level
estimation, may not do separate estimation for new construction or
whatever, but what it does it does well. And, that would actually be the
goal that I would give, that whatever the new redesign less costly
system gives that it would do it well.

I\'m not going to say what to get rid of and what not to get rid of, but
I think that would be my recommendation, whatever it does it should do
well.

From the present design, I would say the reports aren\'t clear, but it
appears that the clusters you are using are compact, in other words, at
the very last stage that you are selecting adjacent housing units. I
assume that was the case, because you are also going with four to five
housing units, which is usually what the optimal answer is when you are
doing compact clusters.

At RTI, for ages, I guess since the \'70s, they\'ve been using
non-compact clusters. Their segments would be about the size of the
segments that you all are presently using, which is about 50 housing
units, but instead of selecting a compact cluster, a systematic sample
from there would be selected.

Now, I think that approach could still be used and, yet, having
recurring designs from the same segments. The advantage of that is that
the inter-cluster correlation is a bit less, so that, actually, in the
optimizations that we did at RTI what we would normally find is that the
optimal cluster size would be more like eight or nine per cluster, which
could yield cost savings. Now, at a guess, it can\'t be done until the
next decade, but it is something to think about for the next decade.

I would say also in general, with respect to the optimization, that I
found the optimization results rather unbelievable. There are some
unique features of this design. The optimization results suggested that
you should do, I believe it was like, one in a quarter households per
segment, which is just a highly unusual result. That suggests very high
household level cost, and one thing to consider in redesigning is,
anything that can be done to reduce the household level cost. But,
besides suggesting that, I did find it difficult just actually believing
the optimization.

I mentioned the under coverage you had earlier. The survey is
benchmarking to CPS survey data. I wanted to caution you about that
twofold. When I first read it, I thought, oh, well, they are just
post-stratifying to age, race and sex data, and then I realized, no, no,
they are not doing that because this is not a person level survey. A lot
of people will say they are post-stratifying to CPS data, and they may
be actually using CPS data to get estimates of age, race and sex, but
what they don\'t know is they are using census projections of age, race
and sex.

Now, in this case, though, I believe what I\'m saying is correct, call
Chip Alexander of the Census Bureau to verify, he was telling me what he
knew off the top of his head. CPS, and most of the census surveys, has a
problem with under coverage. I\'m afraid, it sounded to me like this
RECS already has an under coverage problem compared to the CPS. Well,
the bad news is, CPS itself has a ten percent under coverage problem. If
I understood the reports carefully enough, if I read them carefully
enough, it appears that they have an eight to ten percent under coverage
with respect to the CPS. Well, CPS itself has a ten percent under
coverage.

The under coverage that CPS has appears \-- I\'ve always wondered if
it\'s a census government problem, because I haven\'t observed anything
that bad myself when I do a fresh survey. But, what I would caution you
about is, CPS uses what\'s called the Principle Person Method of
Weighting to get its household level counts, and I find that method
wholly objectionable. I\'m sorry, to be honest I find it wholly
objectionable. I was a discussant about ten years ago on a set of papers
discussing this method, and, in fact, Census doesn\'t like it that well
either, so I\'m saying the household level counts from the CPS are
suspect. I don\'t recommend you using them. You might want to look, I
can give you references for the papers that discuss the method and the
problems they\'ve observed.

Now, one of the things you all were talking about was losing the one
person households and that is a definite problem. You\'ll find that also
discussed in a companion paper, but it\'s the Canadian experience in
losing one person households. LeMaitre, I think, was the one who did
that research. Chip Alexander was one of these papers, and there\'s a
paper from BLS.

Anyhow, what Chip was telling me at that time what he was working for,
and has been working for some time, is to have actual projections of
households being done by the Census Bureau, and he tells me that he
thinks, he says you\'ll have to check and he gave me a number, Leonard
Norry, but he believes that they now have good household level counts
that could be used for post-stratification, and they should be available
by region and division. They might not do, I don\'t know if they do head
of household like male only, the one person, et cetera, but you might be
able to do some kind of racial adjustment.

Another question I had that I couldn\'t go any further with is, you are
using predicted energy expenditures in selections, and I was just
wondering, well, how good a measure is this really. I don\'t know.

For your optimization results, I said I didn\'t particularly like them,
I wanted to mention a couple of things that \-- an approach that Jim
Chromy has developed. You can see it described in a paper by Richard
Sigmund and Nash Monsour in [Business Survey Methods]{.underline}. He
makes it available to outsiders. Jim Chromy is at RTI, and it\'s an
optimization procedure that uses a Koon-Tucker algorithm. It\'s very
good when you have to optimize with many different estimates coming into
the optimization. It appears the optimization here was done an estimate
at a time, this allows you to grow across estimates, in other words,
different quantities being estimated, across domains as well, to have
one model that puts it all together.

When I\'ve used this at RTI, I also had variance components that were
estimated using a package that was available there. I\'m not aware of
that capability anywhere else. I discovered the capability wasn\'t
anywhere else after I left RTI, but I don\'t believe Westhead has the
equivalent software, or anyone else, or even the Census Bureau. It\'s a
program called VariComp, which does variance components, and it\'s a
very, very old program that was a predecessor for SUDAAN. It\'s never
been released to anyone. It\'s just an old primitive program that they
retained because it does variance components. And so, it\'s not used for
very much at RTI, other than variance components.

The response rate for the survey is very good, 81 percent in 1990. In
spite of that, I\'d say, you might be able to do a little better. I
wasn\'t sure if the physical measurements you were making was causing
the problem. The reason I said you might do better, I would say that
would be good if you had to interview a random person within a
household, or every person within the household, but here you are
actually, you are really only talking to one person, so you can deal
with a cooperative person within the household. In other words, they
often talk, the households often have cooperative people and
uncooperative people. Well, because this is really household and housing
unit level, you don\'t have to deal with the individual people, so I
think 85 percent might be possible for this survey.

I noticed an odd little statement in the report that I wanted to bring
up. It\'s a side issue, but I wondered how you were actually defining
your units from the listing stage. It said, because the sampling was
finalized during the summer of 1990, and most interviews were conducted
during the fall of 1990, the units that were first occupied during the
latter part of 1990 would be under-represented in the 1990 RECS. This
was a curious little statement for me, because in listing I would have
my interviewers list all structures, all addresses, so even if something
was clearly a business, it still would be listed and a check would be
made that it wasn\'t, in fact, didn\'t contain a residential structure.
If a structure was under construction, it would be listed, just in case
it was occupied at that time.

And so, I wondered if when they list if they were actually listing all
addresses or just all addresses that appeared to be in occupiable form.

So, I\'ll finish just by going back to the questions you asked. Is it
necessary to redesign the sample every census? Yes, I think it\'s
absolutely critical, if you continue with an area household design, that
there is no choice. You really must do it.

If I had to give something up, I would give the overlap between three
year surveys up, in other words, do a new set of units, forget about
trying to keep people in the sample.

I noted the repeated use of Keyfitz Procedure. In theory, that works. It
may be a little queasy to use it decade, after decade, after decade, but
I didn\'t know any theoretical reason to say don\'t do it, I just
wondered if something bad would happen. I do know that Michigan had used
the technique repeatedly, but I think they felt by the time they got up
to 1980 that maybe they should reselect their sample. So, you might just
want to consider that, how often can you use the Keyfitz Procedure to
keep PSUs in when you redesign with each decade.

I didn\'t have good recommendations for how to update for construction,
that looked like a really hard topic.

On the subject of RDD, as I said, it doesn\'t seem to be a good
interview as is, so it appears to me that if you are going to consider
an RDD approach you are going to have to combine it with something else
to get the kind of information you want. I just couldn\'t imagine that
you would get the kind of information you needed for your modeling
purposes prediction, reporting, et cetera, from a straight RDD design.
So, it appeared that some kind of innovative design was going to be
needed.

And, again, I would prefer to retain the quality as opposed to \-- and
sacrifice maybe a little of the fine ability to report that you might
presently have for redivisions, which I wasn\'t sure how accurately you
could actually predict for divisions to begin with, as you only have
eight PSUs, but sacrifice some of the detail that\'s now possible, but
retain the quality.

CHAIRMAN MOUNT: Thank you, Brenda.

The second discussant is David Bellhouse.

MR. BELLHOUSE: Let me begin by saying that as the first time here that
was a pretty hard act to follow. Don\'t do it again to me.

I only have a couple of things to add. One is, again, the effectiveness
of the longitudinal portion of the sample. The overlap in PSUs between
surveys is used to approve estimates of change.

Now, the surveys currently are three years apart, so, perhaps, it might
be useful to look at correlations between the overlaps between the two
surveys, because the surveys are three years apart if those correlations
are small then you can easily get rid of the overlapping or the
longitudinal portion of the survey with little loss in efficiency in
terms of estimating change.

So, that\'s merely a recommendation, that if you are going to consider
dropping the longitudinal is to actually study the correlations.

My second point is in terms of making a suggestion for reducing the cost
of the survey, and that is to look at a double sampling scheme. In the
first phase of sampling, you take a sample using random digit dialing,
and that would have to use a very simplified questionnaire, and then for
a subsample of those chosen on the telephone you would then go for a
personal interview. You\'d use regression estimation at the second
phase, taking the additional data from the first phase of sampling to
get your regression estimator, and you\'d use that to improve the
estimates based on the data obtained at the second stage.

With the random digit dialing, you might consider, because of \-- you\'d
want to \-- in the personal interviews, you\'d want to have people
fairly close together, so you\'d want to do multistage sampling of
things like the telephone exchange, which would get people modestly
close together.

With random digit dialing, it seems it would be difficult to \-- not
difficult, sorry, modestly difficult but not impossible, I presume, to
over sample for poor people or for new housing. You\'d have to contact
extra people and do screening.

And, my last point is that, if you are going to go the new route and
completely redesign it with something like a double sample, you might
definitely want to go to a contractor to look at \-- with technical
expertise, who would design the sample and then go to other contractors
to collect the data.

Thank you.

CHAIRMAN MOUNT: So, are there any other comments from the committee? I
think it might be sensible, given that this was an unusual session, to
see if we can get comments from the public as well.

MS. CARLSON: I\'m Linda Carlson. I\'m Director of the division that, in
fact, has RECS in it. We have been having quite extensive internal
debate on how to redesign to, essentially, save money.

One of the options we have been talking about is, in essence, keeping
the longitudinal sample or a subsample of the longitudinal, because that
doesn\'t cost us any relisting, and then supplement it with an RDD
sample for either a telephone sample of households, be it households \--
or another alternative is to do a telephone resample of the RECS
longitudinal households, supplement that with an RDD sample, plus we
still have listings we have not used out of \'93 and select new
households out of that.

What would you think of that kind of approach?

MR. BELLHOUSE: If keeping the same households in the sample adds no
extra cost, it would seem to me that you could then reduce cost the
second time around by a telephone sample, just have a personal interview
once and then update it with a telephone interview. And, I think that\'s
done in the Canadian Labor Force Survey.

MS. CARLSON: Yes. That\'s how we were thinking, update that way, because
that saves us \-- we, in fact, know we can get waivers from those
households.

MR. BELLHOUSE: Right, that makes sense.

MS. CARLSON: And then, use it, and you would see no concern using the
old listings to pick new household \-- a very small sample of households
to do a personal interview.

MS. COX: Keep in mind the problem I was discussing with under coverage,
and you probably need to look a little closer at your survey now with
the current procedures to see what under coverage you seem to have.

I would suggest that you check back against \-- see about getting these
housing unit projections that I was talking about, look at your
non-response adjusted weight before your post-stratification to CPS, and
see how you seem to be doing with under coverage.

I\'m particularly concerned about under coverage, because what you would
be missing would, in fact, be mostly new construction, I would think.

MS. CARLSON: Yes.

MS. COX: And, that would be very serious, to be missing that.

MS. CARLSON: We are very interested in the RDD, and we thought what we
would attempt to do is some very small pretest, even starting this fall
ourselves, and literally use \-- to program up a small questionnaire and
start testing it.

What we are concerned with RDD is, in fact, our ability to get the
consumption and expenditures data, which is crucial to the whole survey.

MS. COX: And, I guess it was in some of those questions like that that I
was concerned, because they wind up being almost like long lists, and it
was the capability of actually answering those questions over the phone
that I was concerned about.

MS. CARLSON: Yes.

MS. COX: Now, you could think, with a telephone sample you could
actually, if it appeared possible to actually collect the data, or,
perhaps, a subsample of the data, the old accurate estimate, inaccurate
estimate type thing \--

MR. BELLHOUSE: The double sampling.

MS. COX: \-- the double sampling approach that David here was talking
about, if that appears possible it could be considered. You could even
select your telephone sample within the same areas that you were doing
your personal interview survey.

MS. CARLSON: David, is there any way, using telephone exchanges, we can
pinpoint new exchanges, new construction exchanges?

MR. BELLHOUSE: Other than information from the telephone company, you
wouldn\'t be able to do that.

MS. COX: Oh, keep in mind David is Canadian.

MS. CARLSON: Oh.

MR. BELLHOUSE: Is that a criticism?

MS. COX: No, no, no, I meant knowing you as telephone exchanges might
not be your thing.

MR. BELLHOUSE: No, I don\'t know. We have area codes, too.

MS. COX: There is one thing on new construction, but I don\'t know that
you\'d have access to it. I know in doing projections, the Census always
talks about getting less of new construction, and, apparently, it\'s
very \-- it\'s actually LAN level that they are getting and processing.

I don\'t know if you could get access to that, but they are apparently
doing that as a part of \-- they routinely get listings of new
construction, the Census does, the Census Bureau does. They use this in
projecting. They sample for new construction.

MS. CARLSON: They are getting only from builders over certain sizes.

MS. COX: Oh, is that where it\'s coming from?

MS. CARLSON: Yes.

MR. SKARPNESS: As someone that just purchased a new house, I lived in an
apartment for like nine months, and everybody else that I know that\'s
building lived in some kind of temporary housing, and we kept our same
phone number. So, you know, I don\'t know how that\'s going to \-- you
know, how all of a sudden it shows up that you are either living in an
apartment or you are living in a new house. So, I think there\'s a
little problem there.

MS. COX: With telephones, you just have to screen. You take a larger
sample \--

MR. SKARPNESS: Yes.

MS. COX: \-- then you would ask, have you lived in this home for the
last six years.

MR. SKARPNESS: Right.

MS. COX: It\'s not a bad question to ask at the start of a screening
interview. The income one is a tacky one.

MR. HANSER: The other possibility, if you want to go to permits data, I
mean you could go to local townships and do a sample of the permits data
and then work your way in that way.

You have to be a little bit careful, because permits don\'t necessarily
mean somebody has actually built, right \--

MR. SKARPNESS: Or living there.

MR. HANSER: \-- or living there. But, at least it\'s a place to start,
and then you\'d at least have some real data in terms of new
construction.

I have a question that isn\'t related to the sample design issue, but
it\'s related to something completely different, but maybe there\'s some
answer I can get to.

There\'s been some suggestion about modifying the Consumer Price Index
because of the belief that it somehow isn\'t taking account of
technological change or whatever else. And, I know that relative to
households there\'s been a fairly dramatic change in the efficiency,
both of the structures, the shells, and of the equipment in the shells,
is, does the RECS database or the RECS information inform the BLS in any
way, shape or form about those kinds of considerations?

MR. THOMPSON: The Consumer Price Index uses the RECS data, but not in
that way. They use the RECS data to do some imputations for what the
utility costs are in homes that are rented and utility costs are
included as part of the rent. That changes from \-- the status of that
may change, so that the renter starts paying for his utilities, or from
paying his utilities goes back and it\'s included in his rent, and other
places where \-- and other cases where they don\'t know what the utility
charges are. They use the RECS data in a very sophisticated way,
creating their own imputation models, and that\'s the only way in which
RECS is currently used.

MR. HANSER: I\'m a little puzzled about the imputation, I\'m sorry, they
impute an energy cost for renters from RECS?

MR. THOMPSON: It\'s a part of the rental index, when they have data that
they don\'t know what the utility costs are, and part of the cost for
housing is a part of that index.

MR. HANSER: But, are they certain that the rental costs that they are
getting are exclusive of energy costs?

MR. THOMPSON: They find that out in the interviews that they conduct.

MR. HANSER: Oh, okay, so they know that. Okay.

CHAIRMAN MOUNT: You should identify yourself, I think.

MR. THOMPSON: I\'m Wendell Thompson. I\'ve been Manager of the RECS
Survey.

MR. HANSER: But, is there, then, an issue, you know, in terms of the
same issue that\'s raised around PCs and everything else, that, in fact,
the nature of the \-- you know, the consumption is really essentially
changed because of changes in technology, and that\'s not, in some
sense, accounted for in the CPI. I mean, one of the issues, for example,
is, suppose you have a capital bias as a result of the efficiency
requirements of the various energy acts, so that, in fact, what\'s
happening is you spend more money on the shell and the base equipment in
the house, and in doing so you reduce your overall level of consumption
of energy. You\'ve implicitly made a capital versus operating cost trade
off. It seems to me that if you use the same base for computing, you
know, the consumption in the CPI all the time, and not account for the
change in the capital stock that\'s resulted from the change in the act,
that you, in fact, bias the CPI in some sense.

MR. THOMPSON: In the RECS, we make estimates of what the consumption is
for the end uses and what the expenditures are for the end uses of
energy, but we are not able to separate the effects of improvements in
the shell versus improvements in the equipment.

MR. HANSER: You don\'t even have to do that, I mean, all that it has to
be is that there\'s a clear choice, a clear bias sometimes created by
changing the amount of capital investment being made versus the
operating costs of, you know, living in the house and maintaining
certain temperature and so on. This is the same issue that goes around
PCs, right? I mean, I make a capital investment in a PC, that capital
investment changes the nature of the good that I am, in fact, consuming,
and I end up at some higher level of social welfare as a result. And so,
if I use the same sort of base all the time, I, in fact, am, you know,
measuring a good that\'s producing value and, therefore, overstating the
price index.

MR. THOMPSON: Maybe this is something we could pass on to the consumer
price people. We do have information on personal computers in our
survey.

CHAIRMAN MOUNT: Any other comments?

I would just like to say that I think that more information about how
the stock of appliances changes, and how energy efficiency changes is
very important, and this has been raised before by this committee. Phil
has purported a new spin to it that may be of value in contrast.

So, seeing no more comments, we can close this session and \-- anybody
else from the public \-- so, we\'ll close this session now and we\'ll
have a longer coffee break. Be back at 10:30.

(Whereupon, at 10:04 a.m., a recess until 10:33 a.m.)

CHAIRMAN MOUNT: I\'d like to just do a little bit of business with the
committee about dates for future meetings.

The original plan to meet in October had to be changed because Jay Hakes
was unable to make those dates, so the proposal now is to meet on the
2nd and 3rd of November, if committee members could just sort of \-- the
2nd and 3rd of November.

Seeing no cries of anguish, that\'s the proposal, and the bottom line is
we don\'t have much choice.

And, the April meeting for 1996, the proposal is for the 11th and 12th
of April.

MR. HANSER: When is the ASA summer meeting?

CHAIRMAN MOUNT: Does anybody know when the ASA summer meeting is?

MS. COX: The summer?

CHAIRMAN MOUNT: August 13th through the 17th, is that \-- the ASA
meeting, yes.

MS. COX: What did you say about April? I missed it.

CHAIRMAN MOUNT: Eleven and 12.

MS. COX: Cherry blossom season.

CHAIRMAN MOUNT: Yes, so it sounds good to me.

Okay, so the next topic is estimates of state level natural gas
consumption from RECS, and the presenter is Linda Owens from the Office
of Energy Markets and End Use.

MS. OWENS: Thank you.

I have some slides to show you, but also, Linda, right in front of you
and, Dwight, right to your left are a pile of hard copy print outs of
the slides in case you can\'t see or something or if you want a bunch to
take home with you.

What I\'m going to talk about in the next ten minutes are these state
estimates of residential natural gas consumption and the data and
methodology behind them.

In 1993, EIA did a user needs study for the Residential Energy
Consumption Survey to ask our data users what we could do to improve the
quality of the survey. The suggestion that they made most often is that
we could provide \-- if we could provide estimates at the state level
then the data would be a lot more useful.

However, the survey size of the RECS is too small to provide direct
estimates at the state level, and it\'s financially unfeasible for us to
increase the sample size so that we could produce direct state
estimates.

So, one alternative to the direct estimates is to use model-based, small
area estimation methods to produce indirect estimates of state energy
consumption.

Before I talk about the data and methodology, I would just like to
review the questions for the committee. The first question is, does the
overall methodology approach seem methodologically sound, should we
benchmark the estimates, and, if so, should it be at the level of the
census region or the census division, how should we produce the
estimates in the non-census years, given that on the current schedule
the next time that the RECS and the census will be jointly fielded will
be the year 2020, have we stretched the data too far by basing our small
area estimates on numbers that are modeled to begin with, and then would
the estimates be reliable at the PUMA level? PUMA is a public use
microsample area, and I\'ll talk about that a little bit in a minute.

Okay. One of the basic requirements for small area estimates is that you
need a good auxiliary database that includes a subset of the variables
on the survey of interest. For the RECS, a good database is the Census
Bureau\'s 1990 Public Use Microsample. These data include five percent
of the households in each state, and they also include several household
characteristics in common with the RECS.

The one drawback to them is they do not have information on weather for
those households. For that reason, we had to append data from the
National Oceanic and Atmospheric Administration. The NOAA data include
county level estimates, or county level measures of heating and cooling
degree days. The one problem with this is that the census data do not
contain the county of the residents of the household, but they contain
something called a Public Use Microsample Area, which is a geographic
area of about 100,000 residents. So, in densely populated areas, one
county such as Cook County, Illinois, which contains Chicago, could have
17 to 25 PUMAs or something of that size. And then, in sparsely
populated regions like Wyoming, one PUMA could contain five, or eight,
or ten counties.

So, to attach the county level data to the PUMA in densely populated
areas, we simply assigned every PUMA within a county the same county
level data. And then, in sparsely populated areas where a PUMA contains
several counties, we assigned the PUMA a population weighted mean of the
weather data.

Energy models need both the energy sources used in a household, as well
as the amount used for each end use. The PUMS data include the fuel used
for space heating, but not the fuel used for water heating and
appliances, which are two other end uses of interest. Therefore, we had
to predict the number of households that use natural gas for water heat
and the number that use natural gas for appliances, and we did this with
a non-linear or with a logistic regression technique.

We developed the model based on the RECS data, one equation for each end
use, and then applied that model to the PUMS data. The result of that
procedure is that we have a predicted probability of each household
using natural gas for water heat and another predicted probability of
the household using natural gas for appliances.

Then, we compared these probabilities to probabilities produced by a
uniform random number generator, and if the predicted probability was
higher we kept the household in the sample, if the predicted probability
was lower, we deleted it.

So, once we determined the households to include in the methodology, the
next step was to produce the consumption estimates, and there are three
equations here for the three end uses of interest, space heating, water
heating and appliances.

You can see from the overhead the basic form of the model, and then the
background variables used in each end use.

Again, this was based on the RECS data, and then once we came up with
the output and the coefficients we applied it to the PUMS data. So, the
result was an estimate of the total BTUs of natural gas use for each end
use for every household in the PUMS data.

One way to gauge the accuracy of these results is to compare the results
to the RECS data at the region level and to the State Energy Data Report
supply data at the state level. This first table shows the RECS number
of households that use natural gas and the state estimates number of
households that use natural gas. The total consumption in the four
census regions, according to the RECS, the state estimates and the
supply data, and then the consumption per household according to the
RECS and according to the state estimates.

In the first column, you can see the total number of households is
somewhat different between the RECS and the state estimates. The RECS
data represent the population of the United States as of November of
1990, whereas, the state estimates which are based on census data
represent the number of households in the United States as of April. So,
that could explain why we have slightly more households in the RECS
data. That may also explain why the total estimates are somewhat higher
for the RECS.

As you can see in this second column, that the RECS estimates are
somewhat higher than the supply estimates, and then the supply estimates
are somewhat higher than the state estimates, with the exception of the
West Region. However, the difference in the number of households can\'t
really adequately explain why we have a difference in consumption per
household.

As you can see in the last column there, the Northeast and the West
Regions are close, in fact, the West is right on, but the Midwest and
the South have discrepancies at the household level.

Another way to look at these estimates and see how accurate they are is
to compare them to the supply estimates at the state level, and this
next table shows the RECS, state estimates and the supply estimates for
the ten states that use the most natural gas.

You can see here again the basic patterns, the supply estimates are
somewhat higher than the state estimates, except for Wisconsin where the
state estimates are higher, and in the worst case Illinois has a pretty
large discrepancy of about 90 trillion BTUs.

This information I\'m showing in a different form on the next overhead,
which is a graph of the difference between the supply estimates and the
state estimates. As you can see, the bars going upward are those states
where the supply estimates are higher than the state estimates, and the
bars going downward the state estimates are higher.

Again, you see Illinois stands out as the case with the largest
discrepancy, it\'s followed by Michigan, Texas, California, New York and
Ohio. The one puzzle for us at this point is why Illinois is so much
different than the other states, and why the difference is so much
larger.

At this point, I\'d just like to put the questions back up, and then
pass this on to the discussant.

CHAIRMAN MOUNT: Thank you, and the discussant is John Grace.

MR. GRACE: I\'m, perhaps, the last person on this committee to be
reviewing this paper. My knowledge of sample theory is nearly nothing,
stratification, to me, is something that happens to rocks laid down in
low-energy environments. While I\'ve given a lot of thought to natural
gas supply and production, natural gas consumption only arises in my
household at Thanksgiving when I turn on the oven.

So, my comments will be kind of limited. There are a couple of small
things about the explanation of heating degree days. There\'s kind of a
curiosity about how it is computed, what it means on page two, if no one
is familiar with it, some of the variables don\'t agree in the tables
with the definitions.

The general methodology of using the logit equation for determination of
yes/no, and then the regression on top of it is a familiar construct to
me. It makes a lot of sense for doing this. We have in prediction of the
amount of oil and gas in a prospect the same sort of proposition where
you examine through a logit model whether the prospect is productive or
non-productive, and then using a regression the estimation of how much
is there. So, there\'s some precedent, at least in my experience, for
that.

I would have liked to have seen a little explanation of why certain
transformations of the variables, uses of square root or the use of a
log was explained. Again, this might be better for me, who is totally
outside of the area, then someone for whom it would be obvious.

The same thing applies to the non-linear estimation, a little more
explanation of the algorithm for estimation, which particular algorithm
was used, even if it\'s referenced to one of the subchoices within PROC
ENLIN of what you would want to do.

I would be concerned looking at the variables there that there would
have been multi-colinearity that might have arisen on the right-hand
side in the non-linear estimation, specifically, the multi-colinearity
that might arise between income, age and race, because there were some
curious observations about African American households used more energy
per unit and the relationship between that income, the same thing
between age and income.

You might also look at the proposition of doing a step-wide analysis in
the determination of what the appropriate regressors would be.

I would make a couple of general suggestions. One is, I don\'t know
exactly what the uses of the disaggregated estimates would be put
towards, and understanding what those uses might be might help inform
the method for making that disaggregation.

Also, it might be another way to approach the problem is to look more
carefully at, or just employing some method, information from the
producers of those items that essentially the consumption through which
you are trying to estimate, those involved in space heating appliances,
those involved in water heating appliances, those involved in natural
gas appliances that are neither space heating, nor water heating, it
might give you some sort of orthogonal look at the same problem.

With respect to the lack of coincidence between the underlying
census-based estimate and this estimate, I don\'t know how firmly
entrenched or necessary the determination of a three-year interval is,
but at least with a five-year interval that would accomplish the goal of
putting you in sync with Census, as long as you started in a year that
had zero at the end of it.

And then the final suggestion, which is a totally different way to
approach the problem, and, again, appeals to the minor strength that I
bring to this, would be to look at the underlying RECS data in terms of
spacial analytic techniques and try to make estimations using the
weather data, which is obviously from NOAA specially registered, and
using the spacial components of the RECS data to try and impute values
over areas which may or may not agree with state boundaries, but if they
couldn\'t agree with state boundaries then, perhaps, you could develop
some population weighted centroid for the state and register that to
both the sparse RECS data, which, apparently, doesn\'t support state
level imputation, and then assign the values at the state level, having
done some sort of kreiging and estimation and means and the confidence
intervals in a spatially registered map.

Beyond that, I would ask my colleagues on the committee, who actually do
know something about sampling, and there are several who know tremendous
amounts, and know something about consumption surveys to try and address
more specifically these questions. I\'m sorry my comments have been
limited to such a technically narrow area.

CHAIRMAN MOUNT: Charles, why don\'t we start with you.

MR. BISCHOFF: I know even less about sampling than John, but I would
like to ask a question of econometrics of Linda. Where did the .25 come
from in the Y^.25^ intervals per product of X^.25^?

MS. OWENS: These are based on \-- the model is based on the end use
estimates that we\'ve been using in the RECS for as long as we\'ve been
doing the survey, and I don\'t know, Bob, if you\'d like to pipe up for
this one, since you are the one that has done it.

MR. BISCHOFF: Perhaps, if I could make a further statement. The fact of
the matter is, the estimate of Y is totally \-- for the estimate of Y,
that .25 is totally irrelevant. You could use 7,352 and you would get
the same answer. When you raise the left-hand side the same power as you
raise the right-hand side, it\'s irrelevant. And so, the .25 is
irrelevant. You might as well get rid of it and just use one, and nobody
will ask any questions.

MR. LATTA: My name is Robert Latta. I work on the Residential Energy
Consumption Survey. In regression, the assumption is that the error term
is normally distributed, and they usually like to have a constant
variance. The transformation was chosen because that did the best job of
giving you a close to normally distributed error term and it did the
best job of making the variance constant over the different households,
as far as \--

MR. BISCHOFF: In other words, heterostaticity.

MR. LATTA: Yes.

CHAIRMAN MOUNT: Are you done, Charles?

MR. BISCHOFF: Yes.

CHAIRMAN MOUNT: David.

MR. BELLHOUSE: A couple of comments. One is in the model-based approach,
it doesn\'t look like you took into account the sampling design in the
error structure and, perhaps, there should be a nested error structure
and the reference would be, I think, Batesse, and Fuller in
[JASA]{.underline}, and I can\'t remember the year, several years ago.
If you want to go to a design-based approach with logistic regression,
Binder in 1983 in [International Statistical Review]{.underline} has a
method to do logistic regression which takes into account the sampling
design, and it\'s a model free approach. That might be appropriate to
getting small area estimates also statewide without \-- sorry, statewide
from the RECS survey.

And, there is another paper by Roberts, Rao and Kumar in 1987 in
[Biometrica]{.underline}, which does, right at the end of the paper,
does some small area estimation from logistic regression.

And, what you want to do, essentially, is get crude estimates statewide
from the RECS survey, which gives the proportion of the observations in
the area of interest, use logistic regression to get a model that then
goes back to predict the proportions from the model, and it\'s all
within the RECS survey. You could use the auxiliary data from the PUMS
data.

So, I found the description, it wasn\'t clear enough to me to decide
whether or not these methods were appropriate, so I wasn\'t exactly
sure, because of the description.

If you did go with the simulation method, that is, you come up with a
probability model where you select or reject certain observations, that
seemed to me to be okay to do that, to get the estimates, but if you
wanted to get a variance estimate of your small area estimate, then
you\'d want to go the regular route of domain estimation, which is,
instead of rejecting the data item you set it to zero, then if you take
the weighted average and get the right estimate, and leaving the values
at zero, whatever variance estimation technique you\'ve used, you get
the proper variance estimate. I think if you would just delete the
observations I\'m not sure you get the right variance estimate.

That\'s all, thanks.

CHAIRMAN MOUNT: Campbell.

MR. WATKINS: Two or three comments. First of all, am I right that I read
it that you use five percent of all households in the state.

MS. OWENS: Yes.

MR. WATKINS: Now, I wonder if there\'s not a cheaper way of getting at
the sample in the sense that by and large you don\'t use natural gas for
appliance use, unless you are using it for space heating. Space heating
is virtually a necessary condition.

MS. OWENS: That\'s not entirely true. There are places that use them for
stoves and that sort of thing.

MR. WATKINS: It\'s not entirely true, but for certainly all the \--
probably all the urban areas, it is.

MS. OWENS: Well, New York City is the classic example of one where fuel
oil is a predominant use for heating, but natural gas is a predominant
use for cooking.

We did make some adjustments for PUMA areas where a very small
percentage of the households, under one percent of the households, used
natural gas for space heating, we simply assumed that they would not be
appropriate for the other two end uses.

MR. WATKINS: I don\'t know what the figure would be, but my guess still
is that the predominance is that if you are using it for appliances you
are going to be using it for space heating. I\'m just wondering if that
doesn\'t offer a way to focus the sample more cheaply on those
households by using utility data.

MS. OWENS: Oh.

MR. WATKINS: The second comment is, you know, to look at the variables
on your loaded equation, the expenditure of natural gas is an
interesting variable, in that if you think of it just in an engineering
sense, if you have the average household, from the expenditure data you
could tell whether they would be using gas for something other than
space heating. And, in that way, the expenditure data, if you reconcile
it with the kind of average use for the average household, you can get
some information there on whether the household is using it for
something other than space heating.

The final comment is, do your models, in effect, yield estimates per
household of appliance usage, and then if that is the case, have you
attempted to reconcile that or look at that in relation to what\'s
called conditional demand analysis, and there have been various studies
done of that in the United States, particularly.

MS. OWENS: No.

MR. WATKINS: I think the main author of that originally was someone
called Michael Parti, P-A-R-T-I. And, some of those studies yielding
estimates of unit consumption per appliance might cast an interesting
light on some of your own response.

MS. OWENS: Do you want to address that?

MR. LATTA: Conditional demand analysis is just another name for
regression. Regression applied to energy consumption data, that\'s all
it is.

MR. WATKINS: Well, not \-- it\'s regression certainly, but the condition
of demand analysis is that they base \-- the consumption is given, and
then you have to divide it up among appliances. So, that\'s the
distinction that is made in the description.

MR. LATTA: Michael Parti, we\'ve talked with him over the years, he is
still in the stage of using just straight regression. I don\'t think you
can do an adequate job of it unless you use non-linear regression.

MR. HANSER: I guess I disagree with that. I mean, I guess the concern
that\'s been raised about conditional demand analysis is that the
estimates aren\'t unbiased, and the reason they are not unbiased is
because, in general, they don\'t take account of the choice of fuel in
the level of consumption equation. And, Green\'s technique for including
the probability of the choice of fuel allows you to use a
straightforward linear regression to get at some of that.

I mean, you know, it\'s a tricky issue. If you wanted to do anything to
improve the estimates, one of the simplest things would be to use
something like, you know, what is it, Tiles technique, where you include
past estimates of consumption by category as a kind of prior for doing
the estimate of the coefficients. I mean, that\'s a straightforward
technique, and I think it\'s Tile, and I can\'t remember what \-- it\'s
in Tile\'s Principle of Econometrics that dates back to \'71 or
something like that, but I do think there is a fundamental bias problem
in the sense that what you\'d really like to do is estimate gas
consumption in some sense independent of, in some sense, across all the
range of potential fuels that somebody would be using for their
appliances. That is to say, in some sense you are creating a kind of
bias sample problem, because you sort of say, well, this person has got
gas and, therefore, I\'m going to look at gas consumption, and that, in
the end, creates a kind of sample selection problem. What you\'d really
like to do is say, well, if I had all the potential choices were fuel,
and somebody is going to make a choice about that, what fuel would they
choose and how much would they consume based on the characteristics of
their income and so on.

And, that\'s a problem that\'s plagued, you know, the surveys for a long
time. I don\'t know of any simple way to get around it.

I\'m surprised you haven\'t \-- another issue that I guess I was sort of
surprised at, is that there\'s been this nice work that was done by
McFadden and \-- I can\'t think of his co-author \-- God, I hate when
Alzheimer\'s sets in and I can\'t remember anything, but it dates back
to about 1988, and it utilizes this combination multinominal logit
analysis, along with a subsidiary regression equation to get at the
estimate of consumption by end use.

In some sense, I mean \-- well, I guess the question that I would raise
is sort of like the following, the RECS survey is in some ways the most
carefully done of all the energy consumption surveys that\'s out there
at the residential level. I mean, there\'s no doubt about that. Linda
has done a wonderful job over the years of developing this into quite an
instrument. And, I guess I don\'t see why David\'s suggestion about
utilizing that as the basis for the information shouldn\'t be the
appropriate way to go.

I mean, the PUMS stuff is, it seems to me, you know, really inadequate
in terms of energy consumption, and I think, you know, using some
technique which uses a base, the RECS data, and then only using PUMS as
a kind of supplement to it is probably the most appropriate way to go.

Using PUMS alone strikes me as being, you know, or largely, it seems to
me, you know, fairly risky, at least the way it\'s been described,
particularly, given the variability of the weather data, just changes in
heating degree days alone can produce dramatic differences in gas
consumption for space heating. And, since that\'s the single largest use
of gas that you are going to find, I mean, by orders of magnitude larger
than it would be for water heating, that alone will produce huge amounts
of bias in terms of the estimates of your consumption.

Even taking California as an example, I mean, you know, California,
where I\'m from, has, I can\'t remember, at last count 112 different
microclimates, you know, 42 of them simply within the San Francisco Bay
area. I mean, you know, to talk about applying, you know, any sort of
smearing of heating degree days across a region like that, I mean, it
just doesn\'t make any sense.

MS. OWENS: Right.

MR. HANSER: So, I would stick with David\'s suggestion, I think, of
using the RECS data, and then working out to PUMS is a much more
sensible one then trying to work from PUMS directly.

CHAIRMAN MOUNT: Anybody else from the committee?

I can\'t pass this opportunity to talk about the topic that\'s just been
raised about the choice of appliances. You know, this is the area that
we really have very limited data for, and clearly, the new construction
is the place where that decision is very important. So, that\'s the
first point.

The second point is, I\'d like to go back to something that Charles
Bischoff raised, that when one presents a relatively complicated model
like the non-linear structure, I just am suspicious, that\'s all there
is to it, you know, I would be suspicious no matter where this came
from, because I would like to have an explanation of why that was
necessary. So, when I see a T ratio of zero, sorry, a standard error of
zero, on page nine, I just worry about what is exactly going on.

And so, I think the message is that if you have a non-linear model,
there ought to be an explanation of why simpler models are not adequate.

So, any other comments from the committee? Would you like to reply?

MR. LATTA: The problem we have with trying to model it strictly on
linear regression is that you have \-- your natural gas bill will
include space heating, water heating and appliances, and that some
things like weather are very much influence the space heating load. On
the other hand, the thing that determines a lot of the water heating
load is just the number of people in the household, and something like
their age. So that, if you try to put these \-- try to just use age to
affect water heating, but not space heating, or things like that, it\'s
harder to do if you don\'t use a non-linear model.

And, since we had a mainframe computer and now we have PCs probably that
can do it, it would just be much more accurately to model it if you
could multiply and divide, or multiply and add at different places,
rather than just having this strict structure you have with straight
linear regression.

CHAIRMAN MOUNT: I did not read this section very carefully, but, I mean,
I think that that is the sort of thing that should be written down, you
know, what the philosophy of the model is, the space heating, the one
that Chuck was talking about is the one I was talking about.

MR. BISCHOFF: NGMSZ4, on pages 782 with an \-- extended area of zero to
nine decimal places.

MS. OWENS: I might explain this, is there not a reference category for
these non-linear models, where the standard error for that particular
category is zero? That\'s what they are questioning right now. Do you
know what I\'m talking about, Bob? All right. Well, yes, I can talk
about that, or ask him later.

CHAIRMAN MOUNT: Do you want to make some general replies? You don\'t
have to. This is your chance, if you want to.

MS. OWENS: No. I would just like to hear more from David, not
necessarily now, but about the specific methods that you would
recommend, because we don\'t even have data in some of the states
directly from the RECS.

MR. BELLHOUSE: Right, so it seems to me that what you do is you do it on
the states that you have and then predict it on the states you don\'t
have.

MS. COX: You know, when you do have information on the state, often what
you have is one PSU, which means one location.

MR. BELLHOUSE: Which is also why you use the whole regression for all
50, or, well, however many states you have, to try and improve that one
poor estimate.

CHAIRMAN MOUNT: So, I think that we have no more comments. Anybody from
the back want to comment, since we\'ve got a little time? Anybody else?

Well, thank you very much.

MS. OWENS: Thank you.

CHAIRMAN MOUNT: We can move on to the next subject, the last one, and
this is results and overall plans for customer surveys, and the
presenters, who is going to do the work, Nancy, you are going to do the
work, so the \--

MS. KIRKENDALL: I\'ll start out.

CHAIRMAN MOUNT: Okay. The presenters are Nancy Kirkendall, Office of
Statistical Standards, and Mike Laurence, Office of Energy Markets and
End Use.

MS. KIRKENDALL: Well, there are a lot of new initiatives going on in EIA
right now related to the issue of trying to measure customer
satisfaction, getting feedback from them on what they like and don\'t
like about our products and services, and so, this is really about where
we\'ve started. We\'ve got a long way to go and would appreciate any
comments you have to help us in this area.

I\'m speaking particularly for the Customer Survey Committee that\'s
headed up by Linda Carlson. There are a bunch of other related
committees. There\'s one on developing a customer database. There are
several electronic committees, and there\'s a Performance Measure
Committee that I\'m chairing, so there\'s a whole lot of sort of
interrelated activity in this area.

So, one question is, well, why should we survey customers? Well, we\'ve
got a couple reasons that were given to us in \'93. There\'s an
Executive Order that says that we have to measure customer satisfaction
and determine the expectations of our customers.

In addition, in \'93, the Government Performance and Results Act
dictated that we should all move towards performance-based budgeting,
and part of that is getting a measure of customer satisfaction and using
that for resource decisions. And, we are a pilot project under that Act,
and that means we have to do it sooner.

So, this year our strategic planners have put in a customer requirement
item as a strategic issue, and the first objective is to establish
customer information and feedback program, and the second is to develop
and implement marketing. So, the second one, of course, is finding out
potential customers and getting a better idea who our customers are,
what they like about us, and building our customer base.

We thought that we can identify our primary customers by the way they
get energy information, and I guess the reason for looking at it this
was thinking about how you might maintain lists of customers for use in
sampling.

EIA has a National Energy Information Center. It\'s like a library. We
have all of our documents there. Some people can come in and request
documents, those would be the in-person visits. We don\'t get too much
traffic, people just wandering in, other than EIA staff and other DOE,
because it\'s at the Forrestal Building and you can\'t get through
security.

The NEIC also operates a Telephone Information Center. We have four \--
well, there are four desks. There are usually three dedicated people
manning the telephone lines, and people are allowed to call in, ask for
any energy information that they like, and their requests are \-- you
know, questions are either answered or they are sent publications, that
sort of thing. It was initially set up to take the burden off of staff.
Our survey in January indicates that we still have about 50 percent of
the calls coming in to EIA come in to staff, not in NEIC. So, NEIC is
taking about half the calls, and the rest of the calls come in to staff.

Publications are the one place where we actually have lists right now.
We maintain lists of subscribers to our publications. It\'s because we
have to send the publications. We also can get publication lists from
the general printing office. They sell our publications. They maintain
subscription lists only for people who subscribe to a less than annual
\-- a more frequent than annual publication. So, we have some
information on who gets the information directly. We do not have
information on who received our publications in response to a telephone
call, either to NEIC staff or to other staff.

Another area where we don\'t have much information and we probably
won\'t be able to get very good information is, there\'s a body of
customers who take our information and repackage it and sell it to
others, and those people have indicated a reluctance to let us know who
their customers are, so they are secondary customers we\'ll probably
never find out much about.

In the electronic area, we believe that we\'ll be able to maintain and
record identifiers. People are asked to type in a telephone number and
E-mail address, and at least eventually we may be able to use that
information for frames.

Right now, we have an ad hoc committee that\'s trying to put together a
customer database. It\'s going to start out looking at our publication
lists to make sure that those are included in the database, and we hope
that ultimately we\'ll end up with our customers, our telephone
customers included in there as well. When we have that, then we can move
to, perhaps, more general ways of interviewing our customers.

So, with this effort we started out with our telephone customers,
primarily, because we hadn\'t ever done that before, and the procedure
was that we would have, we started out in NEIC because they are all
conveniently located in one spot, they would, after they helped a
customer they would ask the customer if they would mind participating in
a customer survey. If they agreed, then they\'d be handed off to
somebody else, who would then conduct the interview. So, you\'ll be
seeing results. We did that in NEIC on little one-day surveys. The day
of the survey was predetermined. There\'s no randomization in this
process.

We did a survey EIA-wide in January for three days. In general, the
customers seemed to enjoy being surveyed, those who chose to. We had
about, I think it\'s about a 75 percent response rate. When they made an
excuse about why they didn\'t want to be surveyed they generally say,
well, I\'m too busy to do that right now.

The staff conducting the interviews thought that it was very
interesting. People engaged in lively discussions during the interview,
and they thought that it was interesting. And then, the overall results
that Mike is going to show you, that was something to celebrate to.

So, so far this has been a positive experience, and we expect to
continue it at least for a while.

And, why don\'t I let Mike come up and give you more information about
what we learned in January.

MR. LAURENCE: As Nancy indicated, the survey was conducted over a
three-day period in January, mid-week, Tuesday, Wednesday, Thursday. The
actual period for sampling and interviewing was actually rather
arbitrary. Indeed, the committee was rather concerned that some kind of
an event might occur, either the cold snap that we had experienced in
the previous year in the northeast, or a publication of some EIA report
that was of great interest and produced more callers than normally might
occur.

It was very, very difficult to actually ascertain that we could select a
three-day period where nothing was happening. So, in terms of whether or
not anything actually happened on these three days, from the results of
who actually called, in looking at the demographics, didn\'t appear to
produce any bias.

Individuals who called were informed, either calling EIA\'s program
offices or NEIC, were informed that we were conducting a customer
satisfaction survey and asked if they would care to participate. The
very large majority who, indeed, agreed to participate were then called
back by an interviewer from a different program office or an office
other than NEIC and a ten or 15 minute interview conducted.

The next figure provides some demographics of who we talked to and who
we interviewed. Overall, there were 286 callers, with the slight
majority being callers to NEIC. Most callers were repeat callers. Of the
28 percent first-time callers, these folks were more likely to call NEIC
than a program office. In contrast, the more frequent callers were more
likely to call program offices than NEIC. So, it seems that once people
get introduced to EIA and call back, they go to the program offices for
their information. And finally, the spectrum of callers was very broad,
ranging from 25 percent from those in the research and consulting
industries, down to eight percent calling from internal DOE and the DOE
laboratories.

Interestingly enough, callers from energy and other industries, and
callers from DOE and the laboratories, were more likely to call NEIC
than a program office. Those from research and consulting and from
government, state and local government, were more likely to call a
program office.

The next figure presents the broad results of the survey. The core and
the essence of the survey was to ask two sets of questions, the first
dealing with customer service, the second dealing with information
quality. First, the telephone callers were asked their overall
satisfaction level on a five point scale with customer service, and 98
percent of the callers indicated that they were either satisfied or very
satisfied with their customer service. This general overall question was
followed by a series of five questions asking their satisfaction level
with various specific aspects of customer service, and as you can see,
well in excess of 90 percent on all these characteristics produced
customers who were either satisfied or very satisfied.

Generally, first-time callers were ever, ever more slightly to be
satisfied than repeat callers.

In terms of EIA\'s information quality, overall 89 percent indicated
they were very satisfied or satisfied with the quality of the
information, somewhat lower than for customer service, as you can see.

In addition, following up with five aspects of information quality
produced lower overall satisfaction ratings, in terms of people being
satisfied or very satisfied in contrast to customer service.

As you can see, the issue of timeliness was the area of greatest
concern, with only 73 percent indicating that they were either satisfied
or very satisfied.

There were no systematic differences of the assessments of information
quality between the first-time callers or frequent callers.

Following up, we expected that timeliness was going to be an issue, so
we had built in a number of follow-up questions for those who indicated
that timeliness was an issue, and those were less than fully satisfied.
We asked these people, would it matter if the information you were
provided were less accurate but more timely. And, as you can see,
overall for 56 percent, a majority, this was an issue of concern. Among
those callers to the program offices, fully 70 percent were unwilling to
sacrifice accuracy for timeliness. In contrast, the callers to NEIC
didn\'t find this a particular issue.

And then, we followed up, continuing on the issue of timeliness, whether
or not we produced \-- the question was, if we produced the data on a
more timely basis, but more aggregated, less detail, would that be
satisfactory, and overall 60 percent said that less detailed data would
be a problem, with both callers to the program offices and NEIC
indicating that, indeed, it would still be a problem.

Then finally, as we look ahead to the electronic data transmission age,
we asked our callers whether or not in general they would still want
hard paper printed reports and data if it were available electronically.
As you can see, still an extremely large majority, 69 percent, overall
still want their paper reports, despite the fact that they can receive
the data electronically.

This is going to be an interesting issue to follow up on in the future
as we reconduct this survey, because we are really very early on in the
electronic information transmission age, and we would expect that as
more people become used to it, access to Internet, both by users and EIA
placing things on Internet, that we may see a change in the future.

At this point, I will return it to Nancy Kirkendall, who will continue
the discussion.

MS. KIRKENDALL: That was our telephone survey activities.

We have done some in the surveying of publications. We have a
professional audit review team, which is part of the General Accounting
Office, and they have been doing surveys for a couple of years of
selected publications, and these are customer satisfaction sorts of
surveys, and they are going to continue to do that, at least this coming
year. And, this year they are going to do something a little bit
different as well, they are going to survey the users of some of our
less-utilized publications to see if we can find out whether we can
change our product slate, perhaps, get rid of some publications, provide
the information in a different way. So, hopefully, we\'ll get some good
information from them.

We have had several product-specific surveys. There\'s a survey of the
Monthly Energy Review, that particular survey targets the feature
articles trying to find out something more about what users like and
dislike about the feature articles. We had another survey of Petroleum
Supply Monthly and Petroleum Marketing Monthly, which was trying to find
out what readers liked and disliked about a specific feature article
series, \"The Outlooks,\" which is only published three times a year.

While we had some good information from that survey, we had a terrible
time with response rates. We had only a 33 percent response rate, and
that was with some good hard efforts at trying to follow up. We thought
that part of the problem there was that the sample was drawn from the
subscription list to those two publications and, perhaps, many of the
readers didn\'t bother with those outlook articles and really weren\'t
interested in participating in the survey.

We are also putting reader response cards in all of the pubs, and, of
course, that just gives you whoever has a gripe can send it in that way.
You don\'t get very good responses, nothing generalizable at least.

For electronic users, we still have not actually done a whole lot. We
are still in the planning phase. We have a new home page, and there\'s
been some discussion about whether to put a survey out on the home page
for people to provide us information. We can certainly, at least
ultimately, get lists of our phone numbers and/or E-mail addresses of
people who use our data. There\'s been some work on \-- Linda has been
working on talking to depository librarians to find out at least what
kinds of information requirements they have, what media would they like
to receive, would they use most, and she\'s been working to, perhaps, do
a survey of that group.

So, finally, this is just a subset of questions, and I\'d invite
comments on anything else as well. So far, our telephone surveys have
not had a random element, and do you have any suggestions for a way to
introduce randomization, to get more representative results? Do you have
suggestions for surveying electronic customers? Do you think using the
home page as a survey vehicle is a reasonable thing to do, or do you
have alternative suggestions? Do you have advice on identifying
potential customers, where we can \-- should we use established mailing
lists? And then finally, any advice on the designers, any advice for the
designers of our new customer database system. This system is still very
much in the design stage, so we can actually make use of any ideas that
people have.

Thank you.

CHAIRMAN MOUNT: Thank you, and the first discussant is Jim O\'Brien.

MR. O\'BRIEN: Thank you, Tim.

I guess it\'s clear to everyone that this is a very important topic, and
it\'s an important topic, not only in its own right, but it\'s important
because as the paper indicates on the first page it is the focus of an
Executive Order on activities that should be carried out and which, in
fact, might form the basis for allocation of budgets in the future.

This is what might be termed a low-tech outline of my proposed remarks
this morning.

MR. HANSER: It\'s a remarkable word processor, it just happens to be
there.

MR. O\'BRIEN: It emulates cursive writing, particularly, for those who
do not use the Palmer Method.

On the first item, the background of federal planning and budget
innovations, it occurred to me that there might be many people here who
either were encountering this for the first time or who did not realize
the background.

Going back, say, perhaps, 20 years, almost each new administration that
has come in has felt the need to flesh out the rhetoric of its campaign
promises, cut out the fat, lean, mean fighting machine sorts of things,
with some specific recommendations for the way that the federal
bureaucracy and federal managers can be more effective.

Some of these were ill-fated. One that is of happy memory, because it\'s
a memory, was the zero-based requirement that some of you may recall,
which was known among some skeptics as the \"born yesterday\"
requirement, it required that you begin everything with a complete clean
slate, tabula rasa, and all memory was regarded as irrelevant, and you
treated each activity that you were going to carry out as if it were
being done for the first time.

Well, fortunately, people recognized that that was really quite
impractical. The bureaucracy dug its heels in, in many cases, and it
passed.

However, it left in its wake a residue of skepticism at the very least,
and that, in some cases, as the geologists say, hardened into cynicism.
And so, there has been, I would say, a fair measure of cynicism about
these efforts in the past.

I was very pleasantly surprised to find that when I looked into it this
time that many of the people who were working on these surveys don\'t
have the cynicism or the skepticism of those in the past, they really
are convinced of the importance of trying to get in touch with
customers. They almost seem as if they\'ve been reading W. Edwards
Deming, whom some of you may be familiar with, who put great emphasis on
getting in touch with what your customers want, and who, in fact,
attributed much of the Japanese success, not only to their hard work and
good engineering, but also to their willingness to take chances and
their willingness to find out what customers want and need.

I think that that will give you a very rough perspective on what the
background is, and I hope that this does represent a genuine change.

As you can see, I have become, in a small way, a bit of a convert
myself. Some of my skepticism has been at least diluted as a result of
talks that I\'ve had and from reading this paper.

Comments on the paper and the attachments. The paper, in my judgment, is
quite well written and flows logically from the requirements that are
stipulated by the Executive Order, to the steps that EIA has taken to
identify its classes of customers, to conduct surveys among them, and to
publish results, and those results are summarized in the paper and
provided in more detail in the attachments. I\'m really very favorably
impressed by the response of EIA, particularly, after a visit to the
Census Bureau, which is carrying out some of these same things, and
which in typical census fashion is \-- well, I won\'t say elephantine,
it is more deliberate in its movements.

The response, not only the speed of the response, but the way that EIA
was able to galvanize people to swoop down on all the customers who
called and get the answers back very quickly, and have the results
available in a form for review, was, I found, quite impressive.

I do, however, have a couple of subjective reservations about that. One
is that the questions, in my judgment, seem to expect a great deal from
the respondents, and I would ask you to try to follow me with your copy
of the paper as it was handed out, and, particularly, and, specifically,
look at attachment one, which is a copy of the survey questionnaire,
and, for example, I just picked two examples here, page six, question
15, where they ask, which products should be timely, which tables, how
much sooner do you need the information. I thought that to get
meaningful replies to these would require a very high level of
commitment and of knowledge on the part of all of the inquirers, and
I\'ve seen Master\'s theses at the Wharton School that didn\'t require
as much work as this for the respondents.

The second is, another example is on page eight, question 20, which past
analysis products were most useful to you, which were least, can you
briefly explain why, one and a half lines given for that, and what
topics would you like to see in future analysis products. They appeared
to me to have a very high level of expectation. I hope that I\'m wrong,
and I hope that people have said you are not in touch, you are not au
currant anymore, we know these things, and we can put the down, and will
put them down readily.

The second reservation that I have is that, frankly, the results, in
some ways, seem to me hard to believe. There have been some expressions
of this same viewpoint expressed on other papers this morning, but I\'m
particularly interested in the attachment two, which is the one that was
shown in fancy technicolor, and it\'s the one on the initial findings,
and figures three, four and five, just to pick out three, show the \--
before looking at that, if you look at figure three, the overall
customer service rating, you see that the two highest categories,
satisfied and very satisfied, were the ratings assigned by 93 percent of
the respondents. For the program offices, it was 93 percent, that\'s 33
plus 60. For the NEIC, it was 94 percent, and the overall for EIA was 94
percent.

The customer service ratings is the one shown here, and as you can see
they are very high. In fact, they got even better when my back was
turned. Overall, rating of 98 percent for customer service and for 89
percent for product quality strike me as truly extraordinary, and I
realize that this is in some ways like the farmer who sees the giraffe
and says, I don\'t care what it looks like, it just can\'t be, let me
suggest an alternative perspective.

This is taken from an article in the [Washington Post]{.underline}
earlier this week, the result of a survey by Hart and Teeter on the
general view of the public about the federal government, and its
efficiency and its inefficiency. I would call your attention especially
to this paragraph. Across the political spectrum, the federal government
is perceived as in need of better management. More than six out of ten
surveyed thought the federal government wastes money because of poor
managers. And, as you come down here you\'ll see the greatest weakness
is that it wastes money because it is not well managed. Sixty-one
percent of the people surveyed thought that. It spends too much on the
wrong things, 56 percent, solutions to problems are ineffective, 30
percent, takes too long to solve problems, 29 percent.

And, it seemed to me hard to reconcile that the situation at EIA \--
that EIA is such a polished jewel in such a sloppy marsh that it
achieves extraordinary levels of approval and acceptance by its
customers.

This is not quite as bad as the farmer and the giraffe, because this led
me to make some inquiries, and one of the inquiries was of a young woman
at the Bureau of the Census. For some of you, you may not be aware,
that\'s the mother load. I spent 30 years there. And, this woman is a
psychologist who has taken courses which, apparently, are offered by the
University of Maryland, on surveys of customer satisfaction. And, it\'s
apparently part of the received wisdom known to everybody but me, at
least as of the beginning of this week, was that customer satisfaction
surveys, in general, are strongly biased in favor of the inquirer. This
is particularly so if the identity \-- it is claimed that this is
particularly so if the identity of the respondent is known.

It\'s not clear whether it is higher in telephone cases than in written
cases, but some measure of this can be gained from looking at the
results that are published in the paper for the non-telephone responses.
This is attachment three, Table H5, where the percent responding
positively were in the neighborhood of 80 to 90 percent for the motor
gasoline, in the neighborhood of 73 to 88 percent for distillate fuel,
and slightly higher for propane. Whether the mode, or as the
psychologists call it, the modality, of the inquiry has a crucial effect
on this is not really well documented, but it does appear that at least
to people who respect Teeter and Hart, a fairly well-known polling firm,
that the disparity between their results and the EIA results suggests
that EIA really is quite special.

All right. The next item is the experience of the Census Bureau. The
Census Bureau is also involved in carrying out customer satisfaction
surveys, and, in fact, the Census Bureau has been designated as the
agency of the Department of Commerce which is going to carry out all of
the customer satisfaction surveys for the Department of Commerce. Their
assignment was \-- you know, the questions that they asked are in many
cases almost identical to those that are asked by EIA, but they had to
design questionnaires for 20 different operating units, so the way that
they did it was to have general purpose questions that were asked, and
they had three categories that were specifically tailored to the
products, services, regulatory constraints, et cetera, of the individual
Department of Commerce agencies.

I hold in my hand two samples of the questionnaire, and I will flash up
the part that is different for each one. This is put out by the National
Technical Information Service, and as you can see it is very heavily
slanted toward their technical reports and their specialized services.
These were produced by a computer program. Whenever there\'s a reference
to the agency in this particular one, it says the National Technical
Information Services products and services.

There is another \-- oh, you will note that there is a Section E down
here, Grants and Regulatory Products and Services. In the Commerce
Department, there are certain agencies which don\'t turn out products
that you get just with a phone call. Some obvious examples are Patents
and Trademarks, you don\'t just get a patent by calling up and saying
give me a patent, or please approve this. So, with those, they had to
put more emphasis on the process and on the satisfaction of applicants
with the fairness of the process.

This is one of the agencies which does have regulatory products and
services grants and cooperative agreements. It will give some indication
of a slightly different approach to the problem of one group which is
acting in the name of several others.

The procedure that they went through was roughly the same as what EIA
went through. They had focus groups. They created what they called the
typology of products and services, that\'s what these are, and the
questions were aimed at questions on process in some cases.

The response rates, surprisingly to me, were even lower than the
response rates that were quoted earlier by Nancy. Nancy quoted 33
percent as, I take it, the low point of the response rates. Their
response rates varied from 30 percent to 60 percent, and they put a lot
more emphasis on confidentiality. They made it very difficult to trace
the identity of the respondent when the material was returned.

They targeted 500 completed interviews for each agency, and they sent
out about 1,200 to get the 500, for a total of something like 22,000
that were sent out, and I have the name and telephone number of the
people over there who can provide further detail if it is felt to be
useful.

Next is questions for the committee. These are the questions that appear
on page five of the paper. They were slightly reworded earlier, but I
think that the sense is roughly the same, so I\'ll read from the paper
that you have in front of you. Are there any suggestions for
randomization, that is, selecting the days for the survey at random, so
that results of our surveys of telephone customers are representative.
As I understand the reading in the past, there was a complete census of
all those who called during a period of three days, so it was not
sampled.

And, I would just point out that, I guess the definition, at least what
used to be the definition of a probability sample, is that all members
of the universe of interest have a non-zero probability of selection,
and second, that the probability can be calculated, so that the
weighting can be inversely proportional to the probability of selection.

So, in this case, you could do it two or three days a week. If you
wanted to make it a controlled experiment you could get fancy, but you
could see if the people who called in on Monday are any different from
the people who call in on Wednesday or Friday, and the only thing that I
believe that would need to be randomized would be when you chose Monday,
Wednesday or Friday, in fact, I\'m not even sure that that would be
necessary, pick three random numbers between one and five and use those
and see what differences result. If there is no difference, then there
seems to be no great advantage in randomizing if it costs more. Usually,
it doesn\'t cost much more.

Second question, are there any suggestions for surveying customers of
our electronic products. Well, I spoke to the young guru who was in
charge of electronic products and home pages at the Census Bureau. I
guess his first suggestion was to get the telephone numbers of those who
inquired and call them up. However, after further discussion, he thought
that it would be worthwhile to try the idea suggested here, namely, to
put a copy of the survey on the home page for the EIA, and when people
tried to access EIA data, at the bottom of the request say, please, look
up our home page, and if you would be wiling to provide the information
requested there it would be of great help to us.

It\'s not surprising that he says people who are Internet users are very
responsive, so you ought to get better response rates than you\'ve
gotten in other ways.

The third question is advice on ways to identify potential customers. It
seems to me that there are people, including some people in this room,
who have made almost a specialty of that. It seems to me that this is
really what marketers, the bread and butter of marketing in many cases,
finding potential customers and converting them into actual customers
and users. Some thoughts that occurred to me are that you could get
lists of associations, which are groups of people in the fields of
interest and the fields that would be served by the products of EIA, you
could convene a meeting of professional marketers or representatives of
the American Marketing Association, you could advertise in appropriate
journals, you could organize booths, if you don\'t already do so, at
meetings and conventions of potential customers, or you could mail out
tailored lists of EIA publications that are likely to be of interest to
respondents in the energy surveys, so that, presumably, the people who
provide the information that fuels your data might be interested in
seeing the finished product, if they don\'t already.

Next I have some questions, next and final part, I have some questions
for EIA. Some of these are detailed and specific, and others have,
perhaps, broader significance. One is that on the first page of the
paper, the paragraph one at the top, it says the standard of services
provided to the public shall be customer service equal to the best in
the business. Well, I found that a fascinating idea, and so I guess my
question is, did you find the best in the business, and how did you find
them, and how did you determine that they were the best in the business,
and have you learned anything that you could emulate from finding them?

The second detailed question is, it seems to me that the \-- I found it
hard to distinguish between the emphasis on the satisfaction level of
present customers and efforts to try to get potential customers, or to
reach out to potential customers, and it seemed to me that the first of
those is what these surveys have done, and which the results have been
reported. The second is really a marketing effort, and it\'s certainly
worthwhile. It\'s important, I think, to reach people who might
profitably use the products, but it does seem to me that that\'s a
different question from the question of the satisfaction level of
present customers.

Now, the third is an area, I guess, that is more general. Originally, I
began to jot down the things that I thought were limitations and
problems with surveys of this sort, and when I got to the bottom I said,
well, it ought to be possible to try to say something about where these
would be useful and where they are not likely to be useful. And, I
concluded, I won\'t say I concluded, but it appeared to me that they
would be highly useful for improving products and services of any
individual organization, if they were taken seriously and carried out
well.

However, it seemed to me that they may have serious flaws if they are
used uncritically to compare the performance of two organizations, and
especially if they are used to make allocations. And, in support of
that, I cite this short list of problems. I\'ve already talked about the
biases of the customer satisfaction survey, that is, the received wisdom
that they tend to be overly favorable, received in certain circles.
Second is the possible biases from response sequences. Evidently, the
psychologists have a sort of rule that if people are given a variety of
responses to choose from, and it\'s a written response, they tend to
favor the ones presented first. If it\'s at telephone interview, they
tend to favor the ones presented last. I thought that that might account
for the very high marks here, but it turned out, at least if people
followed the instructions, they presented on the telephone first the
most favorable reviews and the least favorable last. So, I would regard
that as a possible explanation, but not an important one.

Another question that I suggest that EIA might consider is whether the
confidentiality issue and whether the credibility that would come from
an independent contractor would be, perhaps, important, you know,
specifically, would it be worthwhile trying to have one organization
carry out these services for everyone in the Department of Energy.

On the questions relating to accuracy, again, it seemed to me that I
would find it hard to know how to judge, if I were a respondent, I would
find it hard to know how to judge whether the data I got was accurate.
If I knew the answer, I wouldn\'t be trying to get it from EIA, and, you
know, in other fields it would be a major research project to find out
whether the product is accurate.

The weighting by customer importance seems to me to be a very serious
flaw in this. All respondents are created equal, and all respondents
receive equal weight, whether it\'s General Electric, or IBM, or the
corner library, or whoever else, and it seems to me that serious thought
might well be given to something roughly akin to the probability
proportioned to size, that is a weighting proportionate to importance,
if these are really to reflect the contribution to the American economy
of the statistics that are put out by EIA.

I mentioned the low response rates. I had some suggestions for quality
control that have worked out in telephone surveys before. Monitoring
being one, that is, monitoring of the interview in a way that neither
end of the telephone line knows that it\'s being monitored, and the
other is the traditional reinterviews of a sample of the sample.

It is now my pleasure to introduce the second discussant, Dan Relles. I
apologize for taking up a chunk of his time.

MR. RELLES: Thank you.

I guess everybody starts out as a discussant saying that this is a very
important topic, and so I\'ll say it, but I really believe it. I think
this is incredibly important. We had an indication of it by listening to
Jay Hakes\' remarks yesterday, when he talked about the phone calls and
how satisfied customers were, we weren\'t timely enough, but there\'s a
negative tone to this thing which is, we are doing this because there\'s
some laws that say we have to do it. I\'d like to see a really positive
tone injected into this thing, which is basically, this is a vehicle for
really helping you do your jobs a lot better, for helping the EIA
improve.

Jay, in addition to saying how satisfied our customers were yesterday,
also said things like, we have some major decisions to make. Should we
continue our hard copy? Should we be developing products for the home
page? How about Internet versus CD ROM? And, as a pilot project, this
needs to be an example for what a virtual federal statistical agency
ought to look like.

I think the greatest idea in here is the idea of developing a frame, and
I think that\'s really important to do and to finish, and I have a few
suggestions about that. I see four components to the frame. One is the
home page stuff. Another is the mailing lists. Another is the NEIC
channel, sort of a standard number to call into, and the other is
everybody answering questions because their names are published in a
directory.

Regarding the home page, that\'s \-- the other thing that needs to be
recognized is, you are trying to plan three or four years hence, because
this is supposedly going to be a process that lasts for a long time.
With the explosive growth of the Internet and MOSAIC and things like
that, you can only expect that community of users to get more and more
important. And, you can already get information off accessing the home
page that I think is going to answer some of your burning questions,
like timeliness, for example. It\'s easy to get off the home page,
everybody who has access to your home page, and what they\'ve pulled
down from it. So, if there\'s a particular survey that comes out on the
first of the month, and I see 30 people clambering for it on the 1st,
and 20 on the 2nd, and it drips down to zero, well, timeliness is really
important there. On the other hand, if requests were kind of strewn
throughout the month, at least I know I can invest my resources on this
other survey, data source, as opposed to this one. So, whereas, the
questions in the phone interview kind of get at how do you feel about
things, you can actually measure behavior, and I believe having behavior
there in addition to touchy/feely questions would be very valuable. But,
for starters, there\'s information on the home page you can exploit, I
think, already.

The other ways of touching customers, the mailing lists are great, the
NEIC, I think, is a resource that ought to be made increasingly
important in a couple of ways. First, it touches people so it can at
least ask them their name and their electronic mail address, and, again,
the electronic mail address, in three or four years you might find your
customer base is so small without electronic mail addresses, two percent
or whatever, that it\'s just not worth the dwindling resources to try to
deal with anybody outside of the electronic mail medium.

The other thing you can do is stop publishing these damn phone numbers.
This is only encouraging \-- now, I know why you are doing it, you want
to seem like regular guys, you want to seem like you are really in touch
with your user community, but I would not want to be Robert Mannick, who
is on page one, nor would I want to be the people who I find on pages
six and seven. Not only are they pulled away from their work, but you
are losing valuable information about your customers. You want to know
who is calling. You want to know where they are getting routed, and as
long as you are nice guys, you are going to be unable to collect that.

At Rand, we did that about five years ago and the sociology was pretty
interesting. I mean, you mean I can\'t go down the hallway and ask this
guy I know to help me solve this computer problem, and the answer is, we
want you to go through this central channel. So, I say, all right, fine,
I\'ll do that, but I\'d really sneak down the hall and ask this guy
anyway, and he\'d tell me, I want you to go through the central channel.
And, this became the base of a customer database that\'s really been
used quite successfully by our computer department. It\'s enabled them
to \-- since they touch all their users, they can see what kinds of
products are being queried, they can do periodic easy electronic mail
surveys or follow-ups of people, and it\'s become really a vital
backbone for the way this particular organization has functioned, and
it\'s really enabled them also to cut their costs dramatically in the
long run, because once you have that control you can vary parameters of
it. For example, you know, if you decide that the calls are coming in
too slow, you can put a voice mail answering machine on the other end,
or if you decide that what you really need is a contact and an initial
question in order to know how to triage it, you can put a voice mail
message that basically gives your electronic mail address and says to do
it this way. It gives you some control over how you are going to
interface with the users. So, I guess my advice is complete this frame
with all haste, make some minor adjustments to the way you are doing
things, namely, say no when they call you, and type in names and
addresses and, perhaps, the question, and the electronic mail address to
anybody coming in to the National Energy Information System, and sit
back and watch good things happen.

Two parameters I would say really need to be part of this. One is
closure. When a contact comes in, get \-- ultimately, it\'s got to be
fed back into the database what the disposition of that thing was and
when it happened. That\'s a little bit of a chore, but for a management
tool it\'s critical, and, of course, the other backbone I think is
electronic mail.

So, you know, and with data like this I believe you can, not only sort
of do a much more effective job surveying individuals, because you\'ll
know something about the usage habits of the people you are asking and
you can target them questions that are going to supplement the usage
information, but you\'ll also be able to track how the population is
changing over time and the degree to which people are becoming really
much more electronic, which is going to be happening at an incredible
pace.

So, I guess the bottom line is, put a process in place that has a
certain degree of formalism that will carry you into the future, and
that you can\'t improve a process without sort of having a process, and
right now, as long as you are allowing the chaotic interaction with
anybody published in these phone directories, you won\'t have that
process, and it seems like a simple matter to just in the future put in,
contact NEIC for more details and start building up a small triaging
organization.

I believe that any virtual federal statistical agency, which Jay Hakes
has in mind, is going to have this kind of triaging, and what better
place to develop a pilot and a prototype than here at EIA.

CHAIRMAN MOUNT: Any comments from the committee? Brenda.

MS. COX: On the first question, how to sample, I would suggest that you
think of the sampling as sampling of time periods, and you can divide
your time periods into, like, five minute increments, or ten minute
increments, and within those time periods then you could either take a
census or a systematic sample. That would be as opposed to just taking a
solid block at three days. It could be done on a continuous basis with
randomly selected time periods, that\'s how it is usually done.

The second thing, though, is you have to decide what is going to be your
target population. Is it going to be the request, or is it going to be
persons, because if it\'s persons you\'ll have a multiplicity that you
need to deal with. In other words, how often do you call in? And, how
often really will be tricky, if you are trying to get an estimate over
the full year, and maybe you\'ve got a process in place that samples
over the full year, what you really need to know to do persons is, how
often are you going to call in over the next year.

CHAIRMAN MOUNT: Phil.

MR. HANSER: The first thing I\'d like to say is, I\'m a real believer in
speaking to customers. I find most \-- a lot of companies seem to be
afraid to talk to customers, for one reason or another, and so, I\'m
quite encouraged by the idea that EIA would actually want to go and
think about talking to people.

I have a concern about customers, because it seems to me you have sort
of two classes of customers. One class of customer, I would say, is the
sort of free rider, and the other class of customer, it seems to me, is
the individuals who in the end pay your bills. And, I\'m a little bit
concerned that in some sense the survey, and all of this kind of
information, ends up spending all of its time on the customers who are
in some sense free riders and not the customers who end up paying the
bills, and let me see if I can explain what I mean.

It seems to me that a large portion of what EIA does is respond to
requests from the administration, whatever that means, in terms of
future forecasts and so on and so forth. And, in that group, it seems to
me, is, in fact, the people who in the end, by virtue of the nature of
the political process, pony up the money for EIA. And, in some sense,
all of the stuff that you are doing so far that I see is really oriented
towards the folks like myself who are, in some sense, the paying public
but, frankly, were free riders in some sense also. That is to say, our
contribution to EIA\'s, you know, budget probably can be computed by
looking at my Form 1040 that I just submitted, but in the end I\'m not
really a decision maker or influencer, which is going to determine what
your budget is.

And so, I\'m a little concerned that the orientation of all of your
methodology is towards this second group, and my fear is that you\'ll be
in a position where you, so to speak, you sort of win the battle but
lose the war. And so, I don\'t know quite how to deal with that one, but
I think that\'s a very central issue, in terms of this idea of doing
marketing and trying to get at it. You know, it\'s like this problem
about surveying households, and trying to figure out who it is who
actually manages to have control over the pocketbook, right? I mean,
that\'s the person you really want to survey, because they are going to
make the decision about where the money goes, and in the end they are
going to make the buying decision. And, I think you\'ve got the same
problem with this kind of procedure.

I think \-- by the way, I agree with Jim about customer satisfaction
surveys. You know, we used to call them in utilities \"happiness
surveys,\" you know, if you didn\'t have a rate increase any time before
you issued the survey, the survey always came back as, oh, we are
perfectly satisfied with your service. You know, the day after the rate
increase came through, the satisfaction somehow dropped dramatically,
for reasons that I won\'t go into.

I think the issue that most of the marketing people that I know of get
concerned about is not just satisfaction, but the second one, which is
product development. That is to say, there are really sort of four
categories of product \-- you know, information that sort of is a
problem around developing products. The first one is sort of, you know
what they want, and they know what they want, so that\'s the best of all
possible worlds, but there are lots of intermediate stages that occur. I
mean, there are situations in which you know what the customer wants,
but he doesn\'t know what they want, and the net result is that you have
a chance for creating new products. You know, the Sony Corporation
coming forward with Discman is the classic example. The third situation
is, the customer knows what they want, but you don\'t, in which case
doing a customer satisfaction survey doesn\'t really get at that, and
that\'s where the talking to the customer, and just sitting down and
sort of saying, you know, what\'s your problem and how can I solve it
for you, turns out to be sort of crucial. The last one is neither you
nor your customer know what you want, and that happens a lot, don\'t
laugh, that happens quite a lot. And, the question is, who gets more
surprised, you or the customer, by, in the end, what need gets met by a
product.

The last issue, I guess, is, there\'s a lot of talk about, you know, QC,
quality control relative to this survey. You might contact your
customers to find out if they have quality control requirements for
their suppliers or vendors. Commercial companies do that, and then try
to find out what are the requirements of the quality control programs
that they supply that is required of suppliers, and they try to learn
how to participate. So, the EIA might try to find out, of the commercial
firms that they deal with, which have QC programs and what are the
requirements to become a supplier based on those QC requirements.

The other question, of course, that arises is, you are surveying only
the folks that you know use your data, but in some sense you\'d like to
survey all the people who don\'t use your data, and like to know why,
because in some sense that\'s the market that you are not addressing
somehow.

Now, it could be that it\'s just irrelevant to what they do, but I
think, again, the idea of just doing a sort of satisfaction survey
without really understanding fully what the potential market is that\'s
out there really in the end is going to limit the value of what you are
doing.

CHAIRMAN MOUNT: We are running sort of late on time, so I want to leave
time for Nancy to respond. So, John.

MR. GRACE: A couple of very quick things. One, which I guess is
self-evident, the difference between the survey that was headed by Jim
in the [Post]{.underline} and your own, is that the people who think
your work really is wretched haven\'t been calling you, so they are not
going to show up in there, it shouldn\'t be too surprising.

The other thing that might bear some energy, and it wouldn\'t be too
difficult to do, is look very carefully at the products of people who
are repackaging your stuff. Somebody is paying them for something, and
maybe you don\'t compete with them, but maybe you get ideas. Maybe it\'s
timeliness, maybe they are combining with other data or something, but
somebody at the margin is willing to pay Penwell or whomever to do
something that you are doing punitively for free. And so, it might be a
good source of ideas of ways you could change it.

That\'s it.

CHAIRMAN MOUNT: Is there in the back who would like to comment before
\-- oh, sorry, go right ahead.

MR. SKARPNESS: I just have a quick comment on the database or
availability of the data. I\'m not sure, you know, exactly how you guys
are doing that, but lately I\'ve been doing some work in the
transportation area, and there\'s a Bureau of Transportation Statistics,
and it\'s sort of a new group, and they\'ve been very active in trying
to get all this DOT data together, and they\'ve already sent out several
disks, some related to just databases, some related to GIS type data,
and Internet type work, and so, I don\'t know if you have much contact
with them. In fact, I don\'t even know where they are anymore, since
they reorganized.

UNIDENTIFIED SPEAKER: That\'s where I am now.

MR. SKARPNESS: I know you are, I was going to ask, but where \--

UNIDENTIFIED SPEAKER: We\'re at 7th and D.

MR. SKARPNESS: Yes, because I talked to some EIA people and they didn\'t
really know about the group. And, I don\'t know if that\'s the format
that they want to follow, but they\'ve got some experience in that area.

That\'s all.

CHAIRMAN MOUNT: So, Nancy.

MS. KIRKENDALL: Okay. Let\'s see, where to start.

Well, first, in response to Jim, of course, we do think EIA is special,
and that\'s obviously why our customers think we are great. But,
actually, people who call us probably need the information for their
work, and in general we get a lot of comments like, it\'s wonderful to
get a person on the phone. And so, I think that really does have
something to do with the high satisfaction rate, especially the
first-time callers are surprised and pleased to have actually gotten a
person to answer their questions, most likely quite quickly from the
time they started calling.

You talked also about the outlook survey. I think one of the attachments
that you received was the survey of asking about the outlook\'s article
on propane distillate and what not, and that one did suffer from a low
response rate. So, of course, it probably is biased. It only represents
the people who chose to respond to it. There\'s not much we could do
about that. We couldn\'t get anymore of them to want to send us any
information.

On confidentiality, we did tell respondents that their individual
responses would be kept confidential and their names were not supposed
to be put in a database. So, that was supposed to be something we did,
except for those who said that they would like to help us improve the
timeliness of our products. So, we did invite people to participate with
us in improving products and for them we kept their names in a separate
sheet of paper.

Linda looks like she wants to say something.

MS. CARLSON: In addition, the person who was the initial contact is not
the one who did this survey. It was a totally different office that did
that survey, so there was continual switching back and forth.

MS. KIRKENDALL: The question about the best in the business, one of the
things that you are supposed to do, once you start getting a good
measurement process in place, and you actually know something, which we
are not there yet, then you are supposed to do the comparison to the
best in the business. So, we are not there yet. We don\'t have the
information. If anybody has ideas about how to identify who is the best
in the business when we get to that point we\'d really like your ideas.

I talked about confidentiality.

The accuracy question, in your handout the thing that you were mailed,
you\'ll notice that the satisfaction rating with accuracy was lower in
that handout than it was in the slides that Mike showed today, and the
reason is that the first time he prepared the tabulations, he included
all respondents, and in the second tabulation he said, of those who had
an opinion. And, the big difference was accuracy, where people said, I
haven\'t got a clue, I can\'t judge on that. So, the ones who gave us
high ratings, basically, it\'s trust, I guess.

Weighting by customer importance, and that goes back to Phil\'s comments
too, we do have information on who the customer is. We ask what kind of
\-- who are their employers, so we have a little bit of information on
that, but it\'s difficult. If everybody in the federal government called
the people who have the purse strings, they\'d be deluged with
interviews and phone calls, and so we really haven\'t targeted them. And
so, that is a question, how do we target those people in a sensible way,
so we\'ve avoided that and we\'ve been focusing on our free riders.

I\'m not sure what else to do, but ideas are welcome. And, I guess those
were the only ones that I had something that immediately leaped to mind.

CHAIRMAN MOUNT: So, I would like to thank all of the presenters this
morning, Nancy and Mike, and Linda, and the discussants, Dan, Jim, John,
David and the combined presenter and discussant, Brenda.

I would also like to thank the other members of the committee for their
comments, the EIA staff who did a lot of the work for this, and also the
people who organized the meeting, members of the public who stayed to
the very end, and I think we can close it.

Thank you all very much.

For the committee, we are going to go to the Lewis Room.

(Whereupon, the meeting was concluded at 12:22 p.m.)