AMERICAN STATISTICAL ASSOCIATION

\- - -

COMMITTEE ON ENERGY STATISTICS

\- - -

FRIDAY

APRIL 24, 1998

\- - -

The Committee met in the Clark Room in the Holiday Inn Capitol, 550 C
Street, S.W., Washington, D.C., at 9:00 a.m., Daniel Relles, Chair,
presiding.

[PRESENT]{.underline}:

DANIEL RELLES Chair

CHARLES BISCHOFF Member

CAROL CRAWFORD Member

CALVIN KENT Member

GRETA M. LJUNG Member

POLLY PHIPPS Member

SEYMOUR SUDMAN Member

ROY WHITMORE Member

JAMES HAMMITT Guest

**I N D E X**

[Page]{.underline}

Opening Comments from the Chair 3

Recognizing Previous Judges of the EIA Graphics 4

Contest and Announcing Winners, Jay Hakes

**EIA Survey Issues:**

Addressing Declining Budgets 12

Dwight French (EIA)

Discussion: Seymour Sudman (ASA) 36

Questions from the Committee 45

Future Electric Power Data 57

Stan Freeman (EIA), Betsy O\'Brien (EIA)

Discussion: Roy Whitmore (ASA) 71

Questions from the Committee 77

Efforts to Encourage Response in Oil and Gas 92

Surveys

Roy Kass (EIA), Sue Harris (EIA), Jay

Casselberry (EIA), Antoinette Martin (EIA)

Discussion: Polly Phipps (ASA) 115

Questions from the Committee 130

Public Comment

Closing Comments by the Chair 136

**P R O C E E D I N G S**

Time: 9:03 a.m.

CHAIRMAN RELLES: Welcome to the ASA meeting of the Committee on Energy
Statistics. I believe we are supposed to start the meeting by asking if
any EIA senior staff, committee member or member of the public who were
not present yesterday is here and, if so, would you introduce yourself?

(Introductions)

Actually, if anyone isn\'t sure whether he or she was here yesterday,
it\'s safe to introduce yourself again. Those of you who are sure you
were not here yesterday \-- Well, I don\'t think I can be put in jail
for anyone not introducing himself.

This morning\'s session promises to be very interesting. We\'re going to
be starting with Jay Hakes identifying the winners of the graphics
contest and giving them \-- passing out rewards. Then we\'re going to be
having a session on EIA data collection efforts, collecting more with
less resources, and in particular, sort of focusing on the upcoming
efforts to collect electricity data \-- data on electricity deregulation
effects, including the actual status of that effort, plus a discussion
of what data efforts have already been going in the area of oil and
natural gas which might provide some hints as to what we might expect to
see in the electricity data gathering efforts.

So with that, I\'d like to introduce Jay Hakes.

MR. HAKES: I can\'t believe this is true, but apparently this is the
fifth year of the graphics contest. It seems like it just started
yesterday.

One of the things, I think, that\'s clear to all of us is there are
times when you can give people a lot of numbers, and they still don\'t
understand what you\'re talking about, and occasionally you can hand
them a graph and, all of a sudden, they understand. I\'ve personally
observed that in a lot of meetings.

I think that the power of the graphic presentations that we do is a very
important part of our work. So to stimulate innovation and good
practices in this area, we have tried to give special recognition each
year to four particular graphics that are particularly noteworthy, and
it\'s a hard choice, because there are so many good entries,
particularly this year.

We wanted to take the somewhat unusual step of rewarding the judges. You
know, how many times at the end of the game do they call forward the
referees to get recognized? So we thank them for their hard work.
They\'ve done a good job.

I\'m surprised they\'ve identified themselves, because now all the
losers know who to blame, but we actually have nice awards for them, and
I want to call them out in order. If they would come up, I\'ll give them
at least part of their award.

Eugene Burns is first. I see he\'s here. Andy Kydes is next. Theresa
Halquist is next, and Barry Cohen. Barry is not here. Alan Swensen;
Susan Holte is not here; Sandra Smith.

We have Lucinda Gillian; Michael Margreta; Jim Kendall; Tammy Heppner;
Ann Whitfield; and finally, Bob Rutchik.

I would also like to say, apart from being judges, Bob has been one of
the people among the group here that\'s really helped us kind of think
through what is the role of graphics. Dan in one of his first
appearances on the committee gave us an excellent dissection of what
role graphics should play, came up with the novel idea that the
graphical approach should be somehow limited \-- connected with the
purpose for which the graph was made, which I thought was a very \--
something we actually needed to hear. But again, thank you to all of the
judges for your contribution.

This year\'s graphics contest was particularly intense. In previous
years it seems like I\'ve usually sent out a reminder notice saying, now
you do know you get a free lunch out of this, and then we would finally
scrape together a few entries.

This year we had 53 entries spread throughout the organization. We
didn\'t send out any reminder notes this year. So I think that\'s a
healthy sign. We got them from all parts of EIA. We had to do
semi-finalists to break it down, and I saw a lot of my favorite graphs
didn\'t make it. So it was very intense competition.

Somewhere in the package that Bill has given me, I have those, too. The
winners are \-- we\'re not going to let them make speeches either: Craig
Cranston, and Chuck Dale is here to accept for Craig.

I actually would like to say what the graph was, if I can find it.

This regional supply and demand \-- is that yours? They have a graph
that places U.S. dependence on all imports in a global context. I think
that\'s extremely useful, because it\'s a question that we get all the
time. If you give people some pictures, they\'re going to remember the
answer.

Accepting for Christy Hall is Dwight French. This was also from a
publication. It\'s from the [Manufacturing Consumption of Energy Survey
1994]{.underline}, and it\'s a graphic that shows the variability in the
range of natural gas prices paid by selected manufacturers in 1991-94,
and I\'ll commend that effort.

Then accepting for Susan Holte is Mary Hutzler or Jim Kendall. Thank you
very much. We did not restrict the competition to traditional
publications.

So this is a graph that comes out of the brochure that summarizes the
Annual Energy Outlook. These have come in very handy on numerous
occasions briefing public officials and explaining complex concepts to
people. They can be carried around in your pocket. So that\'s a good
effort.

Then the final one actually comes from a presentation that was made in a
briefing at the Gas Pipeline Conference and the Natural Gas
Transportation Conference, and this goes to Phil Shambaugh. Thank you
very much.

I think people have particularly in presentations have been doing a lot
of creative things, and this graphic displays the amount of natural gas
pipeline capacity traded by the capacity release mechanism and is a very
educational graph.

One of the things that we do is we put these graphs right outside my
office. I think we want people who visit my office to be educated about
energy. So we hope that they will read these graphs as they come in and
remember what they learned.

Today we\'re also doing some other presentations. We want to recognize
the new members and our guests. So for Jim Hammitt, Seymour Sudman and
Polly Phipps, we have the widely sought after EIA tee shirts.

MR. KENT: We never had a big enough budget when I was here.

MR. HAKES: I think, when you get thousands of dollars of work for a
cheap little tee shirt like that \--

Then for all the members of the committee, the equally coveted EIA mug.
I\'ll hand it to Dan on behalf of the whole committee, but Bill will
have mugs for all of you, and we hope you\'ll think of us when you\'re
drinking your coffee or your orange juice or whatever you drink to wake
up in the morning.

These are, of course, tokens of the appreciation we have for the efforts
everybody has made in these areas, and we\'re, I think, getting a lot of
recognition around town for the clarity of our graphics.

I feel \-- I\'ve actually been accused of not fighting fair, because
someone from another part of our department will make a presentation in
some old fashioned looking black and white, and it just looks 30 years
old, and then we come in with this sort of snappy color stuff, and they
get upset with us, because, well, your stuff looks better than ours, you
don\'t have a right to do that. But I think we do want to strive for
excellence, and I think we\'re achieving it.

Appreciate the committee\'s help in both intellectually thinking through
the issue and furnishing judges for us each year. The committees are a
combination of the EIA employees and the members of this particular
committee. So that\'s worked real well for us. So thank you very much.

MR. SUDMAN: Jay, can I make a suggestion?

MR. HAKES: Sure.

MR. SUDMAN: I would like to have seen the graphics. Is it possible that
they might be published, and then you could pass them around to sort of
everyone within the organization and we could get copies as well? I
think it might something useful for people to have, if it\'s not a
budget buster.

MR. HAKES: Well, no, it\'s not a budget buster. We can do that. You\'re
asking the question. One of the things we could do is we could make an
announcement on our Web-site of the contest winners and bundle \-- I
mean, most of these graphics are probably on the Web-site anyway, but we
can bundle them together.

We also have them here for anybody that wants to look at them. Next year
maybe we\'ll do some overheads, but that\'s a great idea, and maybe
bundling it up on the Web-site as an announcement item would be very
efficient.

CHAIRMAN RELLES: Yes, I agree. Given a choice between hearing you
describe the graphics versus seeing them, I think I \--

MR. HAKES: I intentionally tried to make that point.

CHAIRMAN RELLES: Okay. Well, I\'d like to get started with our talks
then. Dwight French is going to lead off by giving us an overview of EIA
efforts to deal with costs of surveys. Dwight.

MR. FRENCH: You\'re going to have to wait for a minute while I get this
set up.

A couple of people have already commented to me that they think it\'s
amazing that I would even try something like this. I am affectionately
known as Mr. Neanderthal with regard to technologies in the office, and
I let the staff do most of the work in that regard, but I figured, since
I was going to be talking about addressing declining budgets with
improved survey technologies, at least I ought to make an effort to try
to get something up here on the board.

I don\'t know how to run it. I know how to put the first slide up, but
beyond that I\'m hopeless. Okay. Thank you, Nate.

For anybody who is interested, there are copies of the slides on the
lower righthand corner of the table in the back of the room.

It\'s been heard, and it\'s true, that the Energy Information
Administration has been struggling during the 1980s and 1990s with its
ongoing battle for resources, and across EIA long term reductions in
resources have been quite substantial.

Let\'s take a look a little bit on the macro side for a moment. Well, I
may have to tell you some of the titles. If you \-- The three columns of
figures: The rightmost column is the 1998 budget for EIA. Well, I think
I\'m going to get going, because otherwise I\'ll spend ten minutes here
fiddling around with the slide.

Rightmost column is our current budget. Leftmost column is my best
understanding of the 1980 budget in 1980 dollars, and the middle column
is a simple adjustment for CPI change during the period 1980-1998.

So what you\'ve got there is purchasing power of the 1980 budget in 1998
dollars. I may be a little bit off. I understand the budget in 1980 was
about \$110 million. That was the high water mark for EIA, and at that
point we had about 770 FTEs on board. That\'s government workers. We\'re
now at about 375.

You can see that our total resources, both for paying the FTEs and for
contract dollars, overhead and other expenses, have gone down about
two-thirds in the 18-year period in total purchasing power.

The staff drop of a little over half has about compensated for the rise
in costs of salaries and overhead during that time, leaving overhead and
contract expenses dropping to about one-quarter in purchasing power of
its 1980 level in 1998.

To give a little bit more current and program specific example, in the
consumption area between Fiscal 1995, which is the left column, and
Fiscal 1998, which is the right column \-- these are nominal dollars \--
our contract funds for the consumption survey program dropped by well
over a million dollars a year, and our staff level has gone from 30 to
21, a contract dollar drop of 35 percent, an FTE drop of 30 percent.

So what do you do if you\'re dealing with resource limitations of that
nature? Well, we certainly in EIA have tried to streamline ourselves
over the years. We have cut back some programs, and we certainly try to
do our work more efficiently.

One of the ways that we do try to do our work more efficiently is to
invest in technology improvements that can create, hopefully, big
improvements in efficiency, both in the survey process itself and in how
the staff works on our surveys.

What I\'m going to do in this presentation is examine some of the
benefits and limitations of our use of technology today and the
operations that technology will need to address in the future if we are
going to generate substantial future savings in survey costs. Most of
all of my presentation is going to be focused on the consumption area,
which is where I work.

Certainly, we\'ve had a wide variety of survey operations that have come
under the effects of technology. The office is not what it was ten years
ago or five years ago or even two years ago, and a lot of that has to do
with the administrative way that we do things.

We used to have in the Office of Energy Markets and End Use about nine
secretaries drifting around. As of today, we\'re down to four. That was
five years ago that we had that many. Now we\'re down to four, and their
jobs have changed dramatically.

It used to be everything was done with paper and pencil. Memos were
typed. Reports were typed and retyped and restructured and typeset for
camera-ready to go to print. Now, of course, we\'re reducing the number
of our publications because of the Internet and electronic presentation,
and the stuff that we do is basically done by the statistical staff
virtually to the camera-ready stage.

So the secretaries aren\'t involved that much in that anymore, and memos
are being replaced by electronic communication. But of course, it isn\'t
just the administration area in which we\'ve changed our operations.
There\'s a lot of changes that have taken place in the survey process,
as the next two slides speak to that a little bit, and I\'m not going to
bore you by reading down the list.

Certainly, there are a wide variety of very impressive things that are
being done in the area of survey design, software, data collection
programs, even to outside of technology, the use of specialized subject
matter in areas such as cognitive testing and some others over the
years.

We\'ve had image scanning equipment. Automated querying systems are
coming into being. Some of these improvements, of course, are very
mature, and others are just coming into being.

Overarching all of this is the bringing about of E-mail and electronic
communications and, of course, all of this is underlaid by the use of
the personal computer.

I recall back in the mid-1980s when we started in the personal computer
area, at that time we had about 30 people or upwards of 30 people in the
consumption area. Three PCs were delivered to the office, and we looked
around and we said, what are we going to do with these; we don\'t have
anything to do with these machines. Now we can\'t live without them.

A couple of weeks ago there was some sort of general shutdown of the
LAN, and you should have seen people panic. What am I going to do if I
can\'t get on the machine? I mean, there\'s been an entire culture
change between the mid-1980s and now, as I know you all realize.

Well, in some ways, in fact, we have increased our productivity, and in
some ways we still struggle with this. I want to talk about a few
reasons why advances in technology don\'t necessarily give you all the
costs you expect, and sometimes they\'re really not intended to save
costs.

First of all, a lot of the technologies that we invest in save money
down the line in survey operations, but they have up-front costs,
programming costs.

For example, when we use the Blaise data collection system to input to
the computer assisted personal interview that we changed the 1997 RECS
over to, there was quite a bit of work and quite a bit of programming
that needed to be done up front for that, which took staff time, and
there was extra training of interviewers that needed to be undertaken so
that not only could they deal with the survey, but they could deal with
the machine and how they were going to handle survey operations with
that machinery.

That\'s the sort of thing that, at least at the beginning, you have
offsetting costs which don\'t necessarily allow you, at least at first,
to show a true level of cost savings.

Secondly, some of the things that we do are really intended for purposes
other than cost reduction, such as time savings or improving the quality
of data, something like that; and we will see a little bit more about
that in a few minutes.

Thirdly, a lot of times technology effects are intermingled with other
types of effects, some of which are not associated with the technologies
at all, thus making it difficult to sort out the effects of
technologies. So sometimes it\'s difficult to determine exactly the
extent of the cost savings that you get from technologies.

Fourth, sometimes we find that technologies are addressed to the minor
components of our surveys, minor cost components, and thus while they
may have some significant benefits on particular aspects of the survey
cycle, they don\'t have an overall benefit to a great deal in the total
cost of a survey.

In fact, we have a particular difficulty of this in the consumption
survey area. Why? Because our surveys now are only fielded every four
years. That does a couple of things.

Number one: If you put a new program into place, fine. You do your
programming. You get some savings out of it. You don\'t amortize it
again until four years later. So you have a problem with actually
getting benefits, unlike if you employ something in a monthly or annual
survey and you do something up front, you can crank out the benefits
cycle after cycle and find that you amortize your cost savings very much
faster.

Also, since we only go out every four years, our surveys tend to change
every time we go out, one way or another. That means, if you do have
programming or set-up costs, you\'re probably going to have to duplicate
some of those costs the next time around.

So it\'s not like we\'re running the same thing over and over again.
That means that we have to have probably some of our offsetting costs
occurring time after time.

The other two are really restatements of what I said earlier, in
general.

Okay. Well, let\'s take a look then at a specific example, the 1997
RECS. Between 1993 and 1997, we decided to change the RECS by going from
a paper and pencil interview to computer assisted personal interview,
and the system that we used for the data collection input to that was
the Blaise data collection system.

That was the major technology change that occurred between 1993 and
\'97. Now again looking at the slide, the left-most column shows the
total survey cost, and below \-- Don\'t worry about that line under the
word technology. That\'s just used to separate the two rows on the
righthand side.

I separated out the survey into the components affected by technology
and the components unaffected by technology. So you see about 60
percent, 65 percent of the survey cost there is affected by the
technology, and I\'m going to talk in more detail about that in a
moment. The other part we\'re going to drop out.

The third column from the left is the corresponding cost total \$3.9
million as opposed to \$3.7 for the 1997 RECS, and you see the
components affected and unaffected by technology in that particular
year.

Now I put cost per interview down in the second and fourth columns for
each year, because the sample size in \'97 dropped from slight over
7,000 interviews to slightly under 6,000, and I decided to take a look
at the components based upon cost per interview to try to adjust for
that change.

This was the slide I was most worried about you\'re seeing again. It\'s
in the handout. There are four basic survey components that we can talk
about as being affected by the Blaise technology applied to capping.

The first is interviewer recruiting and training, then data collection,
coding and keying, and finally editing, imputation and file preparation.
Now one would expect that a system like Blaise, which has considerable
built-in edits and allows the interviewers to directly input the data
onto a diskette or onto a file and send it directly to the home office
whence it can be put directly onto a data file, that coding and keying
would be greatly, greatly, greatly reduced, and in fact it is, as you
can see.

Over 80 percent of the cost is reduced. The remaining cost that you see
is basically the cost of recoding some interviewer comments that were
put on the form and putting in some of the household screener
information.

Another operation that you might expect to improving costs is the
editing/imputation/file preparation cost. In a way it does, but in
another way it doesn\'t. You will see parenthetical costs in that row
for 1993.

By the way, the middle column is \-- I forget to mention that \-- is a
15 percent cost adjustment to bring the \'93 costs up to \'97 dollars.
You will see parenthetic numbers in that editing/imputation row.

The reason for that is that the size of individual interviews was also
shrunk between 1993 and 1997 from over 900 data items to a little over
500 data items. So what the parenthetical numbers do is an adjustment to
pro rate that value for a size of 500+ survey items rather than 900+.

When you see that, it looks like the costs have stayed approximately
constant, when you adjust for the size of the individual interviews.
Well, since we were supposed to be getting a lot of editing benefits out
of the survey, why did that happen?

The basic reason was that the interviewers, while they did well on the
electronic part of the survey, had a little bit of a struggle with the
household screener information which was on paper and pencil, and they
made some mistakes which had to be corrected in the central office later
on. So that inflated our costs.

Now we\'ve got one notable savings, one even amount, but if you look at
the per interview costs for those two components, they\'re relatively
small. Where we\'ve got the real cost is in interviewer recruiting and
training and in the data collection operation itself, because we\'re
training a field staff to march around and collect personal interviews.

Those costs actually went up more than inflation between 1993 and 1997.
Reasons: Well, number one, interviewer costs went up faster than
inflation during the period 1993 and 1997. The tighter labor market and
less availability of people to do this type of work means you have to
pay them more.

If you\'ve read anything about the pretest of the 2000 census that\'s
being prepared for right now, they\'re having some of the same
difficulties in certain areas of the country.

Also, reducing the sample size brought about some limitations and some
inefficiencies in interviewer assignments. So when you look at things in
cost per interview, you do get a little bit of anti-benefit of the
reductions that we had to take to try to stay within budget for 1997.

Well, if we had a problem with money and we didn\'t really get cost
savings that are all that impressive on the whole, what did we get?
Well, we certainly got cost savings in coding and keying, as was pointed
out.

We got a lot more complete data. We didn\'t have interviewers getting
lost in the middle of skip patterns in a 100-page questionnaire. We got
cleaner data. The editing operations did, in fact, get slimmed down
enormously for the 1997 data; more timely data.

Earlier this week, I signed out to the statistical methods group the set
of housing characteristics tables for the RECS survey. After they get
through with their review, they\'ll be going up on the Internet. We\'re
talking about 1997 data in April of 1998. Yes, there are benefits that
can be done.

It\'s probably \-- Even taking into account the fact that before we put
them as part of reports, we\'ve saved four or five months at least from
the time that we could put these types of tables out before.

You have to also take into account that the 1997 RECS was not done in
the fall of the year. It was done in the spring into the summer of the
year. So you have to take into account the like four months difference
in the interview times.

So we\'re getting it out earlier, partially because of the earlier
interview time we did in 1997, but still in all there is a substantial
savings in timeliness of the data.

We felt we got efficient use of staff time. We had someone programming
the Blaise \-- the RECS questionnaire in Blaise in-house, and we think
that worked out very well.

The last thing I have listed there is happier interviewers. That\'s
really sort of a misnomer. The way things are in the interviewing field
right now, the genie is out of the bottle. We\'re not going to be able
to go back \-- As long as we\'re in personal interviewing, we\'re not
going to go back to paper and pencil interviewing.

You have to do this now or the good interviewers will just shun you and
go off and do some other type of work, because they want to work on the
electronic equipment. So happier \-- Well, it\'s really you got to do
this or you\'re not in the ballgame anymore.

While we didn\'t see much cost reductions in the RECS, although we got
some other benefits, obviously, the question might be asked what could
we do, if we needed to, to get substantial cost reductions in the
consumption surveys in upcoming years.

The obvious answer, the say ways that it is anytime you have a budget
and you\'re trying to find cost reductions, substantial cost reductions,
you\'ve got to go to the big ticket items. You can\'t hack away little
pieces of three percent here and five percent there in small components
of your survey or even substantial parts of a component that only
represents five percent of survey costs and expect that you\'re going to
do \-- have a great deal of improvement in your survey costs.

Well, there are five things that I could identify that you might want to
try to attack in the consumption surveys, if you are going to try to
save some costs.

First is in rough counting and field listing of sampling units. That is
a tremendous cost for us. Because we\'ve got area based sample, we\'ve
got to send people out even before the interview phase to list units,
and even in the case of the building survey, to rough count in order to
get our second stage units that are selected segmented in order that we
can get a small enough segment size so that buildings can be listed.

Well, potential future breakthroughs in this area include electronic
files of satellite photographs. I was encouraged to see that in 1997 for
the first time commercial payload exceeded government payload in the
marketry area, and there is a great deal of talk right now about
commercial satellite photography.

We have been looking at satellite photography for over ten years as a
possibility. There are cost implications. It\'s very difficult to deal
with that, but perhaps commercialization might change things in that
regard.

Another possible breakthrough might be if the Census Bureau does
implement continuous measurement after its 2000 census and as part of
that has a planned listing of households in the United States, and if
pending legislation gets passed regarding data sharing and
confidentiality for the consumption surveys within EIA, we could be able
to take advantage of that and make perhaps a very substantial
improvement in costs in later year RECS.

The second one is probably the least of the five that we can save costs
in, on-site training of field staff. We have tried before video taping
presentations and eliminating on-site training of field staff for the
1990 RECS.

We were not really pleased with the way things worked out. However,
upcoming technology and video conferencing and Internet interactive
technology may enable us to take a look at this in upcoming years again.

We wouldn\'t be able to save a whole lot of money on this, because we
can save travel costs, but you would probably still have to pay
interviewers per diem for their training that they would get through
interactive means. So you would have to pay that part.

The third one is probably the big one. That is the single most difficult
situation in the consumption surveys. We have done personal interviewing
in both RECS and CBECS throughout their history except for the 1983
CBECS which was done by telephone, and we are looking for a cost savings
in the 1999 upcoming CBECS to once again go to a telephone approach.

There are some real difficulties involved in doing this. Marty can tell
you about the rigorous meeting, four hour meeting, she and several of
the people in the division were in yesterday with the survey contractor,
just working out some of the upfront logistics for trying to figure out
how to do this.

Really, the big savings in trying to eliminate personal interviewing
would be to go to self-administration in these consumption surveys. Now
we\'re not prepared to do that right now, and we won\'t be prepared to
do that for sometime, but you can envision a future where enough homes
and businesses have new equipment and interactive, perhaps through cable
as opposed to PCs, whatever.

We don\'t know how the technology is going to shake out yet, but there
may be ways that we can largely do interactive interviewing with
respondents down the line. Oh, of course, there\'s always a couple of
things to worry about with that.

Number one is response rates, and number two is getting waiver forms,
but even if we have FAX or other means that come in with newer
equipment, we may be able to do that, at least for the commercial
building survey.

As a matter of fact, we\'re considering getting some waivers in by FAX
for the 1999 CBECS. We\'ll see how that works.

The fourth one is energy supplier data collection. Right now in RECS and
CBECS we collect data from the energy suppliers about household and
building consumption and expenditures. We plan in the 1999 CBECS to test
a hybrid approach where we start asking buildings to provide their own
consumption and expenditures information, and only if they cannot do
that will we go to the energy suppliers.

Again, that was part of the rigorous meeting that we were in yesterday,
looking at some of the advance work that we\'re going to have to do to
prepare to do that with the building people.

Finally, designing and disseminating data tables: This is not a contract
dollar savings. This is a staff savings. Most of you, I believe, are
familiar with these enormous publications we\'ve put out.

The reason we do that and the reason we put out enormous numbers of
tables and PDF files corresponding to that on the Internet is that we
collect an enormous diverse amount of information, and we have to
present that to the people in some way, and up to this point it\'s been
by means of tables, because we need to give the people their money\'s
worth.

Well, there\'s questions about using query systems and allowing our
customers to access Micro files or specially set up tabular files within
the electronics and letting people generate their own table.

I have two concerns about that, but certainly querying is something that
we\'re going to be looking at very seriously in the future. The two
concerns are: I\'m not clear whether customers really want to use such a
system. I think Internet users are used to finding what they want, not
being asked to generate what they want, and that could create a problem.

Also, we\'re going to have to find a way to create error estimates to
accompany query generated statistics. I am of the opinion that,
regardless of what we may think about how often data users consider
errors in analyzing data, it is incumbent for EIA as a statistical
agency to see that they be able to do so. So we\'re going to need to
deal with these issues if we\'re going to deal with the table.

Yes, I think by working with some of the things that we have done
already, we can make some incremental gains which can offset things like
the cost of inflation, but big gains, as I said, are going to have to
come from big ticket items. However, I think I\'ve got good news in
that, in order to make some substantial savings, not all of the
speculative things I was talking about in the last five minutes have to
come through.

One or two would be enough to make some substantial gains, and I think
sooner or later we are going to see something of a couple of these
actually occur.

Questions for the committee: I\'d be interesting hearing what
technologies you think that would be \-- present technologies that you
think would be good to consider that we may not have, and what emerging
technologies you think we ought to be focusing on in the next five years
or so.

The second question is whether the statistical community has a
relationship with the business community so that we can encourage the
business community to make improvements and new software and so forth
that would be beneficial to us, and how could EIA be involved in such
interactions.

The third question has to do with the concern about response rates, if
we were to go to a self-administered survey. Just wondered if anyone
knew of recent instances of a voluntary nationally representative,
self-administered survey that is not on a topic of crucial interest,
current interest, to the public.

The reason I say that, energy is not a topic of current crucial interest
to the public, for the most part, that has attained a response rate that
we \-- the minimum we would like to attain, which is 80 percent.

With that, I thank you for your attention, and we\'ll a few of you
discuss from the committee.

MR. SUDMAN: I thought this was a very useful summary of issues facing
the EIA, and I\'m going to attempt to comment on several of the points
raised, but first of all, an apology or a caveat. This is only my second
meeting, and some of the suggestions I give may already have been
considered in the past and rejected, and I shall primarily limit myself,
as the paper does, to a discussion of the Residential Energy Consumption
Surveys and especially the household surveys, because I think there are
key ways in which there can be huge savings if people are willing to
consider them.

Let me start off. My seatmate, Calvin Kent, said, hey, this is not a new
issue; exactly the same problems were facing the agency. And if misery
loves company, it should please you to hear that every survey
organization and every data collecting agency always has the same
problem to face. Indeed, that\'s what statistics is all about. It\'s the
sensible use of limited resources to get the best possible data
available.

Okay. Now here come some fairly radical suggestions, and see what you
think about them.

The first suggestion \-- and this is not my idea. It\'s one that has
been frequently proposed by Leslie Kish \-- is that the EIA give serious
consideration to moving from a very large survey every four years to a
much smaller but continuous survey.

That is exactly what the Census Bureau is thinking about with their
continuous survey. In other words, instead of whatever your sample size
would be every four years, you divide it by four and do it on an annual
basis.

What\'s the benefit of that? This would utilize a smaller staff on an
ongoing basis. It would almost certainly help reduce the large start-up
cost associated with infrequent surveys. The data collectors would be
more experienced, and some improvement in data quality would be likely.

It would also be possible to adapt to new data needs for policy purposes
much more rapidly. The key issue for users would be how would the data
be analyzed?

I assume, without having seen the variance estimates, that many
important variables need large sample sizes to measure small but
important changes. This could be accomplished by summarizing data for
sufficient time periods to make the results reliable and producing
running averages.

Note that to avoid gaps, this annual survey would need to start just
after the completion of a large quadrennial survey, say after the year
2001, and that it can only be accomplished if Congress could be shown
the budget efficiencies of such a proposal, and you could have
reasonable assurance of ongoing budget to do it. That\'s one of the
issues always with ongoing surveys, and I don\'t want to minimize its
importance.

Okay. A second point: Suppose I said that you could cut the costs of
collecting data on the RECS by half at the expense of some small losses
in what you could collect and perhaps very, very tiny losses in data
quality.

You could do much bigger samples or do more them frequently. If you
could cut the cost in half, would you be willing to accept some
reduction in data quality, a little reduction?

If the answer is yes, then I would urge you to consider a method which
has been widely \-- It\'s not new technology. It is essentially the
technology that most people face with budget constraints \-- adopt \--
and that\'s telephone interviewing. That is, adopting telephone
interviewing for the household survey.

It would cut your cost in half. Essentially, face to face interviews
cost about twice as much as telephone interviews. Hey, that\'s a big
difference. Now the question is are you willing to settle for somewhat
less flexibility?

Now I understand you\'re going to use telephones now for the
establishment part of this survey, and the question is have you ever
considered doing it \-- testing it at least. Perhaps if this is way too
radical a notion, how about trying half and half or something of this
kind to see how \-- you know, how the results compare.

My hunch \-- it\'s not a guaranty. I\'m not going to bet my life on it.
My hunch is the differences would be small and that, given the
challenge, you would be able to find ways of doing what you want to do.

It might very well, I would suspect, involve a combination of first
talking to people on the phone and then getting them perhaps to mail
some stuff in to you. The waiver \-- you know, utility bills and things
of this kind, but I think it\'s not at all impossible, and somehow it\'s
not clear to me \-- You know, I don\'t know the complete history \--
that this has been explored or not. Anyhow, that\'s my major notion.

I don\'t see that the obstacles are insurmountable, but perhaps they
are.

Okay. Now the sort of suggestion that\'s been made is that we switch
from a face to face personal interview and go immediately into \-- at
some point in the future, into some form of self-administration with new
technology.

I don\'t think that\'s going to happen. It seems to me that that\'s a
very, very optimistic view. New technology will not solve the problem of
respondents\' interest in the survey and willingness to cooperate.

I haven\'t seen the recent version of this survey, but it\'s a big,
complex survey and, as Dwight mentioned, it\'s not of especially
enormous salience to respondents. I don\'t see, even with the easiest
technology possible, that people are going to be very anxious to do it.

So, you know, I\'m not saying that this is not something that could not
be tested. It is something, certainly, that will work a lot better on
establishments than it will on households, but I don\'t see this as a
solution.

So my suggestion is, rather than sort of looking way ahead and trying to
imagine self-administration, it seems to me that telephone in
combination with mail procedures may be the way to go.

Dwight wasn\'t here yesterday when we had a very, very impressive
illustration of the Web pages of the agency. I was blown away, to tell
you the truth, and it seems to me that there are some real savings that
can be accomplished using the Web to disseminate information, and the
savings will occur \-- and here\'s the controversial part of this thing
\-- the savings will occur if indeed the paper versions either vanish, I
mean, or get charged for.

I mean, if people are going to have to pay for paper, they\'re going to
be a lot more cautious about asking for it. Now, obviously, that means
that some people aren\'t going to be as happy as they are right now, but
if there can be major savings in data dissemination, then these savings,
of course, can go to making the data quality better. So I think they
need to be \-- these issues need to be considered.

Okay. One sort of last comment, because so many things we hear here, I
think, that really apply to the whole area of survey research, and let
me reiterate just one more.

Computers and computer technology don\'t save money. This is \-- You
know, when computer assisted telephone interviewing was introduced,
people said it\'s going to save money. It doesn\'t save money.

Computer assisted personal interviewing does not save money. The costs
are shifted. There are major benefits, major benefits in the technology.
The data quality clearly are better and, as Dwight said, the
interviewers certainly are relieved of a whole slew of clerical tasks,
and they can concentrate on their task, which is getting good data, but
it doesn\'t save money. We see this again.

That shouldn\'t be the way they\'re sold. They should be sold on the
basis of improving data quality.

Okay. Now there were questions that were asked. I\'m going to try to
answer them. Clearly, I think, as we discussed yesterday and today, the
Web is the emerging technology that will have the greatest impact on
survey practice, but interestingly, not, I think, in terms of data
collection yet.

In terms of data dissemination, it\'s already having huge impacts, and
that will continue.

Secondly, does this community have a special relationship with the
business community to facilitate Internet and software applications? I
don\'t think so, but I think we\'re all part of this, you know,
enormously evolving network.

I\'ve just been looking at some numbers \-- at the number of Web-sites.
The number of Web-sites is just astronomical, and the growth rate is
beyond comprehension. I think we just have to \-- It\'s not a question
of sort of special relationships. It\'s trying to keep up with what is
happening.

Finally, you know, are there recent instances of voluntary national
surveys self-administered with 80 percent? The answer is no, and I
frankly don\'t believe that that\'s going to be possible, at least in
the horizon that I see.

Thank you.

CHAIRMAN RELLES: Maybe we should just get it all on the table and then
have you address them.

MS. PHIPPS: I\'d just like to say that I definitely agree about the
self-administered national survey. I think it\'s a real unlikely
scenario, particularly given the survey lengths of these particular
instruments.

One of the things that I would caution about telephone surveys \-- I
think that sometimes there\'s a real difficulty sustaining response
rates, particularly with the RDD samples. I don\'t suppose the
contractor would probably guaranty more than about a 70 percent response
rate, I would think. I don\'t know how negotiable.

I also think you would have to think very seriously \-- I would do some
serious testing on this, and also you would have to cut the
questionnaire quite dramatically, because you\'re \-- Given the length
of the questionnaire, I don\'t think you can \-- I mean, telephone
interviews, I think you can get 30 to 40 minutes at the most, and I\'m
not certain how long the personal interviews are taking now, but they
look \-- If they\'re 500 items, they seem fairly long. So \--

MR. FRENCH: The 1997 RECS went to a little bit over 30 minutes, as I
understand, on the average, because of the drastic cut in the length of
the questionnaire.

So we\'re perhaps where we need to be in that regard to consider
telephone interviewing.

MS. PHIPPS: If it\'s a CADI instrument, then you have the skip patterns
and things that make it shorter for a number of respondents then.

MR. KENT: My comment would be on \-- I think there\'s really two
incredibly excellent suggestions that Seymour made, and one of them is
to see what can be done on telephone surveys; because the processes here
are improving so rapidly, and the techniques are improving so rapidly
that I think there may be some real possibilities of using telephone
surveys, particularly if they do save half.

Then you could even consider paying people to comply, and that would
increase both their interest and their enthusiasm, if they knew that
they were going to do some. So if there are real cost savings, then you
can use some of those cost savings to make a nominal payment to people
like the Nelson ratings will, what, pay you two bucks or something like
that if you send it in, and they get, I understand, very strong return.

The other thing is really explore how you can use your Web for your
data, because most of us who use RECS and these other things \-- we
would rather have it in a data file where \-- and this is how we do it,
you know. We have the data file, and we manipulate that data file
ourselves.

We don\'t really use it in the format you give it to us, if you see what
I\'m saying, because we\'re doing those conversions. We\'re doing those
manipulations.

So I don\'t think, for the real people who use your data, you\'re going
to put any hardship or a great deal of hardship on them if you put it to
them in a data file that\'s one of the standard data files and let them
manipulate and come up with it, and then have somebody available to work
with them if they have problems, kind of like a help line.

So I think that\'s well worth investigating, too. That\'s the only
comments I have.

MS. LJUNG: You went to a new system, and I\'ll bet that there are
start-up costs. Some of those could go down, you know, in subsequent
years. So \--

MR. FRENCH: But, for example, with regard to questionnaire programming
and set-up, when we go out again four years from now, if we were to do
\-- not do continuous interviewing but do another survey, as it\'s done
now, some of the questionnaire will change.

We know that, because issues will change, and people will say, well, you
need to collect this and maybe not that. The questionnaire will change
somewhat.

Now perhaps not as much as it did between 1993 and 1997, because of the
relatively radical change we made in the length of the questionnaire,
but there would be some changes that would need to be made. But you\'re
right.

I mean, doing it the second time around we would reap some additional
benefits on top of what we did the first time around. True.

MS. LJUNG: The other costs will go down.

MR. HAMMITT: A couple of sort of small things. One is you said the costs
of doing it didn\'t go down so much, but the time you could get the data
out was considerably shorter. I\'m not exactly sure how you\'re
accounting for costs, but if a lot of that time saving, the time that
wasn\'t spent, was time staff would have been working on things,
presumably that\'s a monetary cost saving, and maybe that was already in
your editing and so forth.

MR. FRENCH: Well, I mean, that\'s part of how we get down from a staff
of 30 to the staff of 21.

MR. HAMMITT: Some of that should have showed \-- was already reflected
in the cost estimates as well.

I guess I\'m not really clear where you are on whether there\'s a need
for more careful thinking about exactly the purposes and uses of this
survey, because going to the annual surveys would seem to have a lot of
benefits. You could get new information each year, which would be very
valuable to many users, I would think.

On this issue about response rate, it seems there are no real clear
articulation of what a couple of points on the response rate is worth in
terms of other attributes of what you want the survey to provide, and
maybe that can be articulated more carefully to help you evaluate these
choices.

CHAIRMAN RELLES: I think it\'s safe to respond now.

MR. FRENCH: The last shall be first, and the first shall be last.

Regarding response rates, I don\'t think I\'m really concerned about
whether a response rate is 80 percent or 82 percent, whatever, or what
we\'d like to have is 90 percent or 92 percent, but that\'s \-- Well,
there\'s always concern of bias that creeps in, but we have had to face
the fact that our survey response rates have ever so slowly, but they
have inexorably eroded over the years.

Where we could get 90 percent or close to 90 percent \-- The first
commercial building survey we went out with \-- and this is probably not
a fair comparison again, because the first commercial building survey
went out in 1979.

If you remember what was going on in 1979, there\'s a great deal of
difference in the public atmosphere right now. But we got 92 percent
response rate on the first commercial building survey.

Well, actually, we\'re still doing pretty well. I guess we\'re doing
mid-eighties or something like that but, you know, you see this slow
erosion. What we\'re really worried about is, if we don\'t keep up the
good fight, sooner or later, you know, you may see some sort of
substantial erosion if you don\'t keep your guard up. That\'s really
what the issue is, to try to maintain ourselves and improve, if we can,
but guard against the big \--

MR. HAMMITT: If I could just interject, I\'m not sure how many
follow-ups you do with people who don\'t respond the first time, but you
can keep your rate higher by doing a lot of those; but those are
probably a very large contributor to costs.

So if you\'re willing to \-- I didn\'t necessarily mean sacrificing a
few points, but sacrificing ten points or 20 points on the response
rates, conceivably; but that would cut your costs by half. Is that
conceivably a good tradeoff or is it not?

MR. FRENCH: You can\'t ask me as a statistician that and expect me to
say yes it is. I mean, that is not, as far as we\'re concerned.

I would make a couple of other comments about this response rate
business and self-administered surveys.

I\'m probably more optimistic than Seymour and the other people are
around here, for two reasons. Number one, if I would have thought what
the world was going to look like now ten years ago, I couldn\'t have
imagined the situation that we would be in.

I don\'t even want to try to predict what the world is going to look
like ten years from now. Right now, 35 percent of the households in the
United States have a PC. Twenty percent have a modem, and that has
exploded in the last couple of years.

By ten years \-- Now the problem is not just the hardware. The problem,
obviously, is the way that one can utilize it, but I don\'t know what
things are going to be like in ten years. I\'m not willing to tie down
my imagination to that.

I think there may be a possibility of something before I retire, but
we\'ll see about that in ten years, I guess.

With regard to response rates, we\'re struggling even with personal
interview and telephone interview response rates. You\'re starting to
get more screening through answering machines and things like that, that
are making it more difficult, and the computer is still a new toy to a
lot of people, and it has the advantage of you can send it through and,
if somebody wants to do it at a particular time, they can or if they
want to set it aside, do part of it at one time and come back and finish
it, they can.

I mean, there are ways in which self-administered might be appealing to
people. So again, anybody\'s comment, yes, you\'ve got to really
rigorously test this before you jump into the water, but I\'m not
willing to give up on that yet.

As far as the Web to disseminate, yes, we are using that. We\'re
struggling a little bit to use that. We\'re designing and redesigning
and redesigning our Web pages.

We have put out much of our data from the newer surveys on the Web
pages, and in fact, we have eliminated a couple of reports already,
paper reports. The housing characteristics report and the building
characteristics report from the RECS and CBECS respectively are now only
in electronic format.

We combine some of that information now with the consumption and
expenditures information in a single combined format report.

With regard \-- I guess I ought to comment on the smaller continuous
survey. That is a very intriguing idea, and really, what would have
stood in our way right now is the upfront costs. You have to plan, and
you have to very carefully design.

We have been using the same sample design now and updating and just sort
of patching together for the last 15 years in RECS and CBECS. It is
time, certainly following the 2000 census, that we are going to have to
have a comprehensive survey redesign for both of those surveys, however
it is done.

We are taking steps to look at how we are going to approach that
cost-wise and otherwise. At that time would be a good time to consider
the alternative possibility to running the segmented, every four years
\-- used to be every three years; now it will be four years \-- type
survey.

Actually, something was done a few years ago to help out the cost
situation with regard to that. We\'ve done levelized budgeting now,
which \-- So our budget doesn\'t bounce around as we go from this survey
this year to that survey that year.

So we\'ve got a levelized amount that we know where we are from year to
year. So we can use that to perhaps plan a little bit more uniform
continuous type of operation also.

CHAIRMAN RELLES: Thank you, Dwight. Thank you. Polly, please.

MS. PHIPPS: I want to make one more comment and try and caution you on
the \-- You know, self-administered surveys are appropriate for literate
populations, and you need to take that into account. The literacy in the
U.S. demographics has been quite bad when you think of nationally
representative self-administered surveys, because they\'re limited in
that regard.

CHAIRMAN RELLES: I think the suggestions that were put out are just so
important. So many new and novel ideas that ought to get stacked up
against your own in the paper, but I really don\'t expect your sort of
knee jerk responses, immediate responses, to be one that you would
necessarily want to live with.

I think you want a chance to think about it. I\'m going to be writing a
letter to Lynda sort of summarizing the various points and asking for a
formal response from the EIA on these matters, and I think we would
certainly like to hear about your reactions next time after you\'ve had
a chance to think about all of them.

I\'d like to move one step further on the agenda at this point and get
into our discussion of EIA efforts \-- A break is not on the schedule
yet. It is? It\'s not on my schedule.

Why don\'t we take five minutes so we can get some coffee.

(Whereupon, the foregoing matter went off the record at 10:15 a.m. and
went back on the record at 10:31 a.m.)

CHAIRMAN RELLES: About three sessions ago we had a paper by John Macons
on the effects of electric power industry restructuring on EIA data
collection, to which Calvin Kent gave a really excellent presentation.

I guess this is the next step in that process. Stan Freeman and Betsy
O\'Brien are going to be talking about where we are in the data
collection efforts and where we go from here.

MR. FREEMAN: Good morning. I\'m Stan Freeman from the Statistics and
Methods Group, and to my left is Betsy O\'Brien from the Office of Coal,
Nuclear, Electric and Alternate Fuels.

We\'re here this morning to talk about EIA\'s use of focus groups to
identify needs for data in a restructured electric power industry.

As most of you know, the electric power industry is undergoing major
changes. It is moving from a highly regulated industry to one where
there is both wholesale and retail competition.

There will be new actors, such as retail power marketers and energy
aggregators. Traditional actors such as utilities are selling off their
generation. Analysts will need new data to examine the competitiveness
of markets, and data, once readily available to EIA, will be much more
difficult to collect.

CNEAF, the Office of Coal, Nuclear, Electric and Alternate Fuels, has
undertaken a series of efforts to address these concerns. In October of
1996, CNEAF held a workshop in Washington on the data needs for a
restructured industry.

The participants in that workshop urged that EIA carefully and
systematically study the issue. They also suggested that focus groups
with industry and users might prove useful for identifying information
needs.

In March of 1997, as Dan referred to just a few seconds ago, CNEAF
produced a paper which outlined the data issues that restructuring would
raise and presented options for what might be collected. The paper
represented EIA\'s best thoughts at the time of what might be required.

Most recently, CNEAF, in partnership with my office, SMG, began the
focus group study suggested in 1996. One interesting point to note is
that the move towards deregulation in the states has been much more
rapid than anticipated by the 1996 workshop participants. This adds
importance and urgency to the focus group effort.

Why did EIA choose to use focus groups, and what were the benefits?
There were really four major reasons. The first was to bring EIA closer
to the views of the data users and the data providers.

The second was to go to stakeholders who might otherwise be silent in
the process.

The third was to document the data needs to facilitate our clearance of
new forms and revised survey forms through OMB; and finally, to validate
the focus group methodology for EIA establishment surveys.

In all, there were 11 focus groups. The intent was to reach out to all
segments of the industry and users of the EIA electricity data. The
groups were: EIA staff, which was the first focus group, and we used
that primarily to pre-test the questions and the protocols used in the
focus groups.

We went to Federal officials outside of EIA, the state energy offices
and public utility commissions, municipals and cooperatives, investor
owned utilities, nonutilities, the media, consumer organizations, the
investment community, academics and consultants, and Congress.

There were two different sets of questions that were used for the focus
groups, and I have left copies at each of your chairs, and there are
some in the back. They were essentially the same with the exception
that, for the data providers, they were asked about their ability and
their willingness to supply EIA with information in a competitive
environment.

The questions, or as we call them, the protocols, covered five general
areas. They were: Reactions to current EIA data. We wanted to find out
are what we\'re collecting now meeting current needs.

The second area was what information should be collected in the future.
We focused here on what these participants felt would be their core
electric power needs in the future, and what levels of detail and what
frequency would be most useful.

The third area that we were interested in is what information would be
needed for monitoring and assessing a restructured industry. Included
here were questions on assessing reliability and the wholesale and
retail markets.

Fourth, we were interested in the data collection approaches.
Specifically, how could we ensure that all respondents would be treated
fairly.

How could EIA reduce respondent burden and still get the information it
needed.

Finally, we asked the participants about issues related to
confidentiality of the data. What information about electric power in a
deregulated environment should be publicly available in the future, and
what should be held confidential.

The focus group study, which will be up on the Web-site sometime in May,
produced a number of general findings useful in assessing the data
needs. These included: That restructuring is in a transition period. We
don\'t know what it will look like down the road. So EIA needs to remain
flexible as it designs new surveys and revises old ones.

Another interesting comment from almost al the focus group participants
was that EIA should not get ahead of the curve on deregulation. What
they meant by this was that the pace of deregulation is different from
state to state. So that EIA will need to vary its approach, depending on
where they\'re going with their survey data.

Also, and this is of critical importance, EIA will need to come up with
a way to integrate data from regulated and from the deregulated states
into one information system. We need to be able to put our information
together so that it makes sense.

Of course, not unexpectedly, most of the participants said that EIA
should not get rid of any of its current information.

Two other general findings of importance is that electricity markets are
no longer defined exclusively by state boundaries. Markets are now
regional and national in nature, and we need to take that into account;
and as we hear over and over again with EIA data, the data needs to be
more timely.

The issue of timeliness ran the gamut from data on market price for
every hour to annual information. It really depended on who the focus
group participants were and what their needs were.

I now would like to turn over the talk to Betsy, who will go over the
specific requirements that were outlined in four general areas, and then
we\'ll open it up to questions afterwards.

MS. O\'BRIEN: Thanks, Stan. Good morning. I\'ll be discussing the focus
group findings related to establishing the data requirements for the
electric power industry when it is competitive.

First, however, I want to offer a word of caution, because the focus
groups were not and were not designed to be data requirement studies.
They offer general guidelines on areas of interest from a broad spectrum
of survey respondents and data users, and they provide us with
preliminary design ideas, but should not be taken as some kind of an
official endorsement of the need for EIA to collect a particular data
element.

They also provide a good check for us on the ideas that we were
developing internally on what we thought the data needs would be for a
competitive industry.

In this first slide I categorized the information that we found into two
kinds of broad general areas. The first is on analytical issues that
people feel will be important to review in the future and that would
require us to collect \-- make sure that the data was collected so that
these issues could be addressed.

The second area is really an actual specification of data elements that
were identified in a focus group.

On the analytical issues that are shown here, many are related to
comparing things before and after competition, things like are customers
paying more or less when the market is competitive? To understand that
question, they felt that they would need to more detail about the prices
in the components of the price, generation, transmission and
distribution.

Another question was how reliable is the system? How reliable is it
after it\'s competitive versus before, when it was a regulated monopoly?

On the next slide it shows a few more issues. There were questions about
environmental impacts of a competitive industry, and there were also
interests in consumer response, things like elasticity of demand. When
prices change, how responsive will customers be? Also, how many
customers are choosing new suppliers?

Finally, there is a big interest in being able to evaluate the fairness
of the market when it becomes competitive. Was there really open access
to transmission lines? Was there market power that industrial customers
could have more than residential customers?

So we will be looking at these analytical issues to identify data
elements that we currently collect or perhaps that we should consider to
collect, to make sure that these kinds of questions could be answered in
the future.

The next slide goes into the data elements where in certain of the focus
groups, in many of them, they actually identified a certain item, data
item. What we did here was we decided to catalog all these items into
four broad general categories: Generation, transmission, distribution or
customers, and markets.

What we have done is entered these items into an Access database where
we not only catalog it into those broad categories, but we identify the
focus group that identified the item, any suggested respondent that they
proposed for providing that information to us, the reporting frequency
and dissemination frequency for that kind of data, the level of
aggregation we should use when we disseminate it, and also their
comments on how that item would be used and whether or not it\'s already
collected by EIA.

This is really just a helpful tool for us as we proceed to the next
step, to be able to catalog and identify the specific items and the
attributes that came up in the focus groups.

In the next slide, it shows the items that were identified related to
generation. The state PUCs were the only focus group that really
emphasized an interest in still collecting data on cost, which makes
sense. They\'re still focused on regulation right now.

The other focus groups tend to refer more to price information, although
sometimes they confused the phraseology of cost or price, that when it
was discussed, they really were talking more about price.

Almost all the data elements that are listed under generation, EIA or
the Federal Energy Regulatory Commission or EPA already collect. One of
the main ones where there\'s limited collection is in the area of
emissions.

EPA collects it on a subset of utility generators right now, and we
collect information that allows us to estimate the emissions.

On the next slide, it talks about data related to transmission. Again,
most of the items here are already collected by some Federal agency or
the North American Electric Reliability Council.

The main areas where we do not have data are related to price, and part
of these prices are related to changes that will occur in the market,
market price, congestion prices or a transmission price component of a
retail bill, and cost of ancillary services.

Those costs now were bundled into a utility cost and weren\'t separately
ever reported. Other areas that people identified were: Information on
access to really verify that there was open access; and the media was
interested in injuries, kind of a different sort of data element.

On the next slide it shows the items that were identified under
distribution or consumption. In this area we have information right now
on projected demand for electricity through the North American Electric
Reliability Council, and some information that we collect on demand side
management programs that the utilities run on their expenditures and the
savings; but other than that, we don\'t have a lot of the information
that\'s identified here in detail on load profiles and the price
elasticity related to time of demand.

They are things that the utilities themselves would know, but EIA has
not traditionally collected it in that level of detail.

On the last slide on the data elements, it\'s related to markets. This
is the area that we feel is the most new area for us. However, we feel
it also will require restraint on our part, because this area could be
very expensive and expansive in our data collection without some limits.

This is also the area where Stan was talking about people talked about
real time information and timelines that we just have never addressed
before. However, I think we also need to realize that this kind of
information the market will be providing for itself on the Internet,
through the OASIS system, through futures markets.

So we want to make sure that we exercise caution as we look at data
requirements in this area.

On the last slide, I\'m talking about what are our next steps. We
actually have a long range plan. We label it Electricity 2000 or
2000-Plus, where we have a scheme so that, by the year 2001, we would
have a new package of survey instruments cleared and ready to be mailed
out to respondents.

In this next year, I\'ve just listed some of the steps that we would be
expecting to do during the rest of 1998 and the beginning of 1999. We
will be using the seven objectives that were reported by John Macons in
his paper back in March.

Let me just read those to you to remind you of what those were: That as
we go into identifying our data elements, we want to promote equitable
data collection, use new data sources, coordinate data collections with
other agencies, streamline data collections to reduce respondent burden
and cost, identify and protect confidential data, increase electronic
data collections and dissemination, and implement standardized
reporting.

So those objectives will be overlaid as we start to develop a
comprehensive list of data for collection by EIA.

We will be meeting with other agencies, Federal and state agencies.
We\'ve actually already met with the Federal Energy Regulatory
Commission and the Rural Utilities Service, partly right now in
coordination on our confidentiality policy, but also to open up
communication to coordinate our work on our forms redesign.

We plan to meet first with the state of California and New England,
because they are the states that are furthest ahead in having retail
competition.

We\'ll be identifying the actual data elements for collection and
talking to respondents to make sure that they have the information
available in the way that we\'re thinking about collecting it, and we
will start to do preliminary designs of questionnaires and how we will
disseminate that data. Then we need to make a very serious and detailed
look at the resources we\'re talking about, because we do not have
additional funding at this time. So we have to balance our data
requirements within our budget constraints.

So that\'s the conclusion of my talk. Thank you.

MR. WHITMORE: I think I\'ll restrict my comments to the processes and
procedures. I don\'t feel qualified to comment on what should be the
data items and the priorities in terms of the data items to be
collected.

I think, in fact, you\'ve made a very good start in determining the data
needs and the data sources for electric power statistics in this new
environment of a deregulated industry.

I think the use of focus groups to kind of start that process was a good
idea. I commend you for using that approach. I think it is very useful
for getting a general idea of the kinds of data that can be collected
and the sources from which those data items will be available.

I have heard focus groups described as unstructured interviewing. I
think that\'s what you were doing here, and it works well for these
particular kinds of objectives.

I think one of the important things to make that work was to be sure to
convene focus groups with all of the different constituent groups that
would be important in each area, both different groups of data users and
the different groups of potential providers of the data, and it looks
like you\'ve done a good job of trying to cover the waterfront in terms
of different constituent groups that may be important and provide
important information.

With regard to determining data requirements, I would think you would be
wanting to find out from these folks what kinds of data are maybe needed
to support future policy decisions, what kinds of data are needed to
analyze the effects of past policy decisions, in particular, the primary
thing that you were looking at of how are you going to analyze the
effects of deregulating the electric power industry.

You would want to ask people about the importance of monitoring status
and trends, whether or not the current data series that EIA collects are
important in the new environment or are some of those going to be less
important within the new environment. Are there new kinds of data that
are going to become important that were not previously considered to be
important, and what kinds of trends do people want to be able to follow
as the deregulated industry structure is evolving.

It looks like you have addressed those issues. I guess the other data
requirement issue is how frequently are data reports needed. What\'s
needed on an annual basis or a quadrennial basis or maybe monthly basis.
Again, it looks like that your surveys \-- or focus groups addressed
those issues. It would certainly be very important in terms of figuring
that in with the budget requirements to support what people may like to
have.

In terms of identifying data sources, evidently it\'s become clear
through these meetings that the data sources may be changing from what
they were in the past, and it appears that you\'re finding that the data
sources that are available for particular items may depend on the extent
of deregulation, and that may vary by state.

For example, California seems to be ahead of the rest of the country in
beginning electricity deregulation. So I guess a major issue that these
focus groups are suggesting you have to grapple with is where to obtain
the data, and is it going to be uniform across the country as
deregulation takes place at different rates and in different ways in
different places.

Seems to be a major challenge to determine where you will be able to
obtain the required data on a \-- in a consistent way across the
country.

I would hope that you also obtain some information in these focus groups
on where you might be able to obtain lists of the various entities that
are going to be needed.

You will no longer be surveying just electric power utilities, but
there\'s going to be marketers and generators and other companies
involved, and it will necessary to create sampling frames, develop
sampling designs for the different populations that may have to be
surveyed.

Hopefully, these focus groups have provided information as well on how
you might be able to structure a sampling design for some of these
populations that you\'ll have to get at.

Regarding the next steps that are required, it sounds like that you\'ve
kind of gotten a general idea of the kinds of data that are necessary.
You\'ll be wanting to start developing a specific list of data items and
categorize those by the sources to sort of begin putting them into the
questionnaire context and sets of items that would be obtained from the
same sources.

To some extent, you may be reusing data items from the past. To the
extent that you\'re having to develop new data items to address new
issues, there\'s probably some opportunity there, once you have
developed draft items, to do some one on one interviewing and debriefing
kinds of things, cognitive interviewing types of things to test data
items; and you\'re probably going to be wanting to run some small pilot
tests with industry groups to test new questionnaires, new items.

There\'s probably some need for developing new sampling frames and new
sampling designs.

So at this point I would say that you\'ve made a good start. I look
forward to hearing more about the process in the future.

CHAIRMAN RELLES: Other committee members, please? Calvin.

MR. KENT: I have a series of comments to ask \-- or questions to ask.

First of all, how much effort is going into determining what data is
being desired by the businesses, at whatever level, so that they can be
more effective competitors \-- what I\'ll call business intelligence \--
and how much of the information is actually information that the public
needs to know?

I think that, while those datasets overlap, that at least my impression
always was in the oil and gas that we collected a lot of information for
the industry that was a basic benefit to the industry as they tried to
compete against each other, and we were actually doing it for them as a
free service that they would have otherwise had to have done on their
own and paid for it.

Are we trying to make any \-- or is EIA trying to make any distinction
between those two types of data, the business intelligence type data as
opposed to the public information type data, realizing that in many
instances they are the same?

The next question is a fairly mundane one. Is his part of \-- these two
appendices you gave us are part of a larger report that\'s coming out

MR. FREEMAN: Yes.

MR. KENT: Okay. So this has not yet been released. When will that report
be out?

MR. FREEMAN: In May.

MR. KENT: In May. So this is the May report?

MR. FREEMAN: Yes. There will be on the Web a summary of the focus group
report. A lot of the people who were part of the focus group asked if
they could have copies of that, and we\'re going to make it available on
the Web, or if people want hard copy, we could send it to them or a
diskette, that kind of thing.

MR. KENT: We have the whole new independent power producer, whatever
monograph you want to put on them, out there now. At least last March,
these people were being highly resistant to releasing any information.

The public utilities were taking the position, and have continued to
take the position in their literature, that they\'re being unfairly
treated, because all this information is being gathered by them.
They\'re being exposed to the competitive environment, and here are all
the independent wholesale generators and all these other people who are
not required and are not submitting the data, and therefore, have a
competitive advantage because they can hide behind a certain screen of
privacy that the others don\'t have.

I just wonder how that debate is progressing. There was a question about
what FERC might be able to do in this area, and I just wondered if FERC
had taken any action or had gone forward in the last year-plus to try to
deal with the problems of leveling the playing field between the two
types of generators and, if not, if any attempt is being made to try to
get the private sector or the unregulated sector to reveal some of the
same information that\'s now required.

That was a long question.

MS. O\'BRIEN: We are right in the middle right now of developing a
confidentiality policy toward our data collection in the electric
industry. The idea of level playing field which, I think, is what
you\'re talking about \--

MR. KENT: Yes.

MS. O\'BRIEN: \-- treating \-- if you\'re a generator of electricity,
whether you\'re owned by an electric utility company or a nonutility
company, you should be treated the same.

Those comments we received over and over in our \-- We put out a
[Federal Register]{.underline} notice asking for comments on
confidentiality. Whether the parties felt all data should be
confidential or all data should be non-confidential, they agreed that we
should be treating the two parts of the industry the same.

We will be issuing a proposed policy the end of May that we\'re working
on right now to know where \-- how we will treat the data. We\'re just
in the review \-- reviewing comments right now, but that definitely was
\-- is of interest to us to try to do that, and to move generators \--
the information that generators provide that they provide it \-- Our
idea is that they provide it on the same form, that they \-- whether we
get there in a year or a couple of years from now, it makes sense to us
that they \-- the data is reported in the same way, that it doesn\'t
differ by ownership.

MR. KENT: Do you expect that this is going to look somewhat, at least in
general, like the requirements that we put on the private sector in the
oil and gas area? Is that the direction that you all are moving for
reporting?

MS. O\'BRIEN: I guess I don\'t feel ready to answer that right now, but
on your question also about the Federal Energy Regulatory Commission, we
just met with them this week. They \-- Really, a lot of the data that
the utilities were concerned about releasing, their cost information is
reported not to EIA but to FERC.

FERC really is not moving towards a change at this point. They have
those forms in \-- the Form I in for clearance with OMB, and they\'re
not making a change in their proposal, and they said \-- They were
saying that, really, the Commissioners have not \-- don\'t seem to be
focused on the change right now, because the industry is predominantly
still regulated. So they think, for them, it\'s not time yet.

MR. KENT: Some of us might submit that time has passed FERC by on that,
but I guess I don\'t want that comment on the record maybe. But that is
distressing.

So their Form I is going to still be the same one that we\'ve been
looking at and loving for years?

MS. O\'BRIEN: Well, they\'re in the process of clearing it right now,
but the proposal that went in \-- that [Federal Register]{.underline}
notice that went out stated that it would be treated the same.

Now whether in the end \-- How it ends up in the end, I don\'t know, but
they were telling us that the [Federal Register]{.underline} notice that
went out and they were getting comments back on was proposing to treat
it the same as prior.

MR. KENT: So they still will not be getting much information out of the
\-- for lack of a better term, the unregulated section, despite your
statistics that show the phenomenal growth in that as generators and the
fact that more companies are trying to figure out how to spend their
generating capacity out of the regulated and the unregulated sector.
They haven\'t recognized any of this yet?

MS. O\'BRIEN: Well, the nonutilities do not come under their
jurisdiction for filing a Form I. MR. KENT: Well, that\'s going to be a
problem for you all to try to work out, because if FERC continues to say
the only thing we\'re going to be interested in are the traditionally
regulated utilities, then we\'re still going to have this huge gap of
information, because that\'s not going to be where most generation five,
ten years from now is going to be taking place.

I mean, companies are dumping generating capacity all over the place, if
they can possibly do it, and as your own figures show, most of your
growth in generating capacity is taking place outside, for good reason,
I think, you know, to avoid all of this.

MS. O\'BRIEN: I think \-- That\'s why we met with them, and we want to
continue trying to work with them, because as we identify data needs and
if they will not be collected by FERC, it may be important for EIA to
collect that data.

MR. KENT: Yeah, I think it\'s going to be important for somebody to get
the data, but the same agency, whether it\'s you all or FERC. I know
that FERC years ago always took the position the only data we\'re
interested in is data that can be used in the regulatory process and, if
it can\'t be used in the regulatory process, we don\'t care.

So they really weren\'t concerned about data quality as long as they
felt it was accurate for the particular firm that they were regulating,
which led to problems which I assume you all still have, in terms of
using their data or ours. But at the same time, we\'re being told by
OMB, we can\'t ask for the same data if FERC is being \--

I just \-- I don\'t want to go on with this, but \--

MS. O\'BRIEN: If it\'s a different respondent. We already collect \-- On
the EIA 412 we collect the same kind of financial data that\'s on the
Form 1, but it goes to a different frame. It goes to the publicly owned
utilities.

So I\'m talking about that kind of an idea.

MR. FREEMAN: To comment on the issues that Roy raised in terms of the
next steps with our methodology, we have nearly finalized a work plan
that will address the frames issues and the questionnaire design and
testing issues.

We\'re very aware of the fact that we have to carefully examine the
existing frames for electricity at EIA from two perspectives. One is
there are lots of new actors who we may require information from, and
where are we going to get these actors, and how are we going to update
and maintain those frames.

Part of the purpose of our visit, which will be happening within a month
or so, to California and to the New England area is to find out what
kinds of lists are available, how they\'re maintained by the states.
Will we have access to that information? Does that information have any
volumetric data on it, so if we do go to marketers, we\'ll know is that
15 percent of the market of 70 percent of the market. There are lots of
issues associated with that.

We also plan to do two types of testing on any new or revised survey
instruments that we come up with. One will be a series of cognitive
tests with respondents to see if the questionnaires really do work. Now
these may be mail surveys. They may be through the Internet. They may be
sent out electronically. They may be a combination of those things.

We\'re not at the point yet where we know that, but we will be doing
cognitive testing with respondents, probably in states that already have
deregulation.

Once that\'s completed and survey forms are revised, we will go back to
OMB for a clearance for some pilot or pretesting for the same purpose.
So that when year 2001 rolls around, we will have a set of surveys that
we feel reasonably comfortable with, that will be able to get accurate
information in the deregulated states.

So that is on our agenda, and hopefully, we can come back to the ASA as
we proceed to get comments, both on the testing and on what we\'ve
found.

Now in terms of Cal\'s question about the effect \-- about data for
businesses versus information just for the public, I have not been
looking at the data that way, but I think that would be a useful way to
look at it.

If you look at the information \-- If you look at the data needs by the
type of focus group, it\'s pretty clear that investment brokers, for
example, have a very specific agenda about EIA supplying very detailed
kinds of costs and financial information.

They very clearly said, we need this for advising our clients, as
opposed to public utility commissions, especially in the deregulated
states that no longer are going to be telling utilities how much they
can charge but are very interested in being able to say are consumers
better off or worse off.

So that is the kind of thing we have to look at as we come up with our
list of \-- you know, the bottom line, given the resources we\'re going
to have, is what should EIA collect versus not.

The other thing which Betsy mentioned which addresses both your comment,
Cal, and Roy\'s: We\'re going to look at options for information
collection, and we\'re going to try to attach costs to surveys or costs
to units, so we can have some idea of what we will get for certain
levels of expenditure; because, clearly, spending an infinite amount of
money, you can get an infinite amount of data, but given the rather
severe restrictions on the budget, we have to make those kinds of
tradeoffs.

So we are going to try to do some pretty strict costing before we go out
in the field. I don\'t know if that directly answers your question.

MR. KENT: Just to do one follow-up on that, I think we always have to
remember that the \"d\" in deregulation stands for different rather than
disappearing, because we have a different regulated environment rather
than a truly deregulated environment.

One of the things that surprised me was when you said the states are
only interested in costs, because that\'s what they regulate. Well, the
states that are moving towards deregulation, as I understand it, are
going more for caps, you know, earnings caps, and then they don\'t care
what the costs are as long as you can stay within those caps that
they\'ve established.

That, to me, is the more interesting information, and I think costs are
going to become less and less important to EIA in terms of the data that
you collect, because it\'s not going to really matter in the new
regulatory \-- at least, for generation \-- what the costs are, because
who\'s going to care what the costs are. It\'s going to be dispatched on
a marginal cost basis anyway.

What you really need are the prices at which it\'s being dispatched, and
the cost considerations that FERC has been hung up on are not going to
be nearly as interesting questions to be answered.

I just pass that on, that I don\'t think we need to get hung up on these
cost issues anymore.

MS. O\'BRIEN: I was somewhat misleading in that. Most of the state PUCs
that were part of the focus group \-- California was part, but others
weren\'t, and I think it was reflecting more the fact that their markets
had not gone competitive yet, that I would expect those needs would
disappear, just like you are saying.

MR. HAMMITT: I\'d just like to follow up on that a little bit, in that I
think for many purposes knowing about the cost of electric generation
could be quite important as we think about environmental regulations,
further changes in regulation of energy structure.

Prices and costs are different entities. Sometimes they correspond,
sometimes not. So I wouldn\'t say it will not be of any interest. I
think that would be too strong.

CHAIRMAN RELLES: I appreciate your intention to want to keep the
committee informed. I assure you, the committee wants to stay with this
topic, and I hope that you\'ll send each of us a copy of that report in
May when it comes out.

I know it\'s going to take a long time. This is not an overnight
process, but we would like to be kept up to date on your progress. I
apologize on Phil Hanser\'s behalf. He had a conflicting business
commitment. He really wanted to be here, but I\'ll badger him to review
your report when you send it to him, because I know this is right up his
alley.

It\'s time for another break.

(Whereupon, the foregoing matter went off the record at 11:17 a.m. and
went back on the record at 11:40 a.m.)

CHAIRMAN RELLES: Welcome back from the break. This is our last talk,
efforts within EIA\'s Office of Oil and Gas.

Now it used to be to minimize the impacts of deregulation on respondent
cooperation, but it got generalized a bit, perhaps in an effort to avoid
committee darts, to a milder topic, efforts to encourage response in oil
and gas surveys.

So I do note the change, and it\'s going to be presented by, I guess,
Roy Kass \-- are you doing all the talking?

MR. KASS: No.

CHAIRMAN RELLES: \-- Sue Harris, but I think Antoinette said she wasn\'t
going to be part of it. She\'s your slide person.

The talk was initially sort of conceived as something that will fit with
the future of electric power data, because it gives you some lead
experience in what those other people are likely to experience. So even
though the title has changed, I assume there will still be some
discussion about experience that\'s relevant to electricity
deregulation.

MR. KASS: Jay Casselberry is also one of the authors, and he\'s not
going to be talking. He might answer some questions.

In terms of \-- Well, first Sue looks a lot nicer than I do. I am a doom
and gloom guy, and she is a terribly upbeat woman. My thrust was what
can we learn about the effect of deregulation to minimize respondent
noncooperation, and we\'ll talk about some tales from the trenches.

I actually have a really upbeat tale from the trench I\'m going to share
with you in a couple of minutes. But basically, you have two basic
thrusts in this.

What you\'re going to be hearing about from me is the impact of
deregulation and reorganization within the natural gas industry on
natural gas data. I\'ve come to you and spoken to you several times
about this. I\'m going to reiterate it.

We have learned, because having been there, done that, some things work.
Some things don\'t work. We want to share that with our friends in
electricity.

Then Sue is going to talk about other changes facing oil and gas
respondents over and above deregulation and reorganization corresponding
to that, together with efforts that we and the Office of Oil and Gas
have done to encourage respondent cooperation. Then we\'ve to some
questions of the ASA committee.

Okay. There have been \-- Before we change, there have been changes in
responsiveness in the natural gas surveys, and this is what I\'m going
to talk about. Some of them can squarely be placed at the feet of
deregulation.

Because of changes in the industry, we\'re asking the wrong people the
questions. Because of that, as I\'ll get to in a few minutes, the
answers that we\'ve been getting \-- the quality answers that we\'re
getting, while still good quality to the questions, are less and less
appealing in terms of answering the questions that we think we want
answered.

So some of the differences of deregulation are because we\'re asking the
wrong question, and there are differences because of deregulation also
in what companies need on a day to day basis in order to do their work.

Then some are just general trends in the economy. There is a thrust in
the economy, and Sue is going to get into this, mergers, acquisitions,
divestitures, an emphasis on downsizing, an emphasis on increasing
shareholder value, and where does EIA fit in all that.

Now the next one. What\'s the impact of deregulation on EIA\'s natural
gas data? We\'ve got deregulation and restructuring of natural gas
industry. This is a change that\'s happened gradually over the last ten
years.

There has been a break in the link between the physical flow of gas and
the financial flows within the industry. This is absolutely crucial in
terms of measuring things like pricing, and it\'s the center of why we
have the problem, and what have been the effects of EIA\'s ability to
capture information.

Now I have a flow diagram, actually two of them, one I hope you can see.
The first one is the physical flow where I said there\'s a break in
link. Gas flows really neatly.

There\'s a bunch of molecules that come out of the ground. They go to a
gas processing plant probably. They go to a pipeline. They go into
storage perhaps. They then go to a local distribution company, and then
they go to the consumer. That\'s the physical flow.

The financial flow is much more complicated, because now with
deregulation anybody can buy or sell to anybody else, absent some
consumers who don\'t sell. What we have is consumers, much like was
probably mentioned to you in the electricity thing, can either buy from
their local distribution company or cut deals with anyone up the stream
they want.

There is a developed industry called marketers which took ten years to
develop in the natural gas business, but they are not natural gas
marketers. They are energy marketers. They can jump into electricity,
have jumped into electricity, and they\'re set, and EIA does not go to
marketers.

We tried \-- I\'ve talked to you about a prototype survey that we did
surveying marketers, because we thought they would have the information
we needed. In fact, they don\'t.

What\'s been the effects of this break in the link between the financial
flow and the physical flow? It\'s been a reduction in coverage. Over the
years from \'82 to \'96, which just happen to be years that I had on my
disk, we\'ve gone to more than 80 percent coverage nationally to less
than 20 percent.

We have a very good indicator of the price paid by companies who buy gas
from the companies who physically deliver that gas. That is terribly
uninteresting to most of our customers.

Most of our customers want to know questions like what is the cost of
gas to industrial establishments in a state of X or what is a cost of
gas overall. We know what a high bound is. That\'s ours. We know what a
low bound is. That\'s the price at the wellhead.

We know what a surrogate probably is, and that\'s electric generation
costs, though we\'re about to lose that with electric deregulation. We
know that our estimate is biased. We just don\'t know how much. So the
part of nonresponsiveness having to do with the changes in the
regulatory environment have led to that decrease over time in coverage.

The next slide shows the impact of that second part, the impact of the
fact that more and more companies do not want to play our silly games.
We have a rule. If we don\'t have 75 percent of the gas sampled in a
state on the 857, which is a sample survey, we do not publish
information for that state.

There were two ways of solving that problem. One is to say that our
cutoff date for the sample before publication, which used to be the
first of a month, is allowed to slip. Da, da, da, da, da, the first of
the month, the fifth of the month and so forth, but eventually we have
to get the publication out, and waiting until more and more companies
came in worked for the first few years.

Then we simply couldn\'t slip anymore, and we had to say, okay, we are
going to suppress. What this graph shows is the number of states that
had to be suppressed, because after the slipped date or on the slipped
date we still didn\'t have enough gas reported in the state.

It\'s a problem, and it\'s a problem that is terribly problematic in
some states, because in some states you have one or two companies that
are the state. If they don\'t report, we can\'t go with it.

Now those are the horror stories. Let me share with you a great story
that literally happened this morning. I got in, and there was a phone
message, somebody wanting to clarify questions about the annual survey
that they\'re doing.

That phone message was left at eleven o\'clock last night. The person
works on the East Coast. I figure, okay, she couldn\'t sleep. She had
this thing. She called in, da, da, da, da. So I called her back, left a
message on her voice mail saying, you know, I\'m going to be in the
office this morning, hope to hear from you and, by the way, why did you
call at eleven o\'clock at night.

She called me back ten minutes later. She had been in the office since
seven o\'clock, after having left the office after eleven, filling in
our form. Now if anybody could tell me how I get that mindset into the
rest of my respondents, I would love it.

She is excellent. I had never had contact with this woman before. She,
obviously, is very dedicated, and an ideal respondent. Hopefully, the
information she will give me will be pretty good.

Now Sue is going to take over, talking about what we\'ve done in oil and
gas to help increase or at least to stem the decrease in responsiveness.

MS. HARRIS: Good morning. This is my first time giving a presentation to
this committee, and I\'m very happy to be here and have an opportunity
to meet. I talked to Polly this morning and introduced myself.

Roy has already talked to you about some of the changes going on in the
natural industry, and the oil and gas, petroleum area specifically, are
undergoing changes as well.

As you can see, the workplace \-- they\'re decentralizing. Some are
centralizing. I talked to someone on the phone just the other day, and
this one woman who used to file all of our survey forms, some of the
responsibility is now being shifted to the specific sites for the
refineries, and they\'re trying to have people fill out the information
who are closest to it so that it\'s not going from one person to another
person to another person. They\'re trying to gain some efficiencies.

In some areas, they\'re doing the opposite. They don\'t have the staff
anymore. So now people are taking on additional responsibilities, and so
they\'re trying to do 50 things, and they pulled all this together now.
Now you have one contact, but they have a variety of jobs.

So you\'re competing a lot of times with who has priority. Which survey
comes first?

The mergers that are going on in the industry: We went through a
Marathon and Ashland. We have an upcoming one with Shell and Texaco
going on, and every time there\'s a merger, there\'s a decline in the
on-time responses that come in, because they\'re trying to figure out
who should be reporting, how do you consolidate some of the reports.

A lot of times it doesn\'t coincide with how we have our forms
structured. It doesn\'t work with how their structured. So they have to
figure out how to fill the forms out.

The technology: You\'ve heard about technology today from Dwight and
several people. It changes so fast. What you had six months ago isn\'t
what you have now, and we\'re developing certain applications, and in
some cases the companies are ahead of us, but a lot of times they\'re
behind us.

So we offer ways of collecting the data that they just can\'t do. Some
have access to the Internet. Some don\'t. So we\'re trying to figure out
what the best way is to collect the information from them.

Because of all that, our response rates have been going down. Now this
slide shows just for the petroleum supply monthly surveys. We have some
performance statistics here on the trend \-- as you can see in the red
line there, going down \-- on the amount of responses that we\'ve been
getting in on time.

That includes late as well as those that have not responded at all. A
lot of times the reasons for that is because of the mergers, the
workplace changes, the technology, but we\'re also finding that the
strongest downward trend has to do with refineries and bulk terminals,
the larger companies.

We called the companies three or four times to try and get the
information from them, because our publication date hasn\'t changed. So
we have this window that hasn\'t changed either from the due date of the
form to the timeliness, because we don\'t want to lose the timeliness on
the data, but we have this window where we spend a lot of time trying to
get the information.

Some companies wait for you every month to call, and other ones, because
they\'ve had contact changes or the mergers, we don\'t know who to
contact. That changes, and they\'re not prepared.

There\'s any number of reasons that go on for not getting the
information. We\'ve tried conducting second mailouts, and that works for
the annual surveys, but the ones that we have the most problems with are
the monthly. It\'s the recurring ones.

So we\'ve tried second mailouts, and they do get some responses, because
sometimes it\'s the first time they\'ve seen it, for whatever reason.
The forms maybe get lost in the mail or something like that. So we try
second. Sometimes we even have third mailouts, but again it\'s resource
intensive to do this.

Of course, if you don\'t have the information, you\'re imputing more. So
the quality of the data can also go down.

Now I know that the next question is, okay, so you\'ve got this problem,
and what are you going to do about it? Office of Oil and Gas has 32
surveys, and they range from weekly to monthly to annual. There is one
triennial, and I think there\'s a quadrennial survey, but we\'re
predominantly weekly and monthly.

We took the approach in the past, and now, to partner with the
respondents. I think you get more cooperation with honey and not
vinegar, and so what we\'ve been trying to do is get the respondents to
want to file. So how do you do that?

Well, we developed several years ago an electronic reporting option, and
we call it PEDRO. That was a DOS version. That went out, and the
companies \-- We sent diskettes out, and they were able to load it on
their computers, perform some of the editing so that, if they edit the
data before they send it in, then they won\'t be getting phone calls
from us about errors, and it would reduce the amount of time spent on
processing the information as well.

That seemed to work really well. We got a lot of the major companies
wanting to file on PEDRO, but we still have the little ones. You know,
you\'ll never get rid of the paper, and they still FAX the information
in.

Then we used diskettes. We have an electronic filing system that the
natural gas 176 survey uses, and they send the diskettes out that also
have edits on them. This is an annual, and the companies fill them out
and then mail the diskettes back.

The idea here is to get as much of the front end, putting it to the
respondent to edit the information so that we are not having to call
them over and over again regarding error flags, range checks and stuff
like that, but they can fill the information out online and not have to
fill forms out manually, and it would also reduce their burden.

The next bullet is interesting: Respondent designed forms. You know, EIA
has had a policy to accept forms that look similar to ours, trying to
reduce the respondent\'s burden in filing. In the past that\'s really
worked well.

Companies who ran out of forms or didn\'t \-- couldn\'t file
electronically could create their own forms, as long as they looked like
ours. Well, lately we\'ve been getting in some forms that are very
creative.

They range from being cut and pasted together \-- sometimes they\'re
close to little yellow stickies. They like to use spreadsheets, and
everyone has their own spreadsheets. Unfortunately, when the information
comes in, we are not yet set up for processing that information.

So we end up transcribing and then keying it in, but the companies \--
It does make it easier on them, because in some cases they\'re able to
program from a database and just simply lift the data out, put it in a
spreadsheet, print it off and FAX it in.

I think we still want to maintain that for those that can do that, but I
think we need to start standardizing some of this stuff, because the
information \-- We need a win/win situation here where we can develop
data collection instruments that both make it easy on the respondent and
on EIA to process.

The forms have been placed out on our Web-site, and so they can also be
downloaded for the companies who want to print them off and fill them in
and send them in. We\'ve been conducting focus groups. Natural gas has
been holding focus groups and also conducting site visits to talk to the
respondents, to try and find out what their needs are.

This is not just from the dissemination side on what data do you need,
but we\'ve also been working with the respondents; because a lot of
times, the users of the information \-- they want everything. No one
wants to get rid of anything, but they always want more.

Then you go back to the respondents and say, okay, this is what we\'ve
been asked to get, and they can\'t provide it or it\'s very difficult
for them. So there\'s a balance that we need to achieve here.

The recent efforts that we\'ve been working on: We just went through a
reorganization. The Office of Oil and Gas, as a result of the
reengineering effort that actually started in the winter of \'95 for
trying \-- EIA was taking a look at how to do business more efficiently.

In October of \'97 the Office of Oil and Gas consolidated all of their
data collection and dissemination activities into a new division. By
doing so, we were able to take a look at the diverse methods that we\'re
using in petroleum, natural gas and also in the reserves area.

We have seen a lot of over the years how things have \-- how different
we all are, and we\'re trying to reign all of that in now to try and
standardize a lot of that.

The PEDRO, the electronically reporting option \-- we are now creating a
Windows version. A lot of companies like PEDRO, but they don\'t want to
use the DOS version anymore. So they\'re moving more to Windows, and so
we\'re creating a Windows version and, hopefully, that will be out this
year for those.

We\'ve been customizing some of our material. We took a look at what we
were sending to the companies, and in some cases they didn\'t understand
why they have to fill our this government form. They don\'t understand
what you use the information for. They don\'t know why it\'s needed. All
they know is that they get this thing in the mail, and it says this is
the Department of Energy, you know, very standard language and
everything, and you are required by law to file this form.

We\'re trying to take the approach of informing our respondents on how
the information is used, to try and encourage them to report. The first
one that we worked on was the 857.

For the 857 we sent out a brochure, and the brochure is available on the
back table. As Jay pointed out earlier, brochures have been working very
well. You can put them in your pocket.

We sent this one out to the respondents to tell them how the 857 is
being used. We also changed the cover letter that went out. We
customized the letter to account for the company\'s degree of
responsiveness. Not all the companies are as late as some others, and
not all the same. One size doesn\'t fit all.

So we customized the letters, and we also offered copies of
publications. As a survey respondent, you are, of course, entitled to a
free publication.

In October of \'97, there was a workshop that was conducted, and this
was sponsored by the Statistics and Methods Group. The purpose of that
workshop was to try and redesign, offer suggestions on how our survey
forms can be more user friendly, clearer, and how we can make it easier
on the respondents to files.

Some of the things coming out of that workshop were suggestions on for
the annual surveys to do reminder notices, and after a year the contact
person may have changed, and if we send a reminder out a few months
before the survey comes out, then it sort of alerts them to the fact
that something is coming in the mail.

That one, because of the timing of the workshop, we weren\'t able to
implement, but we are planning on doing that for the next mailout, which
is in the fall of this year.

The cover letters, as I mentioned before \-- We\'ve redesigned the cover
letter. So that in the cover letter we thank the companies for their
participation in the surveys, and explain to them the uses of the
information.

We also send them to our Web-site so that they can go out there and see
how the data is being used. They can get copies of the publications and
see immediately, rather than getting the hard copy. So we sent out \--
We give them the Web-site address, and we also sent out our little quick
reference guides to them this year.

We worked with the Office of Management and Budget, and primarily Jay
Casselberry was responsible for that. We got approval from them to make
changes to the survey forms. To us, this was a very big improvement,
because, you know, when you design government forms, there\'s a lot of
regulations where this has to be in this form, and this has to be down
here.

It has to \-- There\'s a standard look and feel to some of the
information that has to be on the form, the confidentiality statement,
certifications, OMB numbers and everything.

A lot of times it hinders the design of the form, because your focus
needs to be on the information you want to collect, not on all the
regulations that are on the form. So working with them, we were able to
get approval from them to redesign the forms. So that was a big plus.

As you\'ve already heard from Dwight and from Stan and Betsy, we are
going to be doing cognitive testing on our redesigned forms in the
future. We\'ve not done that before, and we are going to pursue that.

The future efforts: We\'re in the process of looking for some forms
design software. Our forms are now designed in Corel Draw. Some people
use WordPerfect. Some people use Ventura. We still have some forms on
boards that we send to be printed.

Not all of them are available electronically. So even if a company
wanted a form to fill out, we couldn\'t give it to them. So what we\'re
looking now at is forms design software that is user friendly so we can
make changes easier, more cost effective, and also that, once you design
the paper copy, it also can become an electronic form.

You can place this electronic form out on the Internet so that the
company could then download it, fill it in, and send it back. There are
a lot of issues surrounding that. People still have concerns about the
security issue of the data and everything, encryption.

Some companies have even asked for E-mail. They just want something.
Send it in the mail. I\'ll send it back. They don\'t care about the
security issue or anything. It\'s no concern to them. They just want
something that\'s easy to fill out, that they don\'t have to transcribe.

So we\'re looking at forms design packages. We\'ve looked at something
called Jet Form. We\'ve looked at Forms Tool \'97, and we\'re in the
process of getting Omni Form. All of these are electronic forms and
paper \-- they can create paper versions as well.

So we\'re in the process of looking and standardizing all of our forms.
Once we redesign the forms and the instructions, we\'re going to be
going through the cognitive testing to make sure that it\'s very clear
to the respondents what we want them to fill out.

The forms currently don\'t flow very well. Some of the questions are not
very clear on them, and the instructions don\'t even match the forms in
some cases. So we need to do some work there.

A lot of this work is going to be done, I would have to say, probably
the beginning of \'99. This year we\'re trying to find the forms design
software, and then we want to concentrate on getting everything into a
centralized area that people can access. Then we\'ll take on redesigning
the forms.

Now we have several questions that we would like to ask the committee
for assistance on. We\'d like to see what parts of all of this that
we\'ve been doing in oil and gas can be applied to the electric power
survey, whether it\'s the forms design software or reminder notices or
second mailouts.

We\'re just not sure how much can be applied to their surveys. What
other activities can we do to encourage respondents? We\'ve tried to
partner with them to make it as easy on them to file the forms as we
can.

In some cases, as I\'ve mentioned, they\'re sending in cut and past
versions of forms, but we think what we need to do is find out what
they\'re using, and then come up with a standard spreadsheet, offer them
different types of formats that they can use, but standardize them so
that they\'re not all creating 100 versions of something.

How can we evaluate our efforts to address the response problems? Does
anyone else know of any forms design software? We\'ve been out trying to
search for some cost effective software to evaluate, and if anyone knows
of any other packages that we could be looking at, that would really be
helpful to us.

That\'s the end. Thank you.

CHAIRMAN RELLES: Thank you.

MS. PHIPPS: Thank you. Nice presentation.

I want to also point out that I\'m new to the committee, and I don\'t
have a long knowledge of history of EIA and the survey. So you\'ll have
to take that into context as I speak.

I think there\'s a number of challenging survey issues that were
discussed in the paper and presentation involving nonresponse,
timeliness of response, and completeness of response, and the paper
outlines different methods that the staff have used to address the
issues, including alternative methods of data collection, rapport
building with respondents, organizational restructuring within EIA, and
also the exploring of survey and forms redesign.

The problems facing EIA are similar to those facing all other agencies
collecting data from businesses, in some sense, due to downsizing,
basically time problems in completing the surveys, perhaps greater
respondent burden as some companies just get so much \-- so many surveys
and so much government paperwork.

I think there\'s a negative opinion of government and government
agencies that contributes to problems, and perhaps lack of understanding
of importance of the surveys or the respondents may just simply not
care, one way or the other, about the importance or not.

I don\'t know if it\'s a special problem in EIA. Probably facing other
agencies also is the industrial restructuring and how it\'s affecting
data availability. So your respondents don\'t have a lot of time. Some
of them don\'t have a great interest, and sometimes there\'s no data out
there.

I think the no data problem is a qualitatively different problem than
time or lack of interest. Basically, the inability of respondents to
report the price data, as shown by the dramatic drops in reporting, is a
series problem, and the options there would be probably first to assess
if the current respondent can possibly get the information.

Some of your respondents are providing price data, and maybe they\'re
just \-- I\'d like to know something about that group, the ones that are
providing it. It may not help in the long run, but knowing something
about why that group is able to provide the data, the price data, that
are would be important.

There may be \-- have to be some reassessment of the respondent in
general, assuming here you\'re still trying to get the exact same data
item. Is there another appropriate respondent out there besides the
actual respondent you\'re going to now in the transportation area?

You may also, it sounds like, as in electrical, have to reassess how the
data are collected. Basically, you may have to get the information from
slightly \-- get the information slightly differently, and go to a
totally different type of respondent.

So I think a systematic follow-up of those who do provide the price data
and those that don\'t may be useful at this point, focusing on the
location of records and also the availability of the data.

The overall nonresponse and lack of timeliness, the respondent
cooperation problem, can be addressed in a number of ways, I believe.
Basically, what I\'m going to do is focus now on the questions that the
group asked of the committee, not exactly in the order that they asked
them, and I\'m also going to focus on monthly and weekly surveys and not
on annual surveys; because I think there are just different methods that
you use for annual surveys as opposed to these ongoing ones.

The first question is should OOG be doing things differently or
including other activities to address late and nonresponse problems?

First, let me talk about the mode of data collection. I didn\'t really
see any breakdown of good, late and nonrespondents in their paper by
their reporting method, but my suspicion is that mail is really slowing
down the survey process here.

For monthly surveys, I think that mail is a costly and less efficient
way to go. There\'s a high and increasing cost of mail and a lower and
decreasing cost of telecommunications. I also think that mail doesn\'t
convey that timeliness is essential.

I would encourage you to move yourselves and respondents to other forms
of data collection as opposed to mail. You talked about the electronic
reporting and the E-mail disk. It wasn\'t clear to me here if those
kinds of things are being sent through the mail, you know, if they\'re
getting the disk by mail and sending it back by mail rather than sending
it by E-mail.

I know there\'s some confidentiality problems, but I would encourage you
to experiment with using E-mail to send these diskettes back rather than
using the mail.

Another possibility would be to emphasize the timeliness nature and the
need for data quickly, would be to do automatic reminder notices a few
days before the data are due and possibly a few days when the data
becomes late, and these might be sent out either by FAX or through
E-mail, similar to prenotification in annual surveys has been shown in
the business setting to increase response rate.

Also, here you\'re getting multiple contacts with the respondent, which
has also been shown to increase response rates in business surveys. So I
think that that could be an important thing. You would have your List
Serve and just send out blanket notices about \-- reminder notices.

I think that FAX may convey \-- or does convey timeliness is essential,
both by FAXing the survey out to the respondent and having respondents
return it by FAX. I know that you\'re exploring Jet Form, and that\'s
\-- I don\'t know. I\'ve seen a demonstration. I\'m not certain how well
it does.

I mean, I think it involves an investment in technology here, basically,
the automatic dialing and also the automatic reading of responses when
they come back in, and how well that\'s been tested at this point \--
Also, probably a redesign of the form.

I think there\'s evidence that when you FAX out survey forms, they need
to be in very kind of bold letters for readability purposes.

Basically, I haven\'t \-- Touchtone data is another option which is
fairly widespread. I think it puts a little more burden on the
respondent in calling in, and no one likes to punch in all those numbers
and give you the information. Your forms may not be well suited for it.
You need a few data items only to be reported using it, and that might
be different.

I\'m also a little bit skeptical sometimes of the response rates shown
by Touchtone, because I think there\'s a lot of CADI follow-up that\'s
not reported in the Touchtone response rates from certain agencies.

Also, I guess I didn\'t talk too much about CADI. It\'s certainly, you
know, more expensive than other electronic methods. You have to have
your staff on board. The other question I have on the CADI systems is
that there\'s different surveys that were done on quality of data, and
shows there\'s much more reliance on memory in reporting over the
telephone as opposed to records, which is something you probably don\'t
want to encourage.

So I agree with giving respondents a choice of methods. However, I think
maybe \-- I\'m not certain I agree with, you know, all the different
types of forms that they are able to. I think you do have to really
standardize data and give them a choice of ways to respond, but I\'m not
certain I would do it by a lot of different instruments.

I also would try to track the respondents, the mode that they\'re
reporting by, to assess the success of the methods. Sometimes then you
also just have to make a more systematic effort to move people off mail.

New people don\'t come in, and they just aren\'t allowed the option of
mail to start out with. Old ones are moved off of it in ways, and I
think it\'s just an expensive and \-- hard to get off of it, but it\'s
an expensive and inefficient system.

In addition to electronic or different modes of data collection, rapport
building was used to \-- as a method to work on nonresponse and lack of
timeliness. The brochures, I think, are a good idea, incentives.

Also, I just would caution here more testing to know if these types of
things really make a difference. When I saw the incentives that were \--
or what I call incentives \-- the introductory copy of the publication,
I would talk about a quasi-incentive. I mean, I don\'t think payment is
possible for government surveys.

So, basically, the postcard that was sent out made me a little
suspicious that I was being drawn into being a paid subscriber to a
monthly publication. So \-- and also, I think, to increase response
rates it\'s been shown that incentives have to be sent with the survey
and not after the fact. So I really think they have to be sent up front.
If you want to send the publication as an incentive, send it up front.

The problem, I think, with rapport building, is what \-- in business
surveys is what do we really know about the respondent and respondent
motivation. Basically, you may have someone in a larger company in an
accounting clerk type of population or someone in a higher position, and
then there\'s differences in respondents by the size of the business.

So the more we know about the respondents, the better able we are to
assess what incentives might work. I think it\'s really difficult, and
there\'s not a lot of research out there on how to, you know \-- on
different types of respondents and what might motivate them.

This is also true just about nonresponse in establishment surveys also.
How to approach companies and how to gain cooperation is not very clear,
and it\'s something that would be worthwhile looking into.

I mean, do you go in at a higher level and assess cooperation, and
there\'s research out there in other settings that says, you know, you
need high level cooperation for \-- you know, the group underneath will
follow, but also I\'ve seen instances where companies set kind of
general policies that they are not going to let people in to do their
surveys, but sometimes you call the department, the specific department.
They may respond.

So I don\'t know. There isn\'t really a lot of research out there on
what\'s the best way to go about this, and it\'s something you\'ll have
to consider.

A couple of things that are probably not really rapport building, but I
would emphasize the mandatory nature of the surveys. I mean, that\'s
shown to increase \-- you know, to be definitely tied to response rates.

I also just as I was listening to you, I don\'t know if there\'s some
kind of research that could be done on, you know, what to do in cases of
mergers and instructions you might be able to give out to respondents in
these kind of cases. This is a problem.

So the next \-- One of the other questions was should EIA be conducting
cognitive interviews and focus groups, and I would say definitely yes,
particularly in terms of forms redesign.

I think the earlier methods we talk about, about different kinds of
incentives and how to increase response rates, are really well done kind
of in split sample tests, and I would definitely encourage you to do
that, be able to \-- you know, to really see if things work or not, and
perhaps to be able to assess the cost effectiveness.

I think that split sample tests are also good for field tests of
different instruments that you may have worked on earlier with cognitive
\-- different types of cognitive interviews, and come to the conclusion
that you want to test two different instruments \-- very good, but
cognitive interviews and focus groups, to me, have been most useful in
the redesign work at early stages, basically assessing measurement
error, watching people go through the surveys, seeing the potential
errors.

Also, working that may be what motivates people to complete the forms.

Just thinking about redesign in general, I think the household
literature sometimes isn\'t directly applicable to business. So you have
to be careful of that. There is document design literature out there,
which I think is helpful in knowing how; but some of the things that
I\'ve thought about is, when I was working on these redesign issues, you
do have respondents here that are reporting regularly, monthly, annually
surveys.

You\'re basically changing the form for them, and they need to receive
notification probably that the form is changing. I mean, it just could
be problematic. They\'re used to getting these month after month. They
use the old form. They may pull out the old form to look at as they
answer the new one.

There\'s also some research that\'s been done by the Census. I know
you\'ve been working on the total design method, and that\'s primarily
booklet focused, and I think that could tend in establishment surveys or
all surveys to lengthen the survey.

I know the Census of construction tested some booklets at one point, and
I have articles on them, and they rejected that kind of booklet notion,
and it may not be directly applicable here, but there was more item
nonresponse.

I don\'t think the response rates were too much different, but \-- So
there may be something about government forms and how they look to
businesses that radically changing the structure of them, you may have
to \-- you need to think about.

I would argue always to test, you know, your existing forms against your
newly designed forms, so you have a base as you redesign forms.

How should EIA evaluate current and future efforts to address the
response problem? I think exactly like you have. You\'ve been looking at
things like timeliness, overall response rate, item nonresponse rate;
and if you can add in there \-- and they\'ve talked about this some \--
looking more at the costs and how that \-- That\'s something that should
be taken into the evaluation.

I can\'t tell you any other software packages. It might be worth, though
\-- I don\'t know if there\'s been something recently over the APOR net,
which is the kind of List Serve for a group that we could send out a
question. Questions go out all the time.

There\'s been a big thing recently on Web surveys, and maybe you could
send out a question on forms design software packages. I don\'t think
there\'s been something come across on that.

The lessons to be learned from the efforts from oil and gas to encourage
respondent cooperation: I think still probably the jury is out on the
letters and publications. Basically, you need to follow the units and
see how their responses to your telephone follow-up and to your
materials, the brochure and the other materials, are related to their
past and future response.

I think overall, you know, there\'s just been a lot of work done here in
how to go about increasing response, and I think you always have to look
at the lesson to be learned as you\'re building knowledge in the agency
on nonresponse, investing in learning, and the more you can contribute
and pull together, and I think the organizational restructure is a great
issue. You can pool all your resources and figure out the best ways to
go about a very difficult problem.

MR. SUDMAN: I have one short comment in terms of improving cooperation.
The key factor, seems to me, is a recognition within the firm that the
data are of benefit to them.

Now it seems to me, it\'s worthwhile to try to \-- especially for
important firms that have been reluctant, to discover whether indeed
there is someone within that firm using the data. This you could do from
your list of customers.

If that\'s indeed the case, it seems to me it would be really useful to
get that user within the firm \-- write that person and say, look, will
you help us out; we need the data from your company, would you go and
tell them how useful the data are.

You know, that\'s not a sort of a mass mailing. It\'s a one to one sort
of a thing, but it can be very, very effective if indeed these are
important companies from whom you need data.

MS. CRAWFORD: I wanted to elaborate on a comment that Polly made.
Actually, it concerns slide 6, which is the states with data suppressed
in natural gas consumption because of nonresponse.

I looked at the data. They\'re pretty cyclic over time. I was wondering
if maybe you could know a little bit more about the characteristics of
the states that weren\'t reporting. For instance, March looks like a
pretty bad time of year for them to report. This is the slide I\'m
referring to.

So I had a couple of questions. What is it that\'s going on either in
this industry or with the timeliness of your surveys that causes it to
be so cyclic like that?

Then the other question I had is what are the characteristics of, say,
the eight to ten states that are not responding, say, around March or
so? Is there anything that perhaps they may have in common that could
give you some insight into altering your methods and perhaps increase
response?

MR. KASS: Let me start with the last point first and get around to
thanking Polly for her very extensive and useful comments.

Okay. It is cyclic with a trend, and thinking in terms of states is one
thing, but really we\'re talking in terms of companies, which gets me
back to Seymour\'s comment.

At certain times of the year, the companies who are responding have a
whole lot of regulatory things that they have to do, and in terms of the
bottom line, they have to put their SEC filing forward or they can\'t
stay in business. You know, they have to put their state regulatory
things forward or they can\'t stay in business.

What goes to the bottom of the stack? EIA. And it so happens that the
first few months of the year is when they\'re putting their books for
the past year to bed. So, yes, good insight.

Remembering that we suppress states because the companies don\'t
respond, we have big companies that operate in more than one state, are
crucial in more than one state. When that poor stuckee is overwhelmed by
stuff that\'s necessary for the company, we get walloped. That\'s
what\'s going on there.

The point about using a friend in the company as a lever, we\'ve done
that. We find out from the NEIC who our subscribers are, and it has been
successful. You know, when we have a problem with XYZ Company and we
find that there are a couple of people in the research department, a
phone call either to our respondent saying so and so in your research
department \-- you know, I lie \-- you know, really uses these data or,
conversely, going to the research department saying, look, we got a
problem; I know that you use our information. I know that you probably
focus on your state, and you know, we\'re not going to be able to go
forward with it.

That works when we got friends in the company. What we\'re finding with
retrenchment is that we can\'t find our friends in the company anymore,
you know. They\'re probably there, but we can\'t locate them on our
subscriber list.

The subscriber list goes to a library. We call the librarian, and they
say we don\'t know who uses it, you know. We just get it in. So that
lever is disappearing.

Okay. Some loose ends of what Polly suggested. One point is the fact
that our respondents don\'t have the data, and I emphasized when I was
talking that that\'s a result of deregulation.

CHAIRMAN RELLES: That is actually a fire drill. I\'m told we don\'t need
to respond, although I would add, if we see any elementary school
teachers come in \-- But it was a 12:30 fire drill. So we\'re okay, but
actually we are a little pressed for time.

MR. KASS: Okay. Then I\'ll just in very short measure say, yeah, we know
that there are problems with the companies not having the information.
We know that there is a different universe out there, marketers and so
forth.

The folks who write the bills, the folks who pay the bills know what the
information is. We tried going to the folks who write the bills, came a
cropper because they don\'t have the information.

We\'re talking very generally about scoping out what would be involved
in going to the folks who pay the bills. That would be an immense
effort.

The nonresponse problem \-- Frankly, the nonresponse problem is not due
to the fact that we rely on mail. It\'s due to the fact that the
companies are not putting the information together on the right desk at
the right time.

Once they get it together, they can fill in a form and FAX it in.
That\'s not where the bottleneck is.

Rapport building \-- We try things. One comment about having putting a
publication in the packet \-- at one point we sent publications to all
our respondents, and we found that they didn\'t want them, especially if
you have one contact person who is responsible for a whole bunch of
publications.

They would get a whole armload and they said, we don\'t want them. So we
stopped that. Along with stopping that, we seemed to have lost the
message that, as a respondent, you can get one if you ask. There\'s a
balance there.

In the spirit of timeliness, I thank you again for your comments.

CHAIRMAN RELLES: My apologies for sort of not keeping my eye on my watch
and trying to end this on time. The comments have been really excellent
and, Polly, I thank you. Of course, that added more time, but that\'s
\-- The tradeoff, I think, is worth it.

We\'re actually now at the end, and I\'m supposed to invite comments and
questions from the floor. It\'s one good effect of running over.

I would like to \-- just another two minutes. There\'s a proposal for a
certain meeting date. Based on the scheduling constraints here in
Washington in general and this hotel in general, we\'d like to have the
next set of meetings October 22nd and 23rd 1998, and we would even like
to have the following meeting scheduled to be April 8th and 9th 1999.

Anyone in the room have a problem with that?

MR. KENT: What was the first one?

CHAIRMAN RELLES: October 23-24. I\'m sorry, 22nd and 23rd. All this is
on the record, too. The 22nd and 23rd. Greta?

MS. LJUNG: I have a conflict with that time.

CHAIRMAN RELLES: Okay. Maybe we can talk offline. Then let\'s not
consider it definite, but let\'s \-- Are people otherwise available in
October or November? Could I hear \--

MR. BISCHOFF: I have one thing, but I don\'t know what group and when it
is. I think that was October 8th or 9th. You wouldn\'t want it that
early anyway.

CHAIRMAN RELLES: No, I think that\'s too early, though we would consider
perhaps the later weekend in October or an early weekend in November. I
know Bill would like us to \-- I guess it\'s easiest if we do the 22nd
and 23rd, but I \--

MR. WEINIG: I\'m not sure that we have any options. We can \-- Our
option would be to go perhaps to DOE facilities, which causes other
difficulties, but we would be willing to do that.

CHAIRMAN RELLES: Let\'s just first try to get a sense of conflict. So we
have a conflict on the 22nd and 23rd. Do we have any other obvious
conflicts here at this point? Then we\'ll go ahead and try to schedule
it offline. So there\'s flexibility for the end of October or the
beginning of November?

I will \-- Okay, well, I will assume then that the 22nd through the 23rd
of October through, let\'s say, the 13th of November are currently
feasible dates, and we\'ll see what we can do.

With that, I want to thank everyone.

MR. KENT: When was your April date?

CHAIRMAN RELLES: April 8th and 9th.

Again I hate to schedule things a whole year in advance.

With that, I think I\'d like to schedule the meetings to be closed. I
thank everyone for attending from EIA, especially all of the wonderful
EIA presenters, and our hosts, Linda and Jay and, of course, the
committee.

I think you guys have done wonderful work for us, and I appreciate it
very much. I believe EIA appreciates it, and there will be follow-up
correspondence to ensure that you feel like your comments were carefully
considered, not necessarily followed. That\'s not what we expect, but we
do want to make sure that they get into your debates.

So, thank you.

(Whereupon, the foregoing matter went off the record at 12:41 p.m.)

\- - -