AMERICAN STATISTICAL ASSOCIATION

COMMITTEE ON ENERGY STATISTICS

MEETING

FRIDAY

NOVEMBER 20, 1998

The Committee met in Room 1E-245 of the Forrestal Building, Department
of Energy, 1000 Independence Avenue, S.W., Washington, D.C. at 8:30
a.m., Daniel A. Relles, Chair, presiding.

[PRESENT]{.underline}:

Daniel A. Relles, Chair

Carol Gotway Crawford, Vice Chair

David R. Bellhouse

Charles W. Bischoff

Jay Breidt

R. Samprit Chatterjee

Greta M. Ljung

Polly A. Phipps

Seymour Sudman

[ALSO PRESENT]{.underline}:

Lynda CarlsonBob Jewett

Mary CarlsonRoy Kass

Jay CasselberryInderjit Kundra

Dave CostelloM.T. Lawrence

Ramesh DandekarNancy Leach

Stan FreedmanRei-Pyng Lu

Dwight FrenchRenee Miller

Joan HeinkelLarry Pettis

Bill Weinig

I-N-D-E-X

Page No.

Opening Comments from the Chair

Dan Relles 3

A New Natural Gas Imports Model for STIFS

Presenter, David Costello, (EIA) 4

Discussion: Charles Bischoff, ASA Committee 26

Discussion: Greta Ljung, ASA Committee 30

Procedures to Accelerate and Improve Natural Gas

Estimates: Methodology and Progress to Date

Presenter, Roy Kass, (EIA) 45

Presenter, Inder Kundra, (EIA) 51

Discussion: Samprit Chatterjee 58

Discussion: Carol Crawford 62

Alternatives to Reducing the Cost of RECS: A

Response to ASA Committee Suggestions from the

Spring 1998 Meeting

Presenter: Dwight French, (EIA) 86

DIANA: Disclosure Analysis Software

Sensitive Information in Tabular Data

Presenter: Bob Jewett, (Census Bureau) 114

Presenter: Ramesh Dandekar, (EIA) 130

Closing Remarks 141

P-R-O-C-E-E-D-I-N-G-S

8:40 a.m.

CHAIRPERSON RELLES: Welcome to the meeting of the ASA Committee. I think
I\'m supposed to make a couple of small announcements this morning.
First, any EIA staff or members of the public who was not present
yesterday, please introduce himself or herself.

MS. HEINKEL: Joan Heinkel.

CHAIRPERSON RELLES: Did you get that?

COURT REPORTER: No.

CHAIRPERSON RELLES: Say it again.

MS. HEINKEL: Joan Heinkel.

CHAIRPERSON RELLES: I need to announce that lunch for the Committee will
be held at the conclusion of this meeting across the hall. The NEIC
provided several sample publications for us to look at and take home if
we want, and those are sitting on the table on the right as well as
outside in the hallway. Bill indicated that he hadn\'t gotten back all
the votes for potential dates for the spring meeting, so please make
sure that you get back to Bill on that. And then there\'s also some
sheets over there where you can check off items you might want to get
from the NEIC.

Today is a day where we\'re going to have four essentially technical
talks. I\'d like to introduce the first speaker who\'s going to be
giving a talk entitled A New Natural Gas Imports Model for the
Short-Term Integrated Forecasting System. The speaker is David Costello.

MR. COSTELLO: Good morning. I don\'t know whether anybody else used this
technology yesterday, but I kind of favor it so I\'m going to use it.

Now that we\'ve kind of gotten rid of all the mundane topics like global
warming and stuff like that, I thought maybe it would be a good idea to
get something that\'s on everybody\'s mind which is gas imports in the
United States. This is the title of the work. I put a question mark at
the end because basically I think we have some more things to do before
we have something that will be what we\'re really after. But there are
some interesting things that came out of this work.

So the purpose of doing this, there are several reasons. We have a
meeting every month to talk about the gas forecasts and the data, and in
at least one or two of those meetings the issue came up about how we
seem to be in short-term forecast a little bit \-- we don\'t have a lot
of detail. We do not currently have a lot of detail on imports, and we
may be over-predicting imports because we\'re not taking in account some
of the things that are happening with regards to gas import capacity.

So I basically decided that we probably ought to take some of the
available data on capacity and see if we could do something that sort of
integrates the import sector a little better and perhaps improve the
accuracy and give us the ability to talk about imports a little bit more
intelligently. So that was the main reason for doing it. I think a few
interesting things do show up. We probably have to refine it a little
bit more than it is now.

CHAIRPERSON RELLES: Excuse me. is it possible to measure the flow of gas
coming in?

MR. COSTELLO: Yes. It is. I mean basically every month the flows are
reported essentially to FERC and the Fossil Energy Office in DOE keeps a
detailed database of the flows by pipeline, and they\'re recorded. We
have those. There are a few wrinkles in the data, but we\'ve made use of
those here.

CHAIRPERSON RELLES: I thought the whole goal here was to measure. I
thought your goal is to estimate flows.

MR. COSTELLO: Right. That\'s correct. That is correct. We are going to
estimate the gas import flows. That\'s the main thing. Currently we
really just have a linear function that is time and seasonal factors to
estimate imports to add up the supply components to get a balance in the
short-term outlook. So it\'s pretty simple. I forgot to put the equation
in the back of the paper.

By the way, the paper that I\'m discussing, I think what you have is an
incomplete version.

CHAIRPERSON RELLES: We just passed out the full version.

MR. COSTELLO: All right. Well, a more complete version is available. So
it\'s a very simple relationship. It\'s got an AR(1) term on it but the
fact of the matter is that we just estimate imports over the historical
period and project them out as a way of getting that component of
supply. It has a tendency, as you see \-- it fits reasonably well but
the thing of it is it doesn\'t have a lot of information in it and it
tends to exhibit some bias in the forecast.

But to just look at the recent record of what\'s been happening using
that model, I\'ve shown some of quarterly average forecasts \-- we do it
monthly but I\'m just showing some quarterly averages for recent
history. Beginning with the third quarter of \'97, you see that we did
in fact over-predict it. Part of this, of course has to do with the fact
that you\'re always assuming normal weather and that has something to do
with it. But nevertheless, at least up until the second quarter of this
year, we were over-predicting. This is sort of when we realized we
really ought to take some time to try to do something about this.

With the current model we found we had to analyze how it\'s doing and
try to compensate for it in the forecast using some judgmental factors
which we over-applied in the second quarter, I\'m afraid. But you can
see it\'s a little bit too difficult to deal with it this way. And
hopefully we can make more sense out of the whole thing.

If I just simulate the current model over historical period beginning in
1989 this is what you get, and there are some significant places in
which we don\'t do a particularly good job.

DR. BREIDT: Excuse me. Are these one step ahead forecasts?

MR. COSTELLO: No.

DR. BREIDT: It\'s extrapolating out the whole \--

MR. COSTELLO: No. It\'s beginning \-- Well, there\'s nothing dynamic
about this particular model anyway, but it starts here and just goes
out. It\'s so that the end point is a several year ahead forecast.

MS. LJUNG: Are those end forecasts or are they fitted values?

MR. COSTELLO: Pardon me? No. This is actually a simulation, the
difference being that to the extent that there\'s any lag variables
involved, the solution values to the lags are fed into the next step. As
I say, for this particular model, that doesn\'t mean very much because
there\'s just the deterministic trend in there anyway. So it\'s more or
less like fitted values except for the AR(1) term, but that doesn\'t
really matter.

It\'s clear that we did over-predict as we got towards 1997. That\'s
when we started perhaps worrying about it a little bit and it\'s when
some other people noticed that well, you know, you\'re saying you\'re
going to get these imports but we don\'t really think that the capacity
is there for that. We have been keeping track of capacity but mostly in
the aggregate to make sure that we don\'t generate a forecast that
exceeds aggregate capacity. However, the problem is that just doing that
on an aggregate level pays no attention to possible capacity constraints
in some of the regions.

So in order to try to do this, the first thing we did was say, Well,
let\'s just sort of look at a picture of what imports look like. And
essentially there are five import regions in the United States for all
intents and purposes. The western region, which has two main lines that
go this way, what we call the central region. By the way, this chart was
stolen from *The Natural Gas Annual*, so credit them.

This really has two pieces. One that comes in largely through Minnesota
and another part that comes into Chicago. I guess the Chicago/Detroit
area. And the other one is the northeast. Those are the main points
where Canadian gas comes into the United States. There\'s also a little
bit from Mexico but for the rest of the paper, we decided \-- for this
particular analysis we\'re not really paying attention to that and for
the time being we\'re going to just assume something about those that
they\'re exogenous in this analysis.

There\'s also LNG we export a little bit out of Alaska but we do get LNG
into Massachusetts and to Louisiana. That also we\'re sort of leaving
out of the analysis. So we\'re really just focusing on Canadian gas. So
we\'ve got really four main regions that I\'m basically looking at in
terms of trying to do something with the capacity and flow data.

Just to give a picture of what the flows and capacities look like, I
plotted them here. I really didn\'t get hold of capacity data before
late 1990, so in the previous period I just assumed that the maximum
observed was capacity. I need to go back and see if I can\'t get some
more data, but for the time begin that\'s what I did. Also there were
some instances, that\'s the northeast region \--

CHAIRPERSON RELLES: So the upper curve would be how much gas would be
going at full capacity the entire time?

MR. COSTELLO: Right. One hundred percent capacity is the dark line. The
dashed line is the actual flow reported through that region.

I don\'t know where that came from.

PARTICIPANT: Bill Gates.

MR. COSTELLO: It\'s not an X-rated display, I guarantee. Can you give me
any help? Do you know what\'s going on here? It\'s kind of blanked out
on me. So I\'ve got several charts that sort of just describe what that
data looked like. Since it doesn\'t look like I\'m going to have the
display \-- But if you look at the flows versus capacity by each region,
there are some different characteristics and in some regions there
appears to be times, particularly recently, where capacity has really
been a constraint, particularly the central region which is that second
region from the left that I mentioned.

CHAIRPERSON RELLES: What figure number is that?

MR. COSTELLO: That\'ll be Figure 4. If you look at the way the flows
were looking in \'95 and \'96, there was a fairly extended period \-- I
mean they often bump up in peak periods but this was a fairly extended
period when, by my reckoning, that perhaps the capacity situation there
was a little bit tight.

DR. BELLHOUSE: How was capacity increased? Was that increase in
pipelines or something?

MR. COSTELLO: Well, capacity increase would be either an expansion of
existing line or a new pipeline branching off of some other major
pipeline would be added. That data is recorded. We get that also from
FERC as well. Normally, the stuff is assumed to be going in place at the
end of the year which is why a lot of these charts show jump ups at
around November. I\'m not really clear that in fact effective capacity
really always comes in November. In fact, some of the flow data did pop
up above the rated capacity so in those cases not to dwell too much time
with data, in those cases, I actually assumed the capacity was at peak
flow level and moved it up. There\'s no particular sense in assuming
that it was going to exceed the actual capacity.

Figure 6, which I think is maybe a little bit messed up, what I wanted
to point out with Figure 6 was that if you just look at the flows and
the aggregate as opposed to aggregate capacity in a period of time when
I was pretty sure that there were some constraints on some of the border
crossing points, it wouldn\'t be particularly evident in the aggregate.
So if I was going to use aggregate capacity and if constraints in a
particular region were \-- I\'m going to be without the display so I
guess I\'ll just have to move on. I apologize for the technical
difficulty.

So there are sort of two things going on here. I wanted to get away from
the relatively simple approach that we\'re taking with imports and I
also wanted to know whether I needed to \-- it was essential that we
look at regional capacity. We don\'t have a regional model so we really
have to fudge things a little bit. For example, we may need some demand
factors in there which I did put into the equation. That would have to
come, too. So anyway, that was another question, whether regional versus
national capacity information was particularly important.

So in order to do this, we had done some work previously on a wide range
of basically a whole natural gas framework for STIFS, part of which
involved the wellhead market. On page 11 there\'s a description. I think
Equation Five might be \-- well, slip back a little bit. The basic idea
there was to say that two basic things going on. #1, if we assume that
the domestic and Canadian markets are both supplying gas competitively,
then we could represent the supply side by looking at what amount to
marginal cost functions for domestic and Canadian gas supply. That\'s
one thing. That shows up in the price equations. That\'s represented by
the price equations for domestic and foreign gas.

The other thing is that pipeline systems would be assumed to minimize
cost with respect to providing gas to the distributors of gas in the
United States to the city gate, for example, and, in a sense, in this
particular model, storage management is kind of rolled in with the
pipeline systems in general. So what we determined from that is the
input demand functions for the pipeline/primary supply system is the
demand for new supply of gas and the determination of the shares as
between domestic and foreign. So not being able to point too clearly to
that, on page 11 the QP is the aggregate demand for gas by the pipeline
systems.

The little s function there is the share equation for Canadian gas. Once
that\'s determined, everything else is assumed to be supplied by
domestic. There\'s one equation that we\'re only worried about domestic
and foreign aggregate. There\'s three that we\'re worried about all of
the \-- well, four if we\'re worried about all of the Canadians. It
turns out that for the central and midwest region that\'s described in
the paper, I did some initial regressions to see how they generally
would fit. It turns out that getting aggregate Canadian inputs works a
little better if I combine the midwestern and the central, so I did. So
I\'ve got three foreign sources that I\'m looking at.

So basically I can simultaneously determine the aggregate gas
acquisitions by the primary supply system in the United States and I
have to estimate the supply function for domestic and foreign gas, as
well. It turns out that there\'s a little assumption in here that I
didn\'t mention that is kind of important which is that I\'m assuming
that the Canadian pipelines at the border are equal to what the domestic
supply system can then use to bring gas in and take it to wherever it
needs to go. That may not be really true but that\'s the reason why the
same capacity numbers show up in the input demand functions. That is,
the demand for domestic and imported gas.

Inadvertently, they also belong in the supply functions. They\'re not
there, and that\'s a mistake and I think that actually may explain some
of the problems that I had with the supply functions. But in case you
noticed that the capacity numbers are not there, my feeling is that they
belong there under that assumption I mentioned about the equality of
border import and export capacities.

So with regards to the regions, when I did the regional version of the
model which is really just one in which I estimated three share
equations in which I used specific capacities for each region, I also
put in factors that are particular to those regions that I think was
designed to try to make up for the fact that we don\'t really know what
factors are going on with regards to regional demand configuration.

So heating degree days, for example, or measure of weather intensity,
was geared towards these regions whereas in the aggregate national one I
think we just used average heating degree days.

So then the idea was to estimate all these equations and see if it came
out reasonably well and the answer is yes and no, I think. I didn\'t
summarize all the elasticities but there is a table, Table 1, which
gives some of them, some of the ones that I wanted to focus on. With
regards to aggregate pipeline gas acquisitions, the aggregate price
which is essentially an index of the domestic and foreign price, the
elasticity is about zero. Gas storage is negative but it really doesn\'t
have a significant impact.

Pipeline production is another variable I left out of here. That\'s what
the pipelines will ultimately supply. That\'s the gas that they\'re
actually supplying to people who are going to make the final sales or
placing into storage. There the short run elasticity is about .47 and
the long run elasticity I didn\'t write down here but it\'s close to
one. Heating degree days don\'t seem to have any particular effect here.

One thing about the structure of the equations which is in the back of
the paper that I should mention is that I use a lot of seasonal dummies
because they tend to be fairly convenient for the following reason.
There are a lot of things that affect seasonality in some of these
equations that are easy to show with variables like heating degree days
and there are some that aren\'t so easy to show like some of the demand
factors, for example, like seasonality and agricultural demand and so
forth.

So generally speaking, I put those in but I realize that in a sense what
that would pick up is kind of normal seasonality and so you\'ll notice
that the heating degree day functions are generally taken as deviations
from normal. So there\'s a question as to whether I should be allowed to
get away with that.

Figures 7 and 8 just give the aggregates, the sum of the Canadian import
numbers, one with just the one share equation, that\'s Figure 7, one
where I use the three share equations and add them up. Interestingly
enough, although it turns out that this simulation which is a dynamic
simulation I start at the beginning of \'89 and go through any lag
variables in this model are ceded into the next step so it\'s not a one
step ahead forecast all the way through or a simulation.

Interestingly enough, the national model actually fits a little bit
better which \-- I\'m not 100 percent sure why that is but it\'s worth
looking into. But certainly they\'re both better than the current model
that we\'re using. I think the root mean squared error percentage of the
mean was between 4.2 and 4.5 percent compared to almost six percent, and
that\'s still not all that great. We certainly would like to do a little
bit better than that.

Now, there were some simulations that I did that were sort of designed
to check on a few things. #1, on Figure 9 I simulated beginning in 1998
through the end of our forecast horizon for STIFS which happens at the
moment to be December \'99. Just one thing to note, on the chart I
included the current model, the new model with just the regional
capacity, the new model with just national capacity. Without the colors,
it\'s a little hard to see what\'s going on here but basically I also
included what actual data we had for 1998. The last couple of points
aren\'t really actual. They\'re estimated but I think they\'re pretty
close to the ultimate actual.

Clearly, it seems to me anyway, that the new model doesn\'t do that
badly so far in \'98. It tends to show that in this upcoming winter,
we\'ll probably have, it\'s suggested that gas imports would be a little
bit more than what we\'re currently saying and it really says we\'re
likely to get more gas than what we\'re saying at the end of next year.
Also, it shows, I think, that the national version of this model tends
to show a little bit more gas now, that is for this winter, and somewhat
less in the beginning of the winter starting in late \'99.

The reason for that I think is that where the capacity expansions occur
\-- I think I might have said that backwards. Yes, I said that
backwards. What it is that the regional model shows a little bit more
gas this winter and less the following winter because gas expansion is
concentrated in the midwest/central region this year which has a high
elasticity with respect to capacity expansion and the biggest expansion
next year is in the northeast. So the models aren\'t terribly different
in the way they generally behave, but they do reveal some additional
information if you look at regional capacity as opposed to whether you
do not.

But in any case, I thought it was interesting that compared to what
we\'re saying now we probably are not over-estimating. It suggests that
we may not be over-estimating imports this winter and next winter.

Some of the other simulations simply show that with regards to demand
shocks like very cold winters, one of my concerns was that well, we
might get a lot more imports from these shocks if we just use the
current or the national model than would actually be possible because of
capacity. It\'s very insensitive to this anyway. I think once you take
into account capacity expansions that are planned, not much else
happens. In Figure 12 you see that most of the incremental gas from
demand increases comes from domestic sources.

There was one other thing that I wanted to mention. The gas supply
function has oil prices in it and there\'s a debate as to whether we
should potentially do that or not. Chart 14 shows that so far that\'s a
terrible way of estimating gas prices. They\'re much higher than what
this model would say, but the interesting thing is that I think there\'s
another chart, Chart 16, and this is a simulation of that model over a
much longer period beginning in \'89 all the way through \'97.

There\'s two things about this chart. One is that it\'s not as terrible
until we get to 1998 but I still think we have to revisit this question
of whether or not we should have oil prices in the supply function or
not. Probably not.

The other thing about Chart 16 is that between 1992 and 1994, gas prices
took on a much less seasonal character than they did prior to that
period and there\'s nothing in this model other than the usual sort of
factors on the supply side to explain that, so the interesting thing is
I think the model tends to replicate that pattern. It\'s not a perfect
fit by any means but it just sort of suggests to me that we don\'t need
to go too far afield to explain this particular pattern. I think that
this is a period when storage began to shift around a lot downward and
the storage pattern was somewhat different but you didn\'t need to know
too much about why to more or less capture the price pattern.

After that period though, prices started exhibiting some extraordinary
spikes which are not well explained in the model. In fact, in the later
one, this is actually we think in part due to some supply problems for
alternative fuels, particularly coal and electric power in the south and
southwest. The 1996 one is something we haven\'t done a very good job of
explaining, yet.

But another question arises that well, now we may be getting a lot more
serious volatility in gas so we have to worry about the factors that
cause that and try to either incorporate them or, directly, or at least
have an idea of how the likely variance in gas prices would be affected.

There\'s really just one other thing. On Figure 17 this is working gas
in storage. The model doesn\'t seem to capture completely the entry year
variations on storage. The variation seems to be greater than the model
is capturing. But what I thought was interesting though is that one
thing it did capture \-- and again, this is, as I said, it is a dynamic
simulation beginning in \'89. What it does capture is something that
happened in \'96 was that we had the lowest minimum point for storage
and the lowest maximum point for storage and it seems to have captured
that all right.

So the conclusions that I have are basically that it\'s worth doing and
I think certainly we get a lot more information. We need to consider
some of the things that are included or excluded from the equations and
go back and redo them. It\'s still not clear to me that bothering with
the regional flows and capacities hurts us a lot but I think when we
look at what happens this winter that might be the big clue because we
do have a pretty big expansion in the midwest and I think if flows are a
lot bigger than what the national model is suggesting then maybe we
should take that as a sign. We should go with the region.

Sorry for the difficulties with the display.

CHAIRPERSON RELLES: We have two scheduled discussants for this. Charles
Bischoff and Greta Ljung. Do either of you have a preference?

DR. BISCHOFF: Greta insists that I go first.

CHAIRPERSON RELLES: Charles will get to go first.

DR. BISCHOFF: Well, let me say that my expertise is in general equation
modeling and in forecast evaluation, not in any particular subject
matter involving energy such as natural gas. Thus, the first equation I
ask when I look at these simulations, in particular the ones we would
have seen if the lights had not gone out, the long 10 year simulations
over the entire period as in Figure 1, I believe 7 and 8. Is this any
reason to prefer one model over another or what evidence would lead me
to prefer one model over another?

So I look at Figures 1, 7 and 8 and we see that these are simulations
over the entire sample from 1989 through 1998. And it is considered
significant that the new models both show mean squared errors which are
much lower than the mean squared error in the old model. In one case, 46
percent lower. In the other case, 38 percent lower. Is this evidence
that the new more complex models are better? Well, let\'s think about
what use is going to be made of these models. These models are going to
be used to make forecasts maybe 12 months, 18 months in advance and the
long run simulation doesn\'t tell us anything about how repeated short
run simulations are going to fit. Instead of making a single long run
simulation, I would fit all three models to data up to, say, 1993 and
then make simulated ex ante forecasts 12 to 18 months in advance and
then repeat the exercise with data through 1994 and \'95 and \'96 and so
forth. It is the fit in forecasts like these that I think would matter
in telling which of the models is the best model. An experiment like
this can provide direct evidence about the actual size of the forecast
to be expected. Also, if these forecasts deteriorate over time, they can
provide evidence as to whether the model structure is breaking down.

If, for example, the pass is no longer a good indication of what to
expect for the future, it is important to know this and it is
simulations like these that will tell us. By the same token, I would
caution against putting too much weight on a simulation such as, say,
Figure 9 which starts just in the very end of the period. It\'s very
easy to say, okay, this is our intuition about what\'s happening right
now but it seems to me this is just one of a number of simulations that
we want to look at it and it\'s just as important to see whether the
simulation got off the track in \'93 or \'94 as if it\'s getting off the
track in 1998.

I would not hazard a guess as to whether or not the old model, which is
the simple time trend model with the seasonal, would actually do as well
or do worse. Perhaps the simple model with trend and seasonal does as
well as it does within sample simulation because the trend is pinned
down more accurately by being fit for the whole period and maybe if you
just fit the trend for the half of the period it would go away off the
tracks. I don\'t know and we won\'t know until experiments like this are
carried out.

Nevertheless, I think Dave is to be commended for trying to make a
theoretically more preferable model work. Even if the only parameters
associated with prices, for example, have the wrong signs in the models
where capacity at the regional level do not seem to provide a better
explanation than capacity at a national level, the simulations of the
models do give some interesting implications about what to expect for
the future.

Although the regional model does not seem to provide much advantage over
the national model on the last page of the paper, Dave provides what
amounts to, he thinks, a direct test of the two approaches depending on
whether the national model under-predicts total gas imports from Canada
this winter. However, as I have previously stated, too much should not
be made of this one test.

I want to make one final remark about what to do if the pass turns out
to be not a reliable guide to the future or if, as economists have
phrased it since the 1970s, if there is a regime change. If your best
judgment is that the future will not be like the past, there is no
substitute for just writing down the best theoretical model you can find
and making up your coefficients than estimating a few to put the
forecast on the track and using this calibrated model to forecast. If
this is good enough for the real business cycle people, perhaps it
should be good enough for all of us.

Now, I know that Greta is going to tell you about what to do using a
time series method, so I\'ll turn it over to her at this point.

MS. LJUNG: Thank you very much. I found that paper quite interesting.
Discussing the paper, there are now two models. One univariate model,
the simple one that has a time trend and seasonal component plus an
AR(1) error term and the new model is quite a bit more elaborate. I went
back and looked at the source, Considine\'s paper and it seems like he
was proposing 39 equations and that\'s, in my view, a very big model, I
know econometricians, that\'s big in econometrics and it\'s big in
statistics too, I guess. Personally, I get quite nervous if there are
more than five equations and the reason is I feel that these big models
lack transparency. It\'s difficult to see what\'s going on and also
there\'s multi-collinearity in their selection in the present model. I
think there was some question of the price coefficient having the wrong
sign or something like that could come from multi-collinearity in the
model.

Another problem, as I see it, is the possibility of model
mis-specification. What these multi-variate models try to do is develop
a sort of realistic representation of the system as a whole and I think,
because relationships are often quite complicated, I think this is
something that\'s quite difficult to do.

And another problem, as I see it, for forecasting. If you have a bunch
of these X variables in order to generate forecast, you have to forecast
the Xs before you can forecast your Y variable. So the forecasting
problem isn\'t really solved by that.

What I was going to recommend is taking a look at the old model, the
current model and try to develop it a little more, but I think there is
a relatively simple way to get better unity in this model. Basically,
what I\'m recommending is that you take more of a time series approach.

It will look like a linear time series model and using those models,
like before, basically this would say that the present values are a
linear combination of the past values and for forecasting then we would
use that function that we develop, linear function, linear combination
of the past and this would be a conditional expectation of the future,
given the past.

And this has been shown. It\'s been shown. What we didn\'t have a
particular paper was like a real forecast comparison like Charles
pointed out. We really didn\'t make comparisons out of sample as far as
I could tell. It was more like in sample comparison of mean squared
errors, where we have the fit. That\'s how I perceived it anyway.

So the forecast will be a linear combination of past and the reason it
works well \-- it\'s been shown that it works well in real comparisons
and the reason it works well is that the past, Ys; Y~t~, Y~t~ - 1e and
so on, they serve as proxies for X variables that you don\'t include in
the model. So all those Xs that you like to include, they are implicitly
represented by having the past in there. The degree that they\'re
correlated with the past. You have them implicitly in there. So that\'s
the reason why you don\'t necessarily have to try to pick the right X
variables or anything. That\'s correlated with those Xs and that\'s the
past values of the series.

Now as far as modeling the current series, I think one class of models
would be by Box and Jenkins and seasonal ARIMA models. In the present
case, I believe you would start by removing the trend and the seasonal
factors by difference. You would take the regular difference and on top
of that you would have a seasonal difference and then you would look at
the auto correlation function of what\'s left, the reduced series, the
transformed series and from that what correlation function you would
look at what \-- and try to come up with appropriate model form so you
would have some auto regressive, regular auto regressive and perhaps
seasonal auto regressive. The regular auto regressive is denoted by Φ~1~
and Θ~1~ here. Those would be described as short-term dependence.
Dependence between successive values in the series and then you could
have seasonal parameters which describe correlation between years. I\'m
sure that you would have one of 12 rather than monthly seasonality.

The forecast functions you get \-- and after you have fitted the model,
you can then look at the weight. How are the past values used when you
do the forecasting? And the weight functions are frequently quite
interesting and there\'s an example in Box and Jenkins\' book and
there\'s a paper also on the [Journal of the Royal Statistical
Society]{.underline} from 1972 where they discussed the paper by
Chatfield and Pradura (phonetic.) and they wrote out the forecast
functions and it\'s really quite interesting.

The point is that the forecast function, the way you treat the data, is
determined by the model and the parameter estimates in the model, the
model form plus the coefficients of estimated parameters.

Now the model you use is \-- we could look at that one a little bit so
you have current model you can write it as; Seasonal mean μ~t~, μ~t~
where μ~t~ is the same as μ~t~ - 12 plus a linear trend and seasonal
factor, trend factor plus an error μ~t~ which is an AR(1) error term.
This is the model that\'s being used. Now, if the difference of this
model, you take a regular difference and a seasonal difference, then
what we are left with in this model can be recognized and we have an
AR(1).

The AR(1) factor goes over to the right hand side and on the left hand
side we don\'t really have any parameters. We have this (1-B)(1-B~12~)
but we could write this as parameters Θ~1~ and Θ~12~ on the left hand
side, where in this particular case, Θ~1~ and Θ~12~ are one.

But I sort of like this model actually. I think this would be
sufficient, I believe, for the series you are looking at. If you let
these coefficients go free. So you would estimate the coefficients
rather than forcing them to be one like the current model forces those
to be one. But we could have those as free parameters, estimate those,
and I think that would give you a better fit. Actually, this model, if
we ignore that, we can assume that the fee is not there, this has been
called the airline model because in Box-Jenkins\' book it\'s fitted to
airline data. And it\'s been shown that this particular model works for
many series. In practice it works quite well, I think. So I think it\'s
going to work in this particular case and you may or may not need that
fee but obviously it\'s something that you would look at. After you fit
that, you would see what coefficients are needed. Also, if there\'s any
serial correlation left in the residuals, you could incorporate that. I
think that would be an improvement.

Now, when you fit a model of this type though, it\'s important to use
exact maximum likelihood as opposed to conditional maximum likelihood or
least squares because if you do any kind of conditioning on the starting
values, there tends to be a bias, especially if the true parameter
values are one or close to one. You would really be getting that back,
get that value back. There would be a downward bias. But exact maximum
likelihood avoids that problem.

Once you have the model, then the forecasting is very easy. It\'s just a
simple recursion, a recursive calculation, so the work is really in just
developing the model. Now I haven\'t really analyzed the capacity
constraints and all of that. I haven\'t really given any thought to
that, but basically the forecast and the forecasted value exceeds the
constraint. We would reset the predicted value to the capacity value and
one does that one step ahead and then one can just continue the
recursion and each time it goes farther.

Now as far as expanding the model, one can include an X variable thrice,
or whatever you like to add. The univariate model does assume certain
\-- I mean basically assumes that you have something that\'s stationary
after you do the differencing. So that may or may not be the case. But
instead of expanding, I think my preference would be that you start
small and build up rather than going from a univariate to a very big
model. I would go from a univariate and then look at few important X
variables and then model the serial correlation in the errors. Anything
you don\'t take into account is likely to show up as correlation in the
errors and you can account for those variables by modeling the
correlation structure.

Looking at those graphs, I really think that the model, the univariate
sample, the time series model should fit this series quite well. If you
look at the regional ones though, the central and midwest region, those
series \-- I think one of them you could perhaps do some log
transformation but the second one I didn\'t think that you could really
get a good univariate but perhaps combining the two series like you did,
you may be able to fit a time series model there.

And then I would, like Charles mentioned, really look at the actual
forecast performance by leaving out data at the end of the series. Now,
you don\'t really have very long series. You have long enough to fit and
I think you could easily spare a year or two perhaps and that would
allow you to evaluate the true sample forecasting performance of these
models.

Those were my comments.

CHAIRPERSON RELLES: Thank you very much. Are there other committee
comments? David.

DR. BELLHOUSE: Well, just a question for the presenter. What\'s the
purpose of the forecast? Is it to provide a forecast solely or is it to
provide an understanding of the system because if it\'s just to provide
a forecast, then it seems to me that Greta\'s method is the way to go
because you have a very simple model that does an adequate forecast. If
you want to understand the system, then you\'re going to have to go to
the more complex econometric modeling.

CHAIRPERSON RELLES: Actually, before you come up and answer that
question, can I add a little of my own along the same lines. There\'s a
lot of purposes for fitting models. One is to actually make predictions
when the system is kind of in an equilibrium state in which case the
time series methods are really most appropriate. But the structural
models would be kind of useful if you wanted to do some what ifs and the
what if that yesterday really impressed upon us that\'s important is
what if gas prices sort of go through the roof as might occur under the
Kyoto protocol.

You mentioned briefly kind of the cost elasticities, the dependency on
coal and other things. But you didn\'t elaborate on that. I guess I\'m
really curious to know if you could sort of take these same data series
and whether it\'s possible to try to get some insight into what might
happen if coal prices go way up and, hence, gas prices go way up and
what kinds of insights can we get from the structural model there.

MR. COSTELLO: The answer to the question of what it\'s for. Usually
it\'s for a forecast but then in various visible certain periods of
times, it\'s for explaining what\'s going on in the system. So it\'s
both. I mean the day to day stuff that we do is just the forecasts.
People are interested to know what the balance is from all the different
sources of supply and what we think prices are going to be in normal
times. We really don\'t have to worry too much about explaining a lot if
the system \-- I mean a series of time series equations handles that all
pretty well. Most of the time we\'re okay.

And I\'m not sure that we really get to it in here yet, but those big
spikes in gas prices are huge questions and they don\'t always come up
but they come up often enough that we want to be able to say something
other than prices were high. And the time series, we haven\'t really
gotten it yet but the time series model I think it would be impossible
for it to anticipate it and if it happened, it wouldn\'t \-- might not
add too much to the explanation but if we had the expanded version that
I think Greta was talking about, we might be able to do both just fine.

On the coal question, the thing there is that it really comes through
the electric power model mostly because that\'s mostly what we use coal
for and that\'s mostly what\'s going to be impacted by Kyoto. So part of
the model you didn\'t see \-- I\'ll just sort of focus in on a segment
\-- was how the fuel shares for utilities are determined and if they
change, what happens.

Right now I think what would wind up happening if we were to say listen
to the requirements for coal are going to be essentially shut down to a
large extent, then that means we have to have a lot more gas. We would
be in a position in the model, we would have to force the model to do
that. We could do it by changing the coal price which is probably the
way it would work anyway and, as a result, a great deal more gas would
be used. Then the demand for services of gas would go up and that would
impact the wellhead market.

I think the way things are now though, I think that if we ran a
simulation like that, we would get an impact on gas prices but it
probably wouldn\'t be as big as it probably should be because I think
it\'s almost as though there are some critical points that you seem to
get to, like in 1996 we got to what I consider a critical point with
regards to much lower stocks and the pipeline system having a difficult
time meeting a surge in demand and that\'s when prices went up a lot. We
would get some of that, but I think we\'d have a hard time replicating a
spike like that with what we\'ve got right now.

CHAIRPERSON RELLES: Wouldn\'t there be kind of an expansion of capacity
and might it be interesting to kind of change the capacity levels and
sort of see how the system plays out in a case like that?

MR. COSTELLO: Yes, and that\'s another question that\'s sort of another
piece of analysis that has to be done because in Tim\'s original work we
actually did take domestic productive capacity as an important
endogenous variable and it had some rather clear properties to it but,
more than that, since then we\'ve had an interesting phenomenon I think
in the U.S. which is rapidly expanding productive capacity and a rapid
increase in unused capacity.

So the answer is yes but I\'m a little bit \-- I think we have to figure
out more about why that\'s happening before we try to go back and to
recast that part of the model. That was left out. We\'re assuming
there\'s just no problem with productive capacity in the U.S. But I
think it is an issue.

DR. SUDMAN: May I ask a very, very simple question. What fraction of all
natural gas comes from Canada relative to domestic and how does that
vary by region?

MR. COSTELLO: We have a little less than half of all the gas that\'s
imported from Canada I think goes into what I call the western region.
That\'s basically through the Pacific Northwest and then from some of
the western states down over mostly into California.

One thing I didn\'t mention about the data was there\'s some data
corrections but once you get the data corrections \-- I\'m not going to
go into too much detail \-- there\'s about an equal amount in the
western and midwestern central regions. Combined they account I think
for about a little under four billion cubic feet per day. Three point
something, if I\'m not mistaken. And then the northeast has a little
over two billion cubic feet per day. We import I think close to nine BCF
per day of gas and we consume on average \-- well, we produce about 52
to 53 BCF per day in the U.S.

I don\'t know the \-- during the winter we consume about eight \-- I
forget what the annual average is but it\'s about 60 something BCF per
day. So eight out of 60, about one eighth.

DR. SUDMAN: Thank you.

CHAIRPERSON RELLES: Thank you very much, David. It doesn\'t say break
yet so we\'ll just go right on with our next talk. Roy Kass and Inder
Kundra are going to be talking about procedures to accelerate and
improve natural gas estimates, methodology and progress to date. This
paper will have two discussants, Samprit Chatterjee and Carol Crawford.

MR. KASS: Thank you. I\'m Roy Kass from the Office of Oil and Gas,
Natural Gas Division, and I\'ve been working with Inder Kundra from
Statistical Methods Division. We\'re going to talk about our attempts
thus far to solve a problem we have in the natural gas area. We have
some handouts that are going around. It might be hard to see on the
screen.

MR. WEINIG: Roy, would you use the thing.

MR. KASS: Use the mike. Okay. You all have handouts. You don\'t have to
see the screen. Again for the record, I\'m Roy Kass from the Office of
Oil and Gas, Natural Gas Division and Inder Kundra from Statistical
Methods Division in solving a problem that we have related to getting
timely publication of consumption numbers in our *Natural Gas Monthly.*
I\'m going to give you a general background of what our problem is and
what we\'re doing now and then Inder is going to talk about some of the
approaches to solving that problem we\'ve come up with and tested and
what we intend to do, what the results have been and what we intend to
do in the future.

I\'ve been before this group several times talking about the Form EIA
857 and several of the problems it has. The purpose of today\'s talk is
not specifically about any particular kind of gas consumption which is
something I visited here before but rather a general problem that we
have about the decreasing timeliness of our respondents and the impact
that has on our ability to publish data. The Form EIA 857 is a sample
survey done monthly of approximately 400 respondents nationwide. Over
the years, the responsiveness of the survey has decreased drastically.

To give you anecdotal feel of what it is, eight to 10 years ago every
month we would perhaps have to impute for unit non-response between two
and five companies. They typically were very small. Had absolutely no
impact on what we publish. We would do that in time to get preliminary
tables for a month\'s report in the first week of the report month. Let
me give you an idea of the time frame. The reporting requirement for the
form is that they are to report the beginning of the two months
following the month in question. For instance, August events are
supposed to be reported the beginning of October. So we have a four week
lag before they\'re supposed to tell us. We publish that the month
following that. Again, August numbers are published in the November
publication.

Because of the delay that our respondents have, we have done two things.
One thing we\'ve done is delay the production of the tables so that now
it crunches against our publication deadline. While we used to be able
to prepare tables the first week of the month, now typically it\'s late
the second, early the third week of the month. While we used to impute
for a very few number of companies, even with the delay we now typically
impute because of unit nonresponse something around 40 companies. So
we\'ve got something like 10 percent. I\'ll go over the most recent
information in a few minutes.

So timeliness is a problem. One thing we\'ve done to increase timeliness
for our customers is we\'ve introduced analytic estimates on a national
level so our November publication has analytic estimates based on the
STIFS model through November. We can\'t do this on a state level but at
least people get an idea of what we think the more timely information
is. We\'re not talking about trying to get those kinds of analytic
estimates down to a micro geography. We\'re talking about trying to
impute for the unit nonresponse so that we can develop estimates at a
state level and we still are going to be confronted with suppression
rules. If we don\'t have 75 percent of the gas sampled in a state, we
don\'t intend to publish that state information.

The goal is rather conservative. Our goal is to buy perhaps two weeks in
our publication production cycle to get back to a situation where the
beginning of a month we\'ll be able to prepare tables that our *Natural
Gas Monthly* can go forward with.

The current procedures for imputation are described in the appendix in
the *Natural Gas Monthly*. Essentially they take the ratio of the
current month reported information for companies who do report to the
previous months and apply that ratio to the nonresponding company\'s
previous information. In order to do this, you have to have companies in
so that you can calculate a response and that\'s where part of the delay
has come from. So Inder and I have been working round that problem
trying to get ways that don\'t depend on a large portion of the gas
already having been reported in order to impute for the nonrespondents.

To give you an idea of what kind of problems we\'re up against, I said
that we impute later on in the month. I did an imputation for the August
numbers this week. That\'s 10 weeks after the end of the event. At that
time there were still 44 companies outstanding. They represented 27
states and in nine of those states, if we don\'t get reports between now
and when we go to publication next week, we\'re going to have to
suppress nine of those states. So we\'ve got a problem.

What we\'ve done is we\'ve developed and applied alternative approaches
to preparing estimates that hopefully will let us do our work earlier.
It\'s a work in progress. Inder is going to describe a few general
approaches we\'ve used and the results for a single month\'s data that
was simply selected because it was a shoulder month and any impact of
changes in the direction of change would be stressed.

During this exercise, there were a lot of unanticipated side benefits of
doing this. We found that there were some basic problems in the database
that since have been fixed, but the working database was polluted in
many cases. The one point that I hope Inder talks about but I\'ll just
mention it now. One of the things that we did not do for these tests is
accommodate mergers. So there\'s lots of cases where he couldn\'t match
up past information because it matches on IDs, company IDs when it\'s a
merger and new company ID happen.

Having given that background, I\'m going to turn the discussion over to
Inder who\'s going to actually say what it is that we did and what the
results were.

DR. BREIDT: Excuse me. Is this a pure panel other than the acquisition
and mergers and stuff? I mean do you look at the same 400 companies
every month?

MR. KASS: Every month for a year. The panel rotates every year. About 30
percent turns over. The sampling strategy is certainly an uncertainty
and we aim for at least 85 percent of the gas in a state being in a
certainty stratum.

MR. KUNDRA: Thank you, Roy. Before I start explaining the approaches, I
would like to say something about the sample size. A sample was selected
as a PPS within each state and the measure of size was the total,
sectorial, industrial, residential plus commercial. So as a result of
that, it is expected or it is known that when we estimate the sectorial
estimates probably efficiency is not going to be seen. They are
different efficiency standards will be very high in some cases,
especially within states. And the other aspect of the sample is that
probably we are having four to 14 companies selected in each state and
the certainty companies are so diverse that it becomes impossible to use
the standard adjustment rates for making estimates. So the only thing
left to us was that such format types, to substitute donors that we can
use for the nonresponding companies. For that purpose what we were
looking at, we said why don\'t we go to some methods, so we turned up
six or seven methods in order to get to those donors.

The first method which we used was the one month lag values multiplied
with growth rate and these growth rates were determined from the
certainty companies which were responding in a particular month, in a
particular sector, in a particular state. The second method was using
simply the log value, ignoring the growth rates, as we see from this.
And the third method which we used was the frame year values, that is
two years frame year sample was like in \'96, the frame was \'94. And we
also again used the growth rate from the frame year to the current year.
That means again it\'s based on certainty companies.

The fourth method that was used as the frame year\'s monthly values
along with one month lag data for the companies where monthly data and
frame year was not available. That means we ignored the growth rate in
the fourth one. The fifth one was the one year monthly lag values
multiplied by year to year growth rates computed from the responding
certainty companies within a sector within a state and the sixth method
was in one year lag. That means the data for the \'96 we used for \'95
data. And there we did not use the growth rate.

And the seventh method which we devised was because when you come to the
month of January you do not have the value available for the previous
month for that year. So we said why don\'t we use the frame year\'s
values. Now for the \'98 sample, the \'96 frame values do not have the
monthly values so what we have to do, we have to find the value so we
could proportionate it. So we used a method which we can proportionate.
We\'re showing here that we used again from the responding \-- what?

CHAIRPERSON RELLES: Are those summations within the state?

MR. KUNDRA: Right, within the state, within the sector \-- for the
current year. That\'s what it is.

So matter one to six they were applied to the April \'96 numbers and
what we did, we used a cut-off date 20 days after the deadline so that
we could see where we can match the values where we can publish the
data. And what we found here is that the purpose was to test the concept
of terms inability to generate proper straight level estimate for
publication. The

procedures were all augmented by weighting the imputed values by the
appropriate sample rate. Comparisons were made against information
published for April 1996. That means data which was published in the
Monthly \-- We used the estimates. We compared them. The estimates we
got, the elasticities and their prices were developed using the outlined
methods and compared with those published in the *Natural Gas Monthly.*
If a sectoral estimate was not within 10 percent of the published one,
that meant we compared it. If it is not within 10 percent, we ignored
it. We said, Well, it is not acceptable. The estimates of the price
compared with delivery estimates, the price estimate was found to be
closer to the published ones. The estimates of variance of deliveries
and prices showed a remarkable improvement.

In most cases the reduction in variance was significant, I would say to
the extent of even 75 percent. We did not represent the result here.
This was despite the fact that the 15 sample companies, nine certainty
and six noncertainty, records were not available. That\'s because what I
mentioned, that there were mergers. We could not find, we could not
match it; therefore, we did not use those companies in our estimation
procedure.

If we look at the table results, we can see that one to six method,
that\'s the number of companies that were different, 11, 11, nine, 16. I
think the most problems we were finding in the industrial sector.
Industrial sector I think can probably be explained better that the data
which we copied from the data file probably includes or did not include
which are published monthly. I think he had better estimates about what
happened, that we were getting little bit more accuracy from the
industrial comprised to the non-industrial.

Before I stop, I want to make another point here because the growth
rates are used in method one and method three and method five, it looks
like that the growth rate is not giving us a good estimate, the reason
being that because growth rate is based on very few companies. It might
be possible that in some states only there are one or two or three or
four companies responding and if we compute growth rate from that, they
are going to be biased either upward or downward. So that might have
given us some problem.

In addition to this, we did try to think of the matter using like a
regression or something like that or clubbing the states, pooling them,
but what I feel here is that one thing, the sample is so small, those
sample sizes are so small and the companies are so small that they
probably might become very difficult to use that, so we didn\'t use it
that time and I don\'t know whether we prefer to use it or not because
there will be another problem, how to pull those states and what kind of
characteristics of those companies company by company will have to apply
before we can even come up with some suitable methodology, the
substitution methodology. I don\'t know.

For that purpose, we have questions for the committee. What do you think
of the proposals? That means we did use 10 percent. I don\'t know how.
We just came up by the snap of the chart. There was nothing actually
which we want to use it. The second thing is do you have any suggestions
which I think you probably have, suggestions for improving the
estimates, and do you have suggestions regarding how we may determine
whether to adopt a new imputation method. We would like to have your
guidance for that. Thank you.

CHAIRPERSON RELLES: Thank you. We have two discussants, Samprit
Chatterjee and Carol Crawford. Samprit looks like he\'s most anxious.
Why don\'t you tell the chair about it.

VICE CHAIRPERSON CRAWFORD: I didn\'t know the chair was an option. I
thought he\'s listed first and he goes first.

DR. CHATTERJEE: We follow your orders. I really wanted Carol to go first
so that I could figure out what the problem is.

My discussion is going to be fairly informal and Carol will be the
heavyweight. Kass and Kundra strictly alphabetically discussing a very
important problem in official statistics. That\'s the balance between
accuracy and timeliness. Figures, statistics, official statistics,
professional statistics, very accurate which comes out long after

it is necessary for making decisions is not very useful. So I can quite
understand that attempt to try to not to delay the forecasts and try to
get the figures out as early as possible. And since it\'s dependent on
the survey responses and with the presence of non-response, how to get
estimates or figures of a certain degree of accuracy.

My reaction to your work is going to be basically going through the
document and noting some thoughts which came to me. I\'m curious as to
why the responses have become tardy lately. You speak about such high
good response rates earlier and gradually they\'ve declined.

Is it due to \-- I think EIA maybe or you should look in as to why this
tardiness is coming. Maybe a solution to everything is maybe
electronically getting the responses might improve the rates and so
forth. So maybe that\'s the first thing I would say. Why is this drop in
response rate?

Second, about ignoring all responses for the month that leaves out
information about the monthly production condition. Not using the
returns which you\'ve got in for the month really does not take into
account any kind of production changes and so forth and the idea that
for the particular month for a particular sector, particular state, you
have only small number.

Therefore, the ratio obtained is highly unstable. Maybe for the growth
factor you could use some kind of a combined ratio. I know you\'re
allowing for the heterogeneity between the states of production, but you
might be able to incorporate particular features associated with that
month which would be left out completely by relying on the previous
month.

Your observation that price estimates were found to be closer to
published ones wasn\'t surprising to me because prices probably do not
vary much given the market. About various attribution schemes which you
looked at, the two methods, I was just going to suggest you could, if
necessary, do a little more extensive, you know, but in these cases you
have to stop somewhere but what I was going to suggest is to study the
effectiveness of different methods of attribution, a sampling experiment
could be carried out in the following way.

Take a set of months from which you already have data, randomly omit a
number of responses, and attribute the values by different schemes and
see which procedure is more effective by computing the mean prediction
error of each procedure. This basically carries out a sampling
experiment on the data which you have dropping randomly and just see
which of these things.

About the failures procedures one through six which you have. It looks
in the balance, at least for the two months which you presented data
that procedure six and procedure one seems to be about the same although
the industrial sector percentage error is large in both. Making more
work, but maybe you could use different attribution schemes for the
different sectors in order to balance that. So that\'s the general bulk
of the paper.

And finally the three pointed questions. What do you think of the
proposals? This is my opinion. The proposal seems reasonable. This is
the way anybody would tackle it without building a massive model of
non-responsive. Do you have suggestions for improvement? Always improve.
Continuous quality improvement. Yes. But I don\'t have anything
concrete.

The third question about 10 percent, is 10 percent good or not? I
thought to myself, whenever data is connected and published, they are
supposed to be used for decision making and if figures are off by 10
percent, are decisions made on it? Disastrous consequences. Basically
look at what those figures are used for and what margin of error would
cause terrible decisions to be taken. I think the criteria for whether a
procedure is working effectively or not, that rate of cutoff would be
based on some of that kind of consideration.

That\'s all. Thank you.

CHAIRPERSON RELLES: Thank you.

VICE CHAIRPERSON CRAWFORD: Well, the thing about going last in something
like this is that you run the risk of everything that you\'ve prepared
is now deemed not something that they\'ve already tried and decided not
to do or was already suggested by Samprit. So you\'ll see some of that
here although not completely. I\'ll just go straight to the questions.
What do you think of the proposals? Do you have suggestions for
improvement? And do you have suggestions about how they can choose a
particular education method. So I\'ll just talk about these in turn.

So what do I think of the proposals? I think the goal of the proposals
is a laudable goal. I think, as Samprit mentioned, there\'s always a
trade-off between accuracy and timeliness. Sounds to me like perhaps
their problem has been increasing and will get worse and so it\'s a good
time to start thinking about what to do. I think all of the proposed
imputation methods are very creative and supported by sound statistical
reasoning, so I didn\'t have any problems there. I thought they tried
some things that perhaps they didn\'t think were so intuitively or
theoretically correct but they were simple and if they were going to
work it would be great to know that and that was fine.

I\'m concerned about the accuracy of all of the methods though, and I
had a question about the table. The table reported the number of cases
differing by 10 percent, and I called the published values true values
because in this case they\'re kind of sort of acting like that.

At least they have a lot more information in them. And forgive me for
plagiarizing your table, but I didn\'t know how many cases or companies
went into this test, so I tried to convert them to a percentage because
I couldn\'t tell is this 11 out of 11 that were all wrong or 11 out of
40. So you had mentioned in your report that as many as 40 companies
could have to have this procedure so I divided by 40. How many is it?

Okay. So I\'m not too far off. So I divided by 40 to get a percentage
and you can see in terms of percent error, if you will. I mean here
we\'re almost 40, almost 50 percent up in terms of the estimates. I
don\'t see \-- I mean I guess this one is starting to become accessible
but to me those are pretty huge errors in all of the methods.

The other question I had is which one is it that you currently use? The
first one?

MR. KUNDRA: No. The sixth one.

MR. KASS: Probably closer to #1 than anything else.

VICE CHAIRPERSON CRAWFORD: Okay. I couldn\'t tell that either. I think
one and six are sort of maybe doing the best. A one month lag certainly
didn\'t seem to work very well which is really too bad because it was
the simplest one. So I don\'t know if maybe you want to report things in
a percentage or something other than numbers would have helped me.

Do you have suggestions for improvement? This was the case where you
went up there and you said a bunch of things that weren\'t in the paper
and so maybe we should just like X the slide, but I\'m going to say it
anyway. Do the cases that differ more than 10 percent from the published
values have anything in common? Are there some characteristics that have
to do with where they report or where they\'re from or something like
that could give you some sort of indication as to why the methods
aren\'t working so well?

And then the other thing is you have to use the same method for each one
of the sectors because I thought #6 was actually doing a pretty good job
for commercial and residential and then I\'d probably go up here and do
#1 for the sector industrial. But it wasn\'t clear from your report
whether or not, and I certainly don\'t know the details about whether or
not you have to use the same method for all the different sectors.

This gets to exactly what you said in your introduction that you thought
about and decided you didn\'t want to do, but I still think it\'s
important to try and think about it. I mean the only way you\'re going
to get any better is if you get more creative with the results that you
have, which I don\'t see that you can do, or you try to use some
combined information, either pool by state, if companies are correlated
in some way, to try and use that correlation.

The whole idea behind the imputation method was to use the temporal
correlation in the responses to make a best estimate. If that isn\'t
going to be adequate, then you\'re going to have to supplement that with
some kind of covariate information and the only way you can get that is
through maybe there\'s some correlation among certain characteristics of
the company. Maybe there are some other covariates that you can use that
are important. Those were just my first thought at suggestions although
I know you\'ve probably already tried that.

Another thing that Samprit said, and this is a formal itemization of his
idea, is who are the main users of these data and how accurate do they
need them to be? If there\'s an error \-- what was it? Yesterday someone
decided well, if you\'re doing global warming, you can have estimates
that are up to 30 percent wrong and that\'s okay. Well, maybe this is
another case of that or maybe they\'re just again looking for changes
because yesterday we decided in certain things that while the estimates
themselves may not be very good, the bias was probably the same and so
the change was easier to estimate. But if we can get a feel for who the
users were and what they felt were the trade-offs between timeliness and
accuracy.

And finally, do you have any suggestions regarding how we may determine
whether to adopt a new imputation method or not? I think my suggestion
was that you need to consider both the bias in the responses and the
variability in the estimates. Going back to this table, one of the
things I would have wanted to know is were, say for instance, these 11
that were 10 percent off from the published values, perhaps they were
all high or perhaps they were all low. I\'m sure they\'re probably not,
but maybe if they were that\'s an easier situation to deal with because
if you know an imputation method is always biased on the high side, you
can correct that by adjusting things downward.

So my suggestion would be, and this is a formal detail of the procedures
Samprit suggested although I didn\'t necessarily think it had to do with
sampling scheme. I just thought you could just use the basic study that
you did here but, instead of using the 10 percent cut-off, use all the
values. So take the deviation between the imputed value and the
published value and then look at the bias which is the average of the
differences and then some kind of a means for an error which is the
average of the square differences and then try and take the one that has
the smallest bias and the smallest mean square error.

That way at least you\'re trying to use all the numbers because I was
interested in the table. I mean how would the numbers change if your
cut-off was even 9.5 percent? I mean they could increase a lot. So
anyway that\'s something you might consider.

Just to reiterate, I think it\'s really important to know if any these
methods are systematically biased because if they are then that might
help you to adjust your estimate.

CHAIRPERSON RELLES: Thank you, Carol.

DR. BREIDT: I just had a couple of comments. One is I\'d like to second
what Samprit said. It\'s important to understand the cause of the
non-response and I think that you have a relatively, by many survey
standards, a relatively small number of non-respondents so you could
devote considerable resources to getting at those 40 companies and
getting an answer. Maybe that\'s not practical, but it seems that it
would be.

The second comment is that in the series of imputation strategies it
didn\'t appear that any of those took into account a seasonal effect so,
for example, if you look at one month lag, you\'re imputing a June
effect instead of a July effect. If you look at one year lag, then
you\'re getting that correct effect. But if you just take one month
back, you\'re not really doing that. So maybe you could combine the two
procedures where you\'d have a one year lag directly from that company
plus a seasonal effect estimated from companies in the current month or
something along those lines.

CHAIRPERSON RELLES: Seymour.

DR. SUDMAN: A question and a comment. First of all, a question. The
question is whether or not the companies that are tardy are actually
becoming non-respondents? Are they people who are failing to report at
all or do they simply report late because there would be a different way
of handling it. If they become non-respondents, then it seems to me that
\-- and this is a result perhaps of the fact that they\'ve gotten tired
of doing it or for whatever reason. Then it seems to me some sort of
substitution of those non-respondents may be in order or some different
sort of sampling scheme. If it\'s simply that they\'re tardy, then
obviously the methods that you\'re using I think are clearly the
appropriate ones.

The other is simply to just sort of second what Carol said. It seems to
me that the key issue in terms of people using the data is how do the
imputed data differ from the actual numbers in total rather than \-- You
don\'t report the data by individual companies, so it\'s the actual
difference that matters rather than the fact that the companies are
different.

CHAIRPERSON RELLES: David.

DR. BELLHOUSE: Yes. Actually following up on comments by Samprit, Carol
and Seymour all together. It seems to me that there might be some
further study that can be done if the companies that are not responding
are tardy. That if you\'re getting responses late, you will find out
what you have imputed and the actual result a little later on. The
imputation methods that you\'ve been using, since they\'re based on a
ratio, are implicitly based on a model of regression through the origin.
The ratio estimator is appropriate under regression through the origin
model.

So Samprit\'s idea of the sampling study looked like it was based on
non-response occurring at random. So if you now can follow up the tardy
people and you have the results of what you\'ve imputed, you have the
real results later, you can do the model, the regression through the
origin model, on the sample that you currently have. When the tardy
responses come in, you can see if there\'s a deviation from the model to
see what kind of systematic bias there might be in the estimation
procedure and do the correction in that way.

CHAIRPERSON RELLES: I\'d like to add one more thing about a lot of the
comments kind of look at the 40 companies much closer and I want to add
one more vote in that direction because the gas industry has been
deregulated and to what degree is something of this a lesson for what
might happen in electricity through regulation. Are they not filling out
these forms because of competitive reasons?

Are they unfettered by the federal requirements to send in the forms?
Given the trade-off between the need to keep that information secret and
the order that they\'re supposed to send the stuff in, is it the need to
keep the information secret that\'s dominating for them? I think
visiting a few of those 40 companies and talking to them would be really
instructive.

DR. PHIPPS: I\'d be interested to know also if it is the same 40
companies or are they changing over the 12 months?

AUDIENCE: Right. \--

MR. KASS: I was waiting for all of them to come up. There are several
themes that have appeared. Let me take the easy ones first. Why are
people becoming increasingly tardy? It has to do with what\'s going on
in the industry. Some of it is deregulation, some of it is downsizing,
some of it is mergers. All of it focuses around two things.

1\) the companies outside the beltway have fewer resources to fill in
our silly form and secondly, over the past few years \-- and this we get
from going out and talking to folks \-- the companies have developed new
accounting systems which are less user friendly than they had been
before and getting the information needed for our report is more
difficult.

And the reason for this, depending on which company, there\'s a common
theme. Systems seem to be developed \-- this is my interpretation \--
systems seem to be developed for accountants for purposes that are
important to the accountants, not for purposes that are important for
the poor stuckee who has to fill in forms for statistical purposes.

I have one guy at a company that is active in several states bemoan the
fact that his multi-million dollar new accounting system that went into
effect in January of this year, it took him until this summer to, by
pleading with the programmers to develop the reports needed and he was
six months with no reports.

So why the decrease in timeliness? Filling in forms for us is not a
profit center and the name of the game increasingly is increased
shareholder value. We\'re not there. We have become less important in
terms of the way the companies do business.

Now the flip side of that is that I say it\'s approximately 40
companies. It usually isn\'t the same ones. Right now I have a coterie
of three who have been derelict for a long time. When we get a series of
months where a company doesn\'t respond, we go into what could
euphemistically be called high impact falling out. That includes thus
far getting on airplanes and going down and talking with appropriate
vice presidents and the assembled staff. That gets responses. Sometimes
the responses don\'t last a long time but at least we get caught up.

So we don\'t have the same 40 all the time. Each month it\'s a different
group with a hard core. The problem is that if you have, say, two major
players in the state, one month one\'ll be off, one month the other will
be off. In terms of our customers, each of them accounts for more than
25 percent. Therefore, we\'re not getting the 75 percent. Therefore,
we\'re not going public.

The comment about ignoring current month loses current information is
true. Perhaps we can explore using some combined ratio or some way of
getting an indicator of what\'s going on current month versus previous
month. One thing that was suggested elsewhere is polling states. Right
now we do a state and sector specific analysis. We can pool states and
perhaps get some reasonably stable series of ratios to use.

One point that we didn\'t mention but is in the paper is what we intend
to do once we come up with this thing. We\'re not going to merely take
two or three months and go with it. We intend to apply it to at least a
year and retrospectively we can apply it to past years as well to see if
in fact what we get by doing whatever route we end up taking gives us
anything better.

And the 10 percent good or not. The Natural Gas Division has an effort
currently in place going to our customers to find out what the
requirements are. What information are we providing that they need? Is
there any information we are providing that they don\'t need? Lord help
us all on that one. And what level of accuracy do they need? It\'s
remarkable to me that folks who use our data can\'t come up with a
cogent statement saying I used this and if you can be 25 percent error,
that\'s not good. If you can be 10 percent error, that\'s not good. And
I say well, what is good? Literally now one guy said \-- it was a
national figure number, the number was roughly 1,500. He said I can
accept 10 out of that. Way less than one percent error. He has to be new
in the industry because our revision error is usually greater than that.
So we\'re trying to get an idea of how we can evaluate on a state level
the differences between whatever system we come up with generates and
what we currently generate and then go with it or not.

Okay. The point in our tables was that this is states. We have not
looked specifically at differences between imputed values and reported
values and, as I said, for the most part companies that are tardy do
eventually report. Because of the way the exercise was done, we didn\'t
separate out the weighted and unweighted values to do a case by case
kind of comparison. That\'s something we intend to do in the future.

And the use of supplemental and co-variate information. We explored it a
little bit. The level of effort involved if we could find something
simple would argue use something simple. Are we wed to use one of the
procedures? Absolutely not. If we find that there\'s something that does
real well for residential and another thing does real well for
industrial and another thing does real well for commercial, we drive the
truck. We can say this is what it\'s going to be.

And then the idea of carrying through the exercise and evaluating bias
of mean square is a good one. The point about having regression through
zero is a good one and we\'ll explore that. The seasonal effect. If we
hadn\'t used the ratio of current month to previous month, we would have
lost all seasonal effect and that\'s what we found. So the fact that
there are changes month to month has got to be built into the system
somehow or other. Whether we do it by getting analytic estimates of what
that seasonal effect will be in a month to month variation or actually
implying using reported data to get that difference remains to be seen.

CHAIRPERSON RELLES: Linda.

MS. CARLSON: I wanted to mention something that I hope Joan Heinkel,
who\'s head of the Natural Gas Division, helps me out a little bit. You
made the point \-- are linked to what\'s going on in electricity. In
fact, it\'s almost the other way around in that the electricity, the
fact that they learn from natural gas having this impact and electricity
is ahead of the game and now natural gas has literally a very major
redesign and evaluation project under way for all of their data
collection.

MS. HEINKEL: Right. Well, we\'re trying to do a couple of things that we
have known for a very long time, for example that we are recovering from
our industrial price system. Interestingly enough \-- oh, I\'m sorry.
We\'ve been a little bit surprised in that there hasn\'t, from an
analytical perspective, people are concerned about it but we haven\'t
had a big outcry from a lot of the users and I think what has happened
there is that they use different series to get out of some of the stock
market series and things like that. But what we do have coming up which
I think is causing a great deal of concern internally and I think
increasingly externally is with some of these retail and bundling
programs that we\'re seeing at the state levels. We are likely to start
losing our residential pricing series.

And I certainly think Congress will be very concerned about that and
internally we are extremely concerned about that. And it could happen
differentially. If one state goes completely retail on bundling, we
could lose the entire state whereas we might have all of Montana and
none of New York or something like that. So those are \-- I think some
of this retail and bundling has really provided the impetus or the push
where people are very worried about what might be happening over the
next few years or maybe over the next few months.

I think this has provided a lot of support for just a general review of
the entire program. You know, what does the market need? What do we as
an organization need to basically make sure we can answer policy making
questions? A couple of years ago \-- you saw those slides that Dave was
showing on the wellhead price, the natural gas wellhead price. We\'re
seeing increasingly increasing price volatility and some of that price
volatility does get carried through to, for example, residential
customers although it\'s dampened. But we had one year where in New
Mexico I think they saw probably 50 to 75 percent \-- I may be
over-stating. They saw a huge increase in their residential prices. We
got questions from the Hill. What\'s going on? And we really need to be
on top of this kind of stuff and understand what the policy makers are
going to need as well as generally what the consumers need to be
informed about. So we are trying to take a very broad look at this in
terms of what really makes sense now. There are other things happening.

Roy alluded to the fact that they\'re changing the way they keep
records. Things that were important to them in the past or that they
felt that could provide because they could pass through the costs,
they\'re not quite so cooperative about this and they\'re looking at
different ways of saving cost in their records. So we\'ve got a lot of
interesting questions that we\'re going to be looking at but basically
we are interested in a rather broad based reexamination of our system
and we\'re also interested in \--

Linda has a proposal on the table right now of ways of combining the
electric power and the gas surveys, if at all possible, and Linda is
thinking about going to \-- basically to the marketing segment,
companies like Enron where they market both power and natural gas and
see if there\'s some ways of reducing kind of the overall burden and
getting some interesting new approaches to surveys.

Linda, would you like to add anything?

MS. CARLSON: Yes. Actually, we started for natural gas a series of focus
groups. What is it? A year now?

MS. HEINKEL: It started in March.

MS. CARLSON: Yes. Literally going to both the user, broad-based, users,
respondents in some cases and various state \--

MS. MILLER: Consultants, all kinds of users.

MS. CARLSON: To start and literally begin work on what do you use, what
do you need, where do you think the whole market is going? And then
what\'s also started in the last \-- planning to do it for electricity
and ultimately for natural gas. Staff has been learning essentially how
to do cognitive interviewing and Seymour sat with us talking to Colleen,
one of the \-- who have been training and they\'re starting with some of
the natural gas respondents right now and I\'m going to drag Joan with
me next week or the week after to see the videos. Apparently it was very
painful about listening to how the respondents would respond to these
and what they perceive as \--

MS. HEINKEL: I was \--

MS. CARLSON: I\'m in pain.

MS. HEINKEL: Oh, great. I was kind of pleased and somewhat surprised at
the responses that we did get. I was a little bit worried that people
were not going to be coming to these kinds of interviews. They must have
something to say.

MS. CARLSON: Stan, did you want to throw something in? The last I heard
it was one fifty.

MR. FREEDMAN: But there was one respondent, in Chicago, who had some
very important things that she wanted to say about EIA which had to do
with making the monthly forms \-- similar to the annual forms, and she
would schedule an interview with her, I would come to her office, and
several times during the time of this interview she departed from the
protocol to make sure we heard her agenda of making these two reports
the same, because the monthly was easy to do and the annual was, how did
she put it, a real headache.

And those are anecdotal things from the interviews, so I think that they
are just as important as the non-anecdotal. I\'m not an expert on gas,
but it\'s interesting to me how at the meeting I was involved in, what
they understood how that industry corresponds to their view of that
industry. I think the same thing holds true with every industry that you
look at.

MS. CARLSON: I\'m guessing this is just the beginning for this section.

CHAIRPERSON RELLES: Okay. Well, that was a very nice discussion. I
appreciate all of the comments and let me announce a break which will
last for about 15 minutes. One other question for the formalities of the
reporting. Is anyone here who hasn\'t been in the room before and, if
that\'s true, would you just state your name and affiliation.

MS. CARLSON: I\'m Mary Carlson and I\'m with the Natural Gas Division.

CHAIRPERSON RELLES: Thank you. Okay. See you here in 15 minutes.

(Whereupon, off the record at 10:37 a.m. for a 19 minute break.)

CHAIRPERSON RELLES: The next scheduled talk is a response by Dwight
French to some suggestions made by the ASA Committee in the spring
meetings. Alternatives to reducing the cost of RECS. I had indicated
I\'d be summarizing committee responses, but it sounds like this paper
is about committee comments and responses, so there\'s no need for me to
say anything more. Dwight French will give the presentation.

MR. FRENCH: Thank you, and I appreciate the opportunity to be back.
Perhaps more than some of you do, I tend to go on and on, but I\'ll try
and keep it short this morning. For those of you who are new to the
committee, I am Dwight French. I\'m currently in charge of the energy
consumption area within EIA. The energy consumption area core data
programs consist of three surveys which cover the residential household,
the manufacturing establishment, and the commercial buildings
populations in this country. Together they represent about 55 or 60
percent of end use consumption in U.S. society.

The surveys themselves combine direct collection of information from a
sample of end users themselves regarding the structure of the end use
units. Some of their energy activities, for example, we would collect
information about conservation activities that take place and things
like fuel switching capability in the manufacturing sector.

The surveys are, at least the RECS and the CBECS, the residential which
is the household survey and the commercial building which we call CBEC,
everything is ECS at the end, Energy Consumption Survey. They\'re
complex multi-state surveys and among the most complex operations that
take place in EIA. I don\'t trust the technology. I know it\'s right
here in front of me, but I\'ve got to look to make sure if I click the
button it comes up.

Last meeting there were basically three major comments that were made
when I talked about the residential energy consumption survey. First was
to suggest \-- and this Seymour did in his discussion \-- suggest that
we consider telephone interviewing and also that we consider the
possibility of a continuous measurement process for the residential
survey in particular.

And Greta, I believe it was you that at the end in discussion when I had
mentioned the possibility of using electronic self-reporting for
conducting surveys, you said, Well, wait a minute. There\'s issues that
you have to face up to about‑- it wasn\'t you?

MS. LJUNG: No. I did ask about the cost, that the cost would eventually
go down.

MR. FRENCH: Okay. My recollection was you mentioned something about
problems, whether people would really be able to take a form like that
over an electronic medium and be able to answer the questions and so
forth. No. Okay. Then I recall differently.

MS. LJUNG: We may have to check the record but I don\'t recall that.

MR. FRENCH: Well, maybe it was someone else that made the comment, too.
Okay.

CHAIRPERSON RELLES: Would you like us to work harder to figure out who
it was?

MR. FRENCH: No. I\'ll talk about what I want to talk about. Okay. Let\'s
take the first one first concerning telephone interviewing. I\'ve got a
few things to say about this. First of all, there are some actual and
perhaps perceived things about telephone interviewing that are of
concern and it was mentioned last time that well, you can\'t necessarily
take all these things to heart. Things like concern about response rates
often drop in telephone surveys relative to what happens when you do
personal interviewing which is what historically we have done with the
residential and commercial building survey. Problems with telephones
with electronic gate keepers, answering machines, and also the personal
gate keepers, the secretaries in commercial places that might block you
off from talking with who you want to talk with.

Concern about there\'s a certain maximum length of a telephone interview
beyond which the telephone starts to grow into a person\'s ear and they
say enough is enough. Often in our surveys when we do personal
interviewing we have hand cards and other supplemental information that
we use to try to simplify particular question responses. If you\'re over
the telephone, you either have to work around this or you have to do
some sort of preliminary send out in order for people to have this
information before you actually conduct a telephone interview.

You also have problems potentially over a telephone in obtaining the
waivers to get the supplier information. In our energy consumption
surveys and residential and commercial, we get the characteristics
information from the end user. Historically, we have then gotten a
waiver to go to the energy suppliers to get the energy consumption and
expenditures data.

And one other thing that sometimes comes up is that in fact you do lose
the feel that you get from an on-site interviewer. This may be of
increasing concern as you go through the years and telephone surveys
become more and more ubiquitous and people are getting tired of them. A
telephone survey might be seen as the latest interruption in what is a
continuing series of interruptions over the telephone.

Well, in fact, we are trying telephone interviewing but not in the RECS.
We\'re planning on doing it for the 1999 upcoming Commercial Buildings
Energy Consumption Survey which will be conducted, field work will
probably end about a year from now. Why did we do it? Well, the first
and the most obvious reason, we ran into a funding problem and we had
about a \$1.5 million problem that we didn\'t know what to do with and
we were not going to get additional money so we had to do something
relatively drastic and so we are at this point planning on going to a
telephone approach.

Along with that \-- and this is an associated thing which I\'ll be
talking about a little bit in a couple of minutes \-- we also may need
to collect energy data from building respondents rather than suppliers.
This is something with regard to the deregulation that was being
discussed a couple of minutes ago that we are looking in through and
trying to be a little bit pro-active in this regard also.

Okay. Because of the very different nature of what the 1999 CBECS
apparently will look like, we are going to conduct two pre-tests and, in
fact, have already gone through one pre-test and have some limited
information which I\'ll share in a moment or two. The basic things that
we were looking at in the pre-test were to determine whether in a
telephone access appropriate respondents can be identified and how much
trouble it is, whether in fact you can collect the building data using
computer-assisted telephone interviewing, whether building respondents
can provide energy consumption and expenditures data as well as the
building characteristics data and whether respondents could and would
mail or fax authorization forms to us if in fact they can\'t provide the
energy data and we would need to go to the energy suppliers.

Borrowing from Family Feud, and the survey said, the pre-test. Finding
appropriate respondent has been a difficult exercise in the pre-test. I
guess last I knew we were up to about 50 percent response in the
pre-test or thereabouts and the average number of contacts to get a
completed interview was seven and you\'re going through a lot of getting
to a preliminary person. Oh, I think this person is the proper person
and you call that person and no, they are not and they send you to
somebody else and well, this person is the proper person but they\'re
not around right now and you\'ll have to call back. Then you try to make
an appointment and the person says well, I\'m going to have to check my
calendar and I\'ve got a meeting I have to run off and so forth and so
on.

Some good news. Gate keepers in and of themselves in commercial
buildings don\'t seem to be a major problem. If you can get to an
appropriate respondent, it seemingly can provide most of the building
data. That shouldn\'t be too surprising. Somebody ought to be able to
equally provide straightforward building characteristics, etcetera, over
the telephone equally as well as they might be able to in a personal
interview. And amazingly enough, quite a few can provide energy
consumption and expenditure data. Based on very cursory information from
about 75 respondents in the pre-test, it could be that a substantial
majority of electricity and natural gas respondents can provide such
information. The numbers were actually about 80 percent for electricity
and about two-thirds for natural gas, but I hesitate to quote those
proportions because again, the sample sizes are small and this was a
special pre-test situation.

Another surprise is that the length of the interview, which was about 45
minutes in the pre-test on the average I\'ve been told, didn\'t seem to
be a problem with people. We are really more certain of the quality of
the building characteristics data that we will be getting than the
energy data, and our friends from SMG are going to be doing a validation
study for us in-house where they\'re going to be getting information
from the energy suppliers and comparing it with what the respondents say
so we can take a look at what quality of information we get.

The CATI instrument that we had, which was programmed in-house by our
own Joelle Davis, seemed to work very well and we certainly want to give
kudos to Joelle for that. I say at the end of this, stay tuned for the
next pre-test and information about validation of the energy data. I
shudder to say that because that probably gets me an invitation to the
spring meeting next year.

I\'ll try and mosey rapidly through the rest of this. We have done some
preliminary thinking regarding continuous measurement. Certainly any
move in this direction would be an extended process. What would we have
to deal with? Well, first of all, if we were going to get the way
Seymour had suggested rotating a certain proportion of a sample in,
let\'s say, every year, if we were to get comparable quality data in the
same four year period as we were now conducting consumption surveys and
we were to do it one-quarter each year, we would have to start
immediately after an existing cycle. This is going to have some resource
and logistics implications that we would have to deal with.

We also would have to deal with contacting energy suppliers more often
if we were going to go ahead with our historical manner of getting data
from energy suppliers and, of course, the alternative would be to
perhaps collect energy data from respondents, and we\'re going to check
that out using the \'99 CBECS so we\'ll know a lot more about that, at
least for the commercial sector.

One of the problems we may be having in getting the information from
respondents themselves is that they don\'t have a full year of data.
Historically, we have asked for and have been supplied data actually
over a 14 month period by energy suppliers. We may not be able to get
that from actual end user respondents and that may mean that we would
have to use, if we were going to use that data, to develop some sort of
statistical techniques to offset some of the data losses from not having
12 months of data.

Now, this isn\'t a totally unforeseen thing. In fact, we already do some
of this imputing for missing consumption data from the utilities right
now. But in fact what we have is complete consumption records for the
same time period from some buildings that are a tremendous aid in doing
this type of imputation. We may have less or not at all any of that if
we went to an end user providing energy data mode.

Also, unless the survey was telephone-based, if we were to go back to
personal interviewing, geographic distribution of cases each year would
be an issue. We have limited sample sizes and we really have skeleton
crews in each of our first stage areas for our surveys. If we were to do
one-quarter of the interviews a year, we\'d have a real logistics
problem with scheduling interviewers unless we had interviewers
traveling between PSUs. Alternatively, we could do one-quarter of the
PSUs each year but then we couldn\'t roll up the estimates until a four
year period was over to do national information.

I make the next comment with a rolling estimator. That is, rolling each
year and accumulating four years of data. I\'m not saying that there
would be any problem with this, but if you did want to change
questionnaire wording, it would be an interesting methodological issue
because you would essentially be rolling in any changes over a several
year period following introduction of new wording.

Finally, I would make the comment that it is difficult to know what is
going to happen over the next few years with regard to designing and I
will get to this a little bit more in a moment. But I would say for RECS
the availability of the American community survey and updated housing
address listings after the year 2000 enters into the considerations
because if we have address listings like this, what we may have is a
very real way to get around the costs of survey redesign for RECS but we
may not have good information to deal with the telephone survey. So it
may be that what we would do is decide in this case to use that as a
design and stay with a personal interview for the RECS, although one
might do a continuous measurement type process. That would be a separate
consideration, separate from the telephone interviewing.

However, the availability of this survey and the housing address
listings is only half of the issue. We also have to have, in order to
make full use of this information, the data sharing confidentiality
legislation that is currently pending. That\'s certainly a concern.

I\'ll quickly go over the concerning difficulty in electronic
self-reporting. Certainly response rates and getting supplier waivers
would be difficult and possibly there would be some difficulties in
respondents answering questionnaires that flash up on their screen. You
probably have to use a multi-modal approach if you were considering
electronic self-response. You\'d have to mail out support information,
maybe send some over the Internet. You may have to have a hot line for
questions from people, an 800 number, something like that. You\'d either
have electronic submission. If we needed forms for the suppliers, the
waiver forms, they may have to be faxed in if you have certain people at
home that are faxing.

Obviously I\'m talking about a variety of media which many homes, most
homes don\'t have right now. However, our last 1997 RECS indicated that
35 percent of the homes in the United States now have PCs and a majority
of those have Internet connections and everything you see is that
Internet use and PC use is continuing to explode. I wouldn\'t be
surprised if in the next RECS we found that well over half of the
households in the United States had PCs and maybe Internet connections
might be pushing half of the households in the United States. It\'s just
growing by leaps and bounds.

As an intermediate stage, I would suggest that before we would actually
do an electronic survey what we might end up, especially in the
household sector, is giving respondents the option electronically, to
getting certain forms or support materials comparable to the hand cards
we now use for some of our complex questions transmitted over the
Internet so that they could receive them. Not that we would force it on
them but they would have the opportunity to use something like that.

Well, conclusions. Our future design plans are going to depend upon
statistical and logistical developments in the next few years and there
are a number of them. I have mentioned some of them. The availability of
electronic survey techniques, how fast they penetrate.

How de-regulation in the utility industries plays out and how much we
are going to have to go to self-reporting of energy data or try to
rather than use utility or some other supplier reporting of energy data.
Whether we can get the American community survey and the housing lists
that might be developed from that, whether in fact that will occur.

Along with that, something I hadn\'t mentioned previously. A study that
we\'re doing right now on the possible use of establishment lists as a
frame for looking at commercial buildings rather than going to
commercial buildings directly. We\'re looking at that right now. The
concern about whether or not the confidentiality data legislation
sharing will be passed. How well telephone interviewing works and if we
need waivers from respondents for suppliers of energy data and how we
get them.

Certainly the second bullet there. Funding for redesign and survey
implementation is going to remain uncertain. We have a significant
amount of money planned in upcoming budgets for that. Frankly, I\'m
pessimistic that we\'ll get it. I think it\'s going to be shot down or a
lot of it is going to be shot down. We\'ve got it scheduled to be phased
in over five years. I\'m not really, really optimistic about that but
how much money we get is going to really affect how we think about our
redesigning over the next five years or so.

And, as a result, our thinking and our planning is going to have to be
as flexible as we can possibly make it. Certainly I would welcome any
comments from the committee about the specific things I have mentioned
but also if you have any comments or suggestions on how we might best
maintain flexibility in our survey planning over the next several years,
I\'d be more than happy to hear about it.

Thank you very much.

CHAIRPERSON RELLES: Thank you, Dwight.

MR. FRENCH: Sure.

CHAIRPERSON RELLES: Jay, please.

DR. BREIDT: I\'ve got two comments, two related comments. One is that if
you are collecting data from the respondents, you would probably also
want to collect provider data or supplier data on some sub-sample of
respondents, so that you could more appropriately model the relationship
between the two. Particularly, when you have these problems of only a
limited number of months for the respondent as opposed to 14 months of
data from the provider. So a kind of two-phased sampling design might be
a good idea there.

And a similar idea would apply if you go to some sort of question
wording. Any time you do that, it would pay you to do it both ways, do
it the old way and do it the new way so that you have a means of
comparing those and modeling the differences?

CHAIRPERSON RELLES: Seymour.

DR. SUDMAN: I\'d like to comment on the pre-test, on your telephone
pre-test. I think those results are very, very interesting and really
somewhat encouraging. The first point you made is about the difficulty
with assigning the proper context. Well, heck, join the club.

I mean it\'s always the case and in doing surveys of people in the
commercial field that getting the right person and getting that person
available is very, very difficult but when you think about it, it\'s a
heck of a lot cheaper to do it by phone than it is to try to go back
again and again face to face.

I mean it\'s not that this is a problem which is caused by the phone. If
anything, the cost in doing it by phone are reduced. So it\'s an
advantage. But having said that, it\'s never going to be easy. It\'s one
of the toughest things that all of us face who do surveys of people in
the business world.

And the second thing, of course, also has to do with the quality of the
data you get and again, it seems to me it\'s an issue which is quite
important. I agree with Jay that it\'s very, very useful to compare
those results with supplier results but it\'s not a mode issue. It\'s
not an issue that talking to people on the phone is going to give you
poorer data than talking to them face to face. It\'s an issue of do they
have the data and how good are the data they have relative to suppliers.

So while I think it\'s definitely a very important issue to explore, it
is not again something which argues against calling.

DR. PHIPPS: Also, just on the Internet part. I mean in with the
commercial builders. I mean it just seems to me that that\'s much less
of a hurdle for the commercial suppliers or the commercial building. You
could ask at the end of the telephone interview if they have access to
the Internet and try that the next time in reporting if you want to
increase the timeliness or if it\'s \-- you know, I don\'t know how much
it involves the records that they may need a little bit of time and you
could increase the quality of data by having them do it.

MR. FRENCH: Actually, I wonder if that\'s something we could do in the
second pre-test.

MS. LEACH: Right now can we make all of our materials available via
Internet? If they know we gave them a choice, that is mail them, fax
them, or they could actually get on the Internet. Very few, if any, go
to the Internet. Most of them wanted it mailed, which surprised me.
Recently we faxed them all, they weren\'t mailed. Again, I don\'t know
whether that was another way of delaying the survey. If we mail them, we
don\'t call you back in five days. If we fax them, we call you tomorrow.

DR. PHIPPS: I think just knowing if they have the Internet is useful. I
think you have to kind of wean respondents off things.

MS. LEACH: We\'re trying. So far, making it voluntarily did not help. I
think next time we make them say fax or Internet, specifically, because
the mailing costs us a lot of money.

DR. BREIDT: Just a question. Did you make any attempt to contact these
places by mail before contacting them by phone to try to line up the
appropriate \--

MR. FRENCH: Yes.

MS. LEACH: Yes. We called to get a contact person and a mailing address.
We mail the stuff. We mailed all of the materials and cards, background
expenditures, the whole package. We called them again and ended up
mailing the packets all over again to different people. I personally sat
in. I monitored a couple of phone calls where we were sending out our
third packet and we still hadn\'t gotten somebody at home to interview.

The best part is if you get somebody to give an interview, it\'s no
problem. If you finally get them to sit down with you, it\'s a very good
indicator. But if you\'re trying to find \--

DR. PHIPPS: When you start out, you only have the address. Right? In
your sample frame.

MS. LEACH: Well, these happen to be a model.

DR. PHIPPS: So you have a telephone number.

MS. LEACH: For the new buildings we didn\'t always have that.

DR. PHIPPS: How difficult is it to get the first telephone number?

MS. LEACH: I think we\'re going to try to do more of the pre-screening
to try to get to the appropriate respondent before we actually go into
the field. We\'re going to try that. You have your training interviewers
calling back and making the phone calls and then getting transferred two
or three times. We\'re hoping we have more of an answer to some of that.
Save some money that way. We are having a big meeting on that, on
Monday.

MR. FRENCH: Can I make one comment?

CHAIRPERSON RELLES: Oh, please.

MR. FRENCH: I thank that the people on the committee for all their
comments. They\'re certainly all well taken. The only thing I would
raise any question about is this issue of whether telephone interviewing
and personal interviewing are different with regard to the number of
contacts that are necessary. I\'ll be honest with you. I don\'t have any
information to dispute what you say about it\'s not a mode issue. It
could be. You could get the little old lady syndrome where the person is
on the door and so people say, Well, while you\'re here we want to get
you further along in the process toward getting a respondent, even if we
don\'t get you all the way to it whereas if you\'re on the phone it\'s
easy to say, Well, I\'ll have so and so call you back later. Click. As I
said, I have no information with this. Certainly we\'re getting more
contacts necessary in this particular cycle than in previous cycles but
there\'s two things being confounded here. #1 is the potential issue of
personal versus telephone interviewing and the other is this just
gradual eating away at respondent concern with responding to surveys and
you\'re having to work harder and harder over the years to get the
respondents. So there may be a lot of that in it. In fact, there may not
be a difference between telephone and personal interviewing. It may all
be the increased resistance effect. I really can\'t say.

DR. PHIPPS: I just have one quick comment on the address requirement and
using other people to do the address requirement and then an interviewer
do the interviewing. I\'d test that because I know a study that I worked
on, we had a lot of difficulty. I mean the original people weren\'t
invested in getting the right address and so the interviewer than ended
up having to do a lot of the work. So I\'d test it out.

CHAIRPERSON RELLES: I\'d like to ask a question whether there\'s a need
to panic here. I panicked when I heard seven people.

MR. FRENCH: Contacts.

CHAIRPERSON RELLES: Yes. And Seymour said don\'t panic on that. But what
about the 50 percent response rate? Is that dramatically lower than what
you\'ve experienced in the past?

MS. LEACH: Yes. We\'re very concerned about that.

CHAIRPERSON RELLES: What are the response rates that you have been
experiencing?

MS. LEACH: We\'ve had around 80.

CHAIRPERSON RELLES: Eighty. So there\'s a drop off of over 30 percent.
How much of that might be due to telephone?

DR. SUDMAN: One thing that seems to turn out has to do with the
experience of the interviewers. I mean if you\'re comparing the results
of experienced face to face interviewers with people who are doing
something relatively new on the phone, then the experience of the
interviewers will certainly have \-- I mean this has been sort of a case
over and over again, for example, at Michigan and places when they
switch from face to face to telephone.

They would find response rates lowered because the interviewers weren\'t
as experienced which is not to say that there is not some effect of
phone on cooperation because there is some of that. But some of it has
to do with interviewer experience. But I mean in order to save the
money, you may have to be willing to settle for a somewhat lower
completion rate.

MS. LEACH: For monitoring phone calls, a lot of it is just voice-mail. I
mean, we\'d get voice-mail after voice-mail. We finally had to leave an
800 number for them to call back and we would have to pay to have an
interview. And we got quite a few people on that.

DR. PHIPPS: Did it make a difference in the respondents who\'d been in
the survey? Was it the new respondents that had a lower response rate?

MS. LEACH: It made no difference. We haven\'t been there in three and a
half years \--

CHAIRPERSON RELLES: I\'d like to say something other than okay, come
back next spring and tell us how you\'re doing. And I\'d like to get
your suggestions because a 40 percent response rate doesn\'t sound like
a good thing and because we have a lot of strength here in exactly these
issues. I wonder if there\'s some way we can try to be of help in your
ongoing deliberations?

We\'ve heard only about sort of the high level issues but you\'ve got
zillions of decisions you\'re making on a day by day basis and I just
wonder if you\'ve thought about ways that the committee might be, aside
from sort of coming here and giving a lot of free time, whether there
are some ways we could actually be of help either on an ongoing basis or
perhaps by scheduling a session at the next meeting where we kind of
break out and spend a half day in relatively small groups with different
areas of EIA but you obviously get the survey people, all of whom happen
to be sitting right there. That\'s kind of the least we can do, but is
there more we can do? Have you thought about that?

MS. LEACH: When is the next meeting?

MS. CARLSON: But you can essentially set up an e-mail update for them.
You actually have a history of using the committee in between meetings.

MS. LEACH: Yes. I think you have to get formal results from the
researcher. Some committee members will be going out in March.

CHAIRPERSON RELLES: I just volunteered you.

DR. SUDMAN: I\'m just going to throw out one suggestion which I\'m sure
is going to horrify, shock all the people here because it\'s not
something the feds do altogether. One of the things that seems to be
turning out now, I\'m hearing more and more about this from the academic
sector, is paying people to participate in phone interviews is having a
significant effect on increasing cooperation.

I make that as a statement of fact and no recommendation as to what you
should do. I know the notion that citizens should all feel obligated to
report to government agencies is a long-standing feeling but it may
indeed, one of the things that may well be happening is that some form
of compensation will be required to keep cooperation rates high on
telephone surveys.

CHAIRPERSON RELLES: Linda.

MS. CARLSON: Have you advanced this idea to the census bureau?

DR. SUDMAN: No, no. I haven\'t dared.

CHAIRPERSON RELLES: You can spend your money on additional interviewer
time or you can spend your money on respondent time. Where does it cost
less and does the Constitution say anything about which of those two you
should prefer? I\'m not a Constitutional expert.

Well, if I can volunteer the committee to receive your e-mail, I will
certainly do that. I\'d like to be on the distribution list, too, and I
think we all feel like we\'d want to stay closely connected with this
and some people don\'t sort of carry a card that says I\'m a sample
survey specialist but I think there\'d be a lot of interest in the
committee and I would volunteer to try to sample the committee and make
sure I had a group that could work with you and then we\'d try to have
that culminate with a break out session at the next meeting of a half
day perhaps. We\'ll have to talk about that more.

Well, thank you very much, Dwight. It\'s really exciting to see this.

And now we\'re scheduled for our last talk which is entitled DIANA: The
Disclosure Analysis Software to Protect Sensitive Information in Tabular
Data and Bob Jewett from the Census Bureau is going to be making the
presentation. When I first saw reference to DIANA, I thought that would
be great. We\'ll have a high tech closure on this meeting. I believe
DIANA is a high tech piece of software. I must say I was a little
disappointed when I saw you bringing these large pieces of paper behind
me but I might get over it.

MR. JEWETT: You\'ll survive.

I work at the Census Bureau and our job is to simply gather data from
respondents in business establishments and people like you that publish
statistical reports. I personally guarantee the data that you give us
will be kept secret. We do our best to do that. And one of our favorite
stories is we tell during World War II the Defense Department asked us
for the names of citizens who had Japanese surnames and we wouldn\'t
tell them. That\'s one of the things we\'re very proud of. If they had
threatened to bomb us, maybe we would have changed our mind. But we did
not give them the names of people with Japanese surnames.

All the people at Census Bureau are very aware of this obligation to
keep the data sensitive and we have a certain rule we have to follow
called Title 13 that we have to follow. Very few people know what Title
13 says exactly. Very few people. In fact, if you\'re ever at Census
Bureau meeting one way to really get yourself noticed is to ask people
what does Title 13 say? I mean the exact statement of Title 13, very few
people know it. What it says is that the Census Bureau can\'t make any
publication or data where any individual or business establishment can
be identified.

The key words there are this can be identified. What does it really mean
to identify someone\'s data? Say if someone had a questionnaire, an
agriculture questionnaire, how many pigs did you sell last year? And the
person told us I sold 1,000 pigs. Well, if we published a report that
says we can derive the number of pigs a person sold between 1,000 and
one and 999, if we identified this data or is this okay for us to do or
should we avoid doing things like this?

Well, there are people I\'ve heard say that this is okay. But most of us
think we should not do this. But how about if we can tell the person the
number of pigs a person sold was between 1,001 and say 200? Is that
okay? The degree of uncertainty here is pretty large but this was pretty
close to the true value.

So most people say this is still not okay, that we need to have an
uncertainty interval, at least an uncertainty interval on both sides of
the true value. Something maybe like this would be might be more
reasonable. We have 10 percent here and 15 percent there. But then how
about this case where you have only five percent here but the 70 percent
here. Which of these two cases is better? Here you have 10 percent on
both sides. Here it\'s more on one side. So you have to really think
about these things and identify what do you mean to identify someone\'s
data. I mean if we can argue over what the word is these days, if we can
argue over that, we can sure argue over what the word identified means.

To make things simple, say we want at least 10 percent on every side of
the respondent. Just to make things simple, let\'s assume we have a 10
percent of what identified means. But this is the first real discussion
is to discuss that one word. And so we\'ll assume that 10 percent is our
goal. Then we have the question well, say we want to publish the number
of pigs sold in a certain county. That\'s our goal. And you have inside
this county one respondent who sold 1,000, a second respondent who sold,
say, 500, and the rest, the remainder, sold 100. Two large farms and a
few small ones for a total of 1,600.

The question is could we publish this number and not identify some \--
Well, we could say what if you publish that number, the 1,600, and what
is really being identified? You have respondent A and respondent B. You
know there are some other people. But are you really identifying
anyone\'s data by publishing the number? Well, as most of us look at
that number, we don\'t know what these people are. However, this
respondent B, he knows what he is. He knows he\'s 500. And he knows this
remainder is at least greater than zero. So then he can derive that
respondent A is less than 500. So respondent B can tell a few things
about respondent A.

But you say, hey, but it\'s still okay because this was still 10 percent
over the true value. No problem. This is still okay. Other people would
say this is complete nonsense. This is nonsense because what\'s the
chance this remainder is going to be a zero? I mean come on. If these
guys sold those pigs, they wouldn\'t be in business. They\'d be
statisticians or something. They wouldn\'t be doing real work. They\'d
be doing something else. So we know these guys aren\'t zero. They must
have sold \-- common sense would tell you maybe these guys are at least
50. Just common sense. But if you assume common sense, then you say,
hey, it\'s less than equal to 1,050. Now you\'re in trouble.

So which way is more reasonable? To assume you know nothing and you get
10 percent so-called protection or to assume you have common sense and
you get less protection? This is an issue that you just have to decide.
Now, let\'s just make it simple. We\'re going to assume we know nothing
at all. We\'re going to assume this case here we know nothing and so
this cell here is safe to publish. You might have what you call a
suppression rule. This remainder is like one-tenth of your respondents.
You can publish the cell. This could be your suppression rule.

But the point I want to make here, the main point I want to make in this
whole \-- is that when you form a suppression rule, the way you should
do it is to go back to what you\'re trying to do in the first place,
figure what these words mean, and then decide what assumptions you need
to make before you establish that rule. It\'s a logical process.

Now, if you do this, you\'ll be the first people to ever do it because
no one that I know has ever done this before. Most people I know of just
start with a rule like if this is 10 percent of that, it\'s okay. If
it\'s 20 percent of that, it\'s okay. They just choose some percentage.
If it\'s 30 percent, it\'s okay. They choose a percentage and go with
it. And they never quite think how this rule corresponds to what
they\'re trying to achieve in the first place.

Now, I recall bringing this up one time at a meeting with this person. I
told someone, You have no disclosure policy. He said, \"Sure we do.\" We
have a rule. But I was trying to get at the idea of a policy to go from
what you\'re trying to achieve into this. So that would be my main
advice since your job is to give advice. My main advice to you is
you\'re working with someone who\'s trying to protect the data for their
respondents. When you have a rule like this, for God\'s sake, think what
that rule is achieving. Don\'t just take it as a starting point. Try to
think what is the rule actually trying to get at?

Say we just make this assumption. We know nothing. We make the
assumption that 10 percent is okay. We get this rule. This is 10 percent
of the first respondent. We\'re okay. But these are all things \-- the
Census Bureau doesn\'t quite do it that way but this would be \--
because a simple example, this is going to be even easier using 10
percent instead of 14 1/2 percent or something. So anyway so eventually
you get this rule for what cells can be published and what cells can not
be published.

Then the fun begins because you may have some complicated table like
this where the rows all add to a total and the columns add to a total
and they have some cell here, say right here, and you realize we simply
should not publish that cell. But if you have things, if you just take
that one cell out, you can easily compute it. These things add to a
total. You can subtract those from the total and get it. But they also
add like this. So if you have one thing that you have to leave out, you
probably have things that suppress some more cells.

And now choosing these cells can be very tricky because say you have
even more cells to suppress like this. You would look at this table and
say, as far as I can tell, you can\'t drive these people\'s data. Every
column has at least two things suppressed or all the rows have \-- in
this case, there\'s one person by himself or one cell by itself. So this
appears to be a very safe publication. But you can be fooled because
this cell right here, you can compute this cell.

An example that I gave you in the handout I gave you shows a pretty
simple way to compute this but not that simple but you can compute the
cell algebraically. So by doing some manipulation with this, all the
sums going back and around and forth, you can compute that cell. So the
question is so what because it goes back to our basic rule in the first
place. Try and make sure that no one\'s data can be identified. We
discussed that where identified we said it was going to be 10 percent
but what does the word can mean? Does can mean can in all possible
theory be identified or can after going through all this algebraic
manipulation can be identified? Or does it mean can it reasonably be
identified? If you publish a table like this, is it reasonable to expect
someone is going to take these other cells and do all these algebraic
computations and compute this cell? Is that a reasonable thing to
expect? Are we trying to guard against that?

Well, some people say no. Those people aren\'t in the Census Bureau but
I\'ve worked with some people outside the Census Bureau who aren\'t
afraid of this. They say \-- this cell here can be identified and they
say, Oh, who cares? Someone has got to do that. That to me is a very
reasonable approach. It\'s a very logical approach. If you interpret
this word as can reasonably be identified, this thing doesn\'t matter
but if you interpret it as can possibly be identified, that matters. So
you have to decide how you\'re going to choose to interpret this word.

Now, choosing the proper cells to suppress is a very, very tricky thing
to do. All I can say is thank God I didn\'t have to do it because with
the computer program here\'s how we do it. If we have a table like this
or things added to a total and added that way, you have this thing
called a network like this and all these lines across like this and you
can kind of correspond the table and the network like this cell up here
A happens to be the big thing around like this. B, C and D happen to be
these. And you can convert this thing into a network.

When I first saw this, I thought statisticians are always making simple
things hard. They\'re always taking simple ideas and trying to make them
sound impressive. So this really turned me off. But then I realized what
it is we have computer programs to analyze networks. We have a lot of
specialized programs so if you look at it like this, you can use one of
these specialized programs, it\'ll do the work for you. So if you have a
table up here and you have a suppression here and you want to choose
some more cells to protect this suppression, this cell here corresponds
to what they call an arc inside of here. It corresponds to an arc and
choosing more cells to suppress corresponds to some kind of a closed
path inside the pair.

So all you do is use this formulation and use some program from the
University of Texas and they\'ll tell you what cells to suppress. It\'s
a beautiful, beautiful way of doing things. Some tables are very complex
and you need a lot more than three cells to protect the respondent. It
seems like where there were hundreds of cells suppressed to protect one
cell, really hundreds of cells were suppressed to protect one cell. This
cell happened to be a very, very large important cell and you couldn\'t
choose small cells to do it. So hundreds. And this thing here does it
for you.

So even though I was skeptical at first, this little program that uses
these networks is a wonderful thing. It\'s called a Minimal Cost Flow
Program and it runs very, very fast. I used it one tables where we had
600 rows and like 50 columns which is 30,000 cells and the thing runs.
It\'s a real work horse. But of course it only works on two dimensional
tables. When you get into a three dimensional table‑-

CHAIRPERSON RELLES: I\'m sorry. The decision to suppress those cells is
based on the fact that there\'s one respondent?

MR. JEWETT: Or it fails \--

CHAIRPERSON RELLES: It fails some traditional \--

MR. JEWETT: That\'s right. It failed your rule of over here. Like we
said we\'re going to have a rule where the remainder is 10 percent of
your first respondent. If it\'s less, than I fail that rule. So this one
fails the rule. If you have any questions, just feel free to ask them.

But this only works on a two dimensional table. If the thing happens to
be three dimensional like this, well, you just can\'t do it this way.
You can\'t convert this three dimensional table into a network. You have
to use some kind of fancy linear programming technique. We have a way of
doing that. We have a nice linear program package that we use for the
three dimensional tables. So instead of forming these little things like
this, we form this little cube that says depressed cells. It works real
nice but it runs so slow. I mean it\'s just terribly slow. It\'s just
not usable. For real data we can\'t use our fancy technique.

So a three dimensional table, what we have to do there is we look at
them like in two dimensional pieces. We take this two dimensional table
and do it like this. We take one other two dimensional table that\'s
like this and do it the same way. Once we finish those, we just kind of
check like this to make sure that looking at it in the third dimension
we don\'t have just one cell suppressed inside of one of those third
dimensions. It\'s very much like this table over here. If you look at
this one dimensionally, things are fine. If you look at it two
dimensionally, you see a problem. But over here, if you do it our way
two dimensionally and one dimensionally, maybe it seems okay. If you
look at it three dimensionally, it may have problems. That\'s just a
fact of life.

And so is our way acceptable even though there may be problems? Is our
way a decent way to do it? Well, it all gets back to the word can. Are
you really trying to guard against these really tricky problems? Our way
can have problems but does anyone care? Are we saying can it possibly be
identified or can it reasonably be identified? If we have some table to
say 30,000 cells and a lot of them are suppressed, and someone says Hey,
Bob, I found two of your cells you can compute using the fancy three
dimensional linear programming and ran the program for two hours, our
computer saved your cell. Well, does someone care about that? That\'s
about where things stand.

We have programs that can look at these things and if we have problems
they can find the problems. We can fix them by hand. But we haven\'t got
a real good technique to look at two dimensional tables and do a
thorough job in theory. Now, we can do it on small tables but not on our
tables. Now, Ramesh is going to follow me. I think he\'s going to talk
about a technique that works on tables of entire dimensions, three and
four and five dimensions. But our technique is a real work horse if you
do it on a lot of tables. Most of our tables are two dimensions but the
ones that are two dimensions can still run. It just has this problem in
theory. That\'s just the way it is.

Does anybody have any comments or questions right now?

DR. BREIDT: Would it give you any more confidence to do all possible two
way slices?

MR. JEWETT: Sure, but I just didn\'t feel like doing that. I just
didn\'t \-- that\'s right. That\'s a good question. I thought about
that, but I was under pressure to get a good technique that actually
works by a certain amount of time. I just didn\'t have the courage to do
that. It\'s a good question though. Any more questions like that?

DR. CHATTERJEE: I think that your prescribed three dimensional table,
things might work a little faster if you do the system in integers.

MR. JEWETT: That\'s true. If I had integer programming \-- I\'ve heard
that the best way to do this is an integer program, whatever that means.
But ours does require integer solutions, you\'re right. That\'s true.
Any more comments? I guess that\'s all I have to say.

CHAIRPERSON RELLES: I\'m sorry. Ramesh is going to follow you.

MR. JEWETT: Ramesh is next.

MR. DANDEKAR: Recently EIA converted this synthesis software to work on
the PC and I\'m going to limit my presentation to the PC version of this
disclosure analysis software. Our plans are to make this disclosure
analysis software available to the general public by using EIA\'s
website as well as on the CD-ROM. We have software which is designed to
install the disclosure analysis software on the user-specified disk
drive and the directory.

In addition to the acceptable software code, the user has an option to
install the software documentation two test cases and the software
source code on the separate sub-directory on the hard disk. This
self-extracting software creates a separate icon for disclosure analysis
software in either specific window. I\'m just going to show you a
demonstration of how easy it is to install this software.

Basically to install the software from the CD-ROM we\'ll be going to
this CD-ROM icon under My Computer and there it is basically just
clicking on this instructing software. Most of the installation is very
automatic. There is not much of a user interface required there.

You have basically the welcome screen which asks you where would you
like to install the software? I\'m just going to use the default
directory on the C drive to install the software. User has an option to
install the program files with the entire documentation which I will
label in the PDF format which can be read by using the Acrobat reader.
If you don\'t want to look at the source codes, which is written in
Fortran, these front screens and all that which are in the Visual Basic,
all those are in the source codes and then we have two different tape
data files for a user after they install the software on their computer
to check it out. I\'m basically asked to install all these models on the
hard drive.

In addition to installing the software, you have the option to have icon
to be fit into any of the folders. I\'m going to ask it to install this
in the accessories folder, the icon for this. Basically everything is
unpacked, okay. The software installation is complete. I\'m looking for
DIANA. Okay. At this particular point, the user, assuming he has
prepared all the input files required to run the software on the PC,
there are multiple files required to run the software. Basically, the
user has to specify how different rows are related in these two
different files, how the columns are related. Multiple columns can be
processed at the same time. And your tabular data which basically
explains the cell value, largest contributor to the cell, second
largest, and all that.

In this particular example for this particular data, has 10 different
levels in the parallel one. This is a big table. We need to process \--
basically specified protection rule you want to use which is correctly
stated 15 percent. We can reduce it to 10 percent of all what Bob was
talking about. And next step is to collect into the appropriate file.
I\'ll take data and look at it in the data directory. I\'m going to go
there and I\'m going to basically plug in this file. Row relation file
is in the file to that input. Column relation is in File 3 and our main
data is in the File 4 and just basically start running the software.

Looks like I made a mistake here. Looks like there was some kind of
installation error here.

CHAIRPERSON RELLES: Well, we get a feel for how you \-- I would like to
have you discuss the kinds of problems you see applying this.

MR. DANDEKAR: This particular software has been extensively used at
Census.

CHAIRPERSON RELLES: Yes, but EIA\'s intention.

MR. DANDEKAR: Okay. EIA\'s intention. We are going to make the software
available to all of our users who want to use it, either through our web
site as well as with CD-ROM. We are making some small tests to make sure
that the PC version will do same kind of process.

CHAIRPERSON RELLES: Yes, but the applications area being what?
Suppression of electricity sample data or \--

MR. DANDEKAR: Again, we have not come to that stage yet. Software is
ready to use. We are waiting for \-- like Bob mentioned before, first we
have to determine what can be identified, what is the goal, what is the
policy and once I think we are in that stage, if I\'m not making a
mistake, and then accordingly tune up the software to the user
requirement and then do the processing and all that. But again, I do not
have any input. Would you like to comment about electric \--

MS. CARLSON: Essentially we\'re at the stage, as soon as this is up on
EIA web site and tested, internal marketing of the software within the
various program offices in EIA itself.

CHAIRPERSON RELLES: But who\'s showing interest in using it at this
point? Electricity?

MS. CARLSON: No, not so much the electricity. We\'ve been talking a bit
with petroleum people and the electricity I guess we will be going to
next month. The manufacturing survey actually has a much more complex
problem. We would like it to be looked at and ultimately by someone in
the gas division.

MR. DANDEKAR: Also inter-agency confidentiality group members. I think
that other statistical agencies might also be interested in using the
software.

CHAIRPERSON RELLES: That\'s very interesting. I\'d be interested in the
software from the perspective of the committee. I mean I think it would
be interesting to know kind of what the confidentiality restrictions are
going to imply for using this kind of suppression technology and whether
some of the data that you\'re planning on your current surveys, current
survey redesigns, might make extensive use of this.

I think they might but I just don\'t know. I mean a utility company is
going to be wanting, unless your generating company is going to be
having to report its total generating capacity, it may not want to be
giving out a lot of information about what each of its generators is
doing. But there\'s a lot of value to the competitors to know that and
so it would seem like a candidate for this kind of computation, whether
the numbers are reported in a way where they\'re cost classified and you
need something extensive or whether it\'s just kind of one dimensional.

I just don\'t know for sure. Then again you\'d have the additional
dimension of multiple months worth of information which would now make
it a table and I don\'t know to what degree they\'re going to need to be
reassured that some really sensitive information doesn\'t come out, but
I think it\'s interesting to explore the intersection of new data
collection plans with suppression technology like this and I think it\'s
really wonderful.

MR. JEWETT: One comment that I have is I\'ve seen people sometimes
design tables first, then worry about relations afterwards. If you make
these complicated tables where things inter-relate in very, very tricky
ways, the more ways things add up, the more suppressions you have. And
so you design these really terrific tables and you say I\'m going to get
all this wonderful data, then you start suppressing it. Like it will be
better after you apply suppressions. No one does that though.

MR. FRENCH: When he talks about that, he\'s really talking about our
manufacturing survey which has a number of problems because the number
of complicated problems, among which is the fact that we publish four
different estimates of energy consumption for manufacturing.

Then the problem is one involves a measure of off-site energy and
another is measurement of total energy for heat and power uses, another
one is total energy for all users which include heat, power and feed
stock and so forth, and the fourth is just the non-fuel component of all
this.

Well, the problem is if you add and subtract various combinations of
these, the implicit remainders are the things that sometimes identify
the people because when you take the total heating power usage and you
subtract the purchased energy, that non-purchased in a lot of cells is
just a couple of establishments and you say, well, why do you publish
all these different estimates of energy use? Because people want them.

We\'ve got any number of people who say, No, I don\'t want that stuff.
Another says, yes, I do want the feed stock. Another person says, well,
I want the energy information where you\'ve got the cost information
along with it and we only have cost information for market purchases and
so that\'s another different concept of energy usage that we have. So
it\'s a whip saw between what people want.

MR. JEWETT: What I\'m saying is often times the more you want the less
you get. You went the fancy tables but they get so fancy, all those
inter-relations, you get them all suppressed.

DR. CHATTERJEE: I have some question like in order to protect the
confidentiality, I gather from this, is that some of the cell energies
are going to be smeared as it were, but you can not lead to a proper
identification, but you protect the rule totals and the column totals
are the same so that things can be used. Is that the general way,
because I have seen people looking at this problem. After you suppress
cells and so forth, what property of the table is preserved as it were?

MR. DANDEKAR: I think this particular software, lot of times for
complimentary suppression purposes, if the software goes after a cell
which is very variable in terms of the publication, there is a provision
available in the software whereby you can tell the software to protect
some cells and not to use them as a complement and use the less
important information as a complement. Those kind of options are
available in the software.

MR. JEWETT: What is your question again.

DR. CHATTERJEE: My question is suppose there are a couple of large
entries in some cells. So that would lead to identification of a
particular user. So you\'re going to somehow disperse, smear it, not put
that number there. So that your row totals and column totals are not the
same.

MR. JEWETT: \-- You just have holes in your tables. You just have rows,
columns, and tables for the data, in terms of the data not being shown.
So what\'s there is still correct. What\'s being shown is still correct.
It\'s just a lot is not being shown.

MR. FRENCH: Sounds like he\'s talking about another possibility which is
to inject error into the data in order to \--

DR. CHATTERJEE: At least the marginals or the columns are kept so that
that\'s what \--

MR. DANDEKAR: Introducing random noise in the cells and all that.

MR. JEWETT: People are working on it at the Census Bureau. People are
working on that. Some people like it and some don\'t. But people are
working on that. Does that answer your question?

DR. CHATTERJEE: Yes, it does. But I want to know what your procedure
does in general terms.

CHAIRPERSON RELLES: Sounds like it \--

DR. CHATTERJEE: What?

CHAIRPERSON RELLES: Sounds like it preserves \--

DR. CHATTERJEE: Okay. Yes.

CHAIRPERSON RELLES: Okay. Thank you very much. I assure you the failure
to install did not sort of lessen the information you gave us.

Bill Winig has scripted everything I\'ve said this whole meeting for
which I thank him and eh hasn\'t given me any hints now other than to
say closing remarks. So I\'ll have to ad lib. First I\'d like to invite
questions or comments from the committee at this point. Questions or
comments from the floor. Incidentally, I did forget to thank our
speakers for the really interesting presentation. I really do mean that.
Thank you.

In view of the fact that there are no other questions or comments, I
would want to call these meetings to a close and go off the record.

(Off the record at 12:17 p.m.)