AMERICAN STATISTICAL ASSOCIATION

\+ + + + +

COMMITTEE ON ENERGY STATISTICS

\* \* \* \* \*

FRIDAY

NOVEMBER 5, 1999

The Fall Meeting of the Committee on Energy Statistics commenced at 8:30
a.m. at the Department of Energy, 1000 Independence Avenue, S.W., Room
8E089, Washington, D.C., Daniel Relles, presiding.

PRESENT:

DANIEL RELLES, Chairman

JAY BREIDT

LYNDA CARLSON

THOMAS COWING

CAROL GOTWAY CRAWFORD

JAY HAKES

JAMES HAMMITT

PHILIP HANSER

CALVIN KENT

W. DAVID MONTGOMERY

LARRY PETTIS

SEYMOUR SUDMAN

BILL WEINIG

ROY WHITMORE

[C O N T E N T S]{.underline}

[PAGE]{.underline}

Opening 5

Addressing Accuracy in the Monthly

Forecasting Process

Theresa Hallquist 5

Tom Cowing 19

Jay Breidt 26

Implementing the Graphical Editing

Analysis Query System

Paula Weir 52

Ruey-Pyng Lu 67

Roy Whitmore 71

Dan Relles 74

An Update on Using Cognitive Interviewing

to Evaluate the EIA Web Site

Howard Bradsher-Fredrick 86

Discussing the Charge: What Is the Role

of the Senior Mathematical Statistician

in the Statistics and Methods Group at EIA?

Nancy Kirkendall 105

Committee Discussion 127

Final Remarks and Closing of the Meeting

Daniel Relles 129

[P R O C E E D I N G S]{.underline}

8:24 a.m.

CHAIR RELLES: I would like to begin the meeting of the Friday session of
the EIA ASA Committee on Energy Statistics. The announcements are,
first, any EIA Staff or member of the public who was not present
yesterday, would you please introduce yourselves?

MS. FRENCH: Carol French, EIA.

MR. KING: Walt King, Natural Gas Division of the EIA.

CHAIR RELLES: Welcome. Okay. And that\'s about the only starting
announcements. I\'m pleased to welcome Theresa Hallquist who\'s going to
be talking about something that we all care about, namely statistics,
which we didn\'t have a lot of yesterday but we\'re looking forward to
the morning session on statistics and particularly your discussion of
forecasting.

MS. HALLQUIST: My name is Theresa Hallquist. I work in the Office of Oil
and Gas and this presentation is about a new table called the Initial
Estimates that has been placed in the [Petroleum Marketing
Monthly]{.underline}.

So I\'m not sure if everyone is familiar with the [Petroleum Marketing
Monthly]{.underline} so I wanted to just tell you a little bit about it
first. The [Petroleum Marketing Monthly]{.underline} is a monthly
publication. We place in there price and volume statistics for crude oil
and petroleum products at the national, regional and state levels.

And the publication comes out at least two months after the end of the
reference month. So for instance, November data wouldn\'t be out until
the end of January. And as I know that members of the Committee have
heard before, we know from our customer surveys that the customers would
like to have the information sooner.

And there\'s two items of reference that are put into here and I wanted
to talk to you about. On the first page is a map of the PADDs. PADDs are
regions of the country and I\'m going to use the word in the speech and
I\'ll try to say what part of the region I\'m talking about, but in case
you need to look at it, PADD 1 is the East Coast, PADD 5 is the West
Coast.

And then the last page of this presentation is an example of the Initial
Estimates Table, just to give you an idea of what it looks like. And the
very last month of the prices that you see there are the forecasts that
I\'m referring to.

So because we knew that our customers wanted to have the data sooner,
the Initial Estimates project was born. And basically what it is, is a
monthly forecast of fit and size, crude oil and petroleum products\'
prices at the national and regional level. And to produce the forecast
we use ARIMA, Auto-Regressive Integrated Moving Average Transfer
Function Model.

Transfer Function Models are \-- we use not only the price series own
past history to produce the forecast, but we also use other variables
that we have, that we believe will help us to better forecast the price.

And so for instance, for residential heating oil we use heating degree
days; that kind of thing. And so these variables that I\'m speaking
about, the number of input variables per model range from one to ten
variables. And time series is between six to ten years of history, and
that\'s based on whatever is available.

And I said all this to you about the Initial Estimate to provide you
with a scope of the project. It\'s very big.

So the goals of the project are to publish the Initial Estimates to
(unintelligible) the end of the reference period. So it\'s a
considerable amount of time (unintelligible). And then also, to be
within one percent of final published price for the petroleum products.

So what I wanted to speak to you today about is assessing accuracy of
these forecasts, and there are two stages that we assess the accuracy of
the forecast. Stage 1 is, at the time that the forecast is produced we
take a look at the forecast and make a decision whether or not we want
to publish it.

And so if we determine that we don\'t believe that the forecast is
correct, we can just N/A it: not available for that particular month.
And on average we do suppress three forecasts out of the 40 that are
published in that table every month.

And then the second stage comes along much later. It\'s two to three
months after where once this data are produced using the normal survey
process, and I compare the forecast to the actual price. And at this
point what I would like to know is, are the models performing well and
are there any models that need to be replaced?

So I\'m going to go through Stage 1 and Stage 2 for assessing accuracy.
This is Stage 1. This part I called the Quality Control Report. This
represents what we would see at the time the forecasts are generated and
it\'s part of what we use to determine whether or not we\'re going to
publish this forecast.

So this is 15 months of data and the green line is the PMM price.
That\'s the price that we publish. And this particular one is for PADD 3
wholesale distillate fuel oil for the price there, and PADD 3 is the
Gulf Coast.

And in this model one of the input variables is the Gulf Coast heating
oil spot price. So the very last input there, that price there, that\'s
the forecasted price for fuel \'99. And all of the other prices there
are actual prices over the last 15 months. And the red line, Gulf Coast
heating oil spot price, those are all actuals for the whole country
including the last datapoint.

So I feel confident that I\'ve reached \-- this does not indicate any
reason why we shouldn\'t publish this forecast. So I just wanted to show
you an example of a good one and then I want to show you an example of a
bad one.

Again, the last price \-- this price right here \-- that\'s the price
that we wanted to ask ourselves, do we want to publish that forecast?
Well, there was a problem in the month before and I know what the
problem was.

It definitely was not supposed to be that way and I do not feel
comfortable publishing this forecast because that last month before the
forecast is printed. So this forecast I do not think we should publish.

CHAIR RELLES: What happened that month?

MS. HALLQUIST: It was a problem with the data. It hadn\'t had \-- the
quality of the data was bad, basically.

MR. WHITMORE: That\'s an actual point, whether that actually has some
errors on it.

MS. HALLQUIST: By the way, just so you know, because I don\'t want
anybody to think that we published that price when the second class
price was never published anywhere. It didn\'t go into the PMM.

MR. HAMMITT: Is the second-to-last point \--

MS. HALLQUIST: It\'s an actual that we have at the time when we produce
the forecast.

MR. HAMMITT: Right, and it would be input to your forecast and that\'s
why you don\'t \--

MS. HALLQUIST: Right, right.

MR. HAMMITT: \-- aren\'t suspicious of the forecast?

MS. HALLQUIST: Right, especially since I think that \-- the last
datapoint from Florida forecast has a lot of influence on the forecast.

This is the second part of the Quality Control Report. This would be, if
you look at the dates in the left-hand column, this is six months of
information about the forecast. This would be the forecast for PADD 5,
retail regular motor gasoline price. And we\'re trying to determine
whether we want to publish the May 1999 forecast.

But one of the input variables that was used in this model I have listed
on the right-hand side, the coefficient/T-ratios were not input
variables. And the reason I look at this report is that both the
coefficient and the T-ratio should remain stable over time. It
shouldn\'t change by too much.

Of course it changes some as a price change, but it shouldn\'t change by
too much. So this one sets a flag up in my mind because the coefficient
changed only .776 when it had been around .88 and .89 for the prior five
months while the T-ratio changed for that month.

So this one, I have some concerns about this PADD 5 retail regular motor
gasoline price forecast. And the Quality Report, you\'re supposed to
give the parts in conjunction with one another. So this is raising a
warning flag in my mind and I\'m going to go now to the graph and see
whether the graph also raises a warning flag in my mind. If it sounds
like they\'re both bad I\'m going to suppress this forecast.

So I just wanted to show you an example of that. Okay, so now I\'m going
to move on to Stage 2. Now it\'s two to three months later, the final
published price has come out in the [Petroleum Marketing
Monthly]{.underline}, and I need to know how well the forecast did.

So I just compare the forecast to the actual price and at this stage I
want to determine whether or not the models are performing well, and
maybe I should replace one of the models.

And when I say replace one of the models I mean, you know, maybe one of
the input variables that we use isn\'t the appropriate one to use. Maybe
there\'s one that\'s better that\'s out there. It\'s that kind of thing
that\'s been done in the past in terms of making the models better.

So I use this tool. It\'s a graphical tool. It\'s a box whisker plot and
perhaps you all are familiar with box whisker plots but just in case you
aren\'t I wanted to go through what they are and how you\'re supposed to
use them.

So this is the forecast for PADD 5, retail motor gas price. PADD 5 is
West Coast. And this would be overtime. This would be months of
datapoints would be represented in this box whisker plot. And the
left-hand side is forecast minus actual price in cents per gallon.

And down in the bottom here, this actually just tells you which model
we\'re talking about: retail mogas PADD 5. And what\'s done is, the
datapoints are arranged in order in terms \-- just the forecast minus
actual and what that was \-- put that in order. And then the middle 50
percent of those datapoints fall within that box.

So 50 percent of the forecast for this were between \-- it\'s hard to
work the axis, but maybe -.8 and .77. So it looks good because I\'m
trying to be within one cent. And the horizontal line is
(unintelligible) the median, and essentially, the two endpoints are the
whiskers \-- these two things, right there and right there \-- show the
range of the data.

But consideration is given to the extreme values of the outliers and so
up there, the star, that\'s an outlier. And the outlier is defined as
occurring 1.5 times the height of the box \-- they call it in
(unintelligible) range \-- 1.5 times that. It\'s outside that distance
then, that is considered an outlier.

So ideally you want this to be around zero. You don\'t want it to be \--
the range of the data should be relatively small and you don\'t want it
to be more positive than negative or vice versa. And so this is great,
and I looked at what I saw and thought, where it really became powerful
\-- definitely there is (unintelligible) to determine whether or not \--
which rounds are not very good and need to be replaced.

And then also of course, which are doing very well. This is the accuracy
of selected forecast. Down at the bottom here are 37 different prices
that we forecast for. I don\'t put the crude oil on here. It\'s the only
thing that isn\'t on here that we do publish.

So there\'s 37 of them. The R stands for \"retail\". You can see this
better in your handout in case you can\'t read it up here. So the first
set over there are retail Kero-Jet. The U.S. had one, two, three, and
five. And then wholesale Kero-Jet. Then we move on to motor gasoline,
retail and wholesale prices for motor gasoline.

Then the residential, (unintelligible), T-bill fuel price and wholesale
(unintelligible) fuel oil. I only (inaudible). And so that\'s what they
are. Each one of those box whisker plots represents a different product.
And then the scale on the left-hand side is the same as it was on the
last page \-- forecast minus actual in cents per gallon.

And our eye is immediately drawn, because we\'re humans, to the bad
ones. And that\'s horrible. That is residential diesel for PADD 3 and
(unintelligible) horrible is \-- see, even the median value is about
five cents. So we\'ve been off on this one (inaudible). But the range of
the data goes all the way up to 14 cents it looks like, and I am in the
process of replacing this model.

But let\'s look at the good ones. Look at retail Kero-Jet in wholesale
terms as well. We\'re doing very well. They\'re right around zero. Even
the range of the data \-- in some cases some of those forecasts have
never been more than a penny off. Also you should know when you look at
this that there could be a varying number of datapoints in any of these
different forecasts.

You know, I only put in the datapoints since a particular model might
have been replaced, so maybe it\'s only been in place for five months.
But then in some case there are 18 datapoints represented in one of
these boxes.

So another thing you can do is look at what the bias in the forecast is.
If something is always positive then that\'s not good; meaning that the
forecasted price is always greater than the actual price. It should be
\-- some should be greater, some should be less.

And so you can take a look here and see that there are some cases like
that and these models should be evaluated and maybe we won\'t use the
(inaudible). And then of course, you can see the outlier. Hopefully,
outliers, especially really bad ones, were suppressed at the time when
we were deciding whether to publish them.

I actually don\'t know that. I didn\'t do a study for that. But anyway,
so there\'s some outliers that \-- this one is -7 cents. And so you can
see that \-- but it\'s not too bad. There aren\'t that many outlier
models on this chart so I\'m pretty pleased with that.

So overall I think we\'re doing very well with the accuracy of the
forecast. And so that\'s how I\'m going to conclude. Are we meeting our
goals? We are meeting our goals for timeliness because we established
the procedure to produce the forecast two weeks after the end of the
reference month. So of course we\'re meeting those goals.

But I also believe that we\'re meeting our goals for accuracy. The
majority of our forecasts we do meet our goals for accuracy.

CHAIR RELLES: Okay. Thank you very much, Theresa. We have two planned
discussants for this: Tom Cowing and Jay Breidt. And if either of you
have a preference about going first \-- it looks like Tom is going to go
first.

MR. COWING: Okay, as you\'ll see we\'re letting the expert go second. I
think it\'s fair that I put my cards on the table to start with. I am
neither a statistician nor a time series person. In fact, I have a bit
of an anti-time series bent to my personality, so you should know that.
I\'m going to try to limit that a bit.

To kind of indicate how little I know about time series analysis, I
started reading through the paper when I first got it to see if there
was anything I could kind of grab onto; something that I knew something
about. And I come across a reference to a whisker box plot where whisker
and box \-- and I think the title was capitalized.

And I thought to myself, well I know something about that. This box must
be of box cox fame. And whisker, I\'d never heard of that but you know,
it\'s probably an eminent statistician at Iowa State. And so I kept
reading and discovered the next reference to whisker box plots was not
capitalized.

And then it dawned on me what we were talking about is, and if you look
the plots they are kind of aptly named. They remind me of those little
resistors if you date back to the \'40s and \'50s or a kid building
home-built radios. You\'ve got these little resistors, those little
wires coming out of the end, and they kind of look like boxes with
whiskers on either end.

I think what I will try to do \-- I have a few comments here \-- my
comments, since they will be less technical in nature, hopefully will
complement what Jay is going to talk about, since my understand-

ing is that Jay is a time series person.

As Theresa mentioned, at least in the paper, the written version, the
goal of this project is to produce early forecasts, initial forecasts,
for something like 69 petroleum product prices within two weeks of the
end of the month. That\'s a very ambitious schedule, I think.

The accuracy that they seek is one cent \-- to be within one cent per
physical unit per gallon, per barrel, whatever the units are, in the
forecast accuracy. And then of course, the final published series come
out within three months, I believe, after the close of the month.

So we\'re producing forecasts at the end of two weeks that will be
confirmed or not within three months. And the idea is that there\'s
pressure on EIA to produce these early initial estimates because the
final data takes three months to produce.

So I want to think a minute about those three parameters: two weeks, one
cent per unit forecast accuracy, and three months when the final,
presumably the actual data becomes available.

It seems to me that two weeks is a very ambitious period of time. I
assume that there is some tradeoff here between timeliness and accuracy;
that is, if you can spend three or four weeks you would have an even
better forecast. If you only had one week the forecast isn\'t going to
be quite as accurate.

Now, the pressure for the need for the initial estimates comes, it
appears to me, from the fact that it takes three months to produce the
final series. One could raise the following question.

If you take the resources \-- I don\'t want to eliminate Theresa\'s job
\-- but if you take the resources that were devoted to the initial
estimates to producing these early forecasts and put them into reducing
the lag that it takes to get the actual series, maybe from three months
to two months, you\'re clearly going to reduce the pressure on EIA for
publishing timely forecasts.

So that\'s just a thought. I don\'t know which way you want to go but
there\'s clearly, presumably, it\'s a trade-off here.

I do have a more substantive quarrel I think, with the goal \-- the
forecast accuracy goal of one cent per unit. That\'s an absolute
measure. It seems to me you really ought to talk about a relative
measure. This is one cent in a price which averages in the 40 cents per
gallon range; that\'s a two-and-a-half percent forecast error. If the
price jumps to 80 cents per gallon then it\'s a one-and-a-quarter
percent accuracy error.

So it seems to me the goal really \-- well, I can guess why one cent was
chosen. It\'s a nice sound-bite number to have out there. On the other
hand, I think it really should be in terms of a relative one or two
percent forecast error.

Again, not knowing a whole lot about time series analysis, but I assume
that this \-- I mean, I do know something but it\'s certainly not my
area of expertise. I assume that this statistical procedure works best
when you have a fairly stable series of these trend-wise, and when
there\'s not a lot volatility, and of course there\'s the usual turning
point problem that any forecast has problems with.

And so just to remind you that we are talking about series with a lot of
volatilities and a lot of turning points, let me present the obligatory
slide, and Jay you will be pleased to see that EIA\'s reference here \--
this is from [The New York Times]{.underline} two weeks ago. There was a
story about the resurgence of OPEC plus two \-- meaning what, Mexico and
Norway, I guess \-- that have intervened to kind of re-establish some
market leverage on the part of OPEC.

What is the point here? The point is that \-- here\'s some of these
prices over the last eight to nine years and there\'s a lot of
volatilities and there\'s a lot of turning points.

So I suspect that time series analysis is going to have a problem here,
particularly with turning points. But on the other hand, most other
forecasting procedures also have problems with turning points.

Some information that I would have liked to have known, have been a
little more explicit in the early version of this paper \-- some of this
came out this morning in the presentation, in Theresa\'s presentation. I
would have liked to have known what the specific input data series are
that are used.

You did mention some of those so now I have a better feeling. For
example, heating degree days in the case of residential price forecasts.
Are some of them ever dropped? Why? When? Are new series added? Again,
why? And if you need a new series, think you need a new series, what are
the criteria you use as to which one to grab to try?

And of course, if you\'re going to use these input data series in the
Transfer Function Model, then of course they have to be known for the
period in which you are forecasting for.

And that raises a problem of, well how come you don\'t know the price
that you\'re forecasting but you do know the input data series? Unless
of course, everything is lagged one period and then of course you would
know the current value so to speak, of the input data series if you need
to drive the forecast.

I was very pleased to see \-- I assume that these \-- now that I know
what they are \-- whisker box plots are based on what I call validation
analysis, which is looking at measured forecast error versus actual
values. And I was very pleased to see that.

I think that\'s the kind of evidence you need for any forecast model \--
it\'s kind of a stored validation \-- to be able to get a sense of
whether your models are working well. So I think that\'s a very useful
thing and I was pleased to see that a lot of that\'s being done.

Finally, I don\'t want to get onto this horse, but I\'ll just mention it
in passing. if I were to have done this project I would have picked a
structural econometric model to worry about. But that\'s simply because
I\'m an economist and not a time series analyst.

But it does have some advantages. My guess is that it\'s much more
expensive to do it that way. That\'s obviously, not an advantage. But it
does have one advantage in that it would make it easier to relate that
kind of forecast I should think, to the NEMS model, which is a more
structural, econometric model.

Okay, why don\'t we \-- that\'s kind of \-- why don\'t we just leave it
at that? Again, I thought the validation analysis stuff was very good.

CHAIR RELLES: Thank you, Tom.

MR. BREIDT: Okay. Well, I like the title of this paper which involves
forecasting process, because I think it\'s useful to distinguish between
a forecasting model and a forecasting process.

And the particular forecasting model here is a transfer function model
with ARIMA inputs and ARIMA outputs. But the forecasting process
contains a lot more to it.

There\'s the model estimation, diagnostics, there are these various
quality controls that are built-in, and then there\'s forecaster who can
sometimes exercise expert judgment to suppress a forecast that they know
to be way off because the conditions under which the model was
constructed no longer hold.

And all of this needs to be understood in the context where we\'ve got
multiple series to be forecast in a very quick turnaround time. So I
think these are all important features of the forecasting process.

I have just a few comments on the transfer function modeling. I think
this is a good approach. It\'s a naive approach in some senses because
you really don\'t need to know a whole lot about the underlying
structure in order to be able to construct a useful transfer function
model.

And that\'s actually useful because there is a sort of a cookbook or a
textbook algorithm for model selection and estimation. You can go
through this procedure and construct a very reasonable model using
well-known and well-documented procedures, using more or less
off-the-shelf software. I\'m not sure exactly what you\'re using here.

But the model is fairly easy to write down. It has explicit modeling
assumptions, and because they\'re explicit you can test them. And all of
those are good features I think, in a forecasting model. And it
generally yields good forecasts. It\'s a rich model class. It allows you
to come up with parsimonious models.

It uses auxiliary information very effectively, and because it\'s using
that auxiliary information which you have up to and including the
reference month, it gives you very stable parameterizations. So these
are pretty successful models that don\'t require a whole lot of expert
input to create except in the choice of input series.

I had just a few questions about the transfer function model. First, it
was a little surprising to me that we\'ve got monthly series as input
and output but there\'s no discussion of seasonality. I\'m wondering
where that shows up.

It would be very common to observe seasonal effects in heating degree
days or something like that, I would think. And you often see this both
in the mean and in the variance and interactions between the mean and
the variance. And that didn\'t seem to appear here.

The other thing is a multiple input series. And the usual Transfer
Function Modeling \-- and maybe I shouldn\'t say too much about this
because Transfer Function Modeling is usually Chapter 11 of the
textbooks, so if you teach a time series course you never get to it. But
Theresa has fit hundreds of these things so she\'s probably the world
expert on Transfer Function Modeling.

But the usual story is that if you\'re going to use multiple input
series it\'s a relatively straightforward extension, provided the input
series aren\'t correlated. But if you\'ve got correlation across those
it gets a lot harder to identify the actual model.

And that\'s problematic because if you mis-specify your model, which
there\'s a good chance of doing in a multi-variate context, you can do a
lousy job of forecasting. And even a very simple, naive, univariate time
series model can beat a mis-specified, multi-variate model. So I was a
little concerned about that.

But overall I think that forecasting model, that particular choice, is a
good choice and it works well for the purpose that it\'s being used
here, which is a quick forecast, quick preliminary estimates.

So I\'m going to talk a little bit about the forecasting process now.
And like Tom, I was struck by this very tight criterion for an
acceptable forecast: between within one cent. And there\'s no
probability statement associated with that. It\'s just, we\'re going to
be within one cent; not 95 percent confident that we\'ll be there, but
we just want to be within one cent. An absolute measure with no
associate probability or confidence statement.

Then this business about suppression: the forecast gets suppressed if
both the graph indicates a problem and the ratio reports indicates a
problem. But I\'m a little worried about this because it seems to me
that if you start publishing two weeks in advance users are going to
come to rely on these forecasts \-- not two weeks in advance, a
month-and-a-half or two months in advance.

So I think users will really start relying on these forecasts and no
forecast doesn\'t seem very acceptable once you start relying on those
forecasts. But even if you think that might be an acceptable answer I\'m
not sure the existing suppression criterion is going to be very useful
to users. It\'s a little bit vaguely specified.

In particular, it\'s kind of hard to figure out what the statistical
properties of something like this would be. The forecast is Yhatm from
the Transfer Function modeling, if the graph\'s okay or the ratio report
is okay, and it\'s just nothing otherwise or it\'s not available.

It\'s hard to know what to think of that procedure. So whatever you do
here, I think it needs to be more formalized and more documented than
what was reflected in the paper. So that\'s sort of Stage 1 of the
quality control.

There\'s also Stage 2 in which these box plots show up, and I like box
plots. I think they\'re a simple and effective tool and they show you a
lot of detail about your forecast errors, and you can do comparisons
across similar product types.

Theresa mentioned this business of different sample sizes in the
different box plots, and a lot of software has an option to give a
variable width to the box plots to reflect the different sample sizes.
So that\'s one way to do that.

But there\'s a potential concern here that box plots can hide some
forecasting problems, and if you\'re only looking at these when you get
a new forecast error and you\'re not really looking at the dynamics of
those forecast errors, you can get into trouble.

Here are two absolutely identical box plots corresponding to two
different residual sequences. The light gray line is fine; really no
worries. But the dark, black line here shows evidence of increasing
forecast errors over time. So the dynamics are completely hidden once
you look at the box plots of course.

So it\'s really useful to keep track of those dynamics. There\'s
information in them. And again, that can be looked at graphically.

You could make a more formal test of model adequacy than what\'s given
here in looking at the box plots. You can construct more or less
post-sample, predictive tests that would be chi-squared. You can
construct a formal test.

I\'m not sure you really need to need to go to that trouble. But it
might be worth looking at some more formal tests of model adequacy based
on those forecast errors, particularly tests that might take into
account the dynamics.

This next slide I entitled, \"What Have You Done For Me Lately?\"
because we\'re in a dynamic forecasting context so it\'s really the most
recent behavior that forecasting model that\'s of most interest. And we
don\'t want to be unduly influenced by how well the model performed in
the past.

And when we get to Stage 2 of quality control it\'s basically all of the
forecast errors going into a box plot. And it may be that that model has
performed very well in the past. We don\'t care. We want to know, how is
it doing now, and we don\'t want that historical evidence of good
behavior which has accumulated in the box plot to mislead us about how
it\'s doing now.

So we really have to keep track of the dynamics. If we don\'t do that we
might have a very slow response to some kind of small but persistent
changes in the forecasting ability of the model.

So one possibility is to localize the judgment about the model accuracy
and the idea is, you just sort of reset yourself to neutral whenever you
decide that the model is okay. So the model seems to be doing fine,
throw away that old box plot and start over. Start accumulating new
forecast errors and see if there\'s any indication of failure.

Similarly, if you\'re computing a test statistic you would start over in
its computation, so you just compute it locally based on the most recent
observations. And that\'s usually a quicker way to detect model
inadequacies.

Just a few other thoughts on the forecasting process. You could think
about other kinds of diagnostics. And I was a little confused in the
paper and I\'m still confused after the presentation, about exactly what
goes into the forecast. It seemed to me that there were \-- if you were
talking about reference month M then the forecast was based on real
observations up through M-2, and the input series up through time M.

So it seemed to be a two-step-ahead forecast. But you also had data from
M-1 which was kind of an early close-out number which hadn\'t been fully
edited, and that\'s why we had this weird spike earlier. So it seemed to
me that it was a two-step-ahead forecast, and if we\'re doing a
two-step-ahead forecast then there\'s also a one-step-ahead forecast and
we could look at that as a possible diagnostic.

The other thing is, these ratio reports are showing how coefficients
change with time and how T-ratios change with time. All of those could
be done graphically as well. And you could do tests for breakpoints in
those series.

Any time you have evidence of systematic model failure you need to do a
feedback loop and say, what am I on here? I\'m sure that\'s being done.
And also it might be useful to test against alternative models as
you\'re running along through time, because it\'s kind of hard to tell
how well a model is doing sort of in a vacuum. How well is it doing
relative to some other reasonable model?

And then finally, some extensions of the forecasting process. This is
more kind of musing. You\'re fitting very similar models for dozens of
series, and these are closely related either through geography or for
similar product types. And often it\'s the case that when you\'re doing
the same thing over and over again you can gain some stability and
improvements for error properties of your procedures through some kind
of a hierarchical model.

And if you wanted to go down that road, which I\'m not saying you want
to, it\'s possible to construct a full-blown analysis that would involve
some Bayesian techniques. And that also gives a coherent framework for
bringing in this expert opinion, when you have some kind of intervention
that doesn\'t \-- isn\'t reflected in the normal modeling framework.

So the final remarks. I think the forecasting model here is very
sensible. It\'s a disciplined approach to producing forecasts which are
meeting their accuracy requirements, and it\'s very timely. It\'s a
quick way to get forecasts.

And the forecasting process I think, currently incorporates some useful
monitoring diagnostics. You could extend those, you can prove those. But
in any case, it could use some additional formalization documentation.

Those are my comments.

CHAIR RELLES: Thank you, very much, Jay. Jay and I had a long talk about
statistical approaches on the way to the airport last time. I\'m glad to
see he\'s now a Bayesian.

Theresa, would you like to respond to those comments?

MS. HALLQUIST: There were a couple of items I could clear up. I first
want to respond to something that Tom said regarding possible
improvements to the current process to publish the survey data sooner.

I see that I wasn\'t clear. I know I kept saying two to three months
after the end of the reference period. I do that because the data come
out preliminary, two months later and they come out final, three months
later. And so I prefer to compare to the final number once it comes out.

So I actually do wait the three months when I compare to the forecasted
price. But they do come out in two months. I just \-- in fact the data
just came out for September, this week; those two months. But absolutely
it\'s true that consideration should always be given and has always been
given, for getting the data out sooner.

We have one constraint that for the moment won\'t be changed; which is
that the surveys don\'t come back to us for one month after the end of
the reference period. Maybe that will change over time.

And Jay, I wasn\'t clear in what forecast we\'re using. Those were
excellent comments regarding which month ahead forecast we\'re using. So
let me say that we are using one month ahead forecasts completely.

Early-on in this study we looked at two months ahead forecasts but the
error was too high. We didn\'t want to publish that. So we\'re only
using one month ahead forecasts. So survey data is used through \-- so
the forecast is for \-- that\'s the M+1 month \-- so through M is all
survey data.

But remember I just said it comes out preliminary then it comes out
final. And then actually we have the last month before the forecast is
what I call in the paper \-- I didn\'t say it in my speech \-- but what
I call in the paper the early close-out number. It\'s survey data but
it\'s possible to have error introduced at that point. And so that\'s as
you saw, that\'s why that datapoint went up so high in that one
particular graph that I showed.

Then in addition, the input series are all available through
month-end + 1. We have heating degree days by then, we have spot
variable prices days by then. We\'re actually very fortunate because
those things of course, help with coming up with the forecast.

I want to say that I appreciate the time that you spent and I definitely
will \-- I can already see some changes that I will be making based on
your comments. Thank you.

CHAIR RELLES: Okay. Tom and Jim.

MR. COWING: Theresa, I have one other question which I forgot to bring
up. In the again, the early version of (unintelligible) \-- I assume
it\'s in the most recent version as well \-- you mention there\'s 69
petroleum product prices which need to be estimated early. Of the 69
there are 55 ARIMA Transfer Function Models, apparently, and 14
something else.

So the question is twofold: number one, what models are used for the 14;
and two, is there some generic reason why the 14 have to be treated a
separate way or is this based on model performance of the 14 that caused
you to do it some other way?

MS. HALLQUIST: No. It has to be 68, not 69, first of all, so even that
number\'s wrong. I\'m counting all the models that we produce monthly.
We are \-- the only difference \-- we are considering producing more
forecasts so those are just more \-- these are for propane, actually,
and they\'re not published. We haven\'t made the decision to publish
them.

MR. COWING: Oh, okay.

MS. HALLQUIST: And so that\'s all those are.

MR. COWING: So you produce 55 currently and you\'re thinking about
adding a few more?

MS. HALLQUIST: Yes, number one. And number two is, only 40 actually get
published on the Initial Estimates Table. We produce forecasts for each
of \-- it varies by product \-- but generally we produce each of the
forecasts for each PADD, each sub-PADD, and you might have noticed PADD
4 isn\'t published anywhere in that Table, in the Initial Estimate
Table.

So we have to produce those because we use a weighted average to get the
U.S. forecast and that also we use a weighted average of the sub-PADDs
to get the PADD 1 forecast but we don\'t publish those; partially
because of the accuracy of the forecast, particularly with PADD 4.

PADD 4 is a problem period. It\'s very hard to forecast the price for
it. So the decision was made not to publish anything for PADD 4. But it
has to be there in order to give the U.S. prices.

MR. COWING: You didn\'t mention anything about the one cent absolute in
your criteria, rather than a relative criteria.

MS. HALLQUIST: Right.

MR. COWING: Is that just a nice number that fits in your data well?

MS. HALLQUIST: You might also have noticed that I didn\'t say anything
about crude oil because that doesn\'t apply to crude oil.

MR. COWING: Well, it might but that would be a \--

MS. HALLQUIST: What, whomever crude oil, forget it. There has been a lot
of discussion about the Gulf for this project. I have had people say to
me that I should vary it by product, I should have a relative goal, like
Jay said. It is a very difficult goal to meet. However, it\'s I think,
an important goal.

You don\'t want to put out \-- we could be, and you saw in the box
whisker plots \-- we could be ten cents off. Ten cents off is
inappropriate. I mean, we need to try for that one cent accuracy. Now I
\-- you saw not all of them are meeting one cent and I still believe
that we have very good accuracy of our forecasts.

It\'s one cent overall. Like, I don\'t think every forecast every month
is going to be within one cent. But it might be within two cents. We do
pretty well when you look at the plots there. So it is a target, we\'re
trying to meet it, and if we don\'t we don\'t. But we use that
information to help us to know whether to replace the models.

CHAIR RELLES: Actually, I want to keep this on time. We have five more
minutes in this session so I\'d like to call on David and Jim and Phil
and try to wrap this up, okay? Jim you were first.

MR. HAMMITT: Just two quick points which are really questions to Jay, I
think. On the model diagnostic in the \"What Have You Done For Me
Lately?\" \-- makes a lot of sense.

It would seem to me sort of throwing out all the data at some point and
then letting data accumulate in the new test might not be as good as
some kind of a weighting procedure where you just down-weight all this
stuff. And I\'m wondering if there are standard things done there.

And then the second point was, you talked about formalizing the
suppression criteria. I\'d be interested in your thoughts of why that
would be particularly important, because it seems to me hard in advance
to sort of write down explicitly what rule you want to follow. Because
you might see something that you never would have thought could have
occurred and you might want to suppress it at that point.

MR. BREIDT: Yes. Well, on the first thing about throwing away earlier
data, I was really thinking in terms of box plots there. I don\'t know
how you would down-weight that. But something kind of weighted could
make sense for specificity. But the main point is, not to be overwhelmed
by the historical performance of the model.

And the second bit about writing down formal suppression criteria, I
agree there could be unusual circumstances but the way it\'s worded in
the paper and in the presentation was somewhat vague so I just wanted to
hear something a little more specific about, under what circumstances
would you suppress (inaudible).

I think you want to do that in a disciplined way because I really do
think that people will come to rely on these forecasts. If they\'re
demanding them and they start to get them then they\'ll expect them.

MS. HALLQUIST: It actually has already happened to me where someone
called me and said, where is the discipline?

CHAIR RELLES: Non-existence means something. It\'s not as random.

MR. MONTGOMERY: I think Phil was first.

CHAIR RELLES: No. Go ahead, you were first.

MR. MONTGOMERY: All right. I had three points. The first one was that, I
actually don\'t have any problem at all with the one cent criteria
because your basic series are all the same order of magnitude. They
probably range from 50 or 60 cents a gallon to \$1.20 a gallon, and you
know, so that\'s about one percent for all the series anyway. I\'m not
sure it\'s worth worrying about whether it should be one cent
plus-or-minus a half-cent, because that would be probably all it would
take to make it consistent.

My other thoughts were somewhat more vague though. One was, it clearly
seems that you\'re going to be encountering forecasts you don\'t like
when there\'s something like a regime change or when there\'s a lot of
variability in something that affects your forecast that you\'re not
capturing in your excitees.

I was wondering if \-- and this may not be kind of a sloppy way of
saying what Jay was saying \-- but I was wondering if there was \-- for
example, how you handle a situation where a tax is imposed that would
affect your forecast, or there\'s an oil supply disruption. I mean,
these are things where people really care about the numbers and you\'re
not going to be able to capture them at all.

And I guess my third question was, maybe exactly the opposite, if your
ARIMA model is doing a really good job, could you use it to try \-- to
get back to something we were talking about yesterday which actually I
probably shouldn\'t ask you but I can\'t resist anyway \-- could you use
a model like this to decide if you\'re collecting data on too many
prices because they are not statistically independent of each other?

Could you use this model to somehow take out the auto-regressive part of
it and see whether you could use it to forecast some prices that you\'re
collecting now from other prices you\'re collecting, and maybe not need
to collect those prices if they are, you know, not statistically
distinguishable from \-- if you could forecast them perfectly well based
on prices you are collecting?

I\'m sorry, those were two questions at least.

CHAIR RELLES: Actually, let\'s get Phil\'s thought on the table, too,
and then we can get the last set of responses.

MR. HANSER: I guess there were \-- I mean, I guess this is sort of
techie and I apologize for being this, but \-- I mean, they\'re taping
it seems to me they\'re going on.

If you\'re looking at all the paths together, right, then there\'s all
this unexploited co-variation among the series; that if I\'m doing
individual series estimation I\'m not sort of capturing. And so one of
the things I would have thought you would want to try to do would be a
bar formulation, for example, right from the start.

Because the changes or the moving average price is pretty primitive, and
so if you did some sort of vector auto-regression you probably get most
of the variation there.

Another alternative is to go to something like a Kalmin filter, and you
can set up the Kalmin filter in such a way so that it has a kind of
Bayesian approach in the sense that it allows you to formulate taking
count of the fact that you may have strong priors about other behavior
of the series ought to be.

And there\'s a wonderful book by Harrison and West which is now in its
second edition actually, which sort of tells you how to deal with the
vector series and taking count of this on a national relation.

I\'m not quite going to the words David is about eliminating the series
that you want to forecast, but it seems to me that if you\'ve got all
this data then \-- I mean, it\'s sort of like, gosh \-- I mean, it\'s
sort of like the old simultaneous equations problems, right?

I mean, it turns out I can do wide-scale, just plain regression and do
better than any two-stage or limited information form of the forecast,
unless I\'m willing to go to the full information national likelihood
stuff.

So in that context the reason it works is because you\'re taking account
of the co-variations among the variables. You\'ve got the same issue
here and you\'re not somehow exploiting that data, and it seems like you
could do a significant improvement in terms of the series as a group.

The tricky part is, you may sacrifice temporarily at any point in time,
the individual accuracy of the forecast, right, in favor of the groups
of numbers being better. Because right now you\'re looking at a
criterion which only focuses on an individual theory, not somehow the
series together.

And you have to ask yourself the question whether you only want to look
at just the individual series or whether the overall accuracy of the
series is important. I would think that relative to the oil prices,
where you\'re worried about is forecasting what a national average is,
the group of series might be a more important criteria than the
individual series.

And I apologize for being a nerd.

MS. HALLQUIST: No, thank you, and I will look into that. I can respond
to at least one of your questions.

We\'re very fortunate that we do have information about price
disruptions in the marketplace. For instance, refinery acquisition
across the crude oil. Well, when OPEC has a meeting and they say
something and it causes some kind of cycle price change in the market,
that\'s reflected in the West Texas intermediate crude oil stock price,
which is then used in the model.

So we have that information and as I said, I believe we\'re very
fortunate to have that information by the time we need it in order to do
the forecast.

But also there could be structural changes in the industry and you\'re
absolutely right. If there\'s something that\'s changing that has to
either be modeled for or \-- you have to do something with that. And
hopefully we will be addressing those problems as they come up.

CHAIR RELLES: Okay, well thank you very much, Theresa.

I\'d like to introduce our next speaker at this point, Paula Weir,
who\'s going to be talking about the Graphical Editing Analysis Query
System. I\'m sorry, and her co-presenter, Ruey-Ping. I apologize for not
being able to pronounce your name.

MS. WEIR: Theresa\'s presentation fits very nicely before ours because
she used a lot of visual explanations of data and this feeds in very
well, especially the box whiskers, so a lot of things won\'t have to be
explained luckily, because she\'s explained them and I don\'t have time
to.

In Volume 2 of [Statistical Data Editing Methods and
Techniques]{.underline} \-- it\'s produced by the U.N. Statistical
Commission and Economic Commission for Europe \-- they define graphical
editing as the exploitation of human visual perception\'s abilities to
identify an anomaly, a pattern, or an M-5 relationship in data that may
take much more time and effort to find my analytical, non-graphical
means.

Experiences show that traditional editing on a case-by-base basis does
not often allow us to clearly see the impact of individual datapoints on
aggregate estimates.

I\'d like to give you a brief outline of how we\'re going to go through
this presentation. I\'m going to provide some background of the
motivations of graphical systems and describe some of the other systems
that this one\'s based on, and then show an example of how the series
was used in looking at some data. And Ruey-Ping is going to show a more
recent usage of that data and how certain attributes, certain kinds of
graphics are appropriate for that theory.

In 1990 the Data Editing Sub-Committee of the Federal Committee of
Statistical Methodologies released a working paper that reported the
median cost of editing was 40 percent of all total survey costs. And
this cost is a direct result of over-identification of potential errors
that result in extensive manual review and manhours with unknown or
little impact on survey results at the risk of missing true errors.

The report recommends that survey managers capitalized on the efficiency
of technology while acknowledging subject matter\'s expertise. The
graphical system developed by EIA combines the features of four other
graphical systems that were starting up at the time. Some of them were
in development/research stage, others were actually piloted at the time.

The first of these is the automated review of industry employment
statistics, ARIES system developed by BLS; the second was a prototype
being developed by the Census Working Group; the third, the graphical
macro application being developed by Statistics Sweden; and the DEEP
system being developed by the Federal Reserve Board.

And in particular, from each of these other graphical systems being
developed, we studied each of them and took from each of those, those
attributes which we thought were most important for our data. In
particular from BLS, we borrowed the idea of displaying an anomaly map
similar to that used in the ARIES system.

And this anomaly map summarizes anomalies of aggregate levels through
the use of color, showing relationship of those aggregates. They\'d come
in families; almost parent/child-type relationships.

In addition, it provides drill-down capability to lower-level aggregates
starting at high-level aggregates all the way down to the
respondent-level data.

From the Census Bureau we borrowed the idea of using exploratory data
analysis techniques; in particular, the box whiskers that Theresa has
described. We used box whiskers on change because this is a repeated
survey \-- change since the previous period at the aggregate level \--
so we can actually look at the distribution within the box whisker and
across box whiskers with query capability on those aggregates.

Since the original design we\'ve actually implemented box whiskers even
at the respondent level of the micro data, finding the power of this
visual. Also scatter graphs where we can plot the current period versus
the last period for respondent-level data with query and drill-down
capability. And again, since the original implementation, we\'ve taken
this graphic actually up to the aggregate level too, so we can work in
both directions.

In the scatter graph we use points of high influence, the different
colorations, because low influence outliers \-- even though they are
outliers \-- may not be cost-effective to pursue. We make use of
different symbols for respondents versus imputed non-respondents so that
the same time we can be examining any problems in our implication system
\-- any biases that may be occurring.

We also, from the Census Bureau, they emphasize the importance of time
series in case there are any seasonality in the data.

From Statistics Sweden we borrowed the idea of making use of a Windows
application. When we began this development process we were all on
mainframes and you know, there was some PC everybody shared. But we saw
the direction this was going in, and we were anticipating the various
client/server combination, some solution, and making use of the Windows
for icons tool bars only seemed reasonable in a graphical, buoy-type
approach.

And it was important that what we developed be object-oriented so that
it could be used on more than one survey eventually as the development
increased. And that this process of the ending be integrated with the
total processing system, believing in a total quality management
approach.

The graphics actually, would help us determine the various assumptions
and the limits and the boundaries of the editing process, and that the
time series was very important to this aspect also.

From the Federal Reserve Board we borrowed the idea of using
PowerBuilder Tools as a development tool \-- and it\'s actually what the
application is written in \-- and doing that development using analyst
input. This resulted in a quicker development time. We developed and
showed the users and they\'d make recommendations and we\'d incorporate
those recommendations.

So it was driven by iterative user feedback and iterative user
requirement development. You can never get requirements clearly on the
table at the beginning of a project like this. We also found we had to
make use of a particular a graphic server to increase the useability of
our graphics.

In addition to these four other system we incorporated visualization
techniques, particularly that of the Cleveland Techniques in graphing
and visualizing data; especially playing off the idea that graphics
should be used as an iterative process in this type of exercise.

And making use of some of the smaller things like circles should be used
for datapoints to minimizing documents, and that only two or three
colors should be used in a graphic at a time; no more than four shades
per color. This is why he encourages this in order to enhance perception
and to limit cognition so you don\'t constantly have to be referring to
the legend.

We did a lot of studying about visualization in doing this project. And
then some just functionality things. On this datagraph you might have
higher clusterings so we\'d need a fit line so that you\'d actually get
a clear indication of that trend and not be anecdotal about it. And
providing the ability to select data and regraph to uncluster that data.

Yesterday, I believe in one of the strategic planning sessions there was
a reference to E-edit and I thought that was very timely because most
editing systems do have some sort of score function assigned with it. If
40 percent of your costs are coming for editing you know you have a lot
of edit failures. You have to have some sort of method to determining
what should go first and what should go last.

This score function is probably one of the most famous of the different
functions. And the point of this is you take an individual respondent\'s
previous reference period report, compare it to the current. If you
sample you weight it. You multiply it by an indicator function as to
whether it even failed an edit.

You multiply it by some relative importance of that component. Do you
even care? Is that component more important than other components? This
may be political; this may be statistical. And you divide it by the
aggregate cell; whatever the total is that you\'re estimating.

But notice the point here is that you\'re dividing by the previous
reference period. You\'re trying to validate current period with the
most you had in the old days, with the previous reference period. We\'re
talking mainframe, batch processing.

So building off this idea we built a score function into the graphical
system that we call marginal change. It actually measures each
individual\'s respondent contribution to that total change. So we can
take \-- for our price formulas we were able to decompose it into the
individual respondents, and even easier for the volume.

And this is just a little, quick example showing that a particular cell,
if you look down at the bottom, the cell is changing. This is a
published price by 11.7 cents. We can compose it into each individual\'s
contribution to that change. So we immediately have a very clear
indication of how to prioritize our data that\'s not built like most
systems on the potential of having influence. This is definite
influence.

Once we got this developed we piloted on a survey; it was a monthly
survey of approximately 3,000 respondents. This survey was chosen
because of its complexity and because of its size. We figured if it
would work on this survey it would probably work on just about any
survey.

It was made up of census of refiners and then a sample of reseller and
retailers, so sample rates were required. In the survey, prices and
volumes were reported monthly by product sales, type and state.

This are what we call the dimensions of our data. And the data are
aggregated to over 60,000 various seller, product and sales type, and
published preliminary and final estimates. So this is the whole thick
publication that we produced. It was top-down approach. We\'re keeping
in mind that final product as we\'re doing the individual editing
process.

The data in the system is actually processed mainframe \-- ADABASE,
which is not very friendly to a graphical user interface \-- so for this
pilot we actually download data to a WATCOM people database.

I\'d now like to show a real quick example of how the system is used in
a particular survey. And this is going to be pretty quick just to give
you a flavor of the types of things \-- you can use it for any editing.

Using a top-down approach with our dialogue box we choose an aggregate
\-- this is an aggregate data file, this is company data file \-- that
at some point your costs you just want to look at a particular company.
And some are views. We actually have a selection of different kinds of
graphics.

I\'m going to start with the anomaly map to show you the part we
borrowed from BLS. I can look at just sub-groups of my respondents. I\'m
going to take a top-down approach and look at all my respondents.

And from my product list for this example I\'m going to choose number 2,
distillate, which is a pretty high level aggregate. And I\'m going to
chose total retail, which is also a high-level aggregate because it
actually has many components, but doing a top-down approach I\'m going
to stay at a high level.

This is an anomaly map. You have the U.S. in the center. Orbiting out
you have the regions of the country \-- the PADDs that Theresa explained
\-- and PADD 1 is actually divided into three sub-groups for the New
England, Mid-Atlantic, and Southern part of the country.

So I can see that an anomaly at the U.S. level which is indicated
through the colors of blue or green \-- blue/green being price
increases, green being decreases \-- and various levels of change since
the previous reporting period; the darker the color the more serious was
the problem, so it\'s all fairly intuitive.

You can see that the U.S. level \-- if we have a problem it goes all the
way to our highest level aggregate. In the bottom toolbar when I click
on a particular note it gives me very detailed information; the actual
prices if I really want to focus on the exact numbers.

In this case I could drill down to PADD 1, all the way down to PADD 1(b)
and get that type of information. At this point I\'m going to choose to
go to another dimension of my data and drill down further in terms of
the product of number 2 distillate, its particular components.

And I\'m going to have a product anomaly map, and this is the number 2
distillate that I have collected. It shows me all products so I can get
a cross-dimensional view if there\'s some related problems that I might
want to see at the same time in solving one of my problems.

But I can see basically that it\'s being generated by the sub-product
number 2 fuel oil. So I\'ve now drilled down to my product and I can
select sales categories to see what generated it. I had retail, I now
see the components making up that retail \-- all these still being
aggregate-level data. I see that my problem is coming out of
residential.

At this point I would actually drill down to the respondent-level data
and look at my lowest-level aggregate. But before I do that I\'m going
to show very quickly a different way of drilling down using a box
whisker plot which we call a delta graph. And I\'m going to set these
back up to my highest-level aggregates to show I get the same results.

What I\'m going to get is multiple box whiskers. Going along my Y-axis
I\'m plotting change. Each individual circle within the box whiskers is
an aggregate data point, a geographic area, and I\'m showing across the
X-axis the individual product.

When I click on a particular node in the bottom toolbar it tells me the
exact \-- what that product is rather than a code, and the median
change. So I can track both \-- with he graphic I also have a
spreadsheet to get the detailed information of that.

I see that the one with the most spread is actually this one and down at
the bottom toolbar it is indeed the distillate product I drilled down. I
can go across to see the components of it. This is the highest and that
tells me it\'s number 2 fuel oil. So I\'ve done the same thing using a
different kind of graphic.

Now I\'m going to return to where I was before using the anomaly map,
but instead I\'m going to go down to the \-- return to where I was. I
had selected PADD 1(b). I had selected residential heating oil, which is
number 2 fuel. Now I can actually do the graphic on this datagraph and
look at the respondent data.

And again, on my PC this is just a 133; just on a 180 this comes up in a
second or two. And I now see all the individual respondents and their
contribute to that aggregate line. Along the Y-axis I see people who
were reported in the current period but not the previous, and vice-versa
along the X-axis.

I have the ability to take this data and uncluster it by clicking on it,
and I\'m beginning to see various colors and shapes: circles and
triangles, and reds and yellows. I\'m going to do this one more time to
make it really clear what those are.

And I have, again, a legend telling me that my circles are actually
reported data, the triangles are my imputed datapoints. I\'m showing
various levels of contribution to that aggregate using marketshare.

Over on the spreadsheet though, I actually show this marginal change and
price; the prioritization we talked about. When I click on a particular
node I can see the row corresponding to it; indeed, the one that has the
highest marketshare in this case has the marginal change in price, which
isn\'t always the price. They are related but not always one-for-one
mapping.

I also have, if you can see \-- I\'ll uncluster one more time \-- but
you can see two lines in here indicating, one is a no change line \--
oh, sorry. I hit the wrong button, sorry.

CHAIR RELLES: It\'ll help us then on \--

MS. WEIR: Yes. One is a regression line so you can see the actual trend,
and one is a no change line above, which is a price increase. And below
is a price decrease. So Ruey-Pyng is going to actually show the use of
it in another survey that more recently was being tested.

MR. RUEY-PING: This survey is the natural gas monthly and it\'s one
page. And this survey somewhere is (unintelligible) of this year so you
have idea how the data being collected.

You can see this natural gas survey collect the volume and the revenue,
so (unintelligible) is the volume and the price. So the price is
calculated with the revenue divided by the volume, you get the price. So
there are three factors in here, in the (unintelligible) category, which
is residential, commercial and industrial.

So we will be concentrate on the residential, and using an example from
the (unintelligible). So first of all we can look \-- there is a
(unintelligible) every day. We can look at an anomaly map. This is May
1999 data. And I\'m going to use the Maryland here. The legend will tell
you where is the error, and the yellow is the (unintelligible) and the
blue is no errors.

So I can pick out the Maryland here and that goes back to look at
(unintelligible) and looks at tables. And this table tells me the price
change is 22 percent. And this (unintelligible) is given from the
biggest one. You should get one with about (unintelligible) currently
used.

So it\'s comparing the marketshare and also the price change between
this month and the previous month. Were is the price change larger than
15 percent it says error; if it\'s between 10 to 15 percent. If it\'s
less than ten percent than we say it\'s no error at all.

And then we can go to the detail tables and get \-- to see the company
reports in Maryland. So here we can see all the company IDs. That\'s
where we have confidential data. And then we can (unintelligible) to
tear down (inaudible). And set down the details. This is the detail
slide, so we can go back to look at the tables, find out which company
we just check on and make a follow-up code.

Okay. My progress (unintelligible) variations and used to be 10 or 15
years ago this will be the (unintelligible) analysis. We have all the
data on the mainframe, we run the test, we get back the input, it\'s
about 400 pages. So it\'s (unintelligible) pages and check out the
errors and runtimes to make sure the company we need to follow-up
(unintelligible) reporting numbers.

So GEAQs can help us if that\'s okay. I have all this and then I can
look the data plots. That will give me everything of the elements of the
13 month data, and also it will give me the time series data and tell me
what\'s going on in the company.

And by creating the point here, so here we have all the data. And we
will find something interesting here, within the box whisker criteria,
and you would know which company to borrow. Like here, we can right away
point to this company A, the staff knows how to make a phone call.

My (unintelligible) thing is trying to match what I make a report on
this and the (unintelligible) of this. The story is, if they match with
both have the warning signs or both have the error signs, then we say
it\'s okay because we don\'t have any errors involved.

If there is a mismatch then I summarize the data and give to the certain
managers to do the follow-up. And I did this work for (unintelligible)
monthly. They have to do this for every monthly data for everything.
Which means my staff will run this four times a month.

But if we have this (unintelligible) and we can just equal the data and
assess whether between the check-out or the editing, we just follow the
graphic story and point at which of the companies they can look at.

And the results of my examination is about ten percent, we have a
mismatch of the warning and the errors. But this number (unintelligible)
is sometimes the question of the GEAQs because the number is
interpolated and the other times is the threshold of the edit rule
(unintelligible).

And overall, the ten percent of this gives us ideas. If we want to adopt
the GEAQs to the other surveys we may have to look how to improve the
GEAQ which was accumulating more data and you will have more
(unintelligible) data to do a (unintelligible).

And that will conclude my remarks.

CHAIR RELLES: Thank you, very much. We have two discussants. Roy is
going to go first and I\'ll say some words after that.

MR. WHITMORE: If I could I\'d just as soon make my comments from here.
Can everybody hear me?

First of all I think the idea of doing the graphical editing process is
an excellent idea. There\'s certainly a lot of up-front investment
required to set up the software, but for the types of series of data
collections that EIA does where you\'re collecting similar data over
long periods of time, the graphical process would seem it would allow
you to do two things.

One, reduce the manpower, therefore costs associated with the editing
process; and secondly, there\'s certainly much more that you can do with
the visual graphics kinds of process, so much you can do with the
hardcopy editing, so it might improve data quality as well.

Which of course then drives costs back up, so you may wind up with some
sort of a compromise in terms of cost savings which will have improved
data quality in the process. I think also I think make the job much more
interesting for your staff. And so I think it\'s definitely a positive
development.

There was certainly a good bit of richness in what was presented here
this morning in terms of the kinds of graphics and things that have been
implemented that were not included in the paper. I was glad to see that.
It took away some of my ideas of other kinds of graphics that you might
do. I think they\'ve been done.

The testing that was described in the paper was somewhat similar to what
Ruey-Pyng was describing for the natural gas monthly report. It was
discussing the [Petroleum Marketing Monthly]{.underline} reports and I
was very impressed with the reduction in manhours required to do the
editing. This is a report that was requiring \-- the testing was being
done with three cycles of editing over three months with 60,000
aggregate published figures for each month, so it was a fairly
large-scale test.

And the test was to basically determine whether or not the graphical
system could be set up to accomplish exactly the same results as were
being done with the hard copy editing. And I think that\'s certainly for
each series the right place to start. Now, if we\'ve already gotten
these edits in place that we\'re doing, do we get the same results if we
use the graphical system?

And once you\'ve determined yes, we can get the same results that we\'re
currently getting with the graphical system, then there are all kinds of
enhancements that you can add on top of that.

And what was indicated in the paper is that the results were virtually
identical with \-- like Ruey-Pyng was saying, with about maybe a ten
percent difference in error versus warning flags. So the results were
virtually identical and it cut the editing job from 40 manhours to eight
manhours. So that\'s a pretty substantial reduction in the level of
effort required to do the editing process.

The next step, once you\'ve verified that for any individual data
series, that the graphical process produces the results that you need at
a minimum, namely what you\'re currently accomplishing with the hardcopy
editing, then you can certainly then add on to various enhancements that
allow you to do a lot more with the graphical than you could otherwise.

And I think in terms of looking at differences \-- Paul was talking
about this morning \-- breaking down the estimate into its components
and looking at each individual company\'s contribution to the aggregate
as a quick and clear way to get to which company\'s data are resulting
in the anomaly that you\'re looking at. It seems to be a very powerful
tool.

So I think that\'s about all I have to say on it.

CHAIR RELLES: Let me offer a few comments, too. I guess I didn\'t have a
good understanding of what the new editing process is. My sense is that
the old one, you got a stack of paper and you were told, go ahead, look
for all these errors and find them and take care of them.

And certainly, nobody likes paper. But we could have gotten rid of paper
simply by putting it on a diskette and saying here\'s the diskette; go
ahead and find all the problems. Okay, so now that we\'ve gotten rid of
paper I guess the next question is, how do you want to help the editors?

Well, building a set of indices that said this may be a problem, that
may be a problem. I think that\'s a really good thing to do and
producing a list of indices and sort of setting up algorithms so that
some of those indices you can figure out who to call and what the phone
numbers are and all of that. I think that\'s a great thing to do.

I\'m not sure about the value of giving everybody sort of full
multi-variate analysis capabilities in addition to that though, which is
I think, what you\'ve done. When I look at sort of all those
capabilities of this system I see, you know, substantial redevelopment
of tools like S-plus.

There\'s some really powerful graphical tools in S-plus which I see
here. I guess my hope is that the package that you based all of this on
sort of made it fairly easy to build these graphical tools and to do the
customization that your customers will need. But I guess I\'m kind of
interested to know how much you\'re really re-inventing standard
statistical packages versus how much you\'re just parameterizing what\'s
already there.

I still wonder about sort of, you know, whether full-blown,
multi-variate analysis capability is what you need as opposed to a
simple list of people to call. I think also that ties back to what Jay
was talking about having to do with sort of, the objectivity of what
you\'re doing; what is the algorithm that ends up leading to edit
resolutions.

Perhaps it\'s a little less crucial here because people aren\'t \--
people are just getting data files in the end, but how do you describe
the edits in a way that takes out the subjectivity I think would be kind
of interesting.

And again, you know, I sort of applaud the idea of building these kinds
of Cadillac systems so that they can accommodate any capabilities and
decisions you may reach in the future about how to do it, but I guess I
would also suggest that you think about kind of the Volkswagen
equivalent of it for people to start with and, you know, given an edit
list of people to call, which may be the simplest and quickest thing.

You know, what additional capabilities do people seem to demand? And it
looks like you\'re going to have whatever they\'re going to need in
here, but rather than start off with presenting them with the full range
of multi-variate analysis capabilities and all the potential drill-downs
that could take hours and days to kind of explore; that you think about
the user who just wants to do very simple series of phone calls and try
to worry about, you know, once they get used to that level, sort of how
do they improve what they do by using more of the system.

MR. RUEY-PYNG: Well, we do have the company information built in so
whichever (unintelligible) you can get the company information on the
screen right away. Yes, I like the multi-variate approach if we can
decide about our variants. Like here, if we are interested in the
residential sector data, we may want to see the residential price and
the residential volume (inaudible).

As I could even (inaudible) we can look into develop. And another thing
is the number of months data put into the system. We\'d like to have
more flexibility of importing more data. Depends on how much we really
need it. Could be for 14 months, could be for (unintelligible) months.

If that can be done then it gives us more flexibility to handle
(unintelligible) datasets. And (unintelligible), why we have given
(unintelligible) because we have given. That\'s a period where we did
that last year, for instance. And you can be customized to the different
surveys, especially if EIA has a similar survey, we can group them into
several types.

Then those types of the surveys, (unintelligible) only one set and
another. So we don\'t need (unintelligible), we probably need five or
six up there. And then with (unintelligible) and this can be a high
order (unintelligible). And hopefully you will access more manpower
(unintelligible) editing.

And any suggestion to (unintelligible) enhancements won\'t come until we
spend a little more better (unintelligible) to do the editing.

MS. WEIR: I\'d also like to address the multi-variate approach of it,
and I think it is the building of indexes and the dumping of lists that
has got us into the problem in the first place. That is why 40 percent
of our costs is from looking at these lists.

That the non-multi-variate approach, when you have people responding
many, many data items, they\'re going to pop up many times as a phone
call, then you have to have a process for putting those together and
then a rule, nine out of ten times. So that somewhere in the
rule-building you\'ve lost the details in trying to aggregate it.

Coming up with the appropriate rule we\'ve been trying to do for
beginning of surveys. And the graphical approach is a very smart,
time-driven, data-driven edit. So that you can build in your rules,
visualize rules, and then see if your rules really apply.

So if 40 percent of the data get colored you visually see that that\'s
not really who you want it. This layer here. You immediately see a rule
without writing a rule and you can do it in such a way that it is
repeatable. And I think that\'s the importance of processing the data.
You want a guarantee that another person will come to the same
conclusion and another dataset would use a similar goal.

And I think that\'s more important than the perfect list that you\'re
going to create because we\'ve been \-- survey analysts and
statisticians have been trying to write these rules. There\'s many books
out there: there\'s regressions, there\'s cluster analysis, there\'s
almost every tool in the book done on editing.

And it hasn\'t gotten us anywhere. We\'re still burning up money looking
at lists. And half the time it\'s a type-1 error; sometimes it\'s a
type-2 error, and we don\'t even try and tackle the type-2 errors.

So the multi-variant approach actually builds upon itself to seek
chronic reporting problems like across sectors and biases and things
like that, which is a much more powerful tool in getting your worst
problems solved. Maybe not all your data will be perfectly clean, but is
that your goal in producing aggregate cells? We don\'t release micro
data. So it\'s a whole different paradigm in how we approach it.

CHAIR RELLES: I agree that getting down to indices is throwing away
information and you may want to be able to kind of recover that. But on
the other hand, saying full-blown, multi-variate analysis capabilities
will solve my problem poses \-- that\'s a lot more than you need.

There ought to be sort of an efficient way to sort through it, because
if I looked at all the drill-down possibilities there were, you know, I
don\'t know, four or five different places where I could drill down. And
I have no idea sort of at the outset, which ones to look at, and I could
get mired in looking through multi-variant plots that have nicely
colored residuals and all of that.

And I guess all I\'m really asking for is, when you go beyond the
indices, sort of help me find the pictures I should look at then. Don\'t
just say, okay, and you\'ve got a package here that has all the
capabilities you\'d ever want and yes, start looking for \-- learn how
to use a package and start doing statistics on the raw data.

It\'s a decision your users are going to have to make and I just am
suggesting that you may want to pre-package perhaps, some of the plots
that they look at to make it more easy for them to decide where to go
next.

But I had no problem with the plots. Plots are \-- I\'m delighted to see
plots come into this process.

MS. WEIR: And it\'s meant to be a very structured approach when you\'re
taking the largest, most important cell, the U.S. \-- if you can\'t get
the U.S. right, you know, you\'ve got a problem, and drilling down to
the lower-level aggregate.

So it is very structured how you go about it. It\'s very methodical;
your Windows are left open as you finished a cell. It\'s taken care of,
you have a clear path of what else needs to be done. And the colors
indicate the priority of those items.

So the intent is to give a very clear path of the work to be done and
the work you\'ve already completed.

CHAIR RELLES: Right. We could actually have a nice argument but I
wouldn\'t call it that. I\'d like to give other people a chance to
comment. Joan?

MS. HEINKEL: I\'d just like to know one thing from a user\'s point of
view. I think (inaudible) response problems that we have. And I just
view it as hopefully a more effective and more efficient way of really
going after the ones that make a difference.

You know, we can get a lot of lists in the conclusions, differences, but
it seems to me that this tool puts it in a better perspective of, these
are the ones that really make the difference and these are the ones that
we really need to work on first. I mean, as the years are (inaudible)
one of ours, obviously, and I think, very interesting.

MS. CARLSON: Following on, and I think she\'s (inaudible) strategic plan
(inaudible) EIA has the tradition and the tendencies to look at every
single edit and every single problem, and that takes a phenomenal amount
of time and money. And what \-- the undercurrent that everybody is
saying is \-- and the colors \-- is that, we may over time, just limit
people to do the highest priority edits and leave it at that.

CHAIR RELLES: Jim?

MR. HAMMITT: I just wanted to comment. This thing looks great to me. I
applaud the effort. It looks very useful. One thing that came up is,
Ruey-Pyng mentioned the (inaudible) of having many different versions
(inaudible), a small number \-- four to five different versions of the
different surveys. And maybe that\'s necessary but it would seem
there\'s an advantage to having one.

CHAIR RELLES: We\'ll make this the last comment. Roy?

MR. WHITMORE: One of the types of graphical results that you mentioned I
don\'t think was actually shown here. It seems to me like it would be
very powerful in terms of prioritizing where to start in all that
thinking in aggregate and breaking it down into the components for the
individual companies.

And when you\'ve got like, looking at a difference in price and product,
the plot of the product, the marketshare times, the individual monthly
price for the company, it gives you a very quick visual way of seeing
\-- if there\'s been a large change in price, which company was the
driving force behind that change in price.

And those kinds of plots I think, are the kinds of enhancements that
allow you to quickly identify which ones of those are the companies you
need to call. Being able to look at things like a product in the
marketshare times the price.

CHAIR RELLES: Thank you, very much. Very interesting.

Let\'s see. We had a break scheduled till 10:30 and I\'d like to come
back \-- we\'re four minutes behind so let\'s make it 10:35. But I
really would like to resume at exactly 10:35.

Yesterday, because of my poor time management we did not have the update
on the cognitive \-- the results of our cognitive experiment last time.
So we\'re going to hear about that first, and then Nancy will pick up
around 10:50 and talk about the statistician\'s role at EIA. So please
come back at 10:35 promptly.

(Whereupon, the foregoing matter went

off the record at 10:19 a.m. and went

back on the record at 10:35 a.m.)

CHAIR RELLES: Okay. Well, let me introduce Howard Bradsher-Fredrick. And
I guess Renee Miller is going to manage the slides and my understanding
is Howard\'s going to update us on what happened as a result of that
very interesting session we had at the last meeting where we all got on
the web and you watched us fumble around.

MR. BRADSHER-FREDRICK: Yes, I guess I should begin by thanking everybody
for participating in the pre-pre-test of last spring and getting your
input. It was very helpful. I want to provide you with a bit of a
progress report and finally, to get input from people as to where we
could go from here.

The outline for today\'s presentation is first to review those
recommendations you gave us last spring, then we\'ll talk about the
pre-test with particular emphasis on the findings, and then we\'ll talk
about what we see is the next step.

The principal recommendations that you gave us last spring were to begin
with a reasonable number of pre-tests. You think it allowed cognitive
interviews with interviews with a one-on-one format. The phrase to
exercise questions and planning rather than using EIA terminology. We
should devise the experiment to control for question ordering effects.

And we should revise the demographic questions and shorten what was a
fairly long introduction. We should videotape the pre-tests if at all
possible. And we should review the findings of the pre-test before
proceeding.

Okay, let\'s first talk about the pre-test. We did 17 pre-tests in all,
plus we had 17 subjects: nine of the subjects were EIA staff members;
three were other DOE staff persons; two were from the Bureau of Labor
Statistics; and three were members of the general public.

In terms of the age groupings, it varied greatly from one person being
under 18 to three persons being over 50 years of age. And the
distribution of frequency of use of the EIA site also varied greatly,
from eight of the 17 having never used the site, to six of the 17 who
were frequent users of the site.

Okay, the exercise questions that we used on the pre-test \-- I won\'t
read through these or anything \-- but you can see that the questions
were fairly similar to the ones that we used last spring. And you can
also see that the EIA terminology was pretty much avoided. There were a
total of 11 exercise questions. Those were the first seven and these are
the last four. You can get some sense as to the types of questions that
we asked.

Okay, in terms of the pre-test methodology, the subjects were selected
primarily on the basis of convenience. The sessions were videotaped or
audiotaped as convenient. In a few cases we weren\'t really able to
videotape the subjects. They may have been in their workplace, they may
have been at home, they may not have wanted to been videotaped. Some of
those cases we audiotaped.

Generally a note-taker was present for the session in order to try to
capture some of the high points. We had the opportunity to use the BLS
cognitive lab for some of the tests, and we also used the makeshift lab
at L\'Enfant Plaza, and in some cases we went to the workplace or the
homes of the individuals.

Each subject was given four exercise questions to answer with a maximum
of ten minutes each for each question. Normally the entire session
lasted about 45 minutes. I don\'t think any of the sessions lasted more
than an hour.

And each of the five researchers had a chance to improve their cognitive
skills since each of us did at least three of the one-on-one cognitive
interviews.

Okay, the findings from the pre-test \-- maybe it isn\'t too surprising
to this group that the average number of correct answers to the
exercises was 1.65 out of a possible four. This varied from several
people getting no correct answers, to one person getting four correct
answers. That was one of our summer interns; an obvious over-achiever.

The results showed a fairly weak relationship between the number of
correct answers to the exercises and a few other things that we
measured. These are all self-reported. The subject\'s level of expertise
with the Internet, the subject\'s knowledge of energy terms, the
subject\'s age, and perhaps most disconcertingly, this last one \-- the
subject\'s level of confidence that she or he get correct answers to the
exercises.

It was sort of disconcerting because people in a lot of cases didn\'t
know if they had done well, they didn\'t know if they had done poorly.

More findings from the pre-test were, where a lack of familiarity with
EIA terminology can make it difficult for users to find the right data
and information. Familiarity with EIA publications seem to be very
obviously beneficial to being able to locate appropriate data and
information.

Subjects generally did not read all the buttons on any given page, or
the choices that they had. This may not be so surprising. I started
counting some of the buttons and it seemed like there 56 on the first
page that we had, and some of the second-level pages had similar
numbers.

The petroleum page has 57, the electricity page has 49. So there are a
large number of choices that people can make, and in a lot of cases they
weren\'t reading many of the possible choices.

The subjects also generally found the size of the print and the
coloration to be confusing. In some cases people thought that the size
of the print should be commensurate with the relative importance
perhaps, in finding the answers for the questions, and therefore were
inclined to go after the large printed items.

In terms of the coloration, some of the buttons you may recall are white
on dark red, and some of the people found that coloration something they
found hard to read and consequently didn\'t read the button.

The subjects found the search mechanism to be basically unsatisfactory.
I don\'t know that we\'re any different from anybody else in that
regard. I think everybody tends to find the search mechanism
unsatisfactory.

In terms of the next steps, we think that the future exercises should
provide a greater balance over the fuels. You may have noted that we may
have weighted exercise questions rather heavily on petroleum and gas
questions, so we could probably try to improve on that.

And we think that industry personnel should be included as subjects in
some of the future testing. We haven\'t included them in the pre-tests.

We plan to develop several alternative second and first-level pages
based upon the findings, and to use contractor expertise and technical
support to aid in developing the alternative pages.

And finally, we plan to conduct cognitive tests on these alternative
first and second-level pages to determine the relative useability of
perhaps new pages.

Are there any questions or any advice that you can give? Questions?

MR. HANSER: Just one question. I wasn\'t there and I apologize, but when
you do the testing you try to mimic the response rate that would be
expected given say, a 56K modem and standard servers so that you have a
sense about the speed of the response and given the constraints on the
line or the constraints on the server affect the choices of the
individuals as they go through?

MR. BRADSHER-FREDRICK: We didn\'t do that, but that\'s a good point;
that perhaps in the future we might want to more simulate what people\'s
actual experiences at home might be or in their office place, rather
than using the high-speed lines that we have. That is a possibility.

MR. HANSER: Yes, because I know \-- it\'s my own personal use, if I\'m
sitting there concerned about a 56K modem and I have a slow server
behind it, and I\'m sitting there 20 minutes for the stuff to come up,
it changes how I interact with that system in large ways. Compared to
you sitting there with, you know, Internet speeds and you know, and fast
servers behind it.

So you might want to think about that. You know it limits what, like
folks like the amazon.coms and so on put on a single page in terms of
choices of material because they understand the impacts of that does
concern you.

MS. CARLSON: Well, you did do it (inaudible) place in people\'s homes?

MR. BRADSHER-FREDRICK: Yes. There were a few cases that was part of the
stipulation. It was more by accident perhaps than by design.

MR. HANSER: I mean, you might want to think about, if you do a sample,
of (unintelligible) the sample up by: a) the type of interconnect they
have \-- if they have ESL was it a 56K modem or whatever else; and also,
depending on the material they\'re looking at, what kind of server
you\'re working off of, relative that they\'re going to be accessing.

Because even if you have a fast line, if you have a slow server at the
other end you may replicate the same kinds of problems in terms of the
choices that they make.

MR. MONTGOMERY: Did you follow this at all beyond answering questions,
into how people were trying to get the data table or the data series or
publication in a useable form?

MR. BRADSHER-FREDRICK: Well, the basic methodology, maybe you weren\'t
here last spring for this \--

MR. MONTGOMERY: No, I wasn\'t.

MR. BRADSHER-FREDRICK: But it\'s really asking people continually, as
they\'re going through the pages, what they\'re doing and why they\'re
doing it, and what they\'re looking for and their mental processes that
are involved with actually trying to find that piece of information or
answer that question.

MR. MONTGOMERY: But did you get beyond kind of single questions to, you
know, having people try to get an entire data table?

MR. BRADSHER-FREDRICK: No. The questions were the ones that we showed,
which were really pieces of information.

MR. MONTGOMERY: Right.

MR. BRADSHER-FREDRICK: Typically they had to get to a data table in
order to find the answers to those questions. It may have even made it
more difficult.

MR. MONTGOMERY: Right. I was wondering, since it\'s one of the things I
have difficulty with \--maybe this is idiosyncratic \-- but how they got
to the next stage of getting the data table into a form that they could
use, or they could make sense of the choices you had?

MR. BRADSHER-FREDRICK: This is something else that we\'ll eventually
want to look into because there are surely some problems with the layout
of the data tables, people being able to see the headers and to follow
the columns. This will be something that we\'ll look into. I think that
what we\'re trying to hit upon at the moment is some of the bigger
navigational problems of a higher level.

MR. COWING: I think what might be very important here hasn\'t apparently
been given any consideration, is the learning curve for users over
repeated visits. And so what may not be fairly important is the fact
that there are 56 buttons and the first time in I only looked at five of
them, but rather, what was I looking at before sign-in.

Because if you just looked at the first time you might conclude that you
ought to remove a lot of the buttons. I think what\'s more important is
to think about the learning curve and how many buttons are they looking
at after the sixth visit?

MR. BRADSHER-FREDRICK: So a multiple-visit situation?

MR. COWING: Right. And if you can get them to move up the learning curve
pretty quickly then what they saw in the first visit is not particularly
relevant. But having raised the question, I have no idea how to ensure
(inaudible).

MR. HANSER: The other issue is that you end up having this kind of
segmentation problem which is, the college student who\'s working to
find out the amount of energy consumed in the United States in 1998 for
their term paper versus the fellow who\'s working in a law practice but
he has to deal with law prices all the time, and so he\'s repeatedly
going back.

You know, one of the problems about dealing with any interface is
you\'ve got to sort of deal with both of those segments simultaneously.
You know, what\'s good for the first-time user that makes it easy in
terms of the navigation, other folks go back and say, God you know, get
this crap out of here. You know, I just want to go find my data.

I\'m serious. I mean, you see that all the time. And I don\'t know how
you deal with that in terms of the design, in terms of segmenting the
audience, you know, the market that you\'re going to end up working in.

And it\'s a tough one and I\'m not sure whether \-- you know, whether
you flip \-- are you a frequent user? You know, hit a button and then
suddenly you\'re off to a different kind of setup with the pages; as
opposed to, is this your first-time navigating? You might want to think
about bifurcating the process that way.

CHAIR RELLES: Or you might want to spend more time on other people\'s
web pages (inaudible). Amazon.com, for example. There\'s no reason why
you can\'t copy their ideas. It\'s quite easy to copy. You can add their
pages.

MR. HANSER: Oh, yeah, no, no. And I \-- you know, the trick is, one of
the things also to look at is \-- and it\'s kind of fun is \-- find the
companies that do web design and look at how they\'ve done it to get \--
because sometimes it turns out the best web design are the pages or the
sites that are done by the web designers themselves for their own site,
because they demonstrate all kinds of goodies and it\'s kind of fun to
look at them.

CHAIR RELLES: Calvin?

MR. KENT: Yes, just a couple of comments. The first one is, as someone
with at least a passing familiarity with EIA\'s reports and so forth, I
have no trouble navigating your site and I think I got all of the little
things that we were supposed to get when the quiz came up relatively
quickly because I use it, you know, quite a bit. And this goes back to
the point here.

I find that people on my staff and folks like graduate students who
aren\'t there, have a terrible time finding the data on your site. And
they just don\'t know \-- if I want to know how much oil is imported in
the United States, you know, to get on your site and for some reason \--
and so I think it\'s very important that you focus on the first time or
infrequent users.

Because basically I\'ve got a lot of your stuff just bookmarked, you
know, things that I\'m going to all the time. It\'s just, bing you know,
and I go right to it, and we pass the whole process. And I think
probably for most frequent users they\'re in that same, you know, mode;
that they\'ve got it pretty well broken down.

The other thing is, what have you found \-- you\'ve only done 17 people
\-- but what have you found about the cognitive processes? I mean, so
far, have you come up with anything? I mean, I get interested in people
who are right-brain learners, left-brain learners, people who are
listener-learners as opposed to seeing-learners and all of this. Have we
found out anything?

And then the other thing that I would say that I think jumps out of
this, is that yes, you do need to simplify in using Amazon.com approach
and some of the rest of them, so that you don\'t have quite so many
choices at one time because that does get I think, particularly for
first-time users, confusing.

But I think that you\'re going to have two distinct groups of people.
You\'re going to have the experienced user who knows what he wants and
basically where to get it, and then you\'re going to have the
inexperienced user who just knows EIA\'s got that information on oil
imports.

CHAIR RELLES: Can I get David\'s comments on the table and then we\'ll
just ask you to wrap up so we can move on?

MR. MONTGOMERY: You might answer some of these \-- answer Cal\'s, too,
so I\'ll try to be real quick. I\'ve also found that I can find a lot of
stuff that my colleagues or staff don\'t find on the web site, and
that\'s not because I\'m any good with the Internet but because I\'ve
been familiar with the publications over the years.

And I do think that\'s a problem because it is for some people \-- it
just doesn\'t occur to them where to look for something that they want
to have, and I\'m not sure that that\'s a web site problem.

I keep coming back to this discussion of channels that John Pearson and
John Weiner had yesterday. I thought actually it might be a great help,
because one of the great difficulties I\'ve had, just in terms of
wasting time is, I know what I want, I think I remember which
publications, and I go and find it, I find it for this year, but I
wanted a time series.

And I realize, you know, [Natural Gas Monthly]{.underline} doesn\'t give
me a time series on this. I\'ve got to remember that it was in the
[Monthly Energy Review]{.underline} that I find the time series for this
particular one and go find it. And that\'s a particular difficulty.

I think also it suggests you \-- I think you ought to get beyond asking
people how to find a single data element, to think about how they find
data arrays of the type that they want; whether they\'re cross-sectional
or time series. Because that\'s very hard to do.

And for me, what is \-- the most difficult thing is getting it. Once
I\'ve found the place where it is, just choosing between \-- I mean,
sometimes I wish you just had a FTP site again where I could simply
download the file that I want that contained either the image of the
document or the data series.

Because choosing between \-- I don\'t know what I\'m going to get when I
choose between, you know, HTM or PDF or .XLS. In some cases I can save
them to disc. In some cases I can\'t save them to disc; I\'ve got to sit
there and print them.

All those choices I think, are totally confused and they\'re a level
beyond what you\'ve gotten to yet in terms of peoples\' ability to pull
something off the site.

MR. BRADSHER-FREDRICK: I think that these last two comments from Dave
and Cal were maybe really to the point in the sense that perhaps our
data is organized a little bit too much by publication on the site. And
consequently, if you know the publications, which is something that we
noticed, the person does well with finding the answers.

But to the first-time user, the one-time user, it tends to be pretty
difficult without knowing how we collected the data and how it\'s
organized. Which may be getting back to the infocentric situation that
maybe you heard about yesterday.

But we received other good comments, I think, with thinking about the
high-speed line and simulating people\'s real experiences in the
workplace or at home. And some of the other comments about perhaps
thinking about the frequent user versus the one-time user and looking at
some of the web sites that exist that might be designed perhaps better
than ours and what you might be able to learn from them.

Okay, thank you very much.

CHAIR RELLES: Thank you. I\'d like to introduce our last speaker. Many
of you, I assume all of you, know Nancy Kirkendall who is a
nationally-prominent statistician who has been in the EIA for many, many
years; left about two or three years ago to go to OMB, and fortunately
has come back.

So she\'s been back about four or five months, has been trying to set
her agenda, and wants to talk to us about the Role of the Senior
Statistician in the Statistics and Methods group of the EIA.

MS. KIRKENDALL: I really came to EIA in 1980. I was here from \'80 to
\'96. I started out in the Office of Energy Data Operations and then the
Agency reorganized about \'81 I think, and ended up in the Petroleum
Supply Division for a couple of years and then finally went to the
Office of Statistical Standards where I stayed for the rest of the time.

About three years ago now I went to OMB, and I was there for
two-and-a-half years. I\'ve been back for six months actually; it will
be six months on Tuesday, next week. The time really flies when you\'re
having fun, doesn\'t it?

Now, I view my new role in EIA as one of stimulating technical
discussion and collaboration in the staff; trying to solve problems that
we\'ve got and see the solutions are implemented. So we want to solve
problems and get the solutions implemented. And I\'ve got four projects
that were in the paper that I\'ll go through briefly, and then I\'ll be
interested in any input you guys have.

These are the four projects I talked about in the paper. The first one
\-- evidently somebody talked to you at one of your more recent
meetings, and I think that the advice you gave was, well just go ahead
and do it.

So what we\'re going to do is, I\'ve got a memo that I\'m sending out. I
think I\'ve sent it out to many statisticians \-- perhaps not all of
them \-- inviting comments. I think I\'ll send it out more broadly
pretty soon since I haven\'t got a whole lot of comments. I\'ll send it
to everybody.

Basically, we\'re going to pick a generic definition of response rates;
a nice simple one like it\'s the number of people who responded, or
companies, divided by the total number of people that should have
responded. I think we should have two response rates: one that\'s
related to the number of units that should respond, and one that\'s
weighted based on the impact of the numbers you want to estimate.

And the only challenge that\'s associated with the definition are how to
estimate the numerators, how to estimate the denominators, and how to
figure out what the weighting is.

For many of our surveys I think that\'s really easy and I think the
previous team that looked at response rates got a little bogged down in
details for some of the more difficult surveys.

So if we go ahead and we try to implement it in CCAP \-- that means that
we don\'t have to change our ongoing system \-- it can be changed as
we\'re changing anyhow \-- we can make decisions one at a time as the
surveys move into CCAP and it should at least reduce the conflict, the
results from change.

And then the other thing is, we need an inter-office team of
statisticians to work together to make the decisions about how these
things should be estimated. As I said, I think the early ones will be
easy and we get to later ones \-- the one on alternative fuel vehicles
where everybody and their brother is invited to send a form to somebody,
may be a little harder to come up with a good response group. But we can
save it till later, anyhow.

The next one on the list is a meeting and inter-agency group on
performance measures. This group \-- Jay is on the inter-agency Council
on Statistical Policy. This was at the behest of the largest statistical
agencies. And back in the spring they were invited to share copies of
their strategic plans.

So what this group is done is, we\'ve taken the strategic plans and
we\'ve looked at them and tried to cull out and label the different
measures we use. And we\'ve organized it in terms of a balanced work
chart. I\'m not sure if you can see this. On this one I know the
response rates both. If you\'re interested I can email you copies of
what I\'ve got and I\'d be interested in any comments you\'ve got.

EIA is in this column here. We\'ve got lots of checks. This is going to
be presented to the agency heads at their meeting next week, and I\'m
hoping that it will stimulate some discussion about what we do, how we
do it, and what we ought to do differently, or how we ought to do things
more similarly.

There are some differences in the measures that are actually in the
strategic plan, depending on whether it\'s an agency-level plan like
EIA\'s or whether it\'s a departmental plan. The Department of Commerce
has one strategic plan; the Census Bureau does not have a separate one.

So they of course, don\'t have some of the measures in that plan like
learning and growth and monitoring attitude. The Census Bureau does
that; it\'s just not part of the Department of Commerce\'s strategic
plan. So the level of the plan makes some difference.

In addition to this we have a little memo that describes what we mean by
each category; always nice to know. And then how each agency measures
what they\'ve got (inaudible).

MR. HANSER: I don\'t know all of the initials on that slide; I\'m sorry.

MS. KIRKENDALL: Bureau of Labor Statistics, Bureau of Economic Analysis,
Bureau of Justice Statistics, Bureau of Census, Bureau of Transportation
Statistics, Energy Information Administration, Environmental Protection
Agency, Economic Research Service.

MS. CRAWFORD: I\'m sorry, they don\'t (inaudible).

MS. KIRKENDALL: I\'m sorry, I kept trying to get them to \-- we kept
trying to get them to participate. They never even sent a strategic
plan. They claim they don\'t have a strategic plan. And we couldn\'t get
anybody to participate in the agreements. I kept trying to tell my group
we should take that column out and they just say, well no, we want to
make a point.

MR. HANSER: Yes, you can do that with economics and you\'d have the same
problem.

MS. KIRKENDALL: Economic Research Service; that\'s in the Department of
Agriculture. National Agricultural Statistics Service, National Center
for Education Statistics, National Center for Health Statistics,
National Science Foundation, Survey Resources \--

MS. CRAWFORD: Science Resources.

MS. KIRKENDALL: Science Resources Section \-- something \-- Statistics
of Income Division in IRS, and the Social Security Administration.

MR. HANSER: Thank you. I apologize, I just \--

MS. KIRKENDALL: That\'s okay. One really ought to tell you
(unintelligible).

The other thing that we thought that I should do is some training. I
teach part-time at GW so it\'s not all that impossible to do. And the
two courses that I guess I\'m signed up to do, one of them is on time
series and I\'ll talk more about that, and the other one is an
elementary statistics course to try to educate some of our people who
are non-statisticians.

That one is harder because you really need \-- I don\'t think that we as
statisticians are very good at teaching people statistics in the way
that they actually understand anything about it. I mean, you always here
people say, you\'re in statistics? I had a course in statistics. That
was the worst course I had.

So I\'d rather not teach a course like that; doesn\'t seem particularly
useful. But the other course will be kind of fun and we had a request to
do a time series course. And you heard from Theresa today that we have
some time series applications here; not only hers but we have the
Short-Term Energy Forecasting Group, the Energy Modeling Group. The NEMS
Modeling Group is a bit different. In fact, they\'re all a bit
different.

But I think that that would be a good opportunity to combine some
training and just collaboration. I think that all of those folks could
benefit by learning more about what Theresa\'s done in her early
estimates.

And none of the people who have requested the training is in the NEMS
\-- the modeling area. So I think we\'ll end up getting together and
talking about what we\'d like to do with that course but make it more of
a collaboration and less of a, Nancy standing up and talking in the
course.

The last item on my list is, there are a couple of activities going on
in disclosure limitation and I\'m not real actively involved in either
of them. The Audit Software we\'re sort of \-- Ruey-Pyng is one of the
people involved with this. He\'s back there.

This is an inter-agency effort. Several different agencies have
contributed some money to develop a software product in SASS that would
take tables and audit it to find out what\'s the maximum value and the
minimum value each suppressed cell could take. And that would help you
to know whether the suppression patterns you decided on to protect
confidentiality are doing what you want them to do.

We\'re doing it this way so that agencies can share the software
product. We\'re going to make it available to everybody. And so EIA is
serving as the conduit for that and Ruey-Pyng is one of the technical
monitors and I get it on some discussions, too.

Another thing that we\'re looking at is how to make EIA\'s data from the
residential \-- maybe not residential but the commercial building
survey. I think that we made it available to researchers at research
data sites. So we\'re talking to the Census Bureau and looking at what
other people are doing, that sort of thing.

So that is a set of projects. I am also particularly interested in the
graphical editing. I think that\'s the kind of thing that would be
really helpful to EIA and I\'d like to see it more broadly used. So I\'d
like to get involved in helping to make it work, have it move forward.

Any other ideas?

CHAIR RELLES: David?

MR. MONTGOMERY: Oh, I\'m sorry. It never came down.

MR. HANSER: One of the things that struck me so funny was, thinking
about your time series \-- not your time series but your data general
(inaudible). You know, you could take a lot of the series and bootstrap
them and then use that as a mechanism by which you can look at, these
are the new datapoints. Where does it sit relative to, if we repeatedly
sample the observations to sort of see where things fit.

The technique of bootstrapping has I think, you know, widespread
potential applications in several of the kind of analyses we do get, to
get a sense about just the kind of inherent variability of the data, if
you were sort of like taking it off the web or whatever.

You start wondering about techniques like that may be interesting to
bring into EIA. I\'m not sure much of it technically.

MS. KIRKENDALL: Yes, I\'d have to know more about, I guess the data
series and how they\'d be used. I think that actually I think \--
Doug\'s not here but there was \-- we had a contractor a long time ago
who did some bootstrapping studies with a model \--

MR. HANSER: Um-hmm, that\'s another way to do it.

MS. KIRKENDALL: I think it showed that \-- of course, we underestimate
various (inaudible).

CHAIR RELLES: We are a statistics committee so I guess I\'d like to get
your comments on how you feel about sort of creating a home for
statisticians or statistics at EIA, perhaps in the form of some
statistics seminars or nurturing of summer interns from statistics
departments.

Theresa for example, is in a different section of EIA and yet her
professional development was probably as a statistician as opposed to a
person in an applied area would improve I presume, by having some
difficult statistics presence at EIA.

Any thoughts about that?

MS. KIRKENDALL: I think we can do a lot of that with collaboration. I
think it\'s been good for Theresa to be in her office and work on the
projects that she\'s worked on. And I think that collaborative efforts,
getting statisticians together for example in the time series exercise
will be one way of doing that, and maybe we can broaden into other
problems in statistics as well.

MR. HANSER: Do you have a post-Doc program here like the Bureau of the
Census has a post-Doc?

MS. KIRKENDALL: Census Research Fellow Program?

MR. HANSER: Yes.

MS. KIRKENDALL: No, we don\'t have that kind of program.

MR. HANSER: That might be, you know, something to think about trying to
get going, because you know, a year at EIA for a post-Doc in either
Economics or Statistics or something. And it might be useful.

MS. KIRKENDALL: Yes.

MR. KENT: I would just second that; that\'s why I put my card up.
Because I had good success in getting members of my faculty on with
other agencies on either summer internships or when they\'re doing
sabbaticals or something of that nature, and they\'ve always come back
very much enriched from the experience.

And it\'s certainly something I think EIA would profit from and if
Seymour were here he could talk about what a great time he had with the
Census. And to have some sort of a post-Doctorate or you know, faculty
fellowships either for the summer or for people who come here when
they\'re on sabbaticals, I think would be a real way for you to get some
really good outside talent to come in and either work on special
problems or to even, you know, do some of these other things that you
probably couldn\'t afford to get these people normally. And this would
be a way of getting them.

CHAIR RELLES: Are you trying to (inaudible).

MR. MONTGOMERY: No, no, this exactly the same subject. I think that some
form of a, kind of a post-Doc or a research fellowship would be a great
help. I was going to throw in some observations I had about recruiting
because we have found ourselves competing against BLS and the Federal
Reserve Board in the last couple of years for hiring new Ph.D.s in
Economics.

The basic deal that BLS and the Fed were offering was, we will hire you
and you don\'t have to work for two years; that you can do anything you
please on your own research for the first two years at both of those
places. Which even though we obviously pay a lot more money, was very
attractive to a lot of people that we wanted to get coming out of
graduate school.

And I\'m not suggesting that as a recruiting policy because of the
taxpayer outrage, but something like this might give you a similar kind
of competitive edge in hiring. It would give you an opportunity to bring
people in here who wanted to be around, who had something to contribute,
but also would have more of an opportunity for doing something more
academic in their experience.

CHAIR RELLES: Carol.

MS. CRAWFORD: I have two comments and the first one is about the
graphical editing. I was talking to Lynda and I think we both agree that
we either need a committee member who specializes on some of the more
computational graphic aspects of survey data collection and editing, or
maybe need to invite some specialized speakers in on that topic.

So if you have any suggestions as to who you might \-- who\'s working in
this area or somebody that you\'d like me to contact, then please let me
know.

MS. KIRKENDALL: Actually Paula has done most of the research in that
area. She probably has some ideas. But I\'ll think about it anyhow.

MS. CRAWFORD: And the second thing I have is something that always
strikes me at every meeting that I always talk myself out of mentioning.
And it\'s somewhat related to what Phillip said. It struck me in
Theresa\'s talk that there\'s a lot of emphasis on time series, I think
because you want to forecast in time and because you have a lot of
temporal variation.

But there\'s also a spatial component to EIA\'s data. You have spatial
variations and you also are making predictions in space to some extent.
And so I\'m wondering if there \-- I just feel that that is something
you might want to consider some spatial, statistical methods to help in
the analysis of EIA\'s data.

MR. HANSER: You know, in a (inaudible) \--

CHAIR RELLES: Tom\'s card was up.

MR. COWING: After six years, Phil, you (inaudible).

MR. HANSER: I\'m a slow learner.

MR. COWING: No wonder, no wonder you (inaudible). This is a kind of
minor comment related to teaching statistics, you know, at maybe an
introductory or somewhat higher level, of (unintelligible) statistics
and also econometrics to undergraduates who are not very well prepared.

And I find that using applications after applications after
applications, particularly if it\'s real data as opposed to kind of
made-up data which is placed in textbooks, is really very, very helpful.

Now you can say well, we\'re perfect there because we\'ve got all this
data around. What better place to go for data? But there might be an
argument that says for anybody working in EIA, no one uses Energy data
because that\'s what we get paid to do and that\'s kind of boring to
have to do it more.

So if you\'re really interested in cross-fertilization with other
agencies in this city, then that might suggest you can use data from
somewhere else in order to introduce people to other data sources, to
other statisticians, to other agencies and improve communications across
the mall, so to speak.

MS. KIRKENDALL: That\'s a good idea.

CHAIR RELLES: Phil.

MR. HANSER: Oh, Good. On physical data, not survey data, there\'s been a
huge effort by the SLAC, Stanford Linear Accelerator, in looking at
data, because they have gugalls of data, literally, coming out of the
particle accelerator. And Jerry Friedman there, had a big project
looking at projection pursuit methods for analyzing large datasets and
all.

You might want to just talk to the folks over at SLAC at some point to
sort of see if they\'ve got anything. Because they\'ve been doing
graphical analysis of data in large datasets for at least, I don\'t
know, 20 years. Also, what\'s-his-name at Cornell \--

MS. CRAWFORD: That\'s good.

MR. HANSER: What?

MS. CRAWFORD: (Inaudible) cornell.edu.

MR. HANSER: Yes, Paul (unintelligible), yes, at Cornett
(unintelligible). I have to look at my ASA directory. At one point I
think he was doing a lot of that same sort of thing at a circular slice,
and he developed that software that does the 3-dimensional rotations and
so on. You might want to just bring him in there to sort of look at some
of the stuff that\'s going on.

At one point in time the ASA has sponsored papers of an applied nature
to use in an introductory statistics class. And there was that book by
Judith Tenour, [Statistics of Guiding the Unknown]{.underline}, that was
done, I don\'t know, a long time ago, that\'s got a nice selection of,
you know, kind of things to look at.

MS. KIRKENDALL: Actually, I attended, five or six years ago, at the
Joint Statistical Meeting they had a workshop on teaching Elementary
Statistics. A lot of people got together and they showed some of the
hands-on examples that they use in trying to help people learn. There
was some pretty neat ones.

MR. HANSER: Yes, and something like, I mean, it\'s been a long time
since I taught statistics, but boy, having a little package that\'s
something like a little data desk or a little mini-tab or something. Now
that PCs are everywhere, I mean, it\'s sort of a nice thing to be able
to do. And I\'m sure, since everybody works here, they have this
wonderful capability to sort of play with data, which, you know \--

MS. KIRKENDALL: I think it wouldn\'t hurt to have that ability once they
get out of the class.

MR. HANSER: Yes, right, exactly.

CHAIR RELLES: Cal?

MR. KENT: In this line of thinking, let me just extend it and that is,
if you\'re talking about doing statistics training for people, the
fastest growing area in education right now is certificate programs.

And I think that you would really have something to pass out to people
if you would have certificates in Statistical Analysis that said, after
you\'ve completed a certain number of these courses we will certify and
you\'ve got a little piece of paper but you\'ve also got something that
you can list on a Federal Employment Form and it\'s more than just
saying, I attended a class. And that\'s lost somewhere in a person\'s
resume.

But we literally, at just my little humble school, we do thousands of
certificates every year for people and everything. That means they
haven\'t completed a formal college course, paid tuition or everything,
but we can certify that they have acquired certain competency and they
actually have something. We can give CEUs, you know, for it as well, and
that really increases the popularity because that\'s something that\'s
transferrable and something that\'s referenceable.

So if you really get into this, this doesn\'t say, well here it is and
you all come. But if you set it up in that sort of a formal learning
pattern it would be really a great benefit at low cost to give to your
employees. Of course, they may all go off someplace else if the
certificate begins to have some sort of a cache \-- you know, go over to
EIA and get trained, then come back to Census.

But I still think it would be something. It would be really worthwhile
to formalize a training program, have a certificate, and then it also
helps you design a curriculum; you know, where do people start, where do
people get off, and of this nature.

I\'d be happy to do some more work with you because we do a lot of that
stuff.

MS. CARLSON: We actually are participating in a joint program on serving
methodology but as far as a certificate program, we haven\'t been able
really, to get anybody to participate at this point because the location
is out at College Park and nobody wants to pick up and go to College
Park.

So what we\'re now working with is BLS and NAS who are going to do some
of the classes on-site from their locations, and then we\'ll see if we
can get people to go there.

MR. KENT: Don\'t you have IITV \-- some sort of IITV in college so that
you can bring the class to \--

MS. CARLSON: We don\'t have the appropriate one for what Maryland uses,
what we found out. We did look at that.

MR. KENT: When you\'re in a cross-(unintelligible) situation or
something like that where \--

MS. CARLSON: You\'re beyond my level of competence.

MR. COWING: There are several technologies; they are not always
compatible.

MR. KENT: Yes, I hadn\'t thought about that.

CHAIR RELLES: Okay, thank you very much, Nancy. I guess we\'re coming
\-- oh, Bill?

MR. WEINIG: This is a first. Earlier this morning, Dan, Glenn Arschlady
stopped by. He specifically chose not to speak to the committee but he
asked that he be able to introduce a paper to the committee, which I
would like on his behalf, to circulate.

CHAIR RELLES: Please.

MR. WEINIG: I think that any comments that the committee may have might
be made to him directly, but I haven\'t read it so I can\'t characterize
it one way or the other.

CHAIR RELLES: You have an email address on there?

MR. WEINIG: Yes. And it will go in the record. Well, by doing this it
goes in the record, yes.

CHAIR RELLES: Was he here yesterday but kind of gave up when we went
long?

MR. WEINIG: No, he was here this morning.

CHAIR RELLES: We\'re getting to the end of the agenda. Let\'s see, there
are three things: committee discussion, invitation for public comment,
and final remarks and closing of the meeting. So time for the comments
from the committee, first. Calvin?

MR. KENT: I want to extend my personal thanks and congratulations to you
for having been such an outstanding chair. It\'s been real easy to work
with you, it\'s been a real pleasure to work with you. I think you\'ve
provided really great leadership.

I think that the word that you were used by Jay when he talked about you
said that you were the critiquer extraordinaire. So I looked that up in
my French dictionary and that\'s translates as \"pain in the butt\", you
know.

But nevertheless, I want to really tell you how much I\'ve enjoyed your
leadership. You\'re setting a high mark for Carol to achieve because
you\'ve done a super job and I just wanted to let you know how much I
appreciated that.

CHAIR RELLES: Well, thank you very much. Coming from you that means a
lot to me. It really does. It would mean a lot to me coming from anyone,
but \--

MR. KENT: Particularly me because I never say anything nice about
anybody?

CHAIR RELLES: Actually, if I can have an audio copy of that; I might
have trouble seeing.

Well, all other compliments are welcome, but if you\'ve run out I\'d
invite normal discussion or comments from the public.

Well, I get to make final remarks and close the meeting. My final
remarks have to be about Bill Weinig. And you know, when tennis players
win matches they tend to thank the ball boys and the ball girls. So
thanking people has kind of been cheapened, but in this case Bill
deserves so much praise.

I\'ll just try to be a little objective about things and try to give you
some facts. I do have a bull in a china shop mentality very often and
Bill\'s been instrumental in helping me understand the pressures that
EIA is under and that the various people are under, so that I really,
after talking to Bill, usually figure out and know sort of the best way
to approach people at EIA who are as beleaguered and overworked as I am.
So I give him an awful lot of credit for that.

Secondly, when I started the job I guess I really don\'t work in Energy
so I don\'t have an instinctive feel for what\'s important. I kind of
feel that I\'m capable of assessing whether something matches the
committee\'s skills and interests, but I\'ve really relied on Bill to
set the agenda, propose topics that are what\'s important to EIA.

You know, initially I might have had to do a little bit of editing to
find topics that I thought matched our capabilities. As an example, the
current agenda, I mean, Bill proposed it virtually like it is, and my
job was easy. Yes, this stuff is really appropriate, interesting, and go
ahead and do it. So that\'s the second thing that Bill does.

And then the third thing that Bill does is convinces people at EIA that
it\'s worth the time and effort to prepare their talks for us and to
come in and listen to what we have to say. And he\'s remarkably good at
getting people to volunteer their time at EIA to work with us.

So you know, I\'m being totally objective in giving Bill all of those
credits and you know, in addition to that I\'ve just enjoyed immensely
working with him over the last couple of years, and I just thank him for
all the work he\'s done and for being a friend. So thank you very much.

And I guess at this point I would just like to close the meeting and
thank you all for attending and for supporting me and EIA over the last
couple of years.

(Whereupon, at 11:28 a.m., the meeting was adjourned.)