AMERICAN STATISTICAL ASSOCIATION

COMMITTEE ON ENERGY STATISTICS

\+ + + + +

MEETING

\+ + + + +

FRIDAY

APRIL 20, 2001

The committee met in the Department of Energy Training Facility, 8th
Floor, 950 L\'Enfant Plaza, S.W., Washington, D.C., at 8:30 a.m., Carol
A. Gotway Crawford, Ph.D., Chair, presiding.

PRESENT:

CAROL A. GOTWAY CRAWFORD, Ph.D., Chair

F. JAY BREIDT, Ph.D., Vice Chair

MARK BERNSTEIN, Ph.D.

JOHNNY BLAIR

MARK BURTON, Ph.D.

THOMAS G. COWING, Ph.D.

JAMES K. HAMMITT, Ph.D.

NICOLAS HENGARTNER

W. DAVID MONTGOMERY, Ph.D.

WILLIAM G. MOSS, Ph.D.

POLLY A. PHIPPS, Ph.D.

RANDY R. SITTER, Ph.D.

ROY WHITMORE, Ph.D.

[C-O-N-T-E-N-T-S]{.underline}

Welcome 3

Introductions 3

Opening Remarks 3

Measures of Data Quality 4

Committee Questions 14

Out One Door and In the Other - Knowledge Sharing

In a Period of Transition 40

Committee Questions 55

Recognize Winners of Graphics Contest 104

Adjournment 113

[P-R-O-C-E-E-D-I-N-G-S]{.underline}

(8:31 a.m.)

CHAIRPERSON CRAWFORD: I\'d like to get started. If there is anyone from
the Committee or from EIA or a member of the public that was not here
yesterday, please introduce yourself.

MR. BRADSHER-FREDRICK: I\'m Howard Bradsher-Fredrick.

CHAIRPERSON CRAWFORD: You need to speak into the microphone please.

MR. BRADSHER-FREDRICK: I\'m Howard Bradsher-Fredrick, I\'m with EIA\'s
Statistics and Methods Group.

MR. WEINER: I\'m John Weiner with the EIA and I\'ll be speaking a little
later.

MS. CARLSON: I\'m Mary Carlson with the EIA.

CHAIRPERSON CRAWFORD: Thank you, and anyone who did not sign in
yesterday is asked to sing in this morning at the break before leaving.
Lunch for the committee and guests will be held at the conclusion of
this meeting on the 6th floor as it was yesterday.

My final announcement for the opening of this meeting is that we\'ve
unanimously agreed on the dates for the fall meeting. So mark your
calendars for October 25th and 26th. So we\'ll begin this morning with
Tom Broene on measures of data quality for EIA surveys.

MR. BROENE: Okay I\'m with SMG and also on the Performance Measures
Committee at EIA and I want to tell you about our latest proposed
measures. We just discussed these with senior staff last week, so we
don\'t have any hard data yet. There is always a trade off at a meeting
like this, do you want to get it in advance where you still have time
for input or do you want to see it after we actually have something to
show you.

So this time you\'re getting it in advance. I\'ll tell a little about
why we\'re working on it now. A basic literature search that I did
starting with the CIS current index to statistics and then branching out
from there, past work at EIA, recommendations.

There are limitations. We know the measures we\'re recommending are not
perfect, and we know some of those limitations already, and welcome your
input on that also and the reactions in the questions for the Committee.

Several reasons why we\'re working on this now, one is a strategic plan
calls for it. We\'re on our second strategic plan and when we updated it
a few years ago, we placed a greater emphasis on accuracy.

Moving around on this screen a little bit, down at the very bottom,
standardized design for any new systems. When we are working any new
system such as we were on CCAPS a few years ago, we want to make sure
that whatever system we design will have the measures that we want.

Also down there at the bottom, just because it came in late, is OMB
asked us just a few weeks ago, what\'s our quality policy, and do we
have any quality guidelines, and you have a definition of quality, for
which we said no.

Back up on the top here, some of work that I found going on elsewhere
from the literature search, several people are working on this.
Statistics Canada has been working on this for quite a while. They have
their guidelines up on their web site.

Their third edition came out in \'98, and they recommend a varying set
of measures depending on the data in the audience. At a bare minimum
some measure or rating of coverage, response rate, or imputation, and
the sampling error if it\'s a sampling. That\'s the bare minimum and
then more depending on the data in the audience.

NCES has a checklist that\'s on the output side. When a report goes out
they have either a six- or seven-page checklist signed off, pen on
paper, saying, did you do the editing, did you investigate large
changes, did you use the sample weights? Yes, no, not applicable.

Eurostat has measures. There was a conference at BLS last fall and they
are going to be asking their member countries for underlying data. So
they\'re not going to get the microdata, but if you tell Eurostat how
much of a certain crop you produce, they will want to know more than
just that point estimate. They\'re going to want to know what\'s your
frame, what date it was collected, what was your response rate, how did
you impute, how much by each method. That is what they are going to ask
for.

So that will be with every data point that they request from the member
countries. And they will also ask for evaluations less frequently on
something like coverage.

Those are some of the numeric measures, and while reading the
literature, I also found a couple of things that are nonnumeric, but
that I thought were quite interesting.

One is from RTI and the other from Statistics Sweden. We had Mike Weeks
from RTI come in here last June and conduct a two-day class for us, and
he mentioned that they do regular reviews of all their ongoing surveys.

And the same type of program is in place in Statistics Sweden. And both
Mike of RTI and the article I read from Statistics Sweden said that
these people thought it was very important to do that. At RTI it\'s
reviewed by Mike and at Statistics Sweden it\'s reviewed by a central
group. And so I call that either an annual quality review, although
it\'s not necessary annual, it may be a little bit more frequently or
less frequently, but most of them are annual. Or you could call that a
self-assessment. But it is reviewed by someone else, either another
person or a committee within the organization.

One other thing I saw in the reading is some agencies are advocating a
compiling of written description on an occasional basis, talking about
your staff, what\'s their experience, what\'s their training level, what
cognitive testing have you done. And other procedures including like
your estimation procedure.

Any questions on that? That covers a lot and there is some more
information in the paper that I had up on the web site. But any
questions or clarification on that at this point?

Okay. Past efforts at EIA. Several years ago we had a business
reengineering where they proposed rating all of our surveys on five
components. And you would get a one to five rating on two different
scales. The quality of the data and what we know about that data.

And so for example, if you\'re talking about sampling error, you
probably have a pretty good knowledge of it. You would probably rate
that as a five out of five and then just have to decide where falls two,
three, four, or five in terms of the actual quality of it.

But if you were looking at coverage, it could be that you don\'t know
how good the coverage is, or it could be, as we had for one of our
surveys, we know that we have lousy coverage, because we have the actual
volumes and we know that our price data is not that good. And so you
would probably rate it only a two for coverage of our industrial gas and
actual price data, and we\'re pretty sure of that, so that would be a
five in terms of our level of knowledge.

BR debated this and they decided not to go ahead with that. It was going
to be too subjective and it was going to be too time consuming. Those
are two good warnings.

One other thing I should mention in terms of pass work, is we do conduct
customer satisfaction surveys. We do polls of people that phone into our
support center and also that come onto our web site, and I see those
survey efforts as really independent of the measures that we are
proposing here.

Other efforts that we have done, we\'ve done quality profiles. They are
very comprehensive. The major customers, including this Committee, have
found them to be very useful. But a few years ago I did a customer
satisfaction survey and except the sampling PPS I got the distribution
counts of all of our paper products. And the quality profiles were not
that widely disseminated.

So they do reach a number of people that are important customers, but
they\'re not reaching anywhere close to everyone that probably should be
reading them.

We did have a standards manual. October \'93 was the last time we
updated that and we\'re not sure if that\'s still being used widely or
not. We\'re really seeking a minimum standard set of measures, short,
easy to produce, and quickly understood.

So recommendations. For frames, some measure of volatility and the
number in the frame \-- this isn\'t going to tell us whether or not our
frame is really up to date, but will give us some measure of the work
required. Either the unit response rate or the item response rate,
preferably both, but otherwise one of those two.

For data processing we want to be able to look at our timeliness. We
want to look at the mode. How are we receiving our data? Is it coming in
by fax, by email, by web submission? Imputation, how much of it is being
imputed? And the sampling variants, either externally calculated through
Sudaan or WesVar, or otherwise we would design into any future systems.

And data comparisons, looking at change, either prior survey cycle or a
year ago. Look at revisions. We do have a lot of revisions. We have
initial data and then final data that\'s released later, so investigate
that. And if we have external data, either from another outside source
or from another survey, what comparisons we can make with that.

As I said, these are not perfect. We have avoided doing something that
would require a lot of time or a lot of money. As you heard a little
bit, at least in one of the breakout sessions yesterday we discussed our
budge. There isn\'t a whole lot to go back and validate data and do
reinterviews.

I think our frame measures are not great. Editing activities, I
couldn\'t think of any good ones for editing, so I didn\'t include any.
In the past, in the old standards manual, it called for someone to
evaluate the editing algorithm. I did not want to commit to do that,
either myself or the Committee, nor SMG.

Secondhand data. We do have some. We don\'t have as much secondhand data
as say the Department of Education, but we do have some from states or
other federal agencies, and so that would be dependent on them giving us
background information before we could do anything with it.

So those are some of the limitations that we have seen.

Reactions within the agency. The idea of a self-assessment or a
regularly scheduled quality review, that was well received. And that
appears to be going forward, at least that was the decision I believe at
senior staff last week, that we would work on that. And that we would
start out first by just using it within the agency, but with a goal of
later down the road, I\'m not exactly sure when, but some point in the
future, releasing those results.

Now there was some concern that, how frank are you going to be if you
commit to release all this on your web site? Will you be as honest in
the appraisal if it\'s out there in public? And we had some debate
within the Committee on that topic. Some people really only wanted to
keep it within their office, others said, okay, share it within EIA.
Where some of our surveys already have all this information and more out
on the web site already or it\'s in the appendix in the back of the
publications. So we had some debate on that.

And obviously we also want to minimize the burden on our survey staff or
our contractors who are working on this.

So some of the questions for the Committee are, how important is it that
these measures be released versus only used internally? And for
self-assessment or quality review and annual quality review of some type
of regularly scheduled event, any opinions on what we should add to that
or what do you think that should include?

I have gotten two requests for some type of a composite measure. Two
people in the EIA have asked me for quality index, to somehow combine
different measures and maybe you would have a submeasure to look at all
the frame activity and have something else. But then roll this all
together into one quality index which I have not done yet, but I would
be interested in any opinions on whether or not you think that\'s
appropriate. And of course any general suggestions.

Any questions or clarifications?

CHAIRPERSON CRAWFORD: Thank you, Tom. We have two ASA discussants this
morning. The first discussant is Polly Phipps.

DR. PHIPPS: I have been away from the quality literature for some time
and during that time, as I\'ve looked at the paper and reviewed what Tom
has put together, it certainly has continued to proliferate starting
from the total quality management to the business reengineering, ratings
and indices, best practices, quality guidelines, and I\'m sure there\'ll
be a lot more to come. It\'s an area of continual developments and
progress.

And quality can mean a number of things and encompasses so many varying
concepts, and I think Tom\'s done a really nice job in his paper of
reviewing the existing quality schemes that are out there and also
presenting the history at EIA.

The literature is so broad and most organizations have adapted their
model to fit management models and the types of surveys they do, so
it\'s hard to come up with this kind of one-size-fits-all list of
quality measures.

But the EIA strategic plan focuses on the data. Accuracy is the main
quality type of indicator, and so the critical question is, what are the
minimum quality measures I think that EIA\'s diverse customers need to
know to assess data accuracy.

And as I thought about this, I thought about the measures that were
proposed and what they would tell me about data accuracy and
specifically focusing on the major quantitative indicators that are
proposed and what they told me about the major components of survey
error.

I\'m not going to focus too much on sampling, but particularly on the
nonsampling error including the coverage and the nonresponse and
measurement error.

These kind of quantitative aggregate indicators are only really partly
satisfying to me. What can be produced across all these surveys is so
limited and as a user of the data I would have an interest in knowing
what steps have been taken to assure data quality and reduce error. And
this is more probably in line with the self-assessment that was talked
about or a yearly review.

So I would propose more using these aggregate indicators, but also doing
some kind of assessment of what\'s been done to assure data quality with
like a set of yes/no items or also a set of dates of when certain events
that are difficult to carry out were carried out.

These quantitative indicators are really indirect measures of survey
error and also I\'d like to know in the self-assessment if really any
direct measures were taken to estimate either bias or variable error.

So let me just look at a couple slides here on the proposed indicators.
I think that they\'re solid quality indicators and I would probably
suggest the more tied to this self-assessment review and some slight
variations in the indicators. And I\'m not going to focus on sampling
error here. I think that\'s one of the areas where there\'s probably a
better understanding of the error, but I\'m going to focus more on the
nonsampling error.

For the coverage quality measures that were proposed, there\'s a number
of measures in this area that are proposed, which is a good thing.
Looking at things like the births and the deaths, mergers, and out of
scope or areas that are proposed in looking at the weighted volume. So
you could even compute things like birth rates, death rates, mergers,
that kind of thing.

On the direct side, looking at measurement, the coverage error more
directly, there\'s a discussion of comparison with other surveys or
census data or independent aggregates, which might give some better
sense of direct measures.

In kind of a self-assessment review, as I\'ve talked about these, would
be the more aggregate kind of quantitative indicators. In a
self-assessment review, it would be nice to know what other kinds of
measures have been taken to look at the data over time, just a knowledge
of if an alternative list check or any kind of matching or other
procedure of population units against sample frame issues, if that\'s
ever been done for the survey, and the date that it\'s been done. So I
think it would be nice to get a sense of that in a self-assessment
review.

Nonresponse is another area where there\'s some pretty strong measures
that were proposed. On the indirect side, looking at the unit response
rate, the item nonresponse rate, tied to this perhaps the imputation
rate. As a survey user, the revisions that were proposed would tell me
something about the nonresponse quality. Also, things like the
comparison of survey, unit data from prior surveys, if that was part of
the discussion. It wasn\'t certain if that was one of the things we\'re
thinking of doing in terms of looking at comparisons of actually survey
units over time to look at a different prior survey period.

And timeliness would tell me, as one of the indicators proposed, would
tell me something about the nonresponse quality, not a lot, but just a
general kind of indicator. On the direct measure here, the comparisons
with other EIA surveys, or independent if that could be done, would give
you a more sense of a direct measure of the nonresponse error.

And then again, it would be nice to know just on a yes/no scale if any
more direct indicators events have been carried out in the last number
of years. So if there\'s ever been any kind of link to administrative
data to look at nonrespondents or if there have been any other followup
surveys to estimate nonresponse, and I\'m not trying to suggest more
work, but it could just be a yes/no measure and maybe the date that it
was done and as a part of a review process it would be thought about,
something to look for in the future if it\'s never been done.

The measurement error is where things get really weak here, and that\'s
one of the areas that I\'m most interested in. There\'s really no great
measures proposed to look at all about things, the respondent, the
interviewer, if there is one, the questionnaire, and the mode. There was
some discussion in the paper of an edit failure rate which might give
some indication of measurement error.

Also, there are just some things that aren\'t really measures at all,
but might tell me as a user something about the item nonresponse, maybe
the timeliness tells me a little bit about measurement, what measurement
error might be out there. Just knowing the mode doesn\'t tell me too
much.

Perhaps, something that was discussed yesterday, looking at late
respondent data would tell me something about more the measurement error
that might be tied to both measurement error and nonresponse, but
there\'s really nothing in the indicators proposed that looks at really
what error is out there in the respondent in the questionnaire, things
like cognitive testing, respondent analysis surveys, any type of really
even mode comparisons of the data.

So again, as part of a self-assessment kind of checklist, it would be
nice to know as a user when these have been carried out, if they\'ve
been carried out, and just what dates they\'ve been carried out. That
would give me a stronger feeling about the quality and accuracy of the
data and how I would go about using it.

And also, in the same sense, if there\'s ever been any work done, yes/no
on a record check study or reinterview. And these are just examples,
there are other measures of these that could be used, both indirect and
direct.

So just overall, I think the proposed measures are good quantitative
indicators. It\'s possible there\'s nothing focusing on measurement
error at all, but maybe something like the edit failure or one point
that was discussed in the paper, some kind of an edit, kind of hit rate,
the edit flags resulting in the data change, but it really concerns me
that there\'s no focus at all on the measurement error and I know it\'s
hard.

So I would suggest in that area, and maybe others, just as quality
measures checklist of yes/no items, when things have been carried out,
and then the date they were carried out. So that would be planning for
the future.

And I guess on the composite index, I\'ll let the Committee talk about
that. I think those kinds of things are hard and indexing things does
involve some judgment. I would suggest that the task that\'s been taken,
that you start this out internally and get the process going and then
report it externally is good. Although I think there\'s some external
measures that you just always have to report, like response rate and in
the sampling variants and something about the coverage to your user.
Thank you.

CHAIRPERSON CRAWFORD: Our next discussant is Roy Whitmore.

DR. WHITMORE: I also think that Tom has done a nice job of reviewing
literature and finding out what other folks have done. Certainly that\'s
a very good start. And in my comments I wanted to kind of address each
of the questions that Tom raised and see if I could provide some
opinions as to how to think about those questions.

The first question was whether or not the quality measures should be
released to the public or just used in house to try to improve the
quality of the studies, and I think there are advantages and
disadvantages, no clear yes/no answer. With regard to advantages of
releasing the data or the quality measures to the public, it could help
the public to understand the data better and to interpret the data
properly and knowing it\'s limitations.

It might help protect against the public reaching unsupported
conclusions, particularly if there are frame problems or something like
that, that there are caveats that have to be made. And it may provide
additional motivation to ensure that the quality of the data are really
good, because if you\'re going to have to say something about the
quality of the data and put it out to the public, you\'re not going to
want to say, well, it really isn\'t very good.

On the other hand, releasing all the quality measures to the public
could discourage fully objective evaluation of the data quality,
resulting in positively biased measures of quality. I\'d be tempted to
say that it\'s really better than it really is.

It certainly would require additional effort to figure out how to best
present the quality measures to the public other than the usual things
and response rates, as the deeper you go, the more time you\'d have to
spend in figuring out how to best present things to the public.

And there is the possibility that if you present a lot of quality
measures to the public, you could actually confuse people rather than
enlighten them.

In terms of what measures should be included, let\'s just kind of run
through the ones that have been suggested as a starting point and see
what more we might be able to say about that.

Certainly if there are units on the frame that are not really part of
the population, that certainly needs to be a part of the frame
evaluation. It\'s certainly very important to address the completeness
of the sampling frame and there are various ways that could be done. I
think that\'s extremely crucial.

And we need to address as well the uniqueness of the items on the frame
or multiplicity. When you\'re dealing with population that\'s hard to
reach, you might be dealing with multiple sources of information for the
sampling frame and you need to say something in that case about the
uniqueness or the multiplicity of the units on the frame.

And the other measures that were proposed with regard to frame
characteristics and weighted counts, population size, represented by the
frame I think are a good idea.

And with regard to response rates, the paper suggested doing weighted
and unweighted unit and item response rates. Certainly for unit response
rates that makes a lot of sense. If you go to a survey that has hundreds
of items and you\'re looking at item response rates for all of them,
that gets to be a voluminous presentation. You might want a summary
table of item response rates for key items.

I think there are a number of other things that were considered with
regard to data items that you might want to be looking at for key items
in looking at summary tables.

And with regard to timeliness, the paper talked about using the date and
time stamps. I think that would be \-- I guess the idea is that you\'re
simply recording when each interview came in or was recorded so that you
can say something about timeliness and late respondents versus early
respondents, and that sort of thing, and that\'s a good idea.

And of course sampling variances, the paper talked about how providing
sampling variances for all estimates and you might want to consider a
table of standard errors and relative standard errors for key statistics
as a quality measure. Certainly having the ability for any sample survey
to compute the standard errors is a good idea. But for a quality
measure, you might want to look at standard errors and relative standard
errors for some of the key statistics.

Mode of data collection is certainly valuable to report, especially if
during the multiple mode survey you might want to talk about how many of
the responses were obtained by each mode of data collection. Weighted
and unweighted percentages of imputed items was discussed. That should
go pretty hand in hand with weighted and unweighted item response rates.
You probably need one or the other, not both. And again, in terms of
characterizing the quality of the study, you might want to present that
for a set of key items.

With regard to data editing, and granted it\'s an important aspect of
data accuracy, on the other hand, often there\'s a lot of editing done
because you\'re being thorough and careful and checking things. And be a
little concerned if you had statistics on data editing that that could,
in terms of release to the public, give people an idea that they are
actually of poorer quality than they really are if you report that
there\'s been a lot of editing taking place. I\'m not sure about that.

Some of the things that were suggested with regard to data comparison
seem to me more of analytic outcomes than measures of quality. The paper
talked about reporting changes from cycle to cycle. That seems to me of
being more of an analytic outcome. You\'re talking about changes from
month to month or quarterly or from one year to the next. I\'m not sure
that\'s a measure of quality.

Similarly the paper talked about changes from monthly to annual. Again,
it seems to me like you could certainly have seasonal effects going on
and that\'s more an analytic outcome rather than a measure of quality.

And the paper discussed looking at changes related to timeliness and
that change from initial to final results, and I think that kind of goes
together with the timeliness thing rather than data comparison.

Certainly comparison to external data, or as Polly mentioned,
administrative records data, any sort of outside sources that might have
a more complete sampling frame or universe represented is an excellent
thing to do.

For many of the EIA studies, I think the EIA frame is probably going to
be the most complete and many of the EIA studies are censuses of the
universe as opposed to sample surveys and in many cases I suspect that
there won\'t be outside sources to compare to. And where possible, I
think that definitely should be done and could be very enlightening with
regard to any problems with the frame or the data collection procedures.

Composite measures of quality, it seems like that there are lots of
different aspects of quality and any composite it seems to me is going
to be very subjective and likely to be misleading. This doesn\'t seem
like a good idea.

That\'s as far as I got before I got here, so the rest of my slides
hopefully you can read my writing.

I agree with the conclusion that the methodology suggested by the
business reengineering proposal is too subjective. I don\'t know about
time consuming, but being subjective, it\'s probably not going to be
very informative.

And I think it might also be valuable to have some type of a description
of the process used to develop and test the survey instruments. It has
certainly \-- if you can\'t actually do the nonsampling error
evaluation, having some idea of the completeness and quality with which
the surveying instruments were developed, would say something about what
quality you should expect in terms of the survey responses. And that
would be especially true of any new data series that you\'re developing
as opposed to something that\'s been ongoing and thoroughly tested.

Whenever the study is not a census of the entire population, certainly
there should be some description of the sampling design that was used.
Because probably sampling design is your basis for the
representativeness of the sample. And so for any sample survey I would
encourage to include some kind of a summary description of the sampling
design as part of the quality description of the study.

Many times for sample surveys, you\'re looking at some precision
requirements to determine what sort of sample sizes should be used for
the survey, and in that case it would be useful to compare the precision
requirements that were used to determine sample sizes versus the
precision that was actually achieved in the study. It would not only
provide a measure of quality for that individual survey, but suggest
what assumptions that went into the development of the sample size that
may have been incorrect and how the sample sizes should be adjusted the
next time around to achieve the desire and precision.

With regard to \-- the paper talked about having sampling variances
available. As I mentioned, you might want to present relative standard
errors. Another way of kind addressing the same thing would be to
provide confidence interval estimates for some of the key parameters.
And I think in terms of things that would be easily understood by the
public, that that might be the simplest and most easily understood is
the interval estimates for key parameters to give you a very clear idea
of the uncertainty with regard to the estimates in the study.

And the final thing is that the EIA really wants a minimal set of
quality measures as seemed to be mentioned in the paper. I think that
Stat Canada list is a good place to start. You might want to embellish
that a little bit, but the Statistics Canada minimal set, I think Tom
reported that they had kind of a minimal set, and then expanded sets
that are used in different types of studies depending on what was being
done.

But I think their minimal set is a very good one, especially if you
amplified it just a bit to include measure frame coverage, weighted and
unweighted response rates, and weighted and unweighted item response
rates, and/or percentages of imputed data items, and per sample surveys.
It also included the standard errors and I would say you would want to
look at standard errors or relative standard errors or confidence
interval estimates for key parameters. I think if you\'ve done those
things, that would be a nice kind of minimal set.

Those were my observations and recommendations.

CHAIRPERSON CRAWFORD: Tom, did you have anything else you wanted to say
or add or respond to?

MR. BROENE: What I will say is thank you very, very much, both Polly and
Roy for taking the time to read it and to really evaluate and giving
some excellent feedback.

CHAIRPERSON CRAWFORD: Are there any comments or questions from the
Committee? Yeah, Jim.

DR. HAMMITT: There were a huge numbers of suggestions to initial paper
and then by the discussants, and it\'s sort of easy to ask for more
things, but I think we should also give weight to the idea of not
putting excessive burden on EIA and obviously among the users of the
data, people will have varying ranges of interest in understanding
details of what the data represent and how they were generated.

So I guess I would lean, and then with regard to the public/private
dissemination of this, I would lean I think toward routinely reporting
sort of objective quantifiable measures like standard errors and things
of that sort. Perhaps the imputations, fraction data imputed, computed.
Things like the trends I agree with. I think as Roy has said, that\'s
more of an analytic outcome and if you post the data for several years,
people want to evaluate the trends for their own purposes, they can do
that. I don\'t think that\'s necessary for you to do within the context
of this kind of effort.

Issues about the timeliness of response and the mode by which the
response was made, it\'s not really clear to me how that\'s useful in
evaluating quality. I guess there\'s some sense that the late responses
are not as good as early ones, but unless there\'s a sort of clearer
theory and empirical support, or why that tells us about quality, I
would be inclined not to make that an important element.

And then the other point in think Roy made about describing the sampling
plan. While that\'s sort of not within the set of the quantity of
characteristics, I would like to report like response rates and standard
error, that\'s fairly fundamental. So I would also put that out.

That\'s the kind of stuff I would put out, and you undoubtedly will do a
bunch of other things to evaluate and improve the quality. But by and
large, within our system of government, EIA has been given the
responsibility to do this as well as is reasonably possible and so a lot
of the measure of quality, we count on you guys to do that. And most of
us users don\'t need to know all this detail.

MR. BROENE: Those are good comments. One point of clarification. The
timeliness is really a trade off with the accuracy. The thought is that
if we rush, we release the data sooner and we shorten up the editing
process and we shorten the data collection process, does the quality
suffer? And so it\'s sort of a trade off with the accuracy. It\'s
related to it, almost maybe inversely. That\'s why it was thrown in
there. But I agree with you.

DR. HAMMITT: So it says that if you report data before all of it comes
in \--

MR. BROENE: There was a big rush a few years ago that we placed a real
emphasis on trying to get the data out sooner. Some of the customer
satisfaction surveys we had done showed that people were dissatisfied
with how long it took. And now we want to keep track of both.

DR. WHITMORE: With regard to mode of data collection, I wasn\'t thinking
of that, like you were saying, it\'s not a very direct measure of
quality. But on the other hand, there are some things in the literature
with regard to things that are collected face to face, being more
reliable maybe than things that are collected over the telephone, and
might have some systematic differences with regard to things that maybe
are collected by a touchtone data entry or something like that.

So there may be some known quality characteristics having to do with
mode. But it\'s not a real important measure.

With regard to timeliness, a kind of standard thing that sometimes is
done with regard to nonresponsive biases is to look at, are the late
respondents difficult from earlier respondents? It may be that that
could be used in that context and maybe so one of things when you\'re
talking about this, wouldn\'t be presented to the public, but could be
done within the EIA to look at the late respondents versus early
respondents and see is there any major difference between those
suggesting that maybe the nonrespondents might be different from the
respondents.

MR. BLAIR: Often in looking at methodology reports that address data
quality, I get a sense of sort of a lot of summary measures like
response rate and things about the sampling frame and so forth. But
sometimes for particular variables, particular statistics of interest,
it\'s hard to pull that together and see for a particular item, a key
item, that that\'s of importance, now how do all of these things kind
fit together?

And to kind of go back at a number of points in Roy\'s discussion he
mentioned the idea of key items and key statistics. And I think one
approach that\'s useful sometimes is to select a very small number of
items and really kind of drill down in looking at them in more detail.

Particularly I think things where it is hard to do kind of summary
descriptions like measurement error or even a mode effect that might
effect particular items or be of more import for particular items than
for others.

You can\'t do this on a lot of items, but sometimes I think that it
would make the report more useful if for a few key items you really kind
of pull all of this information together and look for that particular
item, now how do these various quality measures stack up, and what does
it tell me about the confidence I can have in the measure of that
particular item.

CHAIRPERSON CRAWFORD: Are there any other comments from the Committee?
Comments from EIA? Anything else to add Tom?

MS. WAUGHA: One thing I\'ve noticed about working with measurements, is
that often times then the quality is focused on meeting those
measurements and other aspects of quality are ignored. And I don\'t know
if having some of the minimum ones would cause those that aren\'t
included as measurements of quality to be ignored.

MS. KIRKENDALL: Actually, this was a good kind of introduction of what
we might end up talking more about in future meetings. I think Larry
Pettis mentioned last night that data quality is a key issue for EIA
right now. And SMG is trying to figure out how we\'re going to help the
rest of the agency to move along to. So as Tom has presented as one of
the things that we\'re doing.

The senior staff actually, when Tom presented this to them, a number of
them said that they would be interested in external evaluations of the
quality of their surveys and systems. Meaning that we should comment and
we should come and do something. So I think this is going to end up
being a bigger topic for us and we\'ll have a lot more things to share
with you, hopefully by the fall, and maybe before then.

CHAIRPERSON CRAWFORD: We\'re running ahead of schedule, but if John is
ready, I think we could \-- he\'s not? Okay. Are you sure that it\'s
okay to continue? Okay. Yes.

DR. SITTER: Carol, could I make a small comment, a very small one?

CHAIRPERSON CRAWFORD: Yes.

DR. SITTER: The notion of item response rates, if you\'ve just got \--
as mentioned, maybe just looking at a couple when you\'ve got a lot of
items. So a simple thing is just to draw a histogram of the response
rates across items. It gives a very quick overview of not only what the
overall response rate is like, but how much it varies from item to item
which can be quite illuminating sometimes and I don\'t think people
quite realize that. The response can be quite different for different
parts of your survey.

CHAIRPERSON CRAWFORD: That is a good idea.

And so our next topic is Out One Door and In the Other, Knowledge
Sharing in a Period of Transition, and John Weiner is going to take the
lead on that with additional input from Colleen Blessing and Renee
Miller.

MR. WEINER: Technology confounds me. I\'m sorry for seating myself, but
I have a little back issue and if you don\'t want me to have you walking
around and around and have me walking around and around while is speak,
this is the best you\'re going to get.

Let me introduce Renee Miller from Statistics and Methods and Colleen
Blessing from the National Energy Information Center. I\'m John Weiner,
I\'m also from NEIC.

The topic that we\'re going to present, we have teamed on equally and we
decided all to take a bit of it so that you could hear from all of us
and then we could all participate in the discussion. Colleen maybe
delivering from the podium \-- okay, I\'ll pass the microphone on down.

There is a structure in EIA called the Quality Council. The Quality
Council, permit to read, assures that quality, principles, and practices
are an effective integral part of EIA\'s culture, management, and
operations.

The topic we\'re presenting this morning comes out of the Quality
Council realm. It\'s not strictly statistical, but from time to time you
do get topics that are not. We know as well and we value your wisdom and
experience about management issues like this.

When we were working on the Quality Council plan for the 2000-2001 year
last fall, the conversation moved toward staff turnover. We have 386
people working at EIA today and we observed that at this point, almost
80 of them, 78 to be exact, are eligible for retirement right now, which
is 20 percent. And by the year 2005, 173 will be eligible for
retirement, which is almost half, 45 percent.

So the face if EIA is me and that\'s tough. So there are two issues
generating from this staff turnover. First of all is recruitment, and
we\'re doing pretty well and that may be a topic, the whole recruitment
topic that you all want to talk about, but that\'s not what we\'re hear
about.

And the second issue sort of parts us into two parts, and that is for
all the people who are leaving, how do we capture what they know before
they go. And for all the people who are arriving, how do we capture in a
useful way what they can tell us, and also help them make the transition
into the agency in a smooth and effective way.

So my part of this is to talk about capturing the knowledge of those who
depart. We\'re not quite as far along with that as we are with the
topics that my colleagues are going to talk about because we\'re sort of
trying to work on a prototype and we\'re only at the point where we\'ve
developed the first draft of a prototype, but we\'ve shopped it around
to folks and it seems to resonate fairly well. So the help that you can
give us to expand on it will be very, very much appreciated.

One of our entitlements if you will, are the many anecdotes that people
tell us about folks who have left. Oh my God, Diane is gone, Jim is
gone, Jean is gone, Morris is gone, and the names go on and on and on.
Real people. And they took all of it with them. Or at least they didn\'t
leave it in such a way that it could be used.

So we identified some folks in EIA who are of an age and of an
inclination that they are about to leave, or at least to leave within a
year and we had a number of interviews with each of them and worked up
this prototype, which I hope you all have seen.

Let me just go over it briefly. We talked with them about how to share
information in the time before they leave. I want to start with that
because that was fairly straightforward. They can mentor if there\'s
enough overlap. They can create some kind of permanent record. Some of
them would write it all down, some of them would permit themselves to be
videotaped, some of them would make tape recordings. It very much
depends on the personality of the individual.

Some of them would give formal training classes or seminars. One of our
most enthusiastic discussants if you will, was somebody who said, I\'d
like to give some seminars, and it doesn\'t necessarily just have to be
with people in my office. It can be with a wider group of people on
particular topics that I\'m good at before I leave. We thought that was
an excellent idea.

Some of them could return periodically post-retirement as just in time
trainers or, you pick the word, experts, either hired back or back on a
pro bono basis to give consultations or even via email or telephone.
What would they share, what would this all be about?

Well, we broke that down into six or seven categories and I won\'t go
over them in detail because I know we\'ll discuss them later. But
obviously core data competencies, especially what the data really mean
for people who are taking over.

Core analysis competencies, what kinds of training, formal training
might be suggestive for people who replace the retirees, plus any folks
that come on later. What duties were undertaken by the retiree for the
past year. How products are assembled.

Documentation. Ah yes, there\'s the word. Document your systems before
you get out, before they unlock the shackles. And then the whole range
of personal resources, your Rolodex, your important contacts,
bibliographies and lists and the like.

And one of our discussants was a manager. The whole notion of grooming
the successor rather than walking out one day and suddenly there\'s a
vacuum and what do you do. In other words, working with whoever is going
to take over.

Or reorganizing the operation so that when you leave as a manager, the
channels of communication and authority are clear, and maybe the
successors have had an opportunity actually to do some work in the new
structure so that you sort of phase back. This one manager that we dealt
with had, we think, done an excellent job.

So those are sort of the models for how to do it and what to do it. And
the issues are straightforward and we probably could have written them
before we actually started our interviews.

First of all, knowledge sharing gets in the way of real work. And it\'s
very difficult, especially in a situation where there are not a lot of
people and more work to do than can be accommodated, to spend time
actually structuring a knowledge-sharing operation as you are also
moving toward retirement. Your supervisor may expect you to continue to
do your work and there just isn\'t time for it.

We think that departure plans should be as carefully wrought as
retirement plans. And that\'s not only the responsibility of the
individual retiree, potential retiree, but also of the supervisory and
management structure.

We think that another issue is that there may be more than just a
difficulty in finding personal time to do the knowledge sharing, but
there may not be anybody to pass the work on to. But that offers the
opportunity if some of your work as a retiree is not going to be done,
to work within your structure as you are planning your departure to
decide what\'s going to be kept and what\'s not going to be kept and
that gives you a role as a retiree to sort of plan for what succeeds you
when you leave.

It\'s the converse to feeling like a lame duck. You have some stake in
planning for what\'s really important when you go.

And then another two points. Some of our discussants said, you know, you
really can\'t pass instincts and experiences on. And we understand that.
There are limitations. You can\'t clone anybody. But that\'s an issue
that has to be dealt with.

And the final issue that somebody mentioned was that management may
adapt the organization to the new round of employees, so really what
you\'re passing on from the old guard is irrelevant, and that is because
the organization will form around the talents and the strengths of the
new people and not worry so much about the strengths of the old people
that they\'re losing. That\'s also an issue that has to be dealt with.

But I think we think they\'re peripheral to the first two and that is
finding the time to do all this work, and I guess that\'s really the
principle issue, finding the time and the space to do it. Sorry I
rambled on. Let me turn the microphone over to, well, I\'ll have to look
\-- Orientation for New Employees? Okay, that\'s correct. Renee.

CHAIRPERSON CRAWFORD: Tom has a formal discussion I guess on everything,
so maybe we\'ll save him for the end. Okay. So does anyone have any
questions or comments on what John has talked about so far?

DR. BURTON: Most of the strategies that you mentioned for trying to make
sure that information and experiences get passed on, I think are
valuable, perhaps with the exception of bringing retirees back, either
as returned employment or on a pro bono basis. And my experience has
been that that tends to undermine both the confidence and the incentive
that the newer employees have when they know that\'s there. They don\'t
have the reason to gain the maturity themselves. So I kind of am a
little concerned about that. Otherwise the strategies sounded very good.

DR. PHIPPS: I\'m wondering if anything has been done on just generally
thinking about files, old and working files, individual files versus
department files. It seems like when this kind of transition comes, you
tend \-- in my experience there\'s a lot of reinventing of the wheel
that goes on.

And if there\'s any way that both individuals at a division or
department level you could keep the old and the working files up to
date. And also just maybe have a list of what\'s in them. I mean, a lot
of people don\'t know what\'s in those kind of division files, so you go
back and if you\'re going to ten years from do an emergency survey, you
have the documentation for the last. It\'s always been a concern for me
because I\'ve seen just files thrown out and they can be useful in the
long run.

MR. WEINER: That\'s a very good point. One of my other roles in the
agency is records management. Ah yes. And if you want to put somebody to
sleep, come in and talk about records management. But that\'s a key
point and one we did not include and we should.

DR. PHIPPS: I was in a situation once where someone was throwing out
files in the agency and it was on a survey that was going to be
restarted up again and it was all the old files, they were just in the
hall sitting there.

MR. WEINER: And you lost them.

DR. PHIPPS: No, I dusted them off.

MR. WEINER: Dusted them off. Good. That\'s a good idea.

DR. SITTER: I think that overlap of resources to actually make something
out of overlap employees is probably a huge weakness in most
organizations. Not just of retirement, just generally. When you talk
about overlap, you need some real overlap. Two weeks is not overlap. A
month isn\'t overlap. And I think that I\'ve had the unfortunate
experience as an external consultant consulting in long-term projects of
being the only form of continuity.

So I think that that\'s an important thing to have to come from EIA. And
you\'re putting some stress on the individual, but you did mention the
fact that you have to create \-- that the problem is the work still has
to be done. So I think there has to be, in the perfect world, at least
for the people you really do want to pass on their styles and management
skills to a successor, that there is a transition period of where they
actually do that, that is their job is to pass on that material to a
successor. That\'s probably going to be your biggest issue.

MR. WEINER: Yeah. And the reason that we\'re doing this at the level of
Quality Council, which is essentially a representative body of the whole
organization, is that doing this well has got to emanate from a
commitment by senior management. If there\'s no commitment, in other
words, if the work always comes first, and the rest of this is just sort
of peripheral, then I agree with you. Then you\'re lucky if you get two
weeks.

But the long-term, making the retirement decision and the retirement
announcement a positive thing, which is what we want, won\'t happen.

DR. SITTER: Have you given any thought, and I don\'t know how this would
work in this kind of organization, but there are possibilities in some
retirement incentive programs essentially to top off somebody\'s
retirement income for a period. And one way to create resources might be
to create program where a person can essentially retire and then stay on
with the top-off fund during this period so that their successor is in
fact doing the job together with the person, but it\'s not costing you
his entire salary and therefore you actually have the resources.

MR. WEINER: Yeah. The federal government has something called the
reemployment annuitant program where you can keep somebody on board
after retirement.

CHAIRPERSON CRAWFORD: I have one question, and that is that throughout
this discussion and in your writing, there was how to transfer things to
a successor and you mentioned having a successor. My question is, how do
you identify one or what incentives are there for people to actually
succeed in someone else\'s job.

And I\'ll give you an example. I know in our organization there\'s a
senior statistician and his name is Norm and he\'s thinking about
retiring. And what he\'s in charge of, he is in charge of all the
mainframe stuff, the huge cancer mortality data that we collected and
downloading that.

There\'s a couple of issues. First of all, when he retires there no
guarantee that his grade\'s going to be retained. So whoever would take
over this responsibility may not end up getting a promotion of it. And
then the second thing is that \-- I mean, I know that my boss and I were
looking at each other the other day thinking, gee, we don\'t want to do
this.

So what kind of incentives are there for people who take over things
that really are crucial, perhaps records management is one of those.
They\'re really fundamentally crucial, but they don\'t necessarily seem
like they\'re exciting to other people.

MR. WEINER: That\'s a very question, and I don\'t have an answer. In
some cases there are not enough staff hours to make the takeover
complete and in other cases there is no interest on the part of the
folks who are still there to continue to do this and all I can say is
it\'s an issue that the management structure of the particular
organization, as you are doing, has got to deal with.

I mean, you\'re going to have to make a decision when Norm leaves, are
you really going to continue to do that, and you look at the mandates
and usually do we must or do we not must and if we don\'t must, then
there\'s nobody to do it, then it\'s set aside. That\'s the decision
that we have to make.

CHAIRPERSON CRAWFORD: And I think that in addition to all the items that
you\'ve listed here, motivation or reward or something for the person
that\'s coming in and doing these necessary jobs.

For instance, a lot of people are, well what about me, what do I get if
I take over Norm\'s job. I don\'t get any more money. I work a lot
harder at doing something I don\'t necessarily enjoy, and I might be a
big person about that and kind of do that for free, but a lot of people
aren\'t willing to do that anymore. So I guess my suggestion would be, I
think that list of things and the approach that you\'ve gone, is really
excellent, and I think a lot of senior management people are going to
look at this as a template for how to conduct this.

And so that\'s something that I would put on there is the incentive of
identifying these mentorees and giving them something that would make it
in their best interest to take over some of this work and this
knowledge.

MR. WEINER: That\'s a good point and we\'ll include that. Thank you.

DR. MOSS: I just had a couple of I guess questions or observations. It
seems to me many of the measures that you\'ve suggested here really go
to good documentation for just running your organization. And in terms
of kind of the succession, you have products that are put out monthly,
quarterly, annually, on a fairly tight schedule and timeliness is
important to you.

What happens if the person who\'s responsible for one of those products
goes out and gets hit by a car and dies? Do you postpone it for a month,
or is there somebody who will naturally step in and make sure the wheels
keep turning?

It seems to me that\'s a natural kind of framework for looking at how to
make the transition easier. And if you don\'t have that, you maybe ought
to think about it. Not that traffic around here is any worse than it
ever has been, but \--

(laughter)

\-- but you know, this kind of insurance policy is kind of seems to me
to be a natural lead in to succession, unless the coverage you have for
each other is horizontal and not vertical. What I mean by that is if
everybody who\'s in charge of a product kind of covers for the next guy,
then your succession problem isn\'t covered, but if within the team that
does the product, they cover, then you\'ve got the seeds of succession
built there.

MR. WEINER: Yeah. I think that\'s an excellent point. And when we were
pushing 800 FTEs way back in the good old days, there was a lot more
verticality. Now there\'s a lot of the horizontal aspect, that\'s it\'s
difficult to get backups below you because there is no below. But I
think that\'s a good way of positioning this kind of program with a
management structure.

That is to say, you don\'t have to do this all at once if you\'ve got a
well-structured program of good documentation and good backup, then
it\'ll sort of do itself, or at least it will do more of itself than it
would be otherwise. I think that\'s a good observation. We\'ll include
that.

CHAIRPERSON CRAWFORD: I guess we\'re ready.

MR. WEINER: Thank you very much. As I said, this is sort of a two part,
and the second part has to do with capturing the knowledge of those who
are coming in and Colleen and Renee will share that second half.

MS. MILLER: Okay, can you all hear me? Okay. We are going to be talking
about the orientation seminars that we\'ve been setting up for new
employees. We\'ve been calling these EIA 101, and the idea here is to
give information on EIA\'s important activities and products.

And while these seminars have been set up for the new employees, we\'ve
been inviting the not-so-new employees to join us as well. And we had
our first seminar back in October. It was on the history and background
of EIA and Colleen and Mark Mazer, who has been our acting
administrator, gave presentations. They talked about the history of EIA
and the importance of a policy neutral status.

And also at that point as I recall, we all introduced ourselves and
talked about where we worked and where we were from, and I remember this
was back in October during the World Series and so we talked about the
Mets and the Yankees, and everybody kind of got to know each other.

And then our next seminar we called Media Stars and this focused on the
dissemination phase. And we had three presenters, Dave Costello from the
Office of Energy Markets and End Use, Doug McIntyre from Oil and Gas,
and Jonathan Cogan from the National Energy Information Center, and
they\'ve all had a lot of experience in dealing with the media. In fact,
you may have seen Dave Costello\'s been on the evening news talking
about the short-term forecasts.

So they described their experiences and they talked about the importance
of getting data out to the people who need it. And it was really quite
interesting. I think this was interesting also to people who have been
at EIA for a long time as well as the new employees.

We\'ve also had a seminar on the web site and we had Mark Rodekohr who
has been involved from the very beginning on the web site talk about it
and he presented some of the web site usage statistics which are in your
packages someplace. Some of you may have seen them. And he talked about
our popular products and the awards we\'ve one and the recent site
redesign and our plans for 2001. And I think this was also very well
received by both the new and the current employees.

The fourth seminar we had was an overview of the analysis review board
activities and also we talked about EIA\'s forms clearance and
confidentiality procedures and Barry Jaffe from the Office of Oil and
Gas presented a part on the Analysis Review Board and Jay Hasselbury
from the Statistics and Methods Group talked about forms clearance.

And I think this was an interesting combination because we had both the
analysis people and the data people in the room together so I think this
was also beneficial for both the old and the new people.

And we are planning a fifth seminar, we\'re going to call it Ask Steve.
We\'re going to give the opportunity for new employees to ask Steve
Durbin, who\'s the Director of the Office of Resource Management, any
questions they may have on administrative and support issues. So we
think that this should be quite informative.

And then this will kind of complete our first semester of EIA 101 and
then what we plan to do is to ask the new employees, what else are they
interested in. And as you can see, some of these topics could easily be
expanded. We could do a lot more on data collection and analysis that
the people are interested in.

What we were wondering from the Committee, if you could put yourselves
in the position of new employees, and we were wondering what might you
be interested in. So we were interested in your comments. I don\'t know
if we want to do that now or \--

CHAIRPERSON CRAWFORD: We might as well. Any? Tom.

DR. COWING: Is this a required course?

(laughter)

MS. MILLER: Unfortunately no.

DR. COWING: The question really is what\'s the response record and \--
is it pretty good? Or are there things you could do to improve it, get
more of the employees to come in?

MS. MILLER: Yeah, I think it was pretty good at the beginning and then
it started tapering off. And we don\'t really know \-- I mean if you
have ideas on what we can do to get more new employees, that would be
great.

DR. COWING: Well one easy answer is to make it required.

CHAIRPERSON CRAWFORD: Food.

MS. MILLER: Food.

(laughter)

DR. WHITMORE: I was wondering, you\'ve got a whole series of seminars
here, and yet you\'re probably getting new employees in on a continuous
basis. I\'m wondering how often this 101 program would be offered.
Monthly? Okay.

MS. MILLER: Yes, we\'ve been doing this monthly and then we were
thinking of repeating it as we got another birth of new employees coming
through.

DR. WHITMORE: With new starts, they need to start at the beginning
again, and so I would think periodically you would want to have this
repeated, the whole program. And in terms of within our unit at least,
we have an orientation thing that we require of all new employees.

You might want to have a core set of things that everybody \-- what you
feel like everybody needs to know these things in order to be an
effective employee and maybe have the first course of it as a required
thing that every new employee has to have as basic information that
everybody needs in order to be a successful employee. And then maybe
offer that more frequently depending on how people are coming in.

DR. COWING: I differ slightly with the frequency part, but I agree with
the management part. We had a similar issue arise subsequent to some
large turnovers and after a long analysis and debate about what to do,
basically regarding four times a year, some wishing for more frequency
and then we went monthly.

But what you want to do is you want to have a big enough group so that
they can interact, covering the basics and getting them to understand
the whole organization and try to accomplish the part of the thing that
the directors in a corporation are supposed to do, which is to try to
entice the new employees to want to work in their units.

So we are encouraged to sell ourselves to the new employees. You might
try that same tactic with your own folks. Sell them on the work that
they\'re doing and get them interested in the value of it.

CHAIRPERSON CRAWFORD: And I would just add one thing to that, I think
too giving them a contact that they can get in touch with when part of
this really becomes their life. Because at least for me, when I started
in anything one-on-one, I didn\'t have any experience and learned all
this stuff, and some of it pertained to me and some of it didn\'t. And
the more you get out there and you have these experiences then there are
certain particular things that really come home to you and you might
have more questions.

So maybe they could, if they find themselves needing to deal with the
media, maybe they can feel comfortable going to Dave or to Doug and say,
okay, I know I\'m about to have this interview, what advice would you
give me? Or data confidentiality is becoming a real big issue in my
work, who should I talk to about these things?

So this 101 is more of an ongoing thing than it is \-- you know, you
graduate and then your knowledge about all these things \--

DR. BERNSTEIN: Another thing, we\'ve always had the administrative
staff, not just research staff, secretaries, you folks, facilities guys,
anybody new in the organization had to go. And it actually, it\'s added
a lot to the overall effect.

MS. MILLER: Our seminars have just been an hour, so it sounds like yours
were \--

DR. BERNSTEIN: A day and a half.

MS. MILLER: A day and a half.

DR. BERNSTEIN: With a social event in the first day.

MS. MILLER: Oh social events. But they have appeal.

CHAIRPERSON CRAWFORD: Is there anybody here who had taken EIA 101 who
wants to comment? How informative was it? You don\'t have to, just
thought I\'d ask.

MS. WAUGHA: I wasn\'t at the first one on the history of EIA, but I was
at the others, and found the information about sound bites and how
information is utilized by the media was very interesting. Even though I
don\'t expect to be doing anything related to media. And also found it
interesting to find out about the Analysis Review Board because that
actually is something that I feel is important for me to know and which
I really haven\'t been involved with.

The confidentiality issue was something I was familiar with probably
from day one because I do work for SMG; however, I found that the other
new employees, that is something that they were less knowledgeable about
and having some basic understanding that that was useful for them.

I actually have another suggestion too, which is, I think Big
Brother-Big Sister programs and often when you go off to college you
have somebody your Freshman year, at least I did for the other that.
That having somebody who you can go and say, I don\'t know how to do
this, who\'s not your supervisor and maybe not even by a coworker on
team that you\'re working with, but somebody that you feel is safe to go
and say, this doesn\'t make sense to me and I don\'t know what I should
do about it, would be a useful mentoring for new employees.

DR. MOSS: We actually have mentoring like that in our firm. Especially
for the first year or two when people are getting their sea legs kind
of. It is useful. Once they specialized and their careers take different
paths, then the senior people or mentors seem to be more connected to
what they\'re doing, but not in the first year or so.

MS. MILLER: So when they first come in, they\'re assigned mentors, is
that how?

DR. MOSS: Yeah.

MS. BROWN: I just wanted to make a comment on the program. I just wanted
to let you all know that when I started here ten years ago, I wished
that there was something like this in place. A lot of things that you
all are putting on of course I\'ve learned by now, but I thought that it
would have been great if there was something like this when I started. I
think it\'s just fantastic.

One suggestion that I was going to make, was when you do an orientation
kit, now that you know the course is the core classes that a new
employee would need to be taking, include that as a fact sheet in the
kit and then let them know ahead of time, okay in January, you\'re going
to be taking this course, and in February you\'re going to take this
class.

And these things are in place to help you learn more about this
organization. You might want to even include some information about the
ASA meetings. I told Bill that when I first attended an ASA meeting I
said, this is the best thing in the world for a new employee to learn
more about what\'s going on in this organization.

And I know it\'s not sold that way, but when you start hearing about all
the projects that are going on in each office, hear about the comments
from everyone on the Committee, it\'s just a wonderful opportunity for
someone new to our organization to start hearing about all the different
programs that are going on.

When you start working here, you\'re so busy working on your little area
that you don\'t think about what\'s going on in this office or another
office. So that and press conferences is another area that I think is
just great. So for the new employees to start seeing what exactly we do
for the communication, what information we\'re putting out in the media,
and then they would understand the importance of a lot of the classes.

MR. BLAIR: It is important to recognize that while these formal kinds of
things that are being discussed are important, and sound really, but a
lot of learning about an organization and about your job is really kind
of an informal process that happens through personal interactions and
through various conversations, through a lot of unstructured sorts of
things. And that to the extent that that can be kept in mind and things
can be done to encourage that, like brown bags or other kinds of things,
even the physical location of new employes relative to other people. But
you can encourage that kind of informal interaction where a lot of
learning, really sort of unconsciously is taking place and a lot of the
orientation I think happens through those informal channels and should
be encouraged.

MS. BLESSING: I\'m going to talk about the entrance and abuse that we
have with new employees. We all read and we hear that the government is
having a hard time hiring people and that all the good people are going
someplace else. When we started this project before the stock market
went south back in the fall \-- you know, you can read Time magazine and
it says all the good college kids are going for the \$250,000 job,
getting their BMW, waiting for the IPO, and they\'re going to retire at
30.

That\'s not the future if you start at EIA. So we were sort of \-- but
we noticed that all of a sudden after five years of not hiring anyone,
we were getting new employees coming in. And we thought two things.

One, why did they decide to come work here. And two, if we could find
out why they decided to come work here, we could use that information as
marketing info to try to lure other people with those same kinds of
positive things that people told us.

So I came up with the idea that we\'d just sit down with some new
employees and Kathleen Cavanaugh with the Office of Oil and Gas and I
interviewed 11 new employees over a couple months in the winter.

Basically, it was very unstructured. We had some questions, a script
that we followed, but it wasn\'t anything where they wrote anything
down. Most of the interviews were between a half an hour and an hour
depending upon how long they wanted to talk to us. And we wanted to know
why they took a government job, why they picked us specifically, and
then how did that application process go. Because you know you\'ve heard
about, you apply for the government job and three years later they call
you and say you\'ve got it and you\'ve already gone through two other
jobs.

So first we wanted to know how people were finding us to begin with, and
it turns out, this is all in your package, I\'m just going to review
this. But out of the 11 people, seven people went to the Office of
Personnel Management web site, the OPM web site. Two of them came to our
web site. And a couple of people had already been contractors so they
knew us and they were very familiar with what we did.

We asked people what they looked at when they were looking at the OPM
web site basically, or any of the web sites, and they were thinking
about job location and experience level. One thing that we found out
from several of the new employees, especially the younger ones who
hadn\'t worked for the government or worked for anybody before is that
something called \-- I don\'t even know if everybody in the room knows
what a KSA is \-- it\'s Knowledge, Skills, and Abilities.

And there\'s a list of these knowledges, skills, and abilities, and you
have to write something about those specific things that are tied with
your job to apply. And some of the newer employees didn\'t know what, is
it a sentence, is it a paragraph, is it a page. What am I supposed to
do?

And a lot of times, it\'s that the instructions are very clear. So I\'m
not sure we\'re going to be able to straighten that out specifically
because that\'s sort of an OPM thing, it\'s a hiring thing, but we do
know that that can be daunting.

What factors led you to take a job with EIA? We were really enthused
when we heard some of these answers. I think almost the number one
answer was, we do interesting work. And sometimes when we\'re inside the
organization, you\'re kind of caught up in the daily grind, you might
forget that. But from the outside, and actually I think a lot of people
from the inside really do realize that we work on interesting projects.

A number of people knew that we had a good reputation. They had either
been to our web site or read things that we had written. These are some
of the people that maybe had worked in the energy field, but not with
us. They knew that we did quality products and they wanted to work in
that environment.

A number of people said that they wanted an evolving position and I
don\'t know how much in the interview people were told, but John just
said basically have of EIA\'s out the door in five years, so there\'s an
incredible opportunity for people to not only evolve, but move up and
take more responsibility.

A number of people said that they took the job because it was made clear
to them that there training opportunities, and EIA places a high value
on training. And that was something that was very attractive to new
employees and we need to stress that when we\'re interviewing people.

And what factors led you to take a government job? I mean, you read that
people aren\'t taking government jobs anymore. And I was amazed to find
that the same factors that led me to take a government job, and I talked
to my husband last night, that led him to take a government job, were
the same things people are still saying 25 or 30 years later, which sort
of surprised me.

A number of people, almost everybody, talked about stability and
security, and especially now that the stock market has gone south,
government jobs are looking a little more attractive. Now in the paper
you\'re reading about layoffs every day and the security of a government
job is looking better.

A number of people said the good benefits, the health benefits, the
holidays. Although it was interesting, I\'ll skip ahead, when we asked
people, well what are these good benefits, a number of them weren\'t
really clear what they were, but they knew that there were good
benefits.

And so that is a perception, that if you work for the government, you
have good benefits. And I think it\'s true, and so I think we need to
keep reinforcing that when we interview people.

A number of people once they got here realized that we have the
alternate work schedule program which is work nine days and you work \--
you work 80 hours in a two-week period, but you get the 10th day off.
EIA also has a very aggressive telecommuting program where a lot of our
people now have the opportunity to work at home a number of days a week.

Most of the new employees didn\'t know about either of those programs
when they came, but now they say that\'s even more of a benefit to
working here.

A number of people said competitive salary and benefits package, which I
thought was interesting because I think you always here government
employees don\'t get paid as much, you can make more money in private
industry, but the package looked attractive.

A couple people, and this was kind of a surprise to us, were either
fairly new to this country or weren\'t born in this country, and they
said they wanted to make a contribution to the United States. They
wanted to pay for the opportunity to live in this great country, and it
was a public service.

Now they also said we had interesting work to do and it\'s not like
you\'re doing a crummy job to pay back. They took a great job, but one
of their motivations was public service. And like I said before, tuition
support was a very positive thing.

One thing when I was talking to husband last night, one person
mentioned, and this was somebody right out of college said, I wanted to
work at a place where profit wasn\'t the bottom line. I feel like you
make your decision about what you\'re going to do based on what the
issues are and what\'s the interesting work and not necessarily what\'s
going to make you the most money. I thought that was pretty interesting
coming from probably a 22 year old.

We asked them was the job announcement posted long enough. And there
wasn\'t really a consensus. Some people said it was posted way too long
and a couple of people said it wasn\'t posted long enough.

We got a couple of suggestions. One person said, the job that I applied
for had been posted for a year, it just kept being posted. What happens
is, if you have six openings for that kind of generic job, you post it
once and maybe hire a couple people, and then you post it again and you
hire a couple people, and you post it again.

And to a person who is randomly going through the OPM \-- this person
said, I thought you must be desperate at this point because it\'s been
out there forever. And in fact, we had been bringing in people all
along, but it was the same description for an area that really needed to
hire a lot of people. So we just have to be aware of that, that people
might think that it\'s the same job that\'s been open.

One person also commented to us, and I don\'t know if people think about
this, but the job opening closed, all your material needed to be turned
in January 3rd, and he said, that just wasn\'t a good time, who was
thinking about that deadline.

Although I think what we were thinking about was the new administration
coming in there was going to possibly be a hiring freeze, which turned
out to be true, and so I know there were a number of hires in the
January time frame were expedited so that we could get people in before
we had to wait for awhile.

What part of the recruitment process went well? Well for about the first
seven people we interviewed, basically the answer was nothing. They
waited forever. There was one person that said they almost went to
graduate school because they \-- you know, you apply and then you don\'t
hear anything. One person almost went to graduate school because they
didn\'t hear anything. One person was up a week from taking another job
before they heard something.

I mean that kind of thing is just not \-- that\'s that not EIA, that\'s
all personnel, which we need to partner with to make them do a better
job on that. No confirmation of receipt of the application. A couple
said, I tried to call personnel. They can\'t call use directly, they\'d
call personnel and no one would return their calls. Well, it\'s very
frustrating, especially in a tight job market where you may have other
offers, we almost lost people. Oh, we got statements like the
application process was torture.

Then when we tried to sort of expedite the hiring process, the last
three or four or five people that we interviewed, they had much
different impressions of the hiring process. People said it was
efficient, it was impressive, nothing was painful.

One person said, they actually rushed me in. He thought he was applying
for a government job and it would take forever, so he thought he had
three or four or five months to clean up stuff on his desk and the
process went so fast he didn\'t have time to finish what he was doing,
so he was very surprised that the government could pick him up that
quickly.

One person who came from private industry said it was the fastest
federal hiring he\'d ever seen and he was stunned. So we have made some
progress and I think some of that might have been spurred by the fact of
the change in the administration and some of the manager sale that was
coming up. But we do have the capacity to pick people up in a fairly
reasonable about of time, which I think is good.

I think the most \-- I don\'t know, I think it would be counterintuitive
result that we got is, we asked people, did you apply on line? Did you
use the opportunity to apply with email? And you know, everybody thinks
that sounds like a cool idea, doesn\'t it? You know, no paper and resume
and all and you just zap your thing in.

Nine out of the 11 people applied did not apply email electronically.
They applied in person, fax, or by snail mail. And all of them, to a
person, were aware that you could apply electronically, and they
explicitly chose not to apply that way.

And we said, why is that? People like the idea of having their package
and putting \-- you know, you have your old resume and you put in your
\-- and you give it to mailman. They wanted someone \-- one person said,
I run it in, I got a receipt from personnel. I wanted to know that they
had the package.

A number of people said, well fine, I can apply on line, but they you
want my transcripts. I can\'t \-- how does that go in there? Unless you
have a scanner, one person had a scanner at home, but the rest of the
people didn\'t.

A number of people who were applying for higher level jobs had done
professional papers and they needed to submit those. How do the
professional papers go along with that electronic application unless
you\'ve got a scanner.

And would you all trust that if you applied electronically and then you
sent your papers and your transcript later, that those would ever get
hooked together? I wouldn\'t be confident that would happen. And so I
think obviously nine out of the 11 weren\'t either.

A couple of people said that they were concerned actually about the
format that maybe their resume or the application wouldn\'t print out
in. If you print yours out on a really pretty peach-colored paper and
it\'s all formatted in a certain package, and then you print it out and
it wraps and it\'s on recycled, that icky paper, it\'s just not the
same.

So they wanted to present the package. It was very important to them for
the package to look good. And one person actually said that they didn\'t
have access to a PC.

Now the two people that applied on line, they were very confident that
it would work, and it worked fine because obviously \-- well I don\'t
know, maybe 50 applied on line and we only got two of them. We got two
of them. They faxed the transcripts, everything got hooked up, and they
were very confident about it. But it wasn\'t that a number of people
said, oh, I didn\'t know I could apply on line, gee that would have been
a good idea, they said no, no, no, didn\'t want to do it.

So basically from these interviews, we figured out the main thing that
we\'re going to do from this interview is we\'re going to create a
brochure, and I don\'t know if you guys can see it, but it\'s got the
fireworks and the cherry blossoms and the White House. This is an older
version, this is the benefits of working at EIA, but the brochure\'s
going to say, come work with us, EIA, and it\'s got all the benefits
both of working for the government, working or EIA, all these things
that the new employees told us. It\'s at the printer right now, so
we\'re going to be using this to recruit.

And then we also just need to talk to personnel about some of these
issues with hiring and remind people when they are recruiting to mention
some of things like training and alternate work schedule that the new
employees were valuable.

CHAIRPERSON CRAWFORD: Thank you. Our ASA discussant is Tom Cowing.

DR. COWING: Okay, let me start off by telling you that this may be a
mistake because I\'m supposed to comment on the paper, and I\'m not
going to.

What I am going to do rather instead, is to give you hopefully a
different experience and a different prospective perhaps based on my own
experience. And then maybe we can in the discussion bring everything
together. So I hope that what I have to say relates in part \-- well I
mean, I know it relates in part \-- to the report, but I\'m not going to
comment specifically on the report.

My background, briefly, is that I come from an academic department in a
university in upstate New York, and about three years ago \-- I mean, we
knew this was coming, but I\'ll start the story three years ago \-- we
looked around and what we saw was somewhat appalling.

First of all, this is a department in the 18- to 20-person range, had
been as low as 16, had been as high as 23, but reasonably speaking, in
the last ten years, 18 to 20. And what was appalling when we looked
around was that it wasn\'t 50 percent, it was like 75 to 80 percent of
us who were over 55.

So that clearly there was going to be a lot of retirement coming in the
next five to ten years and a lot of it in the next five years.

Secondly, there wasn\'t a lot of diversity. Not only the usual kind of
diversity, but also diversity of opinion, diversity of background.

And thirdly, quite frankly, our productivity had hit bottom. We were
just, in fact, a bunch of old guys waiting around to retire. Somewhat
overstated.

(laughter)

Okay, now every talk needs a title, and with apologies, and this will
become clear in a bit, with apologies to old dogs and cat lovers, let me
entitle this talk, which is in part what the paper was about, the
report, teaching new dogs old tricks. And when I say I apologize to old
dogs I fully realize I\'m an old dog.

However, that\'s not the point that I want to make, or at least the
primary focus I think shouldn\'t be on teaching new dogs old tricks
because those new dogs have their own tricks. And those tricks may well
be better tricks. And so what I want to focus on is rather learning new
tricks from new dogs. That\'s what I want to focus on.

And so the first thing I think, the first point to be made, and I\'ll
just make a couple points I think and I\'ll do this very quickly, is
that if you really think about it in terms of my department\'s
experience, and I think in terms of EIA\'s experience, there is a
tremendous opportunity here to restructure the organization, to reinvent
the organization, reenergize, refocus.

It isn\'t often that you can replace 50 percent of the organization in
five years, or 80 percent of the organization in ten years. And so I
think the very first thing you want to realize is that there is a
tremendous potential here in terms of reinventing, reenergizing,
restructuring the organization.

And it doesn\'t come along very often. This is once in 50, maybe once in
a 100 years type of event. So if you want to take advantage of it, you
have to first realize this potential and then do something about it.

In our case, what we did was \-- one of the other characteristics of our
department, was not only weren\'t we very productive, but we were
essentially trying to do a whole lot of things in terms of economic
fields and we probably weren\'t doing any of them very well.

And so we decided that the new organization ought to be much more
focused. And in fact, our game plan was that we were only going to do
three things, three basic fields in economics. If you wanted to do other
things, urban, maybe environment, the question of the three fields is
still a little bit open. Two of them have been decided on, and we\'ve
moved in that direction. A third one is yet to come.

But if the three that we picked weren\'t ones that you were interested
in as a graduate student, then perhaps you should go somewhere else.
Because we weren\'t trying to be all things to all people. We were
trying to do a very small number of things, but we were trying to each
thing very well. And so that was the focus. And it was this opportunity
to refocus and restructure the organization.

Now clearly, if you\'re going to do this, you need some kind of a
strategic game plan. You need a strategic game plan not only for your
own purposes, something that gives some sense of the focus, what you\'re
going to try to achieve. And perhaps something about how you\'re going
to go about doing it, the implementation so that you can eventually
reach your goals.

So it\'s both internally and probably it\'s something that management
should pull out every day and remind themselves, this is where we\'re
headed and it\'s going to take two, three, four years to get there. And
at every staff meeting maybe take five minutes to remind everybody,
here\'s where we said we\'re going and by golly, we\'re going, we\'re
going to get there.

Because every now and then things come along. People want to do slightly
different things and it\'s easy to lose sight of where you\'re trying to
get to.

Perhaps more importantly, of course this is game plan, the master plan,
the blueprint that you can sell to top level management, and that\'s
something you ought to do very quickly for a variety of reasons, some
rather obvious. But the not so obvious ones is every now and then top
management, a year or two down the pike, may start to renege a little
bit, or perhaps something that might happen more frequently, is second
level management might want to try to change things a little bit.

And so what you do is, and this happened in our case, where the Dean
wanted to change things, but we had sold this to the Provost and the
Dean reports to the Provost. So every time the Dean started to get a
little out of hand, we would march in with our master plan and we would
remind the Dean that the Provost was on board saying, this is what we
were going to do, and that was very effective. So it keeps you focused
and it keeps them focused.

So some kind of obvious ones, the implementation to plan has got to be
consistent with the goals. You\'ve got to constantly be moving towards
those goals and not kind of wasting, spinning wheels off in some other
direction.

Here\'s one that I think is very important. This is back to learning new
tricks from new dogs, which is to say \-- the presumption of a lot of
this usually is that us old dogs who have been around a long time doing
this stuff, we know how to do it.

Well, I\'m not so sure quite frankly. Sometimes the new dogs can do it
better than we did, or than we are currently. For example, in our
department, we started hiring a lot of junior people. Actually what we
did in each of these three areas was hire a senior person first, and
then started to fill in with junior people.

Our thinking was that there was an additional incentive to a senior
person to come on board if they knew they could reshape their group, a
group being in a field of expertise, macroecomonics, or econometrics, or
urban economics, energy economics, whatever.

So we brought in a lot of new people, a lot of junior people in
addition, and one of the very first things we did was, we let them teach
the theory courses. Now why would you do that?

Well, I had been teaching graduate theory, and quite frankly, I learned
my graduate theory 35 years ago. Now I assume, things do move slowly,
but I assume that there has been some change, some new stuff that\'s
come on board and while I\'ve tried to keep up with some of it, I\'m
sure I don\'t keep up with all of it.

So I assume that the new people coming out of graduate school are better
trained than I am in their field, certainly. And so we immediately put
the new dogs to work handing out the new tricks. And I think it worked
very well.

Now there\'s bound to be a little difference between an academic
organization and EIA, so I\'m not saying you ought to do exactly what we
did, I\'m just saying you ought to think about it. Maybe it\'s some of
the things over here that could relate to EIA and hopefully in the
discussion we\'ll kind of try to bring some of those things along.

Measure by results, that\'s an old thing that I keep harping on. What\'s
important are outputs, not inputs. It doesn\'t matter how you do
whatever you\'re doing, what matters is the results. And there are
different ways of doing things, so don\'t look at how somebody did
something, look at what they did, what the results were.

So finally, again with apologies to us old dogs, I think the bottom line
here is, that what we really should be focusing on are the new dogs and
the new tricks, not the old dog and the old tricks.

MR. WEINER: My only comment is that I think that\'s right on track. A
lot of what we\'ve been doing is thinking about, how would I say it,
sort of the back end of the process, and I think the front end of the
process, what we\'re really going to be faced with five or ten years
down the road is something that we ought to be focusing on.

We have a very stable organization structurally, and so some of the
things that you suggest are not trivial, even with a lot of turnover,
but I think it\'s got to happen and I think our mindset\'s got to be
moving in that direction. So I thank you for those comments, they\'re
right on the money.

CHAIRPERSON CRAWFORD: Are there any comments or questions from the
Committee. Randy.

DR. SITTER: I wanted to comment to the hiring discussion. You said that
the people were quite pleased after you expedited the process and you
need to expedite the process. My experience with government hiring is
that it\'s horrible.

And the reason it\'s horrible is primarily because of the disconnect
between hiring and who\'s hiring you.

My efforts at recruitment run around by faculty and graduate students.
I\'m at a university. But the number one thing is talk to them. You\'ve
got to be able to talk to people. You\'ll win out on a competitor
that\'s way better than you if you just are able to speak to them.

And I guess one of the points that I wanted to emphasis is that, you
were able to expedite the process then, can you truly expedite the
process on an ongoing basis?

I don\'t know that that\'s possible within the government the way that
the Canadian government works. I think it\'s impossible. I think that
you deal with an organization there that is absolutely completely
disconnected from the people you\'re applying to and therefore they\'re
a shotgun approach, they\'re impersonal. They do not contact you, the
process is slow, and they treat you like they\'re doing you a favor.
That\'s not the way you get good people. And I think that\'s the big
reason that the governments have trouble attracting people.

It\'s not because you don\'t have a good job, it is an excellent job.

MR. WEINER: I think that\'s changing. We only see it from our rather
narrow perspective here in one smallish agency. But I think possibly for
the reasons that are just not even resources to fill personnel offices
with staff, and that\'s kind of a strange result as there were before.
Much of the authority is beginning to devolve to the operating agencies,
like the EIA. We\'re taking on a lot more in the personnel area in terms
of dealing with our own personnel actions and also talking directly with
applicants and potential hires.

We\'re trying to move the personnel offices in the direction of being a
support to us rather than simply a hurdle we\'ve got to jump over and I
think that\'s working. I think one of the things that we\'re seeing by a
rather impressive group of new employees, as Colleen and Kathy were
interviewing, is that we\'re really beginning to get some direct
communication. Our program of officials are doing that.

And it\'s sort of because that the Red Sea has parted as it were, and
we\'re able to get directly to the other side without going through so
much of a problem.

MS. BLESSING: I just wanted to add one more thing that I forgot to say
at the end of my talk, is that Kathy and I, after we talked to these 11
people, were so heartened about the future of EIA. Because after every
single interview, we looked at each other and said, oh my gosh, we\'ve
got another good person. All 11, to a person, were interesting and
trained and excited and enthusiastic to work here.

And we were just so pleased to see that one after the other after the
other, even though this hiring process can be daunting, and even though
people say people aren\'t coming to work for the government, we got some
really good people. So we were excited about that.

MR. WEINER: You\'re absolutely right in identifying what the problem has
been. I mean, I wasn\'t trying to disagree with you. That\'s exactly
what the problem is.

CHAIRPERSON CRAWFORD: And you are right about the problem, but the
solution that he has is the same one that CDC has taken too and it
really becomes the responsibility of senior management to actually take
the initiative to work with the people in personnel and to have
expectations of them and to actually say, well, have you contacted this
person or have you processed their application, or I really need to have
this person. And just a faster spirit of working together rather than
this adversarial relationship that had developed for so long that
basically just broke down.

DR. SITTER: One semi-related comment, and that is, I got into graduate
admissions because I wanted to recruit graduate students. And the first
thing I realized talking to them is, my web page, and all of our web
pages, are geared for one thing and one thing only and that\'s to
attract graduate students.

Look at your web page and ask yourself, does it encourage people to
apply for jobs here? That\'s where they\'re going. These days, that\'s
where they go.

MR. PETTIS: Steve may want to comment back here, but we were \-- I think
you could say that our personnel system was broken, and probably still
is, and the Department of Energy, but we were a pilot program here where
the personnel office really gave us authority to do the direct work with
the people we were trying to hire and I think we have some pretty good
facts and figures that we expedited the process considerably with that.

And it\'s just, you have ownership for it. You know, you\'re taking an
interest in the people you\'re trying to hire and want to get the
process over with, and it\'s work.

CHAIRPERSON CRAWFORD: Comment? Jim.

DR. HAMMITT: Without going that far, I don\'t understand what rules you
operate under, but I\'m aware of a lot of cases of people applying to
federal jobs and having a lot of direct contact with the people for whom
they would be working, rather than directly through the personnel
system. So I don\'t know what restrictions there are in your ability to
do that.

It seems like Randy\'s main point is that the applicant knows that
somebody in the EIA actually wants them and is sort of keeping track of
the process, keeping in contact with the applicant, that\'s maybe 80
percent of what you want to achieve.

DR. BERNSTEIN: Following along with that is a two-question way to be
sure that you should do that. One is, what are you going to do with it?
Where are you going to send it?

And secondly, and it may be too late if it\'s already in production, but
you might, and maybe it\'s on there, but you might want to describe both
the formal process of sending an application, but also the encouragement
to copy EIA on the application. If you\'re applying the EIA, let us know
you\'ve applied to us.

And encourage that dialog there. Some people may not know and so they
send in the application. You never know that they\'ve sent it in and you
never know if you had a good application. If they copy you on it, if
they\'re allowed, I assume they are, then you at least know what has
gone into the app.

MS. BLESSING: The brochure does not ask any kind of application
information. It\'s more the great benefits and the opportunity to learn
and you live in Washington, D.C., and you work with smart people and you
get transit subsidies. It\'s all more of the benefit package. It does
say to go to the OPM web site or the EIA web site, but it doesn\'t offer
that.

To answer your first question, I think we\'re going to discover more
uses for the brochure, but we do have a series of recruitment, sort of
ventures. People going out to colleges and conferences recruiting. And
we thought that this would be a good take away that people could look
at. On the back it\'s got the web site in big letters so they could go
to the web site and take a look and see what we\'re all about.

And also, this is a mock-up brochure, but we left the back completely
blank so that we could mail it to people. So I think we\'re going to \--
well, what did we ask for, or how many did we ask for on this for the
first print run? Twenty-five hundred, so we\'ll start with that and
we\'ll see. But I think as people go to these recruitment things and
they have them out, people walk away with them.

So I think there are probably a lot of opportunities, but actually the
impetus to hurry up and get this done was that several people in oil and
gas were going out on a recruitment and they said, is it done yet? Is it
done yet? We want the brochure. So there was a demand from the internal
customers to actually have this. And I think we\'ll think of more ideas,
but it was just to have on the recruitment tables because we are \--
Steve\'s sending out a lot of people in the spring and I know we will be
doing it over the next couple of years.

So we needed something like this and I think this is a living document
and we can certainly change it and maybe that\'s a good idea to add
something about the application process. But we\'ll learn as we go.

DR. COWING: To follow-up on Randy and Jim\'s comment, using graduate
student recruitment as perhaps a model here. The old way of recruiting
graduate students, think of them as new employees in a sense, is you
send out offer letters and then you just waited for three or four
months. Because after all, whether they came to your university or not
was their problem, not your problem.

Well we don\'t do it that way anymore. In fact, recently we sent out our
offer letters and then every two or three days somebody from our
department contacted all of our top prospects. And it pays off. We did
this for two or three weeks.

We were a little concerned that we were over doing it. But it was
different people, and so it wasn\'t just the same. It was somebody in
their field, it was me in charge of the graduate program, it was the
chair of the department, so we mixed it up a little bit.

And it looks like it paid off because we picked up four of our top four
U.S. prospects, absolutely unheard of. And I think now that\'s the kind
of thing you have to start thinking about doing.

DR. SITTER: We\'ve been doing something similar for the last five years.
I would say that if you were to ask anybody in Canada what\'s the
biggest and best improved group in statistics in the country, it would
be ours. And I think that that comes from recruitment, both faculty and
graduate students. And we do exactly the same thing.

We stick to one person. We identify the person that would be best, to
contact that person, and we stick to that person. We like them to be in
touch with us to set up a dialog. We don\'t try to oversell ourselves as
being better than everybody else, but to be more informative, help then
with the process.

You\'d be surprised how many people out are a little bit afraid. Afraid
of the job, afraid of the future. And a warm welcome and somebody to
just tell them no, that\'s an excellent job too, that\'s an excellent
organization and I would apply there. That can be a very good
recruitment strategy. It works tremendously well for us. We\'ve been
very successful in it with 50 to 60 percent success rates on our top
applicants.

The other thing is, when they have another offer, they tend to tell you.
You have a chance. You hear about it. You can respond. But it\'s not in
this sort of mercenary way. Here, somebody made me an offer, can you
match it? It\'s more of a dialog and it works very well.

CHAIRPERSON CRAWFORD: Does anyone else have any questions or comments?
Nicolas, I\'m sorry.

DR. HENGARTNER: Since you\'re trying to hire very specialized people, it
would make sense if you happened to make some internships. You know,
sell yourself to potential people that might in the long run come back.
You know, just put out some feelers.

I think that this is often done in organizations, to have them here for
two or three months. Because, don\'t forget, you are going after very
specific qualified people that might have other interests or offers. And
so I think that that is something to maybe consider the low-cost way of
attracting your top qualified people. Since once they\'re here and
working, they\'ll like it.

MS. KIRKENDALL: We are participating in a joint program through the
University of Maryland, University of Michigan, and Westat. And they
have initiated an internship program that all the statistical agencies
are participating in.

We have been doing it for three years, I guess this is our third year,
and we have two students who come in the summer. These are students who
apply to JPSM and they\'re screened. They get really good quality
students who come. We haven\'t had any of them come back full time, but
it\'s still a good program. So I think internships are a good idea too.
And actually we\'ve sort of tied on to JPSM on that one, but it would be
interesting to do a little bit more with internships.

CHAIRPERSON CRAWFORD: Thank you. Thank you all very much. It was a very
interesting topic. Speaking of old dogs, it was inadvertently brought to
my attention last night at dinner that with respect to the ASA
Committee, I am an old dog. And one of the things I\'ve noticed in the
almost five years that I\'ve been on the Committee is the continuous
support that we\'ve received from Larry Pettis.

He has been at every meeting I\'ve been at in the five years I\'ve been
here, and he\'s probably been at every meeting before I came here as
well. And I wish Calvin Kent were here because he could give a much
better speech than I could. But several times I would ask Calvin things
about EIA or the Committee and Calvin would say, oh you know, I don\'t
know, I have to ask Pettis. And if there was something Calvin didn\'t
know, that\'s exactly where he would go.

And so, as most of you know, Larry is going to retire. This is probably
his last meeting with the Committee, and so on behalf of the Committee I
just wanted to say how much we appreciated you, to thank you for your
support and your cooperation over the years that you\'ve helped us.

And to give you best wishes on your future on the golf course or in
front of the fire in the evenings in your house, or wherever your future
may lead you. So thank you.

I apologize to those people that won that graphics contest. We\'re
running a little bit late and you\'ve had to sit on the edge of your
seat, but I would like to proceed with that. And at this time I would
like to ask Larry Pettis, EIA\'s acting administrator, and Nancy
Kirkendall, the acting director of the Statistics and Methods Group to
join me in presenting the winners.

I\'ll let Larry tell you how the process has evolved. I\'ll announce the
winners and Nancy and Larry will describe their graphics and present the
certificates.

MR. PETTIS: Thanks Carol. By the way, for those of you here as you might
know, I had a chance last night at dinner to say what I thought this
Committee had meant to EIA and so, just once again I would like to say I
appreciate your efforts and I know this will be a continuing
relationship in the future.

One of the traditions of this Committee has become now an annual
graphics contest and this was a way for us to promote good graphics in
the EIA. The 2001 contest, graphics could be submitted for a 15-month
period, covering from January 2000 through March of this year.

There were 20 entrants from six EIA offices. We had entrants from all of
our program offices, the Office of Oil and Gas, the Office of Coal,
Nuclear, Electric, and Alternate Fuels, Energy Markets and End-Use, and
the Office of Integrated Analysis and Forecasting.

We also had submissions from the Office of Resource Management and our
National Energy Information Center.

I think you all probably know the processes. There were two rounds of
judging in this. The first was a peer review process within EIA and we
had seven judges, some of who are here, Paul Staller, Mary Ellen Gallby,
Cynthia McLaughlin, Becky McNerney, Mark Rodekohr, Preston McDowny, and
Cue Dufain. And there were also technical advisors to this, Bob
Rutchick, Lawrence Stroud, and Ann Whitfield.

And I know that last night over dinner the Committee pondered long and
hard in making their decisions on these graphics and they may have some
raspberry dessert on them, as a matter of fact.

So now I\'m going to turn it over to Carol and Nancy to announce the
winners

CHAIRPERSON CRAWFORD: Our first winner is Phyllis Martin from ORIF for
the graphic Major New U.S. Natural Gas Pipeline Systems, 1990 to 2000.

MS. KIRKENDALL: The purpose of the graphic was the illustrate the
significant amount of U.S. interstate pipeline capacity that\'s been
added within the past ten years and show the location of the added
capacity.

CHAIRPERSON CRAWFORD: The second winner is Nancy Masterson from OOG, for
the graphic A Primer on Gasoline Prices 2001.

MS. KIRKENDALL: This is an award for a brochure, a primer on gasoline
prices and the brochure explains in layman\'s terms the gasoline market
and answers such questions as, what are the components of the retail
price of gasoline, why do gasoline prices fluctuate, why do gasoline
prices differ according to region.

MS. WHITFIELD: Could I just say something. Yesterday I had a really neat
conversation with the representative from TOSCA. They\'re in Tempe,
Arizona. And what they\'re doing with that brochure is, they\'re
distributing that brochure to all of their marketing people and they are
printing it at their cost, 1 million copies. So I was very happy.

MR. WEINIG: I might add too that Ann Whitfield, who just spoke, is the
graphics person behind the graphics brochure. So she did the graphic. So
every time you see a brochure put out by EIA, it\'s because Ann did the
graphics part of it.

CHAIRPERSON CRAWFORD: The third winner, the co-winners I guess, are Dave
Hinton and Nancy Masterson of OOG for their graphic, their brochure
actually, Propane Prices, What Consumers Should Know.

MS. KIRKENDALL: Okay, this is another brochure. It\'s a consumer
brochure on propane prices that explains in layman\'s terms factors that
influence prices of propane and discusses how the use of propane varies
according to customer, season, and region. The brochures were a hot item
in the graphics contest this year.

CHAIRPERSON CRAWFORD: And the last selection on the floor is Dan Woomer
from ORM for his graphic Percentage of Change in FY-1995 to FY-2001,
Budgets of U.S. Statistical Agencies.

MS. KIRKENDALL: This graphic was developed and used to show that EIA is
the only federal statistical agency that has suffered budget reductions
since 1995, while showing that EIA is one of the smaller federal
statistical agency budgets.

CHAIRPERSON CRAWFORD: In general, we had a very nice discussion and we
found most of the graphs to be incredibly well done and we had fund
doing it. Thanks and congratulations to all those that submitted and
particularly those the won.

I guess now were scheduled for a break, and what time are we supposed to
reconvene? Maybe we\'ll reconvene about 11:10 or so.

(Whereupon, the foregoing matter went off the record at 10:52 a.m. and
went back on the record at 11:20 a.m.)

CHAIRPERSON CRAWFORD: I hate to interrupt our impromptu conversations,
but I would like to reconvene the meeting so we can actually formally
adjourn the meeting. So if you could give me your attention just for
five minutes, we can wrap this up.

All of you should have received a voting sheet for the spring meeting.
If you could turn in your availability dates to Linda Minor, that would
be great. We\'d like to try to schedule it as soon as possible. She\'s
going to try to see if she can\'t get us government rates at the
L\'Enfant Hotel, which is just diagonal, so you don\'t have to drag your
luggage up the stairs.

Does anybody have any other administrative details or comments or
questions? The only last thing on the agenda item would be for me to
again ask you to consider, and this is for both EIA people and also ASA
Committee people, presenting an invited paper at the Joint Statistical
Meetings. This is a really great opportunity that we have, particularly
for EIA, a great opportunity for you to shine statistically and show
everyone at the meetings what you can do. And for the Committee to
present their work on energy statistics at a statistical meeting too.

So, I know I\'ve talked to many of you on a one-on-one situation,
believe me, I\'ll follow up on this and you\'ll be getting emails again
nagging you for confirmation when I get back. So do please be thinking
about things that you might be able to present at this meeting and I\'ll
be in touch then unless anyone has any more suggestions now.

Well, in conclusion, let\'s take the opportunity for any questions or
comments from EIA at this point? Are there any public questions or
comments?

MR. LU: I\'d like to thank you, the Committee, especially yesterday in
giving us some feedback and the inputs on the special project for OMB,
and Roy Whitmore handed me a couple of pages of facts this morning. I
really appreciated that.

And also, another thing is, that\'s a big project and I\'d like to sent
out the PowerPoint file to the Committee members and ask you to give us
input in return. Thank you.

CHAIRPERSON CRAWFORD: Thank you. Okay, if that\'s all, I\'d like to
thank everyone, the Committee and EIA for their participation and
formally adjourn the meeting. Thank you.

(Whereupon, the above-entitled meeting was adjourned.)