UNITED STATES DEPARTMENT OF ENERGY
                  MEETING OF THE AMERICAN STATISTICAL ASSOCIATION
                   (ASA) COMMITTEE ON ENERGY STATISTICS WITH THE
                      ENERGY INFORMATION ADMINISTRATION (EIA)
                                  Washington, D.C.
                              Friday, October 29, 2004

                                                              2
           1    PARTICIPANTS:
           2       F. JAY BREIDT
           3       NICOLAS HENGARTNER
           4       JOHNNY BLAIR
           5       MARK BURTON
           6       MOSHE FEDER
           7       BARBARA FORSYTH
           8       NEHA KHANA
           9       NAGARAJ K. NEERCHAL
          10       SUSAN M. SEREIKA
          11       RANDY R. SITTER
          12       HOWARD BRADSHER-FREDRICK
          13       ROBERT RUTCHIK
          14       NANCY KIRKENDALL
          15       PRESTON McDOWNEY
          16       GUY CARUSO
          17       TOM BROENE
          18       HENRY S. BROOKS
          19       BRENDA COX
          20       GRACE SUTHERLAND
          21       SHAWNA WAUGH
          22
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                              3
           1    PARTICIPANTS (CONT'D):
           2       TOM LORENZ
           3       PHILLIP TSENG
           4       JOHN WOOD
           5       HOWARD GRUENSPECHT
           6       WILLIAM WEINIG
           7       INDUJIT KUNDRA
           8       JOE SEDRANSK
           9       KAREN NORMAN
          10
          11
          12                       *  *  *  *  *
          13
          14
          15
          16
          17
          18
          19
          20
          21
          22
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                              4
           1                      C O N T E N T S
           2    AGENDA:                                    PAGE
           3       Data Analysis on the EIA-826/906           5
           4       Post-Stratification Methodology for       50
                   the 2002 Manufacturing Energy
           5       Consumption Survey (MECS)
           6       Time Series Edits for the Electric       118
                   Power EIA-906
           7
                   If you were King                         133
           8
                   Committee Suggestions                    179
           9
          10
          11                       *  *  *  *  *
          12
          13
          14
          15
          16
          17
          18
          19
          20
          21
          22
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                              5
           1                   P R O C E E D I N  G S
           2                                             (8:37 a.m.)
           3              CHAIRMAN BREIDT:  Okay, I guess
           4    we'll go ahead and start the meeting.  First
           5    I'd like to ask any committee member, guest,
           6    EIA staff, or member of the public who was
           7    not present yesterday to introduce yourself
           8    from one of the microphones?
           9              MR. WEINIG:  If you are --
          10              CHAIRMAN BREIDT:  In that category?
          11              MR. WEINIG:  Directly under a
          12    speaker in the ceiling, it will cause the
          13    thing to wind and break, so we'll use the
          14    other microphone.
          15              SPEAKER:  Okay.
          16              MR. COLE:  My name is Stacey Cole.
          17    I'm from the Bureau of Census, I'll be
          18    speaking here today about the post
          19    stratification we did for the MEC survey.
          20              MR. WEINIG:  Anybody else?
          21              MR. SLANDA:  John Slanda, U.S.
          22    Bureau of the Census.  I'm with Stacey Cole.
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                              6
           1              MS. BUCCI:  Susan Bucci, U.S.
           2    Census Bureau and I'm with Rick.
           3              MR. WEINIG:  Anybody else?
           4              SPEAKER:  Okay.
           5              SPEAKER:  Okay.
           6              SPEAKER:  I think that's it, thank
           7    you.
           8              CHAIRMAN BREIDT:  Okay, thank you.
           9    And anyone who did not sign in yesterday is
          10    asked to do so, out front at the break or
          11    before you leave.  Lunch for the committee
          12    will be held at the conclusion of this
          13    session on suggestions for the spring agenda,
          14    lunch is downstairs as usual, and we'll
          15    probably be discussing a variety of things
          16    including future committee appointments and
          17    so on.  So with that I will turn it over to
          18    Joe Sedransk, who will be speaking on data
          19    analysis on the EIA 826 906.
          20              MR. SEDRANSK:  You're doing the
          21    pushing, or I might as well do it.  This is
          22    of course joint work, so as one typically
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                              7
           1    says when giving the talk, the other -- the
           2    co-author will answer all the questions.  In
           3    any rate, our work is an evaluation of
           4    methodology in current use, but we're not
           5    talking about things we've invented, and we
           6    need to make recommendations to CNEAF about
           7    several things including about methodology
           8    and possibly there's been talk about changing
           9    the sample.  This is just rumor.
          10              But anyway this is the --
          11    background is to estimate -- we're trying to
          12    estimate total monthly sales and revenues for
          13    all non IOU's by end user, first of all IOU
          14    is investor owned utility, and the end users
          15    are residential, commercial, industrial and
          16    other; in all these discussions just forget
          17    about other, we'll just talk about the first
          18    three.
          19              So there are two variables; sales
          20    and revenues.  Other entities such as IOU's
          21    and wholesalers are censused.  So we're
          22    really just talking about the non IOU's
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                              8
           1    essentially.  861 is an annual census, it
           2    covers everyone and it provides a frame for
           3    the 826 which is a monthly survey used to
           4    provide estimates of total sales and revenue.
           5              And to give you an idea, the sample
           6    size were about 150, 160, 170.  There are
           7    over 2000 in the population of non IOU's.
           8    Sample design is cut off sample which was
           9    developed in the early 90s, and at that time
          10    it included the IOU's which are now censused,
          11    and of course there have been additions and
          12    deletions since then.
          13              Sample coverage rates, I cite
          14    below, they depend 76 percent to 88 percent,
          15    depending on the variable and end user, 68
          16    percent to 95 percent by geographic region.
          17    The estimation procedure: very standard,
          18    linear regression through the origin with a
          19    slope different for different geographic
          20    regions.
          21              So Y is the current 826 value for
          22    company I and region R, X is the past census
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                              9
           1    value for that company in the region.  The
           2    only thing to bring to your attention is the
           3    variance, sigma squared2 proportional to X to
           4    the two gamma, and the current usage has
           5    gamma as 0.8.  That was developed in the '90s
           6    long before I had any connection with this
           7    thought to be desirable and I could -- we
           8    could discuss this afterwards.
           9              The current method is simply, add
          10    up the units, the sampled units, to get this
          11    sum for a State.  For example, add up the
          12    sample units and make predictions for the non
          13    sampled ones.  It's possible that there are
          14    some added -- there are some outliers and
          15    that might not be included in estimating beta
          16    hat for the non sampled ones but they would
          17    always be added in for the -- to get the
          18    totals, assuming that there has been a check
          19    for outliers and that the value is correct.
          20              Valuation; what we've been doing is
          21    looking at Standard Exploratory Analysis
          22    using scatter plots, standardized residual
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             10
           1    plots and so forth.  Then I want to point out
           2    the ten regions that are currently being used
           3    for estimation, that is related to beta R's,
           4    they depend on region and the R's here bring
           5    your attention to the fact that NEA is very
           6    large.
           7              These were developed, I think again
           8    in a year -- early to the mid '90s, based on
           9    temperature and climate considerations.  NEA
          10    is very large; it ranges from and -- possibly
          11    diverse from Maine all the way down to
          12    Maryland.  The other thing to bring to your
          13    attention is West, which is somewhere in the
          14    middle of California divide; there are only
          15    three sample companies in that region.
          16              So with this large number of
          17    regions, the sample sizes as you see, as we
          18    go through here and not too -- pretty small.
          19    Four regions are aggregations of the 10 that
          20    we've looked at at various times.
          21    Occasionally, we'll talk about all the data,
          22    and sometimes about all the data except for
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             11
           1    those that have been deemed outliers or
           2    influential and those are the criteria we've
           3    actually -- we've been using.
           4              That's we've been using for
           5    investigation as opposed to what is being
           6    used in the current estimation.  Here are a
           7    couple of -- about the five scatter plots on
           8    top, standardized residual plots down below.
           9    Essentially if you've looked through them on
          10    the website, there's very little that we can
          11    find and suggest that model doesn't fit.  If
          12    you look carefully enough, you'll find
          13    examples where there are things that you
          14    might be a little unhappy about.
          15              But generally the residual plots
          16    look okay.  Here's one, here's another one.
          17    I picked ones where there are decent number
          18    of points, like 14 in this case where you
          19    could see something.  Some of the others, you
          20    can't see much.  If you have only four or
          21    five points, you just can't tell anything.
          22    Here's another one with nine.  Then I put two
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             12
           1    and here's one which looks like an outlying
           2    observation in the top for industrial sales
           3    and it's minus 3 point -- larger than minus
           4    3.5 here and, here's another one, which
           5    probably -- it's probably an influential
           6    point.
           7              Just a quick comment; I apologize
           8    that, there was a gremlin that got in, there
           9    should be a second set of plots and that
          10    should've been on the website.  But a gremlin
          11    got in and the ones, the plots, which were
          12    "the after," these are "the before," this is
          13    the -- you know in another words, this is all
          14    the data, the ones where the things were
          15    removed, never got into the data set, never
          16    got into the -- okay.
          17              Issues and questions; choice of
          18    criteria for deleting, observe values, and
          19    classifying them as outliers, we need to make
          20    a recommendation about that.  Second one:
          21    Composition of estimation groups.  I've just
          22    given you some examples of 11 regions, why
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             13
           1    10, why 11?  It's Hawaii, Hawaii is censused,
           2    and sometimes we call them 10 and sometimes
           3    we call them 11, and that's my fault, it's
           4    really 11.
           5              Smaller number of regions, the four
           6    I cited before, these were examples.  Other
           7    regions could be -- you could use other
           8    criteria for defining the regions based on
           9    temperature and climate.  Another completely
          10    different category is ownership, which we've
          11    looked at.  We don't use the IOUs at all to
          12    make any predictions for the non IOUs.  It
          13    turns out that these seemed to be different,
          14    different in terms of the regression co-
          15    efficients.
          16              But there's a possibility of using
          17    them within the non IOUs, there are also some
          18    sub-sets, including things like the federal
          19    entity, the federal company -- federal
          20    installations.  Unfortunately they may be
          21    different but we can't do much about it
          22    because there aren't very many of them.  I
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             14
           1    think something like six to eight.(b) was
           2    something I was looking for advice about;
           3    it's fairly clear, in some cases how you work
           4    from the ground up?
           5              How would one be using ownership or
           6    geography is sort of clear, but suppose you
           7    just started with all these companies and so
           8    you say, how am I going to form the strata
           9    just from basic micro data?  Not so obvious
          10    to me.  You can get the estimates of slopes,
          11    but how trustworthy they are, not so clear.
          12    So this is just methodology working from the
          13    ground up to form a strata.
          14              Okay, number three, deals with
          15    macro-level. We have a lot of extra data that
          16    we're not making any use of, and perhaps that
          17    can be used to retain precision and reduce
          18    the sample size.  And marked here, remember
          19    there are two variables and three end users,
          20    and here are a couple of plots.  Why don't
          21    you look at the bottom one because the colors
          22    are sharp and the before and after is before
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             15
           1    -- all the data in "the after" is outliers.
           2              Clearly there is a time series
           3    pattern.  What's plotted here are the
           4    estimated regression co-efficients over a 24-
           5    month period starting from January '02 to
           6    December '03.  And so each point is the
           7    estimated regression co-efficient for a
           8    region.  So first thing to notice there's
           9    clearly a time-series pattern for this.  As a
          10    Bayesian, what I would do -- or contemplate
          11    doing, is, suppose I want to make an estimate
          12    for September '03, I might use the data prior
          13    to that to do a forecast, and if the forecast
          14    mean and the forecast variance be the
          15    empirical mean and variance of a prior
          16    distribution to put together.
          17              I mean this is just a suggestion;
          18    we're looking for other suggestions as well.
          19    I want to -- before going on, I want to
          20    point, this is residential, this is
          21    important, I want to point out one thing so I
          22    don't have to go back to this slide.  Notice
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             16
           1    towards September -- let's take any cross-
           2    sectional view, sometimes looks like these
           3    beta hats are very -- they look like they're
           4    estimating the same thing.  In other periods
           5    it looks like there are other groupings.
           6              That there are groupings but it is
           7    not everything together, and some
           8    methodology, which I've been working on for
           9    Years, might be useful for here, in trying to
          10    improve the precision, letting the data tell
          11    which sorts of regression should be combined.
          12    So I'll come back to that.
          13              Here's another one, commercial,
          14    these are again the beta hats, same plot, not
          15    as much seasonality as there was on the
          16    residential.  And then let's look at
          17    industrial and the reason -- one reason is,
          18    this is much less here as well, but note in
          19    the bottom the spike for North-West.  And I
          20    think these plots -- the reason I'm pointing
          21    this out is this is sort of pointed out,
          22    something that maybe very well known to
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             17
           1    everybody in CNEAF, but this is sort of an
           2    interesting thing and I'll come back to it in
           3    a second.
           4              These are comparable plots for the
           5    estimates of sigma square, the residual
           6    variance.  It was my assumption before we
           7    started that in fact sigma square might be
           8    quite stable, and you could use some of this
           9    to improve the estimation of sigma2.  If this
          10    is residential, here's commercial, there's
          11    another sort of wild, but most of them behave
          12    very -- they don't change much over time.
          13              This is the one I wanted to point
          14    to; look at the industrial, look at the
          15    bottom; that's that same region we had
          16    before.  So, in the summer -- spring/summer
          17    of '03, there was both a change in beta hat
          18    and this large increase in sigma2, so
          19    something went on.  And this maybe fairly
          20    useful, we have not tracked this down yet.
          21    I've asked one person who a hypothesis about
          22    what happened, but this maybe a useful thing
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             18
           1    to look at.
           2              Point number four is about use of
           3    company level longitudinal data, and there
           4    are two examples here.  What is plotted here
           5    is again same 24 months, each point is for a
           6    region, and what's plotted is the monthly
           7    value divided by the annual, divided by 12.
           8    And the annual divided by 12 is to put
           9    everything on the same scales, so they should
          10    operate around one.  And the dark thicker
          11    blue line is the estimated regression co-
          12    efficient for that region over the 24 months.
          13              What are the reasons for this --
          14    you've possibly used utility of these plots.
          15    One is to see whether the large and small
          16    companies, all of them seemed out of the same
          17    patterns; that would be a validation --
          18    informal validation of the model.  The second
          19    is if you see a lot of divergence, is maybe
          20    the stratification -- post stratification
          21    isn't any good.  So here's the second one,
          22    this looks even better from my perspective.
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             19
           1    This is a second one, the difference between
           2    these two, these are both residential sales,
           3    one is NWE and one was another.
           4              We've looked at some other -- a
           5    couple of others, which don't look quite as
           6    good as this.  But suggestions looked for
           7    here about what to do with these data of this
           8    sort.  And the last one is what I referred to
           9    before but I want to rephrase now; you have a
          10    choice of a large number of estimation groups
          11    and of course the advantage of that or post
          12    strata, large advantage of that is greater
          13    homogeneity within the groups.  Disadvantage
          14    is small sample sizes within each of the
          15    groups.
          16              Conversely you can have very few
          17    estimation groups and you have a larger and
          18    then you get more heterogeneity but larger
          19    sample sizes.  Methodology that is referred
          20    to, if you've all done your homework and
          21    looked at least at the paper I wrote, alluded
          22    to this research which the data -- a database
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             20
           1    pooling of like regression co-efficients.
           2    What it is, is an extension of the Standard
           3    Shrinkage Methodology.
           4              Shrinkage Methodology is kind of
           5    dumb; it just says which things to put
           6    together.  This method is dynamic and it
           7    basically takes the data and tells you which
           8    things to put together; it works quite well.
           9    So the question here with regard to this is
          10    any suggestions about that or other ways of
          11    choosing between large number of estimation
          12    groups or -- and with a consequent gains and
          13    benefits and other things.
          14              Here are the questions for the
          15    committee; they're just actually a repetition
          16    of the five points I just went through now,
          17    and good point to stop.  The first two are
          18    things for which we need to make
          19    recommendations, the last three, any
          20    suggestions about things other than what
          21    we've talked about or comments about
          22    anything.
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             21
           1              CHAIRMAN BREIDT:  Okay, thanks.
           2    Let me start with quick questions and you
           3    talked about deleting an outlier, so that
           4    really means just setting it into -- setting
           5    it outside the regression, using it as an
           6    observed value?
           7              MR. SEDRANSK:  Yeah, we're assuming
           8    that when such an outlier occurs that there
           9    is contact with the company to make sure the
          10    value is okay.
          11              CHAIRMAN BREIDT:  It's not an
          12    omission?
          13              MR. SEDRANSK:  Yeah.
          14              CHAIRMAN BREIDT:  Okay.
          15              MR. SEDRANSK:  And that's an
          16    important -- I mean, that's a -- I see
          17    Scott's up there -- you know, I mean, that's
          18    a standard part of practice is  going back.
          19    Right now there are programs that do these,
          20    what we call, scatter plot edits and they
          21    have been used heavily in the past, now not
          22    being used very much.  We think they should
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             22
           1    be used a lot more.
           2              CHAIRMAN BREIDT:  And is this cut
           3    off sample maintained or is it pretty much
           4    the same thing?
           5              MR. SEDRANSK:  No in fact that it
           6    was a cut off sample and it's now -- no, it's
           7    not maintained.
           8              MS. KIRKENDALL:  It needs to be
           9    evaluated again?
          10              MR. SEDRANSK:  It needs to be
          11    evaluated.  In fact that was what we were
          12    saying before, as I was saying at the
          13    beginning, there was a rumor about changing
          14    this sample composition.  What happens, I
          15    think it would be fair to say is that,
          16    somebody perceives a problem, maybe in one of
          17    the sectors and in one area and just adds
          18    some sample size to it and it's not done in
          19    an overall principled way, so in fact, it's
          20    not maintained, so this is no longer, you
          21    know, it's no longer a cut off sample.
          22              DR. NEERCHAL:  Why does the beta
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             23
           1    estimate hover around one?
           2              MR. SEDRANSK:  It does --
           3              DR. NEERCHAL:  You explained it but
           4    I didn't get it.
           5              MR. SEDRANSK:  Oh, they do, oh, you
           6    mean in those plots in the end?
           7              DR. NEERCHAL:  Yes.
           8              MR. SEDRANSK:  Oh, yeah, what I
           9    said in the company level plots, what we try
          10    to do, this was the first attempt, was we
          11    plotted the observed value for the month
          12    divided by annual, which is an estimate of
          13    the slope.  But then the values went all over
          14    the place, so the simple idea, divided by 12,
          15    put the annual over 12 on the same scale as
          16    the month.  So it made the plots -- the plots
          17    look better.
          18              MS. KIRKENDALL:  Is the regression
          19    actually based on the annual over 12 also, so
          20    when you put the regression you'd expect that
          21    to be around one --
          22              DR. NEERCHAL:  Oh.
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             24
           1              MS. KIRKENDALL:  To show the
           2    seasonality or whatever, or you could show
           3    market trends, or you could show whatever is
           4    going on. But in this case it sort of looks
           5    like it's primarily the seasonality.
           6              MR. SEDRANSK:  Something I should
           7    say, I know Nick is -- I'll just comment --
           8    just remark; part of this work or much of it
           9    is really a warm up, it's like an exhibition
          10    game for dealing with the more complex 906
          11    and 920 surveys that appeared in the original
          12    title.  Those, the data -- the relationships
          13    are not as nice or nowhere nearly as nice is
          14    what they've showed here.  We were trying to
          15    go through the exercise for something that
          16    was, looked good to get sort of what we were
          17    going to do, sort of down pad.
          18              And then the next thing is to turn
          19    to that, which you presumably will hear about
          20    in succeeding talks.  This is the data, it
          21    really is well behaved and when I talked
          22    about the fact that there are these problems,
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             25
           1    it's not problems but number three, what you
           2    may like to do is get some more precision, so
           3    that you can reduce the sample size; that's
           4    even more important that the other surveys.
           5    Nick?
           6              DR. HENGARTNER:  You talked about
           7    outliers and things like that, and I looked
           8    at the plots and my first inclination would
           9    be to do everything on a log-log scale.
          10    Because if you look at some of the plots
          11    there, I always have one guy way out there
          12    that drives the regression line and just like
          13    rescaling often things look -- it's just that
          14    the type of plots that's what suggest that --
          15    the other thing, I was wondering you said
          16    shrinkage, were you thinking of mixed effects
          17    models or even hierarchal models?
          18              MR. SEDRANSK:  Yeah, you know that
          19    was what I -- yeah, when I used shrinkage, I
          20    mean -- what I was talking about very
          21    specifically, let's take the example of the
          22    beta hats, if you used -- if you looked, if I
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             26
           1    knew where this was, maybe Lorenz can go --
           2    how do I go back?  Go back on -- go back down
           3    there.  I'll give you an example.  Now, I can
           4    go forward, I know where to go as long as you
           5    got me into a -- no, not pushing out, I'm not
           6    going anywhere.  Akay, at any rate, take a
           7    cross sectional view of those beta hats, keep
           8    going, Lorenz, yeah stop there, great.
           9              SPEAKER:  Can you speak to the
          10    microphone?
          11              DR. FEDER:  Can you speak to the
          12    microphone?
          13              MR. SEDRANSK:  No, I'm finished.
          14              DR. HENGARTNER:  But here are those
          15    regions, what you have is you have one that's
          16    on an annual cycle and the other one is on a
          17    bi-annual cycle, and those are the regions on
          18    the bi-annual cycles are the ones in which
          19    you need both heating and air-conditioning.
          20              MR. SEDRANSK:  Yeah, okay no, no --
          21    I say no.
          22              DR. HENGARTNER:  Why?
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             27
           1              MR. SEDRANSK:  Anyway, the point I
           2    was going to try to make a technical point --
           3              DR. HENGARTNER:  Okay, go ahead.
           4              MR. SEDRANSK:  Shrinkage
           5    Methodology, if you -- the standard shrinkage
           6    -- I'll stay over here.  Standard Shrinkage
           7    Methodology, look at September '03 where
           8    there is a kind of a large thing, here we
           9    just say, your estimate for one region, take
          10    NWE for example, will be that regression
          11    estimate, weighted average and that of the
          12    others, it's dumb, it doesn't realize the
          13    fact that some of the data, not that all data
          14    doesn't come from a single source.  The
          15    methodology I have developed let's the data
          16    decide which things should go together.
          17              DR. HENGARTNER:  But you could
          18    actually, since you have the time series, you
          19    can think of modeling this d(t) as a function
          20    of (t).  And you mentioned that.
          21              MR. SEDRANSK:  Yes, that's right.
          22    That was, okay I've told two different
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             28
           1    things, right?  I would do -- that's the
           2    first thing is the time series effects are
           3    probably -- in fact you might do it both ways
           4    actually.  You might have both the time
           5    series and all the --
           6              DR. HENGARTNER:  Yes, the next link
           7    are the time series coefficient.
           8              MR. SEDRANSK:  Yes, in fact you
           9    really want to do this both ways, you really
          10    want to do the cross section and the time
          11    series parts of this.  But sort of not the
          12    dumb shrinkage methodology -- well, shrinkage
          13    methodology is same as like mixed effects;
          14    you don't want to assume that the regression
          15    coefficients in some of these time periods
          16    come from the same source.  They don't.  In
          17    other periods they do.  So the real solution
          18    to this which is really interesting
          19    statistical problem is the time series and
          20    the cross section.
          21              DR. NEERCHAL:  You mentioned there
          22    are only three sample points in the ---
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             29
           1              MR. SEDRANSK:  Lest.
           2              DR. NEERCHAL:  In the Lest?
           3              MR. SEDRANSK:  Yes.
           4              DR. NEERCHAL:  In the frame, how
           5    many are there?
           6              MR. SEDRANSK:  Do we know?
           7              DR. NEERCHAL:  What percentage?
           8              MS. KIRKENDALL:  We don't know.
           9              MR. SEDRANSK:  Do we know off hand.
          10              DR. NEERCHAL:  Are there lots?
          11              MS. KIRKENDALL:  More than three.
          12    I haven't looked at that.
          13              MR. SEDRANSK:  Yeah.
          14              MS. KIRKENDALL:  But that's
          15    probably small.  There's been other
          16    discussion that may be California should be
          17    broken into two regions.  North and south are
          18    very different.
          19              DR. NEERCHAL:  But there aren't
          20    that many IOUs in California?
          21              MR. SEDRANSK:  Non IOUs.
          22              DR. NEERCHAL:  Non IOUs?
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             30
           1              MR. SEDRANSK:  Non IOUs.  There may
           2    not be, there may not be too many.
           3              MS. KIRKENDALL:  We have to look at
           4    the data.
           5              MR. SEDRANSK:  Yes.  There are
           6    more, I'm sure there is more, I'm sure there
           7    are more.
           8              CHAIRMAN BREIDT:  Can you say more
           9    about the background behind this pooling
          10    methodology?  Is there kind of a -- is it
          11    like a model averaging sort of procedure?
          12              MR. SEDRANSK:  Yes.
          13              CHAIRMAN BREIDT:  With prior
          14    subsequent model.
          15              MR. SEDRANSK:  Oh what you do -- oh
          16    it's very simple.  You consider all the
          17    partitions of all those 10 points, put a
          18    prior probability on it, usually equal, and
          19    then within a grouping, within the subset in
          20    the partition, you assume everything is
          21    exchangeable.  It works very well, I develop
          22    -- first person to ever -- this is -- there
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             31
           1    is some literature back to the early 90s, and
           2    it works very, very well.  I now have a paper
           3    to show how to do this for small area
           4    estimation, which is really easy to do.
           5              If you start with the estimates --
           6    this is actually really easy to do because
           7    you -- I'm planning to do it, student who's
           8    worked with me on it is very busy.  And  as
           9    soon as we finish -- the reason this is easy
          10    is if you start with the summary statistics
          11    like the regression coefficient is easy.  If
          12    you want to read the papers, biometrical
          13    paper in '92 and one about 2001 is this is a
          14    sinica paper in '99.  But it really works, it
          15    works surprisingly well this --
          16              DR. HENGARTNER:  It sounds what
          17    Hartigan was also doing.
          18              MR. SEDRANSK:  He assumes -- Okay,
          19    he assumes is there's not a mistake in what
          20    he does, ours is much more general --
          21              DR. HENGARTNER:  I'm glad you say
          22    that.
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             32
           1              MR. SEDRANSK:  No, no, it is, it
           2    is, it is.  It's the same, it's the same
           3    thing, we were actually working on it at the
           4    same time, started working on it at the same
           5    time.  He assumed something, we assumed that
           6    the regression coefficients in the subset are
           7    not exactly the same.  He assumes they are.
           8              DR. HENGARTNER:  Yes.
           9              MR. SEDRANSK:  But once you get
          10    into a subset that they have exactly the same
          11    values, we allow variability in that.  And
          12    that's what the difference is.  There's
          13    unfortunately one of his students wrote a
          14    thesis based on it, he's got a mistake in
          15    that.
          16              DR. HENGARTNER:  Yes, I know.
          17              MR. SEDRANSK:  Oh, you know.
          18              DR. HENGARTNER:  But I mean in some
          19    sense the Bayesian form of clustering.
          20              MR. SEDRANSK:  That's right.
          21              DR. HENGARTNER:  I mean, that's
          22    really what we're doing here.
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             33
           1              MR. SEDRANSK:  That is exactly
           2    right.  And in fact if you could do this
           3    computationally, this would be Bayesian
           4    clustering.
           5              DR. HENGARTNER:  Yes.
           6              MR. SEDRANSK:  If you want a
           7    another reference which is funny, David
           8    Binder who's doctoral dissertation on
           9    Bayesian clustering, and has some early
          10    papers and it -- was something I looked at
          11    when we had started out.  Now this would be
          12    good.  The hurdle is really that with all
          13    these partitions, unless you have a way of
          14    structuring the population, you get too many,
          15    you just, there are just too many of these.
          16              DR. HENGARTNER:  Couldn't we try to
          17    think in terms of the tree, and somehow --
          18    you know split your things in two and then
          19    four like a CART.
          20              MR. SEDRANSK:  You want to try and
          21    do CART.  I haven't actually thought of that.
          22    But that's not --
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             34
           1              DR. HENGARTNER:  Because if you
           2    consider all the all the subsets.
           3              MR. SEDRANSK:  Oh, it's too many.
           4              DR. HENGARTNER:  Forget it, but if
           5    you do it like they do in CART and things
           6    like that, you actually have a prior.
           7              MR. SEDRANSK:  There is a paper by
           8    -- people realized after I wrote my first --
           9    consoniun(?) weren't easy in biometrical.
          10    After I published the first paper, did it for
          11    the binomial and they -- similar work but
          12    similar structure and whatever, but they
          13    recognized that there are some structures
          14    where you can put in the factorial designs,
          15    you only look at sort of some main effects
          16    and some interactions.  Problems such as
          17    this, not so clear how to do it.
          18              DR. NEERCHAL:  One thing you might
          19    want to do is to start by plotting the
          20    Bayesian values against those variables that
          21    you're not using right now.
          22              MR. SEDRANSK:  Yes.
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             35
           1              DR. NEERCHAL:  To see if there is
           2    some pattern.
           3              MR. SEDRANSK:  Yes.  See what's
           4    going on.
           5              DR. NEERCHAL:  Is there is some,
           6    some information there.
           7              MS. KIRKENDALL:  What value of data
           8    that we're not using?
           9              DR. NEERCHAL:  Like some other
          10    characteristics of the business, sales
          11    revenue, and things like that.
          12              MR. SEDRANSK:  Yes.
          13              DR. NEERCHAL:  I don't know what
          14    else do you have there, so.
          15              MS. KIRKENDALL:  Well, one thing,
          16    there should be a relationship between sales
          17    and revenues.  I mean those are reported by
          18    companies, right now we're doing it
          19    independent estimates.
          20              DR. NEERCHAL:  No, no, plot the
          21    beta values against some other
          22    characteristics of the -- whatever you have
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             36
           1    -- let's try to see if there is something you
           2    can pull up and that might help you at least
           3    in terms of cut down the number of subsets
           4    you need to look at.
           5              MR. SEDRANSK:  Anyway let me switch
           6    the discussion because Nancy -- Nancy is not
           7    squirming but probably afterwards we do need
           8    to make recommendations probably about these.
           9    Does anybody know anything about the out --
          10    go back to question number one.  Sort of
          11    anything about looking at other than
          12    influential observations and --
          13              MS. KIRKENDALL:  I do want to
          14    address implementation as one of the things.
          15              MR. SEDRANSK:  Yes.
          16              MS. KIRKENDALL:  That's happened in
          17    the past is that the people who process, who
          18    collect the data and process it, they get it
          19    ready, they do whatever edits they think they
          20    need to, they think it's fairly clean and
          21    then it goes to an estimation program.  And
          22    there's a lot of information that comes out
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             37
           1    of the estimation as a result of these
           2    scattered plots and outliers and what not.
           3    And there has not been a feedback to the
           4    staff.  So one way you could do something --
           5    and then they do more detailed analysis, but
           6    they typically find it when somebody looks at
           7    a total number and it's off it.
           8              So you get an error and it goes
           9    through and really messes up your total
          10    estimate.  And may be caught at the end, or
          11    it may be caught later, and it may not be.
          12    So the thought on the outlier detection was
          13    just take it out and make that outlier so it
          14    doesn't affect your estimate for everybody
          15    else, feedback that information about the
          16    outlier so people can check up on it.  So
          17    it's to try to minimize the impact of one
          18    funny observation on the estimate of the
          19    total that you're producing.
          20              MR. SEDRANSK:  But basically this
          21    is an automatic, sort of an automated
          22    procedure.  Particularly this survey which is
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             38
           1    operating well, it doesn't get a lot of -- it
           2    doesn't get as much attention in some ways as
           3    other's do.  So if you don't -- you've got to
           4    have something that's automated.  But you
           5    know -- it's sort of, it's something strange,
           6    really strange happens that you don't get
           7    blown out of the water.
           8              MS. KIRKENDALL:  I mean sometimes
           9    these are real data to --  there's a huge
          10    changes in the company.
          11              DR. NEERCHAL:  If you classify
          12    something as an outlier, then it will not be
          13    used for the estimate prior?
          14              MS. KIRKENDALL:  Yes, it would just
          15    not be used on data.  It would be added back
          16    in at the end.
          17              MR. SEDRANSK:  Yes.  We always, we
          18    always use it for -- I mean assuming it's
          19    correct, you know it's deemed to be correct.
          20              MS. KIRKENDALL:  But then you do
          21    want to check on it.
          22              DR. NEERCHAL:  I mean if something
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             39
           1    -- some -- they have some special thing going
           2    on that particular data value then it will
           3    marked that as an outlier --
           4              DR. BURTON:  It will still be added
           5    back at the end of it as far as -- if it's
           6    sure to be valid.
           7              MS. KIRKENDALL:  Right actually
           8    that's one thing, there was some discussion
           9    yesterday about over editing.  EIA, we do
          10    editing but we typically do not impute for
          11    failed data.  We try to phone on it,
          12    certainly all the big companies.  So while we
          13    do the editing as an agency, which is across
          14    most of our offices, we do not rely on
          15    imputation for failed data.  So we are a
          16    little different from many statistical
          17    agencies.
          18              MR. SEDRANSK:  Let me switch to a
          19    different--
          20              DR. BURTON:  Would your data be
          21    cleaner?
          22              MR. SEDRANSK:  I am going to ask
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             40
           1    another question which I tried to the get the
           2    -- draw out the answer to.  If in a single,
           3    in an establishment survey and you try to do
           4    stratification -- here we're doing post
           5    stratification, not doing stratification --
           6    but you sort of do probably something like
           7    sales or revenue are probably highly
           8    correlated, profits but pretty not -- not may
           9    be profits, sales and revenues, it's
          10    stratify, if you had the opportunity -- you
          11    would just do something like the cumulative
          12    square root of F method, which is been around
          13    for forty, or forty or more years.
          14              This case you don't have that,
          15    you're looking for common features of the
          16    regression coefficients.  So I just have
          17    question about how you might build up strata,
          18    just forget about climate and temperature.  I
          19    mean this is not smart, just totally forget
          20    about it.  How do you sort of build -- how do
          21    you build up strata from the ground up, is
          22    the only way I can describe it, any ideas?
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             41
           1              MS. KIRKENDALL:  By strata you mean
           2    the estimation?
           3              MR. SEDRANSK:  Estimation groups in
           4    post strata, yes, yes, right.  Or turn it
           5    around and say we're getting a new -- they
           6    were in cut off samples, as far as you're
           7    doing a probability design, performance
           8    strata in the first place.  I'm not
           9    suggesting we're doing this but --
          10              MS. KIRKENDALL:  Actually one thing
          11    for all of these observations, we do have the
          12    state that they operate in.  So we can look
          13    at any different groupings that we think we
          14    need to.
          15              MR. SEDRANSK:  Yes.  It's just I
          16    was thinking of a just primitive method, or
          17    where you start from.  I mean there are
          18    things you can do, you can just get estimates
          19    of the regression coefficients that are just
          20    monthly over annual, you have a whole batch
          21    of them for each company.  I just - I mean
          22    I've suggested that, but I --just curious if
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             42
           1    anyone has any other ideas.
           2              CHAIRMAN BREIDT:  But the
           3    regression coefficients you get at the
           4    company level are, these are based on --
           5              MR. SEDRANSK:  One observation.
           6              CHAIRMAN BREIDT:  Yes.
           7              MR. SEDRANSK:  Per month.  But
           8    you'd actually get a bunch of months of them,
           9    so you --
          10              CHAIRMAN BREIDT:  Then the same
          11    month across the years and that regression
          12    coefficient.
          13              SPEAKER:  Yeah.
          14              CHAIRMAN BREIDT:  We're talking
          15    about.
          16              MR. SEDRANSK:  Well, we got both
          17    those, we've got the same month over the
          18    years, we've got all this, we've got the time
          19    series data and whatever, you could take all
          20    of this and see what -- put it in the cluster
          21    analysis package and see what do you get.
          22              CHAIRMAN BREIDT:  Right, it doesn't
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             43
           1    see that you want to through all the months
           2    together on some of these plots.
           3              MR. SEDRANSK:  I'm not sure if this
           4    is important.  I mean I think the temperature
           5    based thing makes sense, although I think the
           6    ones we've got are -- I think ones we've got
           7    -- I think there are some other units.  I did
           8    that and I'd be surprised -- this one just
           9    looks too large, that north east one.
          10              DR. SITTER:  Well, I mean you could
          11    also use it as a validation, if we came up
          12    with a method to see if it actually is a
          13    decent grouping.  But you can just quantify
          14    the problem, you've got a variance estimator,
          15    obviously?
          16              MR. SEDRANSK:  Yeah.
          17              DR. SITTER:  Okay, it's  going to
          18    depend on the group, throw it into your
          19    favorite search algorithm for finite grouping
          20    searches, clustering, you can just tick away
          21    random groupings.  I mean computers are fast,
          22    you can quickly find out things are
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             44
           1    reasonable.  And I've seen them do this with
           2    genetic algorithms or simulated annealing,
           3    anything that doesn't require any derivatives
           4    or anything.  They're not really fast, but
           5    you know speed isn't of the essence.
           6              DR. HENGARTNER:  Randy we're trying
           7    to group thousands of companies, correct?
           8              DR. SITTER:  No, just states --
           9              MS. KIRKENDALL:  There are only -
          10    there are less than 200 that wer'ee actually
          11    using in this regression, we have about 400
          12    observations if you count the IOUs that are
          13    currently not used to any regression.
          14              DR. HENGARTNER:  So 200 and you are
          15    trying to group them in?
          16              DR. SITTER:  It's not too bad. I
          17    mean you can even use some of the scheduling
          18    methods where they want to schedule jobs to
          19    computers.  There are some very simple
          20    algorithms that are very fast, they are not
          21    going to get you the optimum, but they will
          22    get you something very good.  And they are
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             45
           1    very simple, they just order it on some
           2    reasonable proxy and then they take the
           3    biggest ones, put one in each group, and then
           4    they just tick the next guy and say, well,
           5    which group can I put it in that will
           6    increase the variance the least, and they
           7    just do that bang, bang, bang, bang; it's
           8    very fast.  And you come up with a sub
           9    optimal solution but I'm --
          10              DR. HENGARTNER:  Which is pretty
          11    good though --
          12              DR. SITTER:  But it's pretty good
          13    on average.
          14              DR. HENGARTNER:  I don't have
          15    experience with -- you know that's the
          16    Bayesian method.  But I have the feeling that
          17    if you tried to prove that, you are going to
          18    have a lot of local minima's.
          19              MR. SEDRANSK:  Oh, no, I wasn't
          20    even thinking of Bayesian, this was not
          21    Bayesian --
          22              DR. HENGARTNER:  Oh, that wasn't
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             46
           1    you -- I was thinking of.
           2              MR. SEDRANSK:  I do all kinds of
           3    things, it depends on what day it is.  I
           4    think it like a Bayesian, but I --
           5              MS. KIRKENDALL:  One of the things
           6    that I should say is this the original system
           7    and the estimation methods were done by Jim
           8    Knob who was hiding up there or who is trying
           9    hiding up there, in the active performance in
          10    alternative fuel.  And he presented a lot of
          11    the information to the ASA committee back in
          12    the early the 90s, I think some of it was
          13    when Joe was the chair of the committee.  So
          14    this is an example of a problem that keeps
          15    coming back.
          16              MR. SEDRANSK:  I mean essentially
          17    things are working well here, and this is
          18    sort of a prototype.  But there are these
          19    issues, there is certainly desire to cut back
          20    sample size, may be not so much in this one
          21    as in 906 and 920.  So can one make use of
          22    this time series, information is also issues,
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             47
           1    you know academically it's of interest
           2    whether you can make some use of it.  One of
           3    the problems which we did not allude to is
           4    there was a major in the time series when
           5    you're going to have trouble making a longer
           6    time series because in 2004 there was a major
           7    shift that other was sent away and replaced
           8    by transportation.  So that was a -- so
           9    you've got a problem of different
          10    distribution among those end users after
          11    that.  That's one of the reasons we don't go
          12    after 03, then 02 and 03 are -- you know are,
          13    are clear cut, so we won't have a long --
          14              MS. KIRKENDALL:  Residential may
          15    still be okay.
          16              MR. SEDRANSK:  What?
          17              MS. KIRKENDALL:  Residential may
          18    still be okay.
          19              MR. SEDRANSK:  Residential probably
          20    be okay.
          21              CHAIRMAN BREIDT:  Well, it looks
          22    likes you have spatial(?) structure too.
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             48
           1              MR. SEDRANSK:  Yes, do something
           2    here.
           3              CHAIRMAN BREIDT:  Do something
           4    here, especially in the finding the time
           5    series and the spatial structure.
           6              MR. SEDRANSK:  Yes.
           7              CHAIRMAN BREIDT:  Allowing for the
           8    fact that over time that their ability
           9    changes a lot in states so that you have a --
          10              MR. SEDRANSK:  Yes.  This looks
          11    like a, this is a good problem.
          12              CHAIRMAN BREIDT:  Yes.
          13              MR. SEDRANSK:  I found a similar
          14    one at the census bureau also recently which
          15    I now can't remember, carrying those small,
          16    small area overtime, there's a - they've been
          17    doing it this way, but I detected that
          18    they've got some of the same structure
          19    problem.
          20              DR. HENGARTNER:  Joe, have you
          21    thought of looking at this as a geo temporal
          22    process?  And I'm now thinking, I know the
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             49
           1    location of each of the generators, right?  I
           2    always thought that there is some spatial
           3    correlation instead of just grouping them
           4    into regions, let's look at some kind of a
           5    diffusion those must be correlated in some
           6    ways.  And we have awesome barriers like the
           7    mountain ridges, whatever, I mean we actually
           8    know about the geography.
           9              MR. SEDRANSK:  No, we haven't
          10    because we've looked at really, we're
          11    evaluating their methodology, it doesn't --
          12    by the way it's a good idea --
          13              DR. HENGARTNER:  I mean you're
          14    asking for what goes through my mind --
          15              MR. SEDRANSK:  No, no, that's a
          16    good -- no this is.  I'm down on, --I'm down
          17    geo spatial stuff, I have a student who's
          18    been working on a problem and I don't want to
          19    tell you what troubles he has had.  These are
          20    binary responses though in a geo spatial
          21    setting and I can tell you things that don't
          22    work, lots of things that don't work.
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             50
           1              CHAIRMAN BREIDT:  Okay, any other
           2    comments?
           3              MS. KIRKENDALL:  Okay, you have any
           4    brilliant ideas?
           5              MR. SEDRANSK:  Email.
           6              MS. KIRKENDALL:  Email works.
           7              SPEAKER:  I haven't had one of
           8    those in a while.
           9              CHAIRMAN BREIDT:  So it's now time
          10    to move to our break out sessions.  The next
          11    session will be downstairs on the post
          12    stratification methodology.
          13                   (Recess)
          14              MR. HOUGH:  Desperate to get
          15    started.  I was just informed that there -- I
          16    guess we'll be chairing this ourselves.  It's
          17    a scary thought I know but -- My name's Rick
          18    Hough, I am here from the Census Bureau, I am
          19    the survey manager responsible for the
          20    Manufacturing Energy Consumption Survey.  I
          21    am here to talk to you a little today with my
          22    colleague, Stacey Cole, who's the brands
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             51
           1    chief responsible for the R&M brands that has
           2    MECS as one of its many responsibilities.  We
           3    want to talk to you today a little bit about
           4    the post stratification methodology that we
           5    implemented during the 2002 Manufacturing
           6    Energy Consumption Survey.
           7              Before I begin, I'd like to take a
           8    quick minute and acknowledge everyone who
           9    worked hard on the 2002 MECS, Susan Bucci is
          10    with us today, she's my boss, she's the
          11    brands chief, in charge of the brands where
          12    MECS is conducted, members of my staff who
          13    were responsible for data analysis, Vicky
          14    Haitot, Lacy Loffin, and Eva Snap.  I would
          15    also like to recognize the methodologists on
          16    Stacey's staff, who were responsible for
          17    coming up with a lot of the details that
          18    you'll see in today's presentation, Jeff
          19    Dalzell, Cathy Gregor, John Slanda, who's
          20    here with us today in case any technical
          21    questions would arise, and Bob Struble.
          22              I'd also like to take a quick
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             52
           1    moment to acknowledge our members of EIA who
           2    worked on the Manufacturing Energy
           3    Consumption Survey.  Dwight French is with us
           4    today.  Dwight gave the approval to go
           5    forward with this process, came out to census
           6    few times, listened, gave comments, and then
           7    told us to go ahead, Bob Adler, who was the
           8    other survey manager I saw, my co-survey
           9    manager from EIA who also worked very hard,
          10    and Tom Lorenz who some of you heard from
          11    yesterday, survey analyst from EIA who came
          12    out for census quite a bit to work with our
          13    team on data analysis and preparing reports
          14    and he was a very valuable member of our
          15    team.
          16              Just an overview of what we're
          17    going to talk a little bit about today.  I'm
          18    going to talk to you a little bit about the
          19    background and the goals of the Manufacturing
          20    Energy Consumption Survey.  I'm going to give
          21    a brief description of the economic census,
          22    talk a little bit about some of the aspects
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             53
           1    of the census that affect the reasons why we
           2    Post Stratified, and also some of the files
           3    that are involved with Post Stratification.
           4              Stacey is going to come up and talk
           5    a little bit about the sample design, and
           6    then he's going to get in to the actual Post
           7    Stratification methodology and then we have
           8    some results of what we implemented to talk
           9    about.  And then, finally we have some
          10    questions for the panel that we'd like to
          11    consider, some questions about the
          12    methodology that we implemented for this
          13    survey, and if time permits we'd also like to
          14    talk a little bit about what we might
          15    consider in the future given the time frame
          16    of MECS and how it is ultimately related to
          17    how the census is conducted.
          18              Okay, the Manufacturing Energy
          19    Consumption Survey.  Basically our goal is to
          20    provide detailed data on energy consumption
          21    for the manufacturing sector.  We provide a
          22    variety of information from individual energy
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             54
           1    sources, such as electricity, natural gas,
           2    selected fuels.  We also provide data on
           3    industry as well as geographic.  We report
           4    statistics for some of the energy saving
           5    technologies that are implemented in
           6    manufacturing establishments as well as their
           7    ability to switch to alternative fuels in
           8    certain situations.
           9              The survey was initiated in 1985,
          10    it was conducted every three years until the
          11    survey year 1994 and it's been conducted
          12    every four years since that point.  The
          13    economic census, just some general
          14    information, provides a detailed portrait of
          15    the nation's economy.  We provide data on
          16    industry as well as geographic, we provide a
          17    variety of maniputable formats for data users
          18    such as CD-ROM's, we have an American Fact
          19    Finder on our website where the data user can
          20    go in and get data and create their own
          21    tables.
          22              The census is conducted every five
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             55
           1    years; it is conducted in years that end in
           2    Two and Seven.  It is conducted using the
           3    North American Industry Classification
           4    system.  This is the system that was brought
           5    on to replace the old SIC.  The 2002 census
           6    was the second census conducted using these
           7    classification definitions.
           8              Okay, and the census provides a
           9    comprehensive update of the classifications
          10    that are contained in the Census Bureau's
          11    business register.  A lot of the companies
          12    within the business register do not get
          13    Census Bureau questionnaires every year.
          14    Therefore, the update path that we have from
          15    them to update their classifications is done
          16    through the census.
          17              I want to talk a little bit about
          18    the 2002 economic census for the
          19    manufacturing sector.  The census bureau does
          20    not mail every establishment in the country a
          21    questionnaire.  We have what's called a
          22    non-mailed file, it is defined as single unit
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             56
           1    manufacturing establishments with less than
           2    five employees and these establishments were
           3    not mailed questionnaires.  For the year
           4    2002, this file contained approximately
           5    160,000 establishments.
           6              The Census Bureau uses
           7    administrative data from other government
           8    agencies to estimate for these establishments
           9    and they account for approximately 3 percent
          10    of the published totals.  Now, if you have a
          11    non-mailed file, then you have a mailed file,
          12    and the definition of the mailed file is the
          13    ones you don't -- the ones you sent
          14    questionnaires to.  They are the single unit
          15    establishments with more than five employees
          16    and all multi-unit manufacturing
          17    establishments were mailed questionnaires.
          18              DR. HENGARTNER:  Now how do you
          19    have those addresses?
          20              MR. HOUGH:  Other government
          21    agencies.
          22              DR. SITTER:  Homeland Security.
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             57
           1              DR. HENGARTNER:  Good answer.
           2              MS. DODDS:  We do some surveys in
           3    the years between the census and many of the
           4    addresses come from those, and others would
           5    come from other government agencies, taxing
           6    agencies.
           7              DR. HENGARTNER:  IRS.
           8              MR. HOUGH:  Okay, the mailed file
           9    contained approximately 200,000 manufacturing
          10    establishments for the survey year 2002.
          11    Okay, the data that's collected during the
          12    census from the manufacturing establishments
          13    -- we collect operational data on employment,
          14    receipts, inventories, costs, assets,
          15    purchase services, et cetera.  Among the cost
          16    data we collect from the establishments are
          17    cost of fuels and cost of electricity.
          18              And when Stacey comes up, he's
          19    going to read for quite a bit to this cost of
          20    energy, that is what we use to define the
          21    cost of energy.  We take the cost of the
          22    fuels reported and the cost of electricity
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             58
           1    reported by the establishments and that is
           2    how the cost of energy is defined.  I'm going
           3    to defer now to Stacey and he's going to
           4    start off by talking a little bit about the
           5    sample design.
           6              SPEAKER:  Which one's up?
           7              MR. COLE:  Good Morning.  Basically
           8    my area was responsible for going to in
           9    developing a sample design for the MEC
          10    survey.  The key goal of MECS is to produce
          11    detailed aspects of energy consumption at the
          12    national level for industries as well as
          13    regional level for industries.  The target
          14    population is basically the mail panel of the
          15    -- every portion of the census.  Single
          16    location establishments with more than five
          17    employees and all multi unit establishments.
          18              The mail file was used as just
          19    said, every establishment in the mailed file
          20    was classified to a specific industry code,
          21    and everyone has to sign cost of energy data
          22    which came from historical sources.  For many
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             59
           1    cases the cost of energy data came from the
           2    2001 annual survey manufacturers, for other
           3    cases it came from the 1997 government
           4    census.  So in some cases, it's very, very
           5    old.
           6              We used a salvation approach to go
           7    through the sample, there were 473 specific
           8    manufacturing industries, way too many to
           9    cover in detail for MECS but fortunately
          10    energy use is concentrated to a great deal.
          11    There are 37 industries that are of high
          12    interest to EIA, for the most part these
          13    account for almost half of the energy
          14    consumed in manufacturing.  The balance of
          15    the -- the remaining 440 -- 400 and whatever
          16    is -- 436 whatever it is -- industries were
          17    collapsed to higher levels, so between both
          18    groups that cover the can clear the entire
          19    sector of manufacturing.
          20              And also there was a desire for
          21    some states and some sub- national level
          22    data, so we collapsed the stage to Census
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             60
           1    Bureau regions.  There are 267-industry group
           2    by recent strata, there was one that doesn't
           3    exhibit basically an empty set.  There are no
           4    establishments.  The allocation was also
           5    challenged, we have a sample of only 15,000
           6    establishments.
           7              The establishments both the
           8    collection unit with the mail unit as well as
           9    the collection unit, the collection unit as
          10    well as the sample unit, and also the
          11    collection unit.  We have a sliding scale of
          12    reliability constraints, basically we think
          13    the cells that have the most amount of
          14    activity are the cells of the highest
          15    interests, so we tried to do a really good
          16    job in the cells with a lot of activity.
          17              The smaller cells have fairly high
          18    C V constraints to inflation.  I believe the
          19    C V constraints range from 2 percent to about
          20    10 percent.  Again the big cells had higher
          21    constraints.  The section approach basically
          22    is, an independent sample was drawn from each
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             61
           1    one of the 267 stratum, within each stratum,
           2    we've a PPS approach.  The establishment size
           3    does matter, establishments that have a large
           4    consumption of energy had a -- a very high
           5    problem with selection.  Again, the energy
           6    data we had for such level came from the year
           7    19 -- 2001 ASM or historical data.
           8              We also imposed a minimum
           9    probability of.02, this helps to prevent in
          10    the cases where we end up with large weights
          11    and surprisingly large data, which causes
          12    problems for estimation as well as the
          13    variances.  Estimation is really a four-stage
          14    process, I only have three on here, but the
          15    first stage is obviously to collect the data
          16    and like most surveys, we do not get all the
          17    mailed cases in.  There's a small amount of
          18    all that -- not a small amount, there's about
          19    a 25 percent non-responses rate to a survey.
          20              We mailed out 15,000 forms and we
          21    got about 12,000 back in.  Historically we
          22    have done a non-response weight adjustment to
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             62
           1    MECS and this time we did the same thing.
           2    The first and third board there, I think,
           3    we've done historically in MECS.  The middle
           4    board is something basically brand new.  In
           5    non- response weight adjustment process, we,
           6    no, go back a second, we did it by stratum.
           7    With each stratum we said, we broke down in
           8    to sub-stratum, we said within each stratum
           9    there are certainties and non-certainties,
          10    response rate varies between the two
          11    subgroups, and we wanted to go through and
          12    adjust their weights separately.  That's also
          13    an enhancement for 2002 MECS.
          14              It is basically a standard form of
          15    the non-response weight adjustment, where the
          16    numerator is basically equal to the weighted
          17    data from the entire sample on a sample frame
          18    and the denominator was the observed data
          19    from respondents, I'm sorry, it's the data
          20    trivial to respondents.  So every response in
          21    the cell that did the adjustment factor is
          22    equal to 1.  In most cells that one didn't
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             63
           1    happen.  So at that point we adjust the
           2    sample weights to reflect non-response.  The
           3    post-stratification, why we do this?  We are
           4    concerned about sample frame.
           5              Basically we are using a 2002 mail
           6    file, we are worried about the coverage of
           7    the target population, we are worried of the
           8    classification of the el, and we are worried
           9    about the cost of energy data being accurate.
          10    The coverage, we know that the mailed file
          11    includes records that will ultimately not be
          12    in your factory when your mail questionnaire
          13    is censused, they'll come back in
          14    re-classified into wholesale, retail or
          15    construction or other areas.
          16              Initially there were cases data
          17    being mailed as retail/ wholesale for census,
          18    they can be classified in manufacturing.  You
          19    know, that's a problem.  In addition, we have
          20    cases that -- the classification within
          21    manufacturing will have changed between 1997
          22    and 2002 and this affects the thin strata of
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             64
           1    variability, and the cost of energy data is
           2    also a problem because it's quite volatile
           3    year to year, in addition for most of the
           4    records, the last testimony of the
           5    consumption is five years old.  So, we had
           6    all those concerns.  And that prompted us to
           7    go through and examine the idea of Post-
           8    Stratification.
           9              Fortunately, we had the Econ census
          10    as a source to go through and do the post
          11    stratification.  It provides a comprehensive
          12    collection of all factors; it allows us to
          13    update the classifications of everybody in
          14    the Econ directorate, and also we do collect
          15    the cost of energy for all manufacturing
          16    establishments.  So we were able to go
          17    through and basically re-assemble the frame
          18    based upon the more current information,
          19    which is what we did.
          20              For each stratum, we identified the
          21    eligible units in the stratum, we summed
          22    their cost of energy to the stratum level, we
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             65
           1    also developed an estimate of that from the
           2    sample cases in that stratum, so we were able
           3    to do a comparison between the target
           4    population energy data and the sample
           5    estimate of that, and here's a graph that
           6    shows the comparison between the actual and
           7    the estimated total.
           8              For the most part, the estimates
           9    are pretty close to the actual data, but
          10    there are a handful of cells where the
          11    estimate is quite a bit higher than the
          12    actual total from the census.  Look at those
          13    cells and determined that -- that cause was
          14    caused primarily by the big weight and big
          15    data problem, had some cases we thought they
          16    were quite small, but in reality they end up
          17    being quite large.
          18              So we had a handful of cells where
          19    the estimates was quite a bit greater than
          20    the actual population total.     We are
          21    looking for a function that will go back and
          22    adjust the sample weights of the sample
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             66
           1    cases, so that the weight adjusted estimate
           2    will equal the controlled total.  We wanted a
           3    function where we didn't adjust the weights
           4    of a certainty cases.  They are a
           5    self-representing cases during the selection
           6    process and we want them to remain that way
           7    during the estimation process.
           8              We also wanted to avoid a problem
           9    of lowering weights on non-certainties to the
          10    point where they became less than one, which
          11    is kind of a hard concept to understand,
          12    weight less than one.  So we develop a
          13    process where the K is our factor, K is
          14    linear, it goes to the point 1.1 and 1.1 is
          15    the axis, it swings round that at a point,
          16    and it's the purple area, if we have the
          17    estimate as less than the population, we have
          18    to adjust the weights up.
          19              In the blue area, we are saying
          20    basically the estimate is greater than the
          21    population and we have to adjust the weights
          22    downward.  That's the graphic display.  And
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             67
           1    here's the actual formula that we used,
           2    basically numerator represents the amount of
           3    energy data associated with establishments
           4    are not part of the sample.  Basically it's
           5    the cap end of the total, the first time is
           6    the published total, and the second time is
           7    what we observe in the MECS sample.  So
           8    basically the numerator is what was not in
           9    the MECS sample.  And the denominator is
          10    basically an estimate of the numerator.
          11              So hopefully, our ratio is close to
          12    one, and if it's one, we are really happy.
          13    The sample was -- after we get the case for
          14    each stratum we go back and we adjust the
          15    sample weights further by the K.  Just look
          16    at the top of the slide there, you can see
          17    that the certainty cases, the cases where
          18    WI=1, are never going to be adjusted.
          19    They're going to remain -- retain their
          20    sample weight of one, and never will it get
          21    to miss an adjustment, but it's not a linear,
          22    but not proportionate.
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             68
           1              In my example I have got myself --
           2    got cable.8.  The first by the weight of two
           3    goes done by only 10 percent to 1.8 but the
           4    weight of 20 goes down by almost 20 percent
           5    down to 16.2.  So basically the
           6    establishments with higher sample weights are
           7    more impacted by the adjustment than the
           8    cases with small sample weights.  And I just
           9    said that, okay actually in truth when we
          10    classify the population and develop a control
          11    total, we had almost a 1,000 establishments
          12    that switched strata within manufacturing.
          13    It's a little high than we expected but they
          14    have often verified.
          15              The overall impact on the estimate
          16    of the energy at the national level was to
          17    provide the down load by about 1.8 percent.
          18    About half this stratum was just up and half
          19    went down.  So we are pretty pleased it
          20    wasn't anything systematically suggesting
          21    that we had a upper bias, redundant bias and
          22    for the cells that were adjustable, that were
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             69
           1    adjusted, the average absolute adjustments a
           2    little under 4 percent well, 3.7 percent.
           3              So we weren't making major
           4    adjustments on most of the cells.  There were
           5    a handful of cells that went down by a 30 or
           6    40 percent.  The color adjustment, this has
           7    been done in the past, this is not a new
           8    activity for MECS.  The sample frame is a
           9    mail file, it does not include those were
          10    identified after we mailed out the census and
          11    not they are not part of the mail file, they
          12    are not put in the estimates and in addition
          13    there are cases that were mailed as retail or
          14    wholesale who were reclassified in the
          15    manufacturing.  Those cases again not in the
          16    sample frame, so they are not included in the
          17    estimate right now.  That's a problem that we
          18    are trying to address.
          19              Here is a frame, nice big graph and
          20    a pretty picture, and those are the sample we
          21    picked originally, again the sample of frame
          22    was a mail sample.  We have cases that became
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             70
           1    ineligible, these are cases that died, went
           2    on ---- or duplicates, cases where not
           3    manufacturing.  We didn't know them, that
           4    they were advanced, so we drew a sample of
           5    them.  So our sample represents the number of
           6    deaths in the actual frame, so these cases
           7    can be removed from MECS, and they were.
           8              But the births and the -- in the
           9    transport and manufacturing were not in a
          10    sample frame and yet they are still part of
          11    the total population.  We like to have them
          12    included in the estimates but we had no
          13    sample of those cases.  So we basically went
          14    through identified them all for about 15,000
          15    births and incoming transfers who were a part
          16    of the total population, which is one of the
          17    cases we did have cost of energy data and we
          18    developed a revised stratum cost of energy
          19    control total by simply adding the energy
          20    cost associated with these cases to the
          21    originally control total and develop a final
          22    adjustment where we had a revised control
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             71
           1    total cost of energy going up by the original
           2    control total.
           3              And then we get a final way and we
           4    applied that to establishments in each cell.
           5    This is done at each one of the cells levels.
           6    About 3/4 of the strata were adjusted upward.
           7    The remaining quarter, they were no births or
           8    there no incoming transfers, cases like
           9    chemical plants and refineries, pulp mills
          10    are not likely to have incoming transfers and
          11    their adjustment was about 3 percent.  So
          12    that point we finished the process we had
          13    gone through, we are adjusting our response
          14    that does it for the economic census and then
          15    adjusted for the coverage of the original
          16    sample frame and that's where its finished as
          17    we are right now.  These are the questions we
          18    got and I see these were in right ballpark,
          19    are we committing heresy here.
          20              Isn't that all that you recently
          21    found and we forgot something; we miss a big
          22    issue, and are your alternative that maybe
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             72
           1    more appropriate.  And the real one that I'm
           2    looking for is, what do we tell the EIA and
           3    their users about what we did, so they can
           4    figure out how to use data.  That's it.
           5    Questions, comments?
           6              MR. HOUGH:  You are the chair.
           7              MR. COLE:  I'm the chair.  Direct
           8    all the questions to Rick.
           9              MR. HSEN:  Stacey, you astounded
          10    them.
          11              MR. COLE:  I'm sure, I astounded,
          12    oh yeah, the great minds here I'm sure I
          13    astounded the great minds.
          14              DR. SITTER:  You chose post
          15    stratification, you really calibrating, I
          16    mean you using post stratification in
          17    calibrating.
          18              MR. COLE:  That right.
          19              DR. SITTER:  You look at
          20    calibration then?
          21              MR. COLE:  No, we do not look at
          22    calibration methods.
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             73
           1              DR. SITTER:  Some of the
           2    calibration methods will allow you to put
           3    constraints on them, which is essentially
           4    what you have, that is you have a constrained
           5    calibration problem.  I think I can probably
           6    come up with two or three references.  You
           7    can take a look at them.  I'm not saying that
           8    we better --
           9              MR. COLE:  Is that done by a
          10    variable, if it was a one calibration for the
          11    entire stratum, I didn't --
          12              DR. SITTER:  I think anything is
          13    possible, as just viewing it as a calibration
          14    problem may give you some insight as to where
          15    you're sitting, you may in fact be in
          16    calibration, if you can, I don't know.
          17              MR. COLE:  Okay.
          18              DR. SITTER:  I can give you a
          19    couple of references that I know of.
          20              MR. COLE:  Okay, appreciate that.
          21              DR. HENGARTNER:  Randy, I don't
          22    know that literature, how sensitive are these
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             74
           1    methods to randomness, I mean the way to get
           2    their attention random variable.
           3              MR. COLE:  Yes.
           4              DR. HENGARTNER:  Because the
           5    frames, well, we estimated them, the frames
           6    change into both. Does that have an impact?
           7              DR. SITTER:  Of course, what are
           8    you really meaning, are they really sensitive
           9    to that or something like that?
          10              DR. HENGARTNER:  I mean the analogy
          11    is it like errors and variables problem or
          12    not.
          13              MR. COLE:  I wouldn't classify
          14    myself as an expert.  John, he is my wizard.
          15              MR. SLANDA:  Well, we looked at the
          16    weights and the impact on variances --
          17              DR. FEDER:  Okay.
          18              MR. SLANDA:  And how they did that,
          19    we treated, we looked at -- we used the total
          20    cost of energy and then we used electricity
          21    as the variable from the actual survey data
          22    because we needed something to like a
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             75
           1    by-variant along with that and we did
           2    variances on the -- before and after the non
           3    response adjustment and after the K
           4    adjustment and about 80 percent of the
           5    variances went down.  But our big question
           6    became, how would me measures bias and that
           7    was something we hadn't come up with yet and
           8    we are so interested in knowing how to
           9    approach and attack that problem.
          10              DR. FEDER:  Actually you also
          11    talked about -- felt that the stage for my
          12    question is, could you do stimulations to
          13    examine the bias issue, because by looking at
          14    your formula it's not apparent to me that
          15    there is no bias here because of the
          16    selection issue because you are changing the
          17    weights in a disproportional way, so
          18    obviously the Horowitz-Thomson is out of the
          19    window, so I would recommend doing some
          20    stimulation and see again, if you get good
          21    results I think I certainly would say that
          22    the good paper here.
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             76
           1              Now I -- then you revise you
           2    approach, because I haven't seen something
           3    quite like it but it might make sense.  And
           4    actually not just bias or variance, I would
           5    look at them into -- together, because any
           6    calibration method might introduce some bias
           7    but reduce there miniscule error and this
           8    after all would be after, I mean, a little
           9    bit of bias is okay, I learnt from Randy.  I
          10    have questions for Randy although quite a bit
          11    older I mean his contribution is just obvious
          12    so --
          13              DR. SITTER:  Yeah, a bit
          14    intimidating he already had a PhD in
          15    mathematics.
          16              MR. SLANDA:  Well, I attempted to
          17    actually look into some of that simulation
          18    and I was wondering, have an idea I could
          19    just bounce off, see it works but I was
          20    thinking of using the actual survey data to
          21    get a regression but then also modeling some
          22    type of error term around the regression so
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             77
           1    that when we impute back, so then everybody
           2    not in the sample then would have much more
           3    variability as opposed to just having measure
           4    of size there because if we have the measure
           5    of size and we are doing simulation on that,
           6    then we get funny results too.
           7              DR. FEDER:  One of the things that
           8    you could also do -- stimulation is pretend
           9    that 20 percent you assembled that responded
          10    and 20 did not respond, take them out and try
          11    to figure a method that would give you
          12    anything close to what you actually have.  So
          13    --
          14              MR. SLANDA:  So label about 20
          15    percent of the population?
          16              DR. FEDER:  Yeah, I mean if you
          17    stimulate no response by making some you
          18    know -- you play God, you make them no
          19    respond, and then you see you will have to
          20    predict them the right way, you know and try
          21    to make it a bit informative, make that no
          22    response -- some known attributes of those --
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             78
           1              DR. HENGARTNER:  For example more
           2    non responses of the smaller companies will
           3    probably what we tend to generalize though,
           4    the step, you know, the assumption that
           5    moderately violated.  The same thing is that,
           6    we had the discussion about your estimate
           7    with those weights, are you using the highest
           8    estimate or the Horowitz-Thomson?
           9              MR. COLE:  Horowitz-Thomson.
          10              MR. SLANDA:  Horowitz-Thomson
          11    estimate.
          12              DR. HENGARTNER:  Okay, there was
          13    discussion and Randy suggested how it might
          14    be better for some cases.
          15              DR. FEDER:  Randy also suggested to
          16    you some calibration methods and once you
          17    calibrate if you include in you calibration
          18    constraints the population side, then you are
          19    in fact doing high.
          20              DR. HENGARTNER:  Yeah but then
          21    because you weights are random then you get
          22    really the errors and variables problem and
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             79
           1    then you have to worry about attenuation
           2    which is a few random weights.
           3              DR. FEDER:  It's a very complex
           4    thing.  That's why I think if you do
           5    something in relation it could be some
           6    cancellation of some of these errors.  I
           7    think it's and interesting approach affecting
           8    the issue but you know, that the people that
           9    represent themselves on the weight response
          10    would be not change their weight but go until
          11    --
          12              DR. SITTER:  I'll give you why the
          13    simulation is probably your best route I mean
          14    I stayed so bluntly that this confront into
          15    the class of constraint calibration problem,
          16    if you look at the papers that I would give
          17    to you none of them actually proved any
          18    reason, any consistency results under any
          19    reasonable framework as to what a consistency
          20    result should be.  Certainly they left the
          21    simple size go to infinity but what you
          22    really need, they discussed the consistency
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             80
           1    probably more realistic framework but it
           2    hasn't really been proved yet.
           3              And it really comes down to that
           4    constraint that is you -- because of those
           5    constraints, you're disproportionably
           6    adjusting the weights and that's really going
           7    to play a role and that you know, if you just
           8    sort of set up an obvious fixed framework as
           9    N goes to infinity, you are fine but you have
          10    to imagine a situation where those
          11    constraints are also not staying fixed with N
          12    and that I don't think anybody has actually
          13    come up with a ---- it's not that they can
          14    prove it, it's that you could come up with
          15    such a thing but it would be sort of made up
          16    with you know, what does it mean
          17    realistically to say as your sample size
          18    increases or your number of strata increases,
          19    what happens to those constraints.
          20              That's an issue, I like, in your
          21    situation your constraints are the certainty
          22    versus non certainty.  So you wanted to set
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             81
           1    up a asymptotic framework, you have to have
           2    some idea of what happens with those
           3    certainty versus non certainty weights as
           4    your sample size increases.  And that really
           5    impacts some products, so you are really
           6    stuck with simulations.
           7              DR. FEDER:  In this case your point
           8    is even more important because we are dealing
           9    with a very ---- population which is also the
          10    measure of size, the energy consumption, so I
          11    think when you do this disproportionate wages
          12    adjustment, it might have an attenuated
          13    effect and I think it is really worthwhile to
          14    examine.
          15              MR. HOUGH:  I have one other
          16    question to you, I certainly have time, we
          17    were able to do this this time because this
          18    -- MECS is currently conducted every four
          19    years.  The census is conducted every five.
          20    This is the first time that the two surveys
          21    fell in the same survey year.  So we were
          22    able to pull a sample from a mail file and
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             82
           1    stratify to a final file.
           2              In the future for example, in '06,
           3    we will pull the sample from the completed
           4    '02, which is what we stratified to here.
           5    But things like Stacy talked about will
           6    happen.  We do an annual survey on
           7    manufactures, which is a sample of the
           8    census, that kind of fills the gaps between
           9    census years for manufacturing data.  So the
          10    question would be, if we consider doing this
          11    in the future and the two surveys don't
          12    coincide, is there anything we should
          13    consider if we think about stratifying to
          14    another sample of the census.
          15              So we would want to use, say the
          16    annual survey manufactures are the control
          17    total?  Is there anything we should consider
          18    in doing that, I mean those actual control
          19    totals will now contain sampling errors is
          20    one thing, the estimates will themselves
          21    contain sampling error and the ASM itself is
          22    not really set up to give point estimates
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             83
           1    although they are fairly good.
           2              DR. SITTER:  Well, I think you are
           3    going to run into, I mean conceptually run
           4    into same issues that we saw yesterday with
           5    the Natural Gas essentially.  I mean there is
           6    going to be a change in the relationship over
           7    time because you sort of have certain
           8    determined your stratification on the basis
           9    of old information and the relationship is
          10    going to change over time.
          11              So looking at it back over time and
          12    what's happened you may see a pattern as to
          13    how those things might change.  I am speaking
          14    incredibly, vaguely your problem is much more
          15    complicated then the one that was presented,
          16    well, ultimately that's going to be the
          17    problem, it's a lag problem.
          18              MR. HOUGH:  Right.
          19              DR. FEDER:  And in '06 it will be
          20    -- the census data would be as old as
          21    possible because in '07 you're going to have
          22    another census --
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             84
           1              MR. HOUGH:  Correct.
           2              DR. FEDER:  So, but could you
           3    analysis the '06 data once you get the '07
           4    census in at least it will be more up-to-date
           5    than the '02 census.  Or reanalysis it and at
           6    least assess the impact of the uncertainty by
           7    the way, but I would at least try to analysis
           8    the '06 MECS with the '07 census.  We have
           9    been -- our surveys of individuals are a
          10    problem because we use census data which is
          11    only 10 years in the -- analyzing unlike in
          12    Canada and so we sometimes use a 10-year-old
          13    census to calibrate our population estimates.
          14    And as you know, there was quite a surprise
          15    because the last census -- some of the
          16    demographics, clearly we face the same
          17    problem.  But they do projections for the
          18    inter census years, I don't know if anything
          19    is prevalent here.
          20              DR. HENGARTNER:  The problem is of
          21    course because the countries addressing,
          22    right, so it's -- we make our justification
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             85
           1    based on signs or something like that?  What
           2    are you using for the first justification?
           3              MR. HOUGH:  For the control tables?
           4              DR. HENGARTNER:  Yes.
           5              MR. HOUGH:  Well, we would use the
           6    -- rederive the cost of energy from the
           7    annual survey manufacture, the most
           8    reasonably the survey manufactures.  So we
           9    will mail the 2006 MECS at the beginning of
          10    '07 and as we process that, the annual
          11    survey, I mean the 2006 annual survey
          12    manufactures will be processed.  So by the
          13    time we get to where we are ready to publish
          14    our estimates, we will have an '06 ASM data
          15    base to reestablish the control totals.  But
          16    my question was that since we were able to do
          17    it this time to the census which is the
          18    universe which doesn't contain any sampling
          19    error and doesn't you know, it contains all
          20    those smaller establishments.
          21              In the ASM, the smaller
          22    establishments are sampled with weights, so
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             86
           1    they're representative of other smaller
           2    establishments, so the control total would
           3    contain some level of error.  Whether or not
           4    we should consider that when we think about
           5    doing this again, you know, we could analysis
           6    the data certainly and look at.
           7              DR. FEDER:  Can I go back to the
           8    original question of the weight adjustment.
           9    Looking at the formula, I see one thing, I
          10    think someone, Nick said that it's likely or
          11    that as many small companies won't respond
          12    and the larger ones will but let's assume for
          13    the moment that the opposite happened.  So
          14    many companies that had to weight of close to
          15    one do not respond, the other company that
          16    had the weight close to one which are more
          17    like them, their weight will me adjusted much
          18    if you look at the formula, which ones will
          19    get adjusted, the ones that had the weight
          20    far away from one.  So I'm worried about bias
          21    here.
          22              And that's why a simulation of
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             87
           1    their first, can I propose, may not be the
           2    best because may be the way the population is
           3    it's not so bad and my concern is not so
           4    real.  If you take the other kind of
           5    validation by throwing away 20 percent of
           6    your sample, just saying 20, some part of the
           7    sampling, see the impact, you will able to do
           8    it and I really like what Rick suggested
           9    doing based on the weight.
          10              Make it's own respondents, two
          11    exercise, one with those that have weights
          12    are close to one and study the impact and one
          13    which would tend to be large companies.  Self
          14    representing, close to being self represented
          15    and then another exercise when you make the
          16    non-respondent fictitious with churn the
          17    respondents to ones that have weights far
          18    away from one and see the impact.  Because in
          19    theory, this estimate in my opinion would be
          20    general bias just because of the phenomena --
          21    knows nothing about because it's tends to not
          22    alter the weight of units that have weights
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             88
           1    close to one and those might be the ones that
           2    do not respond.  So we are not treating them
           3    in the same way, that's my concern.  It might
           4    not be a problem with your population, that's
           5    why I --
           6              MR. HOUGH:  The reality of the
           7    survey is that the coverage rate is about 89
           8    percent, so we are getting those large
           9    companies and the majority of the
          10    non-response is absolutely, definitely in the
          11    higher weight and smaller establishments --
          12    so I think if we do this simulation, we
          13    assume that some of the certainty cases are
          14    non response, you are going to see a lot
          15    different look to these ways.
          16              MR. COLE:  We know who the non
          17    response were.  And we could tell them to
          18    look at their size, and we could probably go
          19    through and say, well, looking at the
          20    respondents can we stimulate in our response
          21    to be a mirror image of the non respondents.
          22              If we say that half of the small
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             89
           1    cases didn't respond and look at the
           2    responding small cases and say well half of
           3    them had not responded either, what would
           4    have been the impact on the weights.  We do
           5    have -- most big cases do respond, most small
           6    cases don't respond.  Well most of the
           7    respondents are small.  But it think if we go
           8    through just stimulate random non response, I
           9    don't think it was random.  I think there was
          10    a pattern with small cases not responding.
          11              DR. FEDER:  So use what you know
          12    about the non response --
          13              MR. COLE:  Yeah.
          14              DR. FEDER:  Like the weight
          15    distribution and actually that might --
          16              MR. COLE:  Good point.
          17              DR. FEDER:  Support this approach
          18    because then you are saying that companies
          19    that have big size, their weight is close to
          20    one, their weight shouldn't be changed, so
          21    may be it's okay, I think we need to validate
          22    this but if you don't put this into
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             90
           1    consideration just on general principles,
           2    this is a biased approach but it's not if
           3    your big companies respond, big companies
           4    that have the weight close to one.
           5              MR. COLE:  Right.
           6              DR. FEDER:  And again a bias is not
           7    always so bad if we tell the miniscule error.
           8    That's why we have look that miniscule error
           9    are notified.
          10              DR. HENGARTNER:  Well, except that
          11    in this case the bias can be a problem
          12    because errors and variables, we have found a
          13    big consistency.  That means you will not
          14    estimate what you want to even
          15    asymptotically.  And so unless you are
          16    careful with what you are doing and that is I
          17    think what Randy is saying is that you need
          18    to careful how you do your asymptotically in
          19    the first place.
          20              So usually bias is another small
          21    problem because it's vote of one over the end
          22    and I went over the sample size where
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             91
           1    variance would have been throughout of the
           2    sample size though ---- but in cases where we
           3    need to prove consistency I feel worried
           4    about that a little more.
           5              MR. SLANDA:  I remember one NAICS
           6    by region that we are looking at, there was
           7    only one or two of them that was like that
           8    and they had the certainties reported and
           9    then there were some non certainties that
          10    were shifted in from another NAICS code
          11    region I think that's what I should say and
          12    they were non reporters and it ended up
          13    getting or actually maybe non reporters but
          14    they were non selects because they were in
          15    the population but not necessarily in the
          16    sample and the measure of size for the cost
          17    of energy was higher than that -- for one
          18    company was hirer than for any of the
          19    certainties.  So that cost just in that one
          20    cell it caused a little problem but the other
          21    cells were good.
          22              DR. HENGARTNER:  The other cells
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             92
           1    were good, okay.
           2              MR. SLANDA:  We had to look at that
           3    for a while.  It can happen.
           4              DR. HENGARTNER:  You need to be
           5    careful here.
           6              MS. WAUGH:  One of the other
           7    thoughts and it s related to the fourth
           8    question about metrics is that EIA publishes
           9    this data along with shadow tapes of the
          10    relative standard errors and I think it would
          11    be useful if the committee could address what
          12    tables EIA must want to publish that would
          13    communicate to the users about the errors
          14    associated with the data.  And the other
          15    thing you should know is that when they
          16    publish the data they may withhold it because
          17    of non-disclosure or they may not publish the
          18    data due to quality, so if they have a
          19    relative standard error of fifty, they won't
          20    publish the data.
          21              DR. SITTER:  Then there's no
          22    problem.
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             93
           1              MS. WAUGH:  Well, actually our data
           2    users may consider that a problem.
           3              DR. SITTER:  That's too hard, I
           4    think.
           5              DR. FEDER:  One of the things that
           6    I know in my limited experience, this is just
           7    a suspicion is that a lot of what we do in
           8    variance estimation is still based on the
           9    unrealistic assumption that the weights are
          10    in fact the inverse conclusion probabilities,
          11    in other words the Horwitz-Thompson weights
          12    and we ignore post stratification and
          13    response adjustment, to some extent, and
          14    implication variance although some people on
          15    this committee have done some work on that,
          16    but we don't practice what we preach on that,
          17    and there are a lot of things that we don't
          18    do, and we need to and I think a reality
          19    check is really a good idea.
          20              And again, I want to iterate again
          21    that part of it can be done by doing the
          22    exercise I've proposed, but you're right.
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             94
           1    But in practice people used the weights and
           2    say "if I have a weight of 2 that means the
           3    rate of incursion was 0.5.  And if the weight
           4    is smaller than 1, if someone say there is a
           5    problem, how do you defer that as one of our
           6    probability, one of the probability has to be
           7    taken a lot.  So I can't even, I mean you
           8    know, we fake it, sort of, I am as a I mean
           9    it's quite incomplete, it's state of the art,
          10    Randy am I correct in saying that. I believe
          11    that --
          12              DR. SITTER:  Oh yeah.
          13              DR. FEDER:  State of art is of
          14    variance.
          15              DR. SITTER:  No, I mean that in
          16    some cases, they'll -- you'll use replication
          17    methods like a jackknife or --
          18              DR. FEDER:  Yes.
          19              DR. SITTER:  A bootstrap or balance
          20    bootstrap and recalculate all of your
          21    adjustments as well and then hope that
          22    captures that.  But I don't think there is
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             95
           1    anything close to theory, if you were to do
           2    that in any realistic situation --
           3              DR. FEDER:  Uh-huh.
           4              DR. SITTER:  Like you're suggesting
           5    or even worse, I mean I've seen cases where
           6    they have got non-response adjustments, item
           7    non-response imputation and then some sort of
           8    calibration or - called G-weights,
           9    Generalized regression weights and then, you
          10    know, you just, okay, we'll just throw a
          11    replication method at it and recalculate them
          12    each time.  But you know, they can't, it
          13    can't possibly -- I mean somebody I don't
          14    know.  Well okay then if somebody now uses
          15    the data and they decide to take some of your
          16    vector of outputs and calculate I don't know
          17    correlation between two of them, is this
          18    going to be right?
          19              DR. FEDER:  And the issue of weight
          20    trimming, which people use to reduce the
          21    error and then sometimes the need hides under
          22    the carpet, the bias that it introduces, so
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             96
           1    one is to look at that and again simulation
           2    or replication methods can help, but it's a
           3    big -- Shawna, it's big problem.  I think,
           4    definitely it's something that we should pay
           5    more attention to and not just on the MECS
           6    Survey but the general.
           7              MR. HOUGH:  Well, where is the
           8    chair.
           9              CHAIRMAN BREIDT:  Okay, I will take
          10    over here.  Anymore questions from the
          11    audience?
          12              MR. FRENCH:  I was going to say one
          13    thing for context and that is, when you were
          14    mentioning the issue of thinking about post
          15    stratifying through ASM, I presume it's still
          16    50-60,000 establishments somewhere in that
          17    line.
          18              MR. SLANDA:  Yes.
          19              MS. WAUGH:  Yes.
          20              MR. FRENCH:  Whereas the MECS is
          21    15,000 and four percent of 100,000 and there
          22    is something else I want to say about the ASM
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             97
           1    too.
           2              MR. SLANDA:  It's optimized.
           3              MR. FRENCH:  Yeah that's true it is
           4    optimized for economic rather than energy so,
           5    I don't know whether that would have anything
           6    to do with the thought of using a post
           7    stratification where the numerator is now at
           8    different type of estimates based on a
           9    different type of sample other than a --
          10              DR. SITTER:  But it's done
          11    essentially independently, correct?
          12              MR. COLE:  Yes.
          13              MR. FRENCH:  Yes.
          14              DR. SITTER:  And you are really
          15    falling into the sort of two phase sampling
          16    idea, I mean if you would do the simplest
          17    possible calibration on a previous survey
          18    based on a bigger sample, therefore you think
          19    it is more accurate than you're really in a
          20    two-phase situation.  So I'm sure that that's
          21    essentially what you are ending up in and now
          22    sort of ignoring the variability of that
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             98
           1    other survey amounts to saying that it's
           2    sample size is been in order, - is in order
           3    larger.  If that's not true, or if you don't,
           4    if you feel the variability of that maybe an
           5    issue then --
           6              MR. HOUGH:  Well, I'm comfortable
           7    with that definitely the size of the
           8    establishment and energy use are correlated
           9    and the ASM will pick up the largest
          10    establishments because those are the ones
          11    that have the most effect on economic data.
          12    So we will get a large percent of our cost of
          13    energy from the ASM.
          14              DR. SITTER:  Uh-huh.
          15              MR. HOUGH:  I believe that to be an
          16    accurate statement.
          17              DR. SITTER:  But I'm talking about
          18    the translation in the variation really, you
          19    know which can have to do with the weights
          20    and other things, so.
          21              MR. HOUGH:  Right.
          22              DR. SITTER:  But it -- the
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                             99
           1    calculation of it's impact is easy, because
           2    they're independent.  Well, easy.  If it were
           3    a calibration it would be easy, you have a
           4    lot of constraints and so forth, but in
           5    principle, you just sort of have two pieces
           6    of a non-linear function, do a Taylor Series
           7    expansion, you can sort of roughly, get a
           8    feel for what the variance that you're
           9    missing, what variance you are dropping out,
          10    and you may be able to answer the question
          11    what I'll be doing if that variance is
          12    relative.  The part I'm estimating or can I
          13    estimate it, harder estimates there, things
          14    like that.
          15              DR. FEDER:  Go ahead.
          16              MS. WAUGH:  Could you post stratify
          17    on the certainty cases and not on the
          18    non-certainty cases, since that might be
          19    common to both surveys?
          20              MR. HOUGH:  That would be different
          21    from what we did here.  Actually, basically
          22    what this post stratification does is it
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            100
           1    measures how effective, what did you have to
           2    surpass that mission Kaleen.
           3              MR. COLE:  How well the ----
           4    population, so referred to, they're already
           5    self representing.  I'm not sure you would
           6    have anything you can do productive.
           7              MR. HOUGH:  Unless their energy --
           8              MR. COLE:  This sounds like --
           9              MR. HOUGH:  Unless there is such
          10    as, like for example, we pull a sample, if
          11    the 2005 ASM data is available to us, we take
          12    the nearest piece of data to derive and
          13    measure size.  Now if the Survey year 2006
          14    shows some spike or decrease in one of the
          15    strata's of these certainty cases, then you'd
          16    - - that would definitely have an effect,
          17    because the initial measure or size that we
          18    sample on and wait on would be from the 2005
          19    ASM.
          20              And if some condition and whatever
          21    is going, the largest 1000 have to drop out
          22    or shut down, that would affect the strata to
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            101
           1    the point where the control total from '06 is
           2    drastically different, then that may have an
           3    effect.  But if the strata remains relatively
           4    level from '05 to '06, the certainty cases
           5    are driving those estimates.  I don't know if
           6    the weight effect would be, can you think of
           7    anything?
           8              MR. COLE:  I think that the process
           9    should, not only adjusting the weights of the
          10    non-certainties there is no impact for
          11    certainties, they are basically wiped out, in
          12    a way that it's not there.
          13              MR. HOUGH:  Okay, it's different
          14    from one doesn't mean we have a bad estimate
          15    itself, it means we have bad samples on
          16    certainty but the total being driven by
          17    certainty, in that case, you still have a --
          18              SPEAKER:  Such as what?
          19              MR. HOUGH:  You still have a very
          20    good estimate, even though we had bad samples
          21    on certainties.
          22              DR. FEDER:  Rick, you asked this
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            102
           1    question about what to do just when the
           2    survey data are old and you know, like an old
           3    thing, - I want to mention some things that
           4    my colleague Jim Kromy, once -- he had some
           5    thoughts, he said to me, you know the census
           6    data, I'm talking about the Census of
           7    population is conducted every ten years and
           8    what do you do, that's nine years after the
           9    Census, we use the Census Projection and we
          10    use them to calibrate our large scale survey
          11    which has about seventy thousand respondents,
          12    I'm talking about the household survey of
          13    drug use, which is a distant theory has
          14    unbiased estimates.
          15              So why do you calibrate an unbiased
          16    large scale sample which is - supposedly has
          17    small min - small variants, no bias and you
          18    calibrate it to census that, I know your
          19    colleagues are doing a terrific job in doing
          20    the interferential projections, but they are
          21    old, nine years old and what we do and that's
          22    what we do in practice, we calibrate it and
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            103
           1    we treat the Census numbers as being
           2    non-random.  No error on variables made, and
           3    is that correct or not and there's a lots of
           4    talks about it and one thing that I said why
           5    don't we do the next year when we have the
           6    census data go back and check our
           7    assumptions.  And I would suggest the same
           8    thing to you.  We do a lot of think--
           9              MR. HOUGH:  I think that's the best
          10    we've got.
          11              DR. FEDER:  Because we are dealing
          12    with reality and uncertainty in a complex
          13    statistical estimation that sometimes
          14    requires maybe some refinement.  But those
          15    questions are not only with your situation;
          16    we face them in our cases too.
          17              DR. HENGARTNER:  Isn't there going
          18    to be a problem with your suggestion that by
          19    going to next year's inventory or census,
          20    some companies may not exist anymore at that
          21    time?
          22              DR. FEDER:  But if you go back four
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            104
           1    years there is a greater problem, you have
           2    one either way.
           3              DR. HENGARTNER:  All right.
           4              DR. FEDER:  Yes.
           5              MR. HOUGH:  It's always an issue
           6    with the survey, the timing.  You're always
           7    drawing the sample problem, the data census
           8    that will be updated as we process our
           9    survey.
          10              DR. SITTER:  Do you have a public
          11    perception issue, is that one of the reasons
          12    that you're calibrating or is it not a matter
          13    to you, because I think that in some of the
          14    cases that Moshe is talking about it's
          15    probably a public perception issue.  That is
          16    there are some key variables and if you put
          17    out estimates that's different from the
          18    previous census, that is you put it out and
          19    they take your weights and calculate it and
          20    you are way off on some key demographic
          21    characteristics or something--
          22              DR. FEDER:  Well they --
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            105
           1              DR. SITTER:  That brings in things
           2    into question.
           3              DR. FEDER:  Our published estimates
           4    are on drug use, mental health and things
           5    like that, so there's no issue there, but
           6              DR. SITTER:  Well, there is --
           7              DR. FEDER:  But we have tried to as
           8    a quality measure to see, what if you go back
           9    and estimate.  In fact well, I cannot talk
          10    about specifics, but I've done some
          11    experiments and I didn't like the results and
          12    I talked about it, because, it has to do with
          13    I think with the state of the art of
          14    calibration which is still evolving, because,
          15    I remember actually again if I might digress
          16    a little bit, when I came to STATS Canada I
          17    was new to statistics and taking courses from
          18    Randy, I -- people were still doing
          19    regression type of calibration where negative
          20    weights were generated.
          21              We're talking about something which
          22    is quite recent, we're talking about
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            106
           1    mid-nineties and people said "Hey how can I",
           2    I mean you're worried about the weights
           3    smaller than one, people said "How can I have
           4    a negative weight?" and I remember a
           5    situation, we did some projection for certain
           6    region of Canada, where Dwight had turned up
           7    and we said, "you know you don't have to go
           8    down because of calibration", he said "no, no
           9    that's wrong", so people said "you know, you
          10    can do smaller estimation if you have
          11    negative weights because some small areas get
          12    negative estimates of total of a positive
          13    quantity", so then all my colleagues who were
          14    much more advanced in this topic than I was
          15    came up with wrecking ratio adjustments or
          16    what we call in Britain, SPREE and some other
          17    methods of doing calibration that guarantees
          18    positive weights.
          19              Then people were happy for a while,
          20    then they said, no, weight has to be greater
          21    than one.  Okay so I think and then you go to
          22    cross extreme weights and they had leverage
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            107
           1    on the various estimates, we are still moving
           2    I think as a -- I call myself a young
           3    statistician because I haven't been doing
           4    statistics for that long.  I see a lot of
           5    progress in that but it's still quite in
           6    evolution.
           7              DR. SITTER:  And more recently
           8    there's a empirical likelihood methods to
           9    guarantee positive weights, there is some
          10    rich aggression methods, there is quite a lot
          11    of work on this where you want these weight
          12    adjustments to have constraints.  I think
          13    there is some literature there but --, and I
          14    think that a lot of that literature is driven
          15    by real problems but I don't think that it
          16    addresses the real problems.
          17              I think they are still at this
          18    stage where they're talking about theoretical
          19    exercises.  So at least your problem is hard.
          20    Well that's good news for some things, you
          21    come in and you say gees did we do it right
          22    and then people say wow it's hard.  That's
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            108
           1    good there.
           2              DR. HENGARTNER:  The last question
           3    I have is, why can't we sync those two
           4    studies?  I mean they were in sync, now that
           5    sync works, why don't we just keep it.
           6              MS. DODDS:  They just barely work.
           7              MR. FRENCH:  And keep the census
           8    and MECS relative during the same year.
           9              MR. HOUGH:  We have to go back to
          10    the back there.
          11              MR. FRENCH:  I think we've already
          12    expanded the interval between the MECS year
          13    to the point where we are worried about
          14    relevance, so it's every 4 years now, and for
          15    it to be in sync with the census, it would be
          16    every 5 years --
          17              MR. HSEN:  And we don't have an
          18    annual manufacturing survey in the interim.
          19              MR. FRENCH:  Correct it's a problem
          20    that, right now I don't think we know what
          21    the --
          22              MS. DODDS:  Yeah in fact I thought
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            109
           1    your association with MECS --
           2              MR. FRENCH:  Yeah and expand the
           3    sample size --
           4              MR. HOUGH:  I mean, this is going
           5    to exist, this situation is going to exist,
           6    like I said every time we do this survey,
           7    this is always going to be the post
           8    stratifying to a new control total, it's
           9    always going to be an option, the way the
          10    surveys are done and the way our surveys are
          11    done.  Next time we come across, we came up
          12    with this method for when we had the census
          13    this time, that's not going to happen for 20
          14    more years.
          15              DR. FEDER:  But I think we are
          16    getting here into the design issues, may be
          17    we should put it on the committee's agenda
          18    for next time because you could think about
          19    many ways maybe like a rolling survey where
          20    every one of those 4 years or 3 years you do
          21    part of the sample so you have a way to
          22    measure change, maybe a panel, a survey
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            110
           1    whenever a company is observed, maybe at
           2    least a couple of times we can measure change
           3    more effectively, I mean there is a lot of
           4    things so, maybe what we should do is put it
           5    on next meeting in April, I think it'll be
           6    and discuss not just the estimation issue but
           7    also the design issue.
           8              DR. SITTER:  How stable are things
           9    over the years?  Have you looked at patterns
          10    in terms of the relationships from census to
          11    your survey?  I mean in some sense having it
          12    badly coordinated with the census can give
          13    you sort or more data in a sense that you
          14    know, you -- if things are reasonably stable
          15    then you have some relationship between how
          16    things deteriorate from the census over time.
          17    I mean we are talking about long time windows
          18    here and I don't know your industry that well
          19    but did you have some data?
          20              MR. HOUGH:  Well the energy data is
          21    going to be correlated with the production so
          22    the industry has increase in revenue; we
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            111
           1    usually see the energy usage go in the same
           2    direction.  Over the past 4 years it's been
           3    going, it's been on a decline and that's the
           4    result that we've had this time.
           5              MR. COLE:  Is that quantity-based
           6    or is that value based?
           7              MR. HOUGH:  It has to be value
           8    based, that's what it is based on -- that's
           9    what the measure and size is based on values,
          10    that's all -- that's all we've got but then
          11    MECS collects quantity detail as well as,
          12    dollar expenditure so - -
          13              DR. SITTER:  Yeah and your economic
          14    cycles are quite often quite a bit quicker
          15    than your census.
          16              MR. FRENCH:  And that doesn't get
          17    into the issue of the fact that MECS collects
          18    things like non-purchased fuels and other
          19    things that's used in very different places,
          20    the costs numbers which are based on
          21    purchases --
          22              MR. HOUGH:  We collect quite a more
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            112
           1    -- we collect and publish quite a bit more
           2    detail then we sample.
           3              MR. FRENCH:  I just want to throw
           4    another monkey away.
           5              MR. HOUGH:  We don't want to talk
           6    about what Dwight just said.
           7              SPEAKER:  So are we done?
           8              SPEAKER:  Yes, we are.
           9              SPEAKER:  All right.
          10              CHAIRMAN BREIDT:  So our first
          11    summarizer will be Moshe Feder who will talk
          12    about post stratification methodology -- hey.
          13              DR. FEDER:  With a very interesting
          14    description of a problem with the MECS
          15    surveys, say I've to look there, it's the
          16    Manufacturers Energy consumption Survey,
          17    which uses the economic census conducted
          18    every five years on business they did it
          19    October 7, to construct a calibration
          20    weights.  And in 2002, the two surveys
          21    occurred at the same time because the MECS is
          22    conducted now every four years, it used to be
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            113
           1    every three years since 1985, and 2002 was
           2    unique in the sense that every 20 years of
           3    the current design was to survey, as I said,
           4    the survey and the census coincides.
           5              So there was an opportunity also to
           6    use the economic census in calibrating the
           7    weight.  Now, the, what we heard is about an
           8    approach to, first of all, I should go
           9    through the steps of the weight adjustments
          10    and there are three of them.  First they do
          11    the non-response adjustments, which is a
          12    simpler simple ratio adjustment that to
          13    calibrate the weight to the census -- to the
          14    weight of census total energy consumption.
          15              The next step is a post
          16    stratification adjustment and that's followed
          17    by a coverage adjustment, because there's a
          18    typical, I guess with this kind of
          19    establishment as per new companies that are
          20    created, that are not covered by the survey.
          21    There's also some companies that transferring
          22    the manufacturing, which in the census says
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            114
           1    that is also a birth.  There's also death,
           2    but the deaths are caught by the center
           3    because when you, a survey or -- a company
           4    turns out to no longer be in business or not
           5    doing manufacturing that you can account for
           6    that.  So all those things lead to a coverage
           7    adjustment.
           8              And the goal of the study was to
           9    come up with a certain weight adjustment,
          10    that will have two features.  One, it will
          11    calibrate to the census total weight of
          12    consumption figures and also they wanted to
          13    guarantee that units that were self
          14    representing in the sense that their analytic
          15    weight was equal to one, their weight
          16    wouldn't change.  So what they proposed is a
          17    new weighted weight, which is actually, it's
          18    in your handout, it's equal to 1 plus K times
          19    the old weight minus 1.
          20              And if you look at the formula as
          21    to weight is equal to one, that the
          22    adjustment is one and the weight remains the
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            115
           1    same.  And there was some concern about the
           2    statistical property of those adjustments and
           3    what impact it will have on the standard
           4    errors.  Somebody mentioned that EIA
           5    publishes this every table, a scheduled table
           6    of standard errors and that also used in
           7    determine -- determination whether figures
           8    should be released or not based on quality
           9    and other consideration.  So the committee
          10    was asked a few questions related to that.
          11              And also another important question
          12    that was raised by the presenters was that
          13    2002, what happened here like this perfect
          14    storm of the two, two things happening at the
          15    same time, the survey and the census will not
          16    occur again for another 30 years, so what do
          17    you do, with the censile information is old
          18    like the next time MECS is going to be
          19    conducted is '06.  But the last census then
          20    would still be '02, so what do you do about
          21    that.
          22              And there were some other related
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            116
           1    questions and a lively discussion about what
           2    would happen here.  And if I may quote, one
           3    of the committee members said, the problem is
           4    actually hard and some of it is not clear and
           5    we were also asked, have you seen this method
           6    before, and the answer was not exactly like
           7    that.  It was mentioned that the calibration,
           8    the concern about calibrated weights being
           9    greater than one is actually quite common.
          10    It's done in most long surveys and are
          11    procedures to do it.
          12              So a committee member suggested to
          13    look at that and maybe use some other
          14    calibration method.  But the question still
          15    remain, how do you evaluate the current
          16    method.  There was some question whether
          17    there is or isn't a bias, and a simulation
          18    might help here, another suggestion that was
          19    made was to perhaps take the 100% of the
          20    sample, set aside, say 20% of it.  Use the
          21    80, treat them as if they were
          22    non-respondents, apply the method to the 80%
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            117
           1    and validate your prediction for those
           2    missing units using the new weights.  I wont
           3    go in to the mathematics but there were some
           4    issues that if units that have the weight
           5    close to one are different from the ones
           6    whose weights are far greater than one, then
           7    a bias would occur, so that's one way to
           8    check that.
           9              Let's see if, yes, there was some
          10    issue here about the -- about the weights,
          11    the randomness of the weights should be
          12    treated as error and variables issue which
          13    can lead to bias even asymptotically, as was
          14    pointed out and that's a hard problem and
          15    this was mentioned that in all practical
          16    applications, this is the problem and there's
          17    no clear solution to that.
          18              And there was some other discussion
          19    about using the ASM, which is independent of
          20    the survey in doing the weight adjustments
          21    and the calculation, it shouldn't be very
          22    hard because of the independence.  So that's
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            118
           1    another suggestion that was made by someone
           2    in the.
           3              Audience.  Finally, the discussion
           4    in particular was the issue of the census not
           5    being at the same time as the survey.  It was
           6    mentioned that they had the design issue and
           7    maybe the survey could redesigned, maybe
           8    having a part of the sample data in each
           9    year, rather than having it once every four
          10    years or so, what I am -- we were wondering,
          11    perhaps it could be put to the agenda for the
          12    next meeting in April or in the Spring.
          13              CHAIRMAN BREIDT:  Any questions,
          14    comments or corrections?  Okay, thanks Moshe.
          15    So next Nagaraj will summarize Thursday's
          16    break out session.
          17              DR. NEERCHAL:  We were looking at
          18    the EIA 906 and the one that Joe was talking
          19    about earlier and the data is being entered
          20    through an internet data collection, that's
          21    IDC, IDC systems.  Another -- basically
          22    monthly data is being entered and the people
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            119
           1    are asked to enter their information on fuel
           2    consumption, generation and stocks of fuels
           3    on every month.  And within -- I think within
           4    45 days of the end of the month they're
           5    supposed to enter the data and the -- while
           6    they're entering the data you want to give
           7    them comment, so to help them -- the idea is
           8    to help them enter the correct data.
           9              So that we can kind of cut down
          10    your edit work later on and so that you get
          11    the data in, person who is supplying the data
          12    give you as correct a data as possible, that
          13    is the overall objective.  This gives rise to
          14    a very complex database, because every plant
          15    has several types of prime movers and each
          16    prime mover has several types of fuels that
          17    it can consume and so on.  So we have to --
          18    they have to fill in a form like this with a
          19    lot of different units going down there and
          20    different fuels for each unit and so on.  So
          21    it gives rise to a very complex database.
          22              The current system I think, the
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            120
           1    current edit that is being applied right now
           2    is essentially data plant taking up data
           3    plant and compute the mean and standard
           4    deviation for that over a 24- month period.
           5    And compute an interval like mean plus or
           6    minus 1.5 standard deviation, and if the
           7    entered data is outside that, then flag it
           8    and invite them to correct the data points,
           9    and give them mild warning saying that the
          10    data you have entered is not correct or not
          11    in the range perhaps and so on.
          12              So one of the problems of this
          13    system is that you know, if you have
          14    autocorrelation in your data for example, you
          15    might miss it, or seasonality in the data you
          16    might miss it.  And of course you know, there
          17    are some modification and this is what I told
          18    you is the simplest form of the edit.  There
          19    are some, you know further checks on the --
          20    this system, this method to incorporate
          21    seasonality and so on but still it seems to
          22    me, you know, the autocorrelation part might
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            121
           1    be missed completely.
           2              So that's why the proposal on the
           3    -- in the paper right now is to look at
           4    something like, simple exponential smoothing,
           5    that we capture the autocorrelation.  And you
           6    know, help you pick up some outliers that you
           7    may not normally consider outliers.  So
           8    number of questions to the committee on this
           9    one is to -- I think the primary question is
          10    how to choose the best alpha.  That is one of
          11    them and alpha will be a smoothing parameter,
          12    or another -- another way of putting up the
          13    same question is that, are you happy with the
          14    alpha you chose?
          15              And also another aspect that I know
          16    whether you have a handout here, I think if
          17    you look at the picture, if you compute the
          18    smoothing parameter for every plant and make
          19    a histogram, that sort of looks like a
          20    bimodal thing like this for some of the
          21    parameters.  And why do we get lots of zero
          22    alphas and lots of 1 alphas and what does it
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            122
           1    mean and so on.
           2              In this case it so happens that if
           3    you have alpha that is very close to zero,
           4    then you're sort of back to the mean edit
           5    situation.  And if alpha very close to 1,
           6    that means that you don't trust the mean very
           7    much because you want to startup
           8    autocorrelation in the data.  So some of the
           9    -- some of the decision, I mean there was a
          10    lot of discussion, I think, in this case I
          11    think committee really acted like a
          12    committee.  I think we really did not have a
          13    solution for any of the bullets.  So we have
          14    a lot of suggestions today, and that's more
          15    like the all the, you know, all the non and
          16    in the PHD committee, all the non-chairs do
          17    that, right?  Chair is the only one who
          18    worries about getting it done.  So -- that is
          19    what we were acting like the other committee
          20    members.
          21              And so, the issue I think is that
          22    -- you know think overall we felt very
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            123
           1    comfortable with the proposal.  Because
           2    especially it picks up alpha and you know
           3    that it is really kind of an improvement on
           4    what you're doing right now.  You're not
           5    really completely throwing away what is being
           6    done and starting something new.  It is an
           7    improvement on what is being done right now,
           8    so it seems like it is something definitely
           9    that remains to be explored.  But one of the
          10    critical issue here is, we don't know who is
          11    right.
          12              You know, suppose you apply this
          13    old edit as well as the new edit and the old
          14    edit kicks this out and the new edit kicks, I
          15    mean does not kick it out, we don't know who
          16    is right, who is wrong.  That is one of the
          17    critical issues.  It may be worthwhile coming
          18    up with a data set that you may want to say,
          19    okay, we all agree that these are the correct
          20    values, now let us evaluate the different
          21    edit systems.  It may be a worthwhile
          22    exercise to spend sometime coming up with a
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            124
           1    "test database" where you feel comfortable in
           2    accepting some values as the correct values.
           3              Now, that is one of the things, and
           4    as far as estimating alpha right now as one
           5    of the heat I would say is that the smoothing
           6    parameters are estimated using SAS.  You
           7    know, I'm sure there are many softwares that
           8    can help you in that one but I think SAS is
           9    being used, but then again you know, it seems
          10    most of the simple things like kicking it out
          11    and kicking it in and things like that are
          12    probably done on a spreadsheet and you have
          13    to combine the SAS output with the
          14    spreadsheet, send it to somebody else and so
          15    on.
          16              So one of the suggestions might be
          17    to use a simple estimate that is -- can be
          18    done in like a one line for moving average
          19    smoothing parameter, we can write down a
          20    formula for you know, getting a decent
          21    estimate for the alpha, instead of using a
          22    sophisticated software.  So that is something
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            125
           1    that, if you do that, then your ability to
           2    manipulate lot of these things will probably
           3    go up.  The other suggestion was to consider
           4    a grid of alpha.
           5              Not just you know, if the estimated
           6    alpha is something, then simple estimate is
           7    some 0.3, then go down a few, go up a few or,
           8    you know, and see well how this, how many,
           9    what percentage of the flagging and so on.
          10    Yet another suggestion we got was, perhaps if
          11    you look at some of the plots there are too
          12    many large numbers, lots of zeros and large,
          13    you know, it's kind of a oscillating thing.
          14    So one might want to consider some
          15    transformation of the data, so that something
          16    that is not being buried somewhere, some are
          17    data points.  Then that is some of the
          18    information we got.
          19              But we ended up discussing a lot
          20    about, you know, how can we incorporate some
          21    of the other information that is available,
          22    like, the type of fuel for example.  Use of
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            126
           1    major fuel used by the plant or --
           2              MS. KHANNA:  Weather.
           3              DR. NEERCHAL:  Weather perhaps, and
           4    other information that you know, some
           5    auxiliary information that can be used for
           6    edits and there were a lot of suggestions and
           7    I think fortunately we were -- we had both
           8    the people there, they seemed to be nodding
           9    and saying that maybe we can try that you
          10    know, we have no idea what we were saying
          11    right or wrong, but I think, when you can see
          12    them nodding, I said, okay, not too bad.
          13    Because we didn't have the data in front of
          14    us so, some of these things may be just a
          15    pipe dream.
          16              For example, one of the slightly
          17    more complicated suggestion that came out
          18    was, for certain types of plants there is a
          19    special pattern that kind of goes on line and
          20    come -- goes to zero for a while, comes back
          21    on line for little bit and goes back to zero.
          22    Now we know, and the database also tells you
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            127
           1    whether that is, you know, the plant you're
           2    trying to edit is of one of that type.  So
           3    you may want to build in a memory sort of, we
           4    know, are you, speaking plan for example.
           5    You know we may do -- build in another one,
           6    so you just entered zero, is it, you know,
           7    supposed to be zero that, so some special
           8    edits like that, some addition, additional
           9    suggestions on that one.
          10              So overall I think committee seems
          11    to be very excited with the project.  I mean
          12    it was lot of excitement and the, and I
          13    think, specific question about model
          14    selection, I think we were basically
          15    skirting.  That may not be that important an
          16    issue to look at in this case.
          17              I think this specific bullet apart,
          18    whether should we consider mean absolute
          19    prediction error or mean square error?  I
          20    mean that may not be that critical in this
          21    case, I think there are lot of other issues
          22    that will be, you know, better to focus your
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            128
           1    energy on, that's the overall feedback based
           2    on it.
           3              CHAIRMAN BREIDT:  Okay.  Thanks
           4    Nagaraj.  Are there comments or questions or
           5    corrections?
           6              DR. FEDER:  I wasn't at the meeting
           7    but I looked at the handout and I want to
           8    make a few comments.  Just that they're all
           9    focused on the time series adjustments.  One
          10    thing is said, if you look at the model, it
          11    assumes stationarity of this, no allowance
          12    for snow and there's a method by
          13    Holton-Winters which, I mean it convinces(?)
          14    it's in Howry's book on structural time
          15    series models, which is also very simple mind
          16    it, more than, very simple.  It just allows
          17    for a slope and there's another version, it
          18    also allows for seasonality which might be
          19    appropriate here.
          20              I would like to not have too many
          21    alphas but I would, because of their pattern
          22    that Nagaraj was talking about with some
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            129
           1    plants phasing in and phasing out to do
           2    production, maybe stratified by something and
           3    then just treat the ones that exhibit more
           4    regular pattern, perhaps with or without
           5    seasonality, I don't know enough about those
           6    plans to say one way or the other.
           7              And finally the selection of the
           8    alpha are we seeing as (?)IBEC in Canada had
           9    a forecasting issue when we also dealt with a
          10    very short time series.  Actually we ended up
          11    using something like a variant of the common
          12    filter but the way to do it, also with
          13    Holt-Winters by using a penalty function.  By
          14    that I mean to following.  You start with the
          15    series what you know and every -- for every
          16    time point you look at the Holt-Winters on,
          17    the exponential smoothing, best estimates for
          18    that time point, look at the difference,
          19    that's called the innovation between what you
          20    observed and what you predicted, you square
          21    it, you edit up, that's your penalty.
          22              Of course you look to minimize
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            130
           1    that, you want your prediction to be what you
           2    observe.  And the alpha that is best -- that
           3    minimizes that is your choice of alpha.  It's
           4    sort of an objective way of coming up with
           5    alpha which is very easy to program and we
           6    had the satisfactory results with that.
           7              But I think having too many alphas
           8    defeats the whole purpose, because it will
           9    tend to favor I think the last observation
          10    rather than a smooth estimate.  But certainly
          11    I think because of the nature of those time
          12    series, I would allow for a slope and for
          13    seasonality.
          14              CHAIRMAN BREIDT:  The exponential,
          15    some of it is allowed for a local linear
          16    change, so it's --
          17              DR. FEDER:  Yeah that's -- that's
          18    just included in the Holt-Winters.  I mean --
          19    Holt- Winters is exponential -- or maybe I'm
          20    wrong about the terminology.
          21              CHAIRMAN BREIDT:  Yeah, not on
          22    perimetric change, but local linear changes,
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            131
           1    that's what the exponential smoothers on.
           2              DR. FEDER:  In the model they had
           3    there was no allowance for that one --
           4              CHAIRMAN BREIDT:  What I mean is a
           5    stochastic change, a local linear change.
           6    Not a perimetric change.
           7              MS. KIRKENDALL:  It is adaptive,
           8    it's just -- if there is a real trend in the
           9    day, then it's not good enough.  But if it's
          10    just some changing over time -- then it --
          11              DR. FEDER:  Oh yeah.
          12              MS. KIRKENDALL:  It is a change.
          13    We haven't seen many trends in this area.  We
          14    see some seasonality in the data.
          15              DR. FEDER:  Okay.
          16              DR. HENGARTNER:  Explicitly more
          17    than the seasonality of them, smoothing the
          18    residuals using the exponential weighting
          19    might actually be a good idea.  Because to
          20    control better the variance, that way.
          21              MS. KIRKENDALL:  I think we --
          22              DR. NEERCHAL:  One of the
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            132
           1    suggestions the committee had was to look at
           2    some seasonal versions of the moving average
           3    and we're not --
           4              MS. KIRKENDALL:  Yes, here I think
           5    we had a brief discussion about this and one
           6    of the things to remember is that, these
           7    edits on at a facility level, so you'd have
           8    to the seasonality on whatever structure that
           9    you're talking about, each facility
          10    separately.  And so this is not you know,
          11    like you're doing cross section across
          12    facilities.  So that just increases, you
          13    know, and that's an enormous preposition.
          14              DR. FEDER:  We'd have too many
          15    problems because we did with, but why can't
          16    you assume the same seasonality within the
          17    structure?
          18              DR. HENGARTNER:  Well, it is like
          19    what Joe was talking about earlier today,
          20    cluster residual companies.
          21              CHAIRMAN BREIDT:  These are issues
          22    that were discussed.
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            133
           1              DR. HENGARTNER:  Okay, I'm sorry.
           2    We had a good discussion here, too.
           3              CHAIRMAN BREIDT:  Okay, any other
           4    further comments or questions.  Since we've
           5    already had a warm up break, I suggest that
           6    we don't take a break now and just move right
           7    into the next section, that's all right,
           8    which would be the discussion on --
           9              SPEAKER:  Yeah, I understand, yeah
          10    I understand, that's right.
          11              SPEAKER:  This is the last chance
          12    to --
          13              CHAIRMAN BREIDT:  So this is the
          14    session, if you were king brought by Howard
          15    Gruenspecth.
          16              MR. GRUENSPECHT:  That's one way to
          17    do it.  Anyway I was in West Virginia
          18    yesterday, that's why I was not here.  West
          19    Virginia is a fairly affluent state, which --
          20              SPEAKER:  Yeah, you would never get
          21    involved in it.
          22              MR. GRUENSPECHT:  It is important
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            134
           1    and they were having a conference and they
           2    asked for representation.  I did talk with
           3    him last night, he indicated that this
           4    meeting is so great.  So future meetings,
           5    hope we won't be able to talk about, but it's
           6    nice to be here with you.  But you know, as
           7    you know we're periodically reassessing how
           8    we can continue to fulfill our mission?
           9              And again it's a tough place to be
          10    King or Crown Prince or whatever role we play
          11    here because, you know, we do have this
          12    demographic issue that in fact you know where
          13    lots of technical skills combine the needs in
          14    the areas of expertise, it is a growing you
          15    know, interdependence and nationalization of
          16    the world economy and the energy
          17    environments.  I think that -- try and look
          18    at this years blow out in the oil markets
          19    where I think really demand, well,
          20    particularly Asia that was not anticipated I
          21    think by anybody and the data available was
          22    quite poor.
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            135
           1              There's a lot of concern about the
           2    quality of international data in this area.
           3    There's a joint oil data initiative underway
           4    to try to improve it.  So there was lot of
           5    pressure to improve the quality of data.
           6    There is unrelenting demand for product
           7    promos, service reports, data and model
           8    improvements.  I guess, yesterday you heard a
           9    lot about the STEO, the regionalization of
          10    the Short Term Energy Outlook, there's a lot
          11    of interest in that.  There's a lot of
          12    interest in natural gas data, we are planning
          13    a new natural gas production survey.  And
          14    then so what, rising demand and then a pretty
          15    tough budget situation, which I'll elaborate
          16    on a little bit.  EIA has been doing a lot of
          17    -- for in some sense we've been doing it on
          18    an unsustainable basis, in that we've been
          19    living beyond, we've been spending more than
          20    our current budget in each of the last couple
          21    of fiscal years.
          22              That's been done by reducing what
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            136
           1    is called priority year on cost and balances.
           2    So money from earlier years that was carried
           3    over, that we were spending, but if you look
           4    at how we've been delivering our program,
           5    we've actually been spending a fair amount
           6    more than we're budgeted on, on a yearly
           7    basis.  That how we can continue, roughly 60%
           8    of our budget supports staffing and staff
           9    related costs and increases in those areas
          10    are driven by Congressional action on pay.
          11              Certainly we have a well-qualified
          12    staff and we want them to be well paid.  But
          13    in the environment where last couple of years
          14    that has been, maybe what, 4% each year, in
          15    that area and I don't know where they will
          16    come out this year, it's one of the last
          17    things they'll do.  It will probably 3% I'm
          18    hearing, 3 or 3.5%.  But this cost of
          19    pressure in that area and it's also a very
          20    tough budget environment.
          21              EIA can try to swim upstream in
          22    that tough budget environment, because we're
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            137
           1    a very small fish, so that's a good news I
           2    guess.  If we were a big piece of the
           3    domestic discretionary budget, it would be
           4    very hard to be going in a different
           5    direction then that whole overall budget and
           6    that is a strategy that the administrator and
           7    I are trying to pursue and I think that, at
           8    least in the administrations budget proposal
           9    for this fiscal year that started this month,
          10    without a budget yet.
          11              The administration only has
          12    proposed a pretty substantial increase for
          13    us, for about $81.1 million to $85 million,
          14    which for us is a good increase.  Congress
          15    has not acted on it, but there's no guarantee
          16    that that success, the administration would
          17    propose significant budget increases for EIA
          18    in 3-4 years.  That Congress would provide
          19    the money, we're in a tough league here,
          20    we're in the interior budget.
          21              So you know we're competing with
          22    smoking the bear and keeping national parks
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            138
           1    open and the forest service, fighting fires
           2    in the West, and all kinds of, you know,
           3    other programs that are important too.  So we
           4    think EIA's important, obviously other
           5    programs have other constituencies, I think
           6    they're important.  So we always need to
           7    think about how to deal with, you know, best
           8    employing the resources that we have, how to
           9    make trade offs between products if we have
          10    to do that.
          11              How to balance, you know, meeting
          12    immediate needs and requests of today with
          13    still being our capability to meet tomorrows,
          14    our questions, how to deal with the
          15    turnovers, some of our personnel, they're
          16    interested in, there are just a lot of
          17    questions.  How to deal with domestic versus
          18    international data?  How to deal with the
          19    split between surveys and analyses.  How to
          20    prioritize from other surveys?  If indeed we
          21    run into trouble, how to choose between
          22    changes and periodicity and sampling
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            139
           1    strategies that might save some resources
           2    versus perhaps terminating some surveys.  And
           3    we have in the 2005 budget, even with the
           4    increase from $81.1 to $85 million, proposed
           5    to terminate one survey in the electricity
           6    area.
           7              Again there's always an issue about
           8    how to deal with these questions.  And we're
           9    dealing with them on an everyday basis,
          10    sometimes we get tunnel vision, there's a
          11    real advantage in bringing folks in from the
          12    outside who might have some ideas in these
          13    areas.  So I just like to rally throw it
          14    open, there are lot of challenges before us
          15    in running what I think is, a pretty good
          16    program at EIA or a very good program at EIA.
          17    But we do have challenges and if you were the
          18    king or queen or crown prince, or whatever it
          19    is, what would you be thinking about, are
          20    some of these questions?
          21              And we can break it up into the
          22    technical expertise area, so how to trade off
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            140
           1    among products, adds opportunities for cost
           2    savings, which are always welcome.  I think
           3    our staff and you always like to talk, yeah,
           4    I think it is about cognizance and a sense of
           5    frontier that you try to get to, we're
           6    trading off cost on the one hand and then you
           7    have quality and timeliness, which are the,
           8    you know, very important dimensions for EIA
           9    and I think we in EIA naturally always tend
          10    to think about ways to improve quality,
          11    improve timeliness, well, the discussion
          12    around, this program is about that.
          13              We also need to think potentially
          14    about ways to save money, holding quality and
          15    timeliness constant.  It's not as much fun
          16    for out staff or maybe for you to think in
          17    those terms, but again if you have ideas in
          18    those area, it will be very helpful.
          19              MR. BLAIR:  You just touched on
          20    staff and I think that obviously the issues
          21    of quality and timeliness are important, but
          22    I think staff frustration, staff burden, how
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            141
           1    staff reacts to trying to do the same thing
           2    or trying to do more unless there is also
           3    something that you want to factor into the
           4    equation.  I know, few meetings ago, there
           5    was a big discussion on the issues of
           6    recruiting and maintenance and losing senior
           7    managers and so forth.  So I've -- I just
           8    recommend that it should not get lost in the
           9    equation.
          10              MR. GRUENSPECHT:  Yeah, it is
          11    interesting.  We have done some, that's
          12    right, we have been very interested and over
          13    the past year have introduced some new ways
          14    of recruiting staff.  We used it for our
          15    entry level, each office as they needed
          16    people would put out an individual
          17    announcement, which then had to be cleared
          18    through the departments personnel system.
          19    And there was a very high cost for each
          20    announcement.
          21              Even though we realized pretty
          22    early on that, at the entry levels which is
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            142
           1    out of, either out of under graduate or just
           2    out of gradate school levels, we really
           3    weren't looking for people who had, it didn't
           4    matter so much if you're going it pretty well
           5    in gas activity, you just have to have a lot
           6    of oil and gas expertise.  As along as you
           7    went through with the activity, you didn't
           8    really need to have a lot of, look through
           9    the expertise in those levels.
          10              We're really looking for, you know,
          11    we'll teach you the field expertise, we
          12    really want the best math staffs, the best
          13    survey staffs, the best operations research
          14    analysts, the best economists.  So we went to
          15    a EIA wide recruiting at those levels,
          16    although again at the higher levels, we're
          17    still recruiting office by office with
          18    individual announcement.
          19              I think we had a fair degree of
          20    success with some of that recruiting, we
          21    managed to bring some good people and I think
          22    we're going to try that approach again, we've
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            143
           1    also increased our outreach, because these,
           2    our recruiting announcements, we're not
           3    sending what information the various
           4    contacts, the colleges and universities
           5    indicating that we put these announcements
           6    out.  Again trying to, you know, build
           7    interest.  And we've gotten a lot of, I think
           8    a lot of pretty good people.  So we're able
           9    to recruit, which is good.
          10              We don't have some of the
          11    incentives I guess, that some of the other
          12    statistical agencies offer and some of the
          13    others may have direct hire authority, Nancy
          14    were you in with a session with some of the
          15    public agencies.
          16              MS. KIRKENDALL:  It doesn't look
          17    like, I think they're trying to, there's an
          18    interagency effort to try to get direct hire
          19    authority, I'm not sure right now.
          20              MR. GRUENSPECHT:  Right.
          21              MS. KIRKENDALL:  I think that the
          22    agencies have been grouped, I think we'll do
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            144
           1    better if we all go together and say, we need
           2    direct hire authority.
           3              MR. GRUENSPECHT:  Then they will
           4    want to bypass some of the --
           5              MR. BLAIR:  Sure.  No, that also,
           6    instead of having sort of a ---- that one of
           7    the things you want to do when you get
           8    various budget problems, whatever is
           9    maintained, your core competencies and so
          10    that you get them.  Whatever happens in the
          11    future you've got to maintain that core and
          12    personality is a big part of that.
          13              MR. GRUENSPECHT:  Do you have any
          14    thoughts on the -- you know the use of
          15    contractors versus in-house staff?
          16              MR. BLAIR:  Well, being from a
          17    consulting organization --
          18              MR. GRUENSPECHT:  Well, we do a lot
          19    of both by the way.  We're saving about 60%
          20    of our costs here for the federal staff and
          21    the costs directly related to that, and
          22    probably you know, most of the rest is
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            145
           1    contracted out.
           2              MR. BLAIR:  Yeah I don't have a
           3    general -- maybe someone else does.
           4              MS. KHANNA:  I wanted to say or
           5    what should be done -- but I don't know
           6    without making any difference in the cost of
           7    doing it in-house versus doing outside, but
           8    one of the problems you see with a contractor
           9    is, you have no control over what's happening
          10    within the contracting agency.  For example,
          11    if I was working with Abt consultant and you
          12    know I worked in a particular project and
          13    then 6 months later, Abt gets the same
          14    continuation of the project, it maybe
          15    somebody else.
          16              And you have no control over the
          17    composition of the team that actually does
          18    the work, whereas if you do it in-house, not
          19    only are you keeping the skills, especially
          20    determined, you're trying to recruit and
          21    build up the skill levels of the younger
          22    staff members.  You know, there's a long term
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            146
           1    payback in doing it in house whereas it maybe
           2    a short term payback in doing it through a
           3    contract, and maybe that's something you may
           4    want to think about when you take that
           5    decision.
           6              MR. BLAIR:  Go ahead.
           7              DR. BURTON:  It's getting close to
           8    the -- the advantages that come with
           9    contracting are more pronounced when there's
          10    a great deal of variability in project
          11    natures and in workload.  When you have a
          12    very consistent set of tasks, that you're
          13    doing through time, the advantages of
          14    contracting out are not nearly as big and
          15    it's my sense that most of what happens here
          16    happens with a little very predictable
          17    regularity so that that probably the
          18    advantage is that a lot of front seat
          19    contracting wouldn't be -- wouldn't
          20    necessarily be realized in this specific
          21    setting.  That's kind of what I have.
          22              MR. BLAIR:  There are some ways to
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            147
           1    control for that and I know that has given
           2    some projects with different contractors is a
           3    problem, you know, but keep moving staff
           4    around and shifting up from one project to
           5    another, but I -- I think there's some
           6    contractual ways of dealing, it is with some
           7    of that.  The other perspective that I would
           8    take on it and this is very similar to a
           9    perspective we take on you know, we
          10    subcontract out something as opposed to doing
          11    it ourselves.
          12              Again goes back to sort of, what is
          13    it you're trying to develop internally as
          14    skills, what competencies are you trying to
          15    maintain when you contract out.  Is it
          16    something that, you know is an area that you
          17    know, you're not trying to build in
          18    internally, by means of a particular project
          19    normally, but I think in the back of your
          20    mind it should be, you know you're
          21    contracting out something that you should be
          22    building the capability for internally.
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            148
           1              DR. FEDER:  Howard, to your
           2    question, if I were king, I would still be a
           3    member of this committee, but I want to quote
           4    one of my colleagues from Australia, David
           5    Steele.  He said that, "If you have poor data
           6    then the best estimation method is still not
           7    going to get you very much but if you have
           8    good data, even decent estimation will be --
           9    will give you almost as good."  What that
          10    tells me is that if I were king and I had to
          11    decide where to --
          12              MR. GRUENSPECHT:  You're going to
          13    the priority question?
          14              DR. FEDER:  Yes, I would say look
          15    at your data, first of all use of
          16    administrative, already available data is of
          17    course the cheapest and sometimes the most
          18    efficient way and anything that can be done
          19    there should be -- but the -- we talked today
          20    about some issues on some surveys that led us
          21    to think that maybe design issues should be
          22    looked at.  And for myself, I'm actually more
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            149
           1    of a data analyst and I get to look at the
           2    data when it's already been collected.  It's
           3    sometimes used after the data had been
           4    collected and it's already done.  And it
           5    drives me sometimes crazy when I look at it,
           6    when I see that certain data, the
           7    questionnaire had been changed, you cannot
           8    even look at consistency because trends are
           9    sometimes much more important than levels.
          10              There is a lot of interest now to
          11    know why we see a spike in -- hopefully it's
          12    a peak, which is going to come down, but you
          13    know, cost avoidance so and so, you raise the
          14    issue of international data in it's quality
          15    and I think maybe there are some sources and
          16    some kind of relationships that EIA can build
          17    with other organizations in other countries
          18    to help there.
          19              And I would put my emphasis or best
          20    part of my effort in improving data quality
          21    prior to data analysis, just because, and of
          22    course the simulation is important because it
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            150
           1    helps you get more money.  But good data is
           2    the foundation for that and not necessarily
           3    data on this and I'm saying it although
           4    personally I'd like to do data analysis, but
           5    that's my viewpoint.
           6              DR. SITTER:  Now, I think that on
           7    that issue, I probably as king would be more
           8    inclined to cut surveys than quality as a
           9    general rule.  It has two basic advantages.
          10    The worst thing you can do is deteriorate on
          11    average overall.  Secondly you know, when you
          12    state your budgets, you state your budget
          13    requests on, well okay, under this budget we
          14    can do this, otherwise we can't.  I mean --
          15              MR. GRUENSPECHT:  Well, that's what
          16    we tried to do.
          17              DR. SITTER:  That's all you can do.
          18              MR. GRUENSPECHT:  I think this is
          19    something that really -- I really started and
          20    I -- were in the budget request it spells
          21    out, at this level we drop in this survey,
          22    we've not done that before.  Because I think
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            151
           1    that, you know that, every, lot of agencies
           2    and a lot of people are been in there, sort
           3    of a streamlining process for a long time
           4    now.  And I mean maybe early on that the
           5    solution is different.  Because I think that,
           6    you know, you come from fat times to lean
           7    times, areas start to trim, I mean you can
           8    become more efficient, you can decrease costs
           9    and it's a good thing.  But my feel has been
          10    over my time in this committee that you
          11    passed that.
          12              There's a lot of fact determined in
          13    terms of we could do things quicker, faster,
          14    more efficiently, decrease cost, and it
          15    doesn't seem to really be there.  And the
          16    budget you're talking about doesn't, you
          17    know, say, well we're going to have to have
          18    to drop a position or something, these are
          19    millions of dollar changes, right.  When they
          20    drop your budget, they drop you by enough
          21    that may cost the entire, an entire survey
          22    easily.  So now that's my consensus.
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            152
           1              DR. BURTON:  You're first this time
           2    and then we'll be --
           3              MR. BLAIR:  Kind of to second
           4    Randy's comment in that, it also maybe less
           5    disruptive internally in a sense if you, you
           6    know, just have to cut out a piece of you
           7    know, as hard as that is, rather than trying
           8    to tinker with a whole bunch of things, this
           9    is in terms of how well you can do it, that
          10    might also be done with that.
          11              DR. BURTON:  I know, I'm probably
          12    preaching to the choir but I was talking with
          13    one of the staff yesterday and I mean a lot
          14    of the reason I think the EIA has been able
          15    to do as much as it is done, as well as it is
          16    done it, with a actual decline in real
          17    dollars, so over the last decade it is
          18    because of we increased productivity this
          19    largely, attributable to technology and
          20    communications.  Those opportunities for
          21    productivity I think have been fully taken
          22    advantage of, the ability to ring anymore out
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            153
           1    in terms of additional productivity, my guess
           2    is its simply not there.  I'm telling you
           3    this but I'll tell it to anybody else you'd
           4    like me to.  So I think if you're going to go
           5    forward and keep doing what you're doing,
           6    much less do anything additional, if you got
           7    anymore resources.
           8              DR. HENGARTNER:  I mean, my
           9    comments come from discussions we've had
          10    yesterday and it is I think a fact that many
          11    of your surveys, survey the same companies
          12    for different purposes.  And there is maybe a
          13    little bit in efficiency of being able to
          14    say, I just kind of pin each company once a
          15    year or once a month, so whatever the
          16    frequency is that we need to, I get all the
          17    information on both oil and electricity and
          18    whatever else we're asking.
          19              Now the problem with that is that
          20    this is going to require lots of money to
          21    change the e, and the equation is, is it
          22    worth the energy in the long term to do that
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            154
           1    or not.  And I'm in no position to advise on
           2    this, but there is probably this one little
           3    sliver one can -- I can see maybe getting
           4    some increased efficiency.  And in terms of
           5    dropping surveys, it's higher, I mean it's
           6    always going to be someone who uses that they
           7    can well feel betrayed, but you're king.
           8              DR. HENGARTNER:  Those who
           9    complain, you know, --
          10              MS. KHANNA:  That could -- that
          11    could be turned around into an opportunity
          12    because the next time that someone from
          13    Congress asks for some information and we
          14    just don't have it because we terminated the
          15    survey, we say, oops, that's what.
          16              DR. HENGARTNER:  Well, no -- I mean
          17    I'm ---- that it dropped four months of that
          18    survey.  I mean DLS dropped four months of
          19    the monthly survey.  That's a whole, that's
          20    irreplaceable.
          21              SPEAKER:  Rightly, four months.
          22              DR. NEERCHAL:  I think that's a
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            155
           1    good suggestion but I think it largely
           2    assumes that there is a certain amount of
           3    coordination going on in the company side,
           4    because you might have a survey --
           5              DR. BURTON:  That's a different
           6    form.
           7              DR. NEERCHAL:  Yeah, once I was
           8    dealing with, you know, second floor and the
           9    other one in the fourth floor, and they don't
          10    talk to each other then it's probably not
          11    going to be much of a use coordinating from
          12    one side.  I wanted to mention a couple of
          13    things about the recruiting.  One of the
          14    things I feel that you can really help in
          15    your recruiting is to let the younger people
          16    like graduate students and the younger people
          17    who know the kind of work that is going on in
          18    EIA.  So it'll be -- I think this has been,
          19    this idea has been thrown around in this
          20    committee before, if the project sponsor a
          21    session in ASA perhaps with the EIA work, the
          22    kind of statistical work that is going on
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            156
           1    here.
           2              So that you really let the
           3    students, the graduating students know that
           4    there is an interesting work going on in the
           5    agency and some things are, you know, that
           6    doesn't have to be in the ASA.  It could be
           7    some of the smaller satellite conferences, or
           8    even in the modeling area there might be some
           9    conferences, where, which is suitable.  So
          10    you really kind of showcase your work and
          11    with plenty of interesting things going on
          12    here and that is one of the attraction for
          13    the students when they graduate, go ahead to
          14    work in.  They do have lot of choices now and
          15    statistics really is selling.
          16              DR. SITTER:  Well, on the
          17    recruiting issue.  So here's a question, are
          18    you having trouble recruiting still?  I think
          19    last time it just came up, the economy was
          20    blooming right.  So I know that at least all
          21    the students I know seemed to have as many
          22    choices as they wish.  We had master
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            157
           1    students, master students from Vancouver,
           2    turned down flights to the East Coast because
           3    they just can't travel that much for
           4    interview.  So it's not the case, that you
           5    are having trouble.
           6              MR. GRUENSPECHT:  I always think
           7    about, the troubles that I focus on are
           8    always the internal ones of moving paper
           9    through the department systems.  It -- my
          10    sense is that we're getting some pretty good
          11    applicants.
          12              MS. KIRKENDALL:  Yes, I think we've
          13    been getting pretty good applicants.
          14              MR. GRUENSPECHT:  And so far we
          15    think we are all right.
          16              DR. SITTER:  Because I have --
          17              MR. GRUENSPECHT:  If you're giving
          18    us a heads up, that's the environment is --
          19              DR. SITTER:  It's not changing.
          20              MR. GRUENSPECHT:  Not changing.
          21              DR. SITTER:  So that just means you
          22    -- I mean I haven't seen it change, I mean I
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            158
           1    think that there are still lots of jobs and
           2    the students have lots of choices.  It just
           3    sounds like you know you've an attractive
           4    choice now --
           5              DR. HENGARTNER:  There's always a
           6    problem that you require US Citizenship.
           7              MS. KIRKENDALL:  Yes.
           8              DR. HENGARTNER:  And if you look
           9    at, the school of graduate students, half of
          10    them are out.
          11              MS. KHANNA:  More than half of
          12    them.
          13              DR. SITTER:  Yeah, that -- that's a
          14    big problem.
          15              DR. HENGARTNER:  That is a problem.
          16              DR. SITTER:  Yeah.
          17              DR. HENGARTNER:  And if that could
          18    be alleviated or it could be somehow changed,
          19    that would help a lot.
          20              DR. SITTER:  I mean even Los
          21    Alamos, hires Canadians.
          22              MS. KIRKENDALL:  Well what's
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            159
           1    interesting because that's what changed here
           2    in the last year is that now even our
           3    contractors have to be US citizens, well, uh.
           4              MR. GRUENSPECHT:  Yes, complicated
           5    there.  Certainly an issue that we're looking
           6    at, very hard, there's officially --
           7              SPEAKER:  Some conflicting items
           8    right there, I would say but access of
           9    foreign nationals to certain systems, you
          10    know, in headquarters, it's a problem and in
          11    our case if you can't have access to our
          12    systems what're you going to do.
          13              MR. GRUENSPECHT:  So it's almost,
          14    it's not that foreign nationals are ruled out
          15    as contractors, but it's just access to
          16    systems issue.
          17              DR. NEERCHAL:  Of course, the top
          18    of the line is the airport security, you need
          19    that.
          20              MR. GRUENSPECHT:  So there are --
          21    that is the challenge.  You know, both --
          22    there are challenges in the contractor area
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            160
           1    as well.  It hasn't gone to the point of an
           2    absolute bar, but it's a problem.  --- 3b
           3              MR. BLAIR:  I'd like to reinforce
           4    the point that Moshe made about having the
           5    data versus sort of what you can devote to
           6    analysis and to give that is not just your
           7    analysis of course but it's the analysis of
           8    other users and that having the data out
           9    there, you know is important to them.
          10              But I think more broadly yesterday
          11    in a lot of the different parts of
          12    discussions we had, we were I think as
          13    committee saying, we think, you know, this is
          14    I guess primarily with an OMB directive and
          15    that's saying like, you know, you guys, you
          16    know do a good job now and you aren't
          17    appreciated sort of for what you're doing
          18    though, you know, hopefully that's not in all
          19    quarter and that most of you are appreciated
          20    and that you have a reputation for excellence
          21    and when it comes to having to make choices,
          22    I still kind of go with sort of Randy's
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            161
           1    approach, it's better to cut out a survey
           2    however painful that maybe and whatever
           3    constituency that might affect than to tinker
           4    here and there.  If you start to have a
           5    reputation for sliding in terms of quality
           6    and in terms of excellence, that takes a long
           7    long time to recuperate from.
           8              So, if I'm, you know, looking at
           9    choices and also trying to keep in mind the
          10    long-term, I think that things, you know, are
          11    very hard to reverse once they start to --
          12    once they start to slide.
          13              MS. KIRKENDALL:  I think over the
          14    last ten years a lot of those changes in
          15    sample sizes is to get a lower precision.
          16    Those have already been done.  So we are much
          17    lower than we've got now, you don't know if I
          18    have anything much left.
          19              MR. BLAIR:  Yeah, if you're not to
          20    the point of diminishing returns you're
          21    awfully close to.
          22              CHAIRMAN BREIDT:  John?
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            162
           1              MR. WOOD:  Just on the diminishing
           2    returns, you know, you casually said there
           3    might be some efficiencies in going to a
           4    company to get all the EIA data.  Many of the
           5    companies, let's say Exxon Mobil have many
           6    times the size of EIA and vastly many times
           7    the size of the budget.  There would be some
           8    efficiency, if you were going to run an oil
           9    survey and a big oil production survey and a
          10    gas production survey, that would be
          11    efficient to go to the production accounting
          12    department, but if you were going to go to
          13    Exxon Mobil for all the types of information
          14    that EIA actually surveys, you would add, I
          15    would say significant cost and an efficiency
          16    to do so because you know they have this
          17    diverse interest on most of all the interior
          18    department and all the department of energy,
          19    etcetera.  So there are limited areas where
          20    you could still potentially gain something,
          21    or just a blanket go to Exxon and ask them
          22    all the questions that you want to know is
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            163
           1    not very practical.
           2              DR. SITTER:  No, I just -- it
           3    didn't fall down when I told it to you, you
           4    know what I say.
           5              CHAIRMAN BREIDT:  I just wanted to
           6    second or maybe third what Randy and Johnny
           7    have been saying that we really don't want to
           8    die by a thousand horizontal cuts, you need
           9    sharp vertical cuts, you want to share that
          10    pain, so you want someone else to suffer for
          11    it, and see how they react to it and I don't
          12    know how --
          13              MR. GRUENSPECHT:  But what I'm
          14    asking, how do you prioritize what you pick,
          15    is that the answer?
          16              CHAIRMAN BREIDT:  Well, yeah and
          17    for example some of your surveys are now used
          18    in derivative products; I mean make them
          19    hurt, so --
          20              MS. KHANNA:  But that's what the
          21    economists are saying.
          22              MR. GRUENSPECHT:  What a good
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            164
           1    example that is, they're actually making us
           2    hurt because of the fact they're being used
           3    in a way that we had not anticipated, we have
           4    to go back and look at our processes and
           5    procedures and some of the things that we've
           6    done that, you know make our work more
           7    efficient like having a lot of people work on
           8    something part time.
           9              If it gets to be very financially
          10    important, then confidentiality is well too,
          11    maybe we should analyze this more, well then
          12    you have, you know, less efficiency in the
          13    group, you know you take one system that was
          14    producing that survey and others, you say,
          15    no, this only has to be done separately, so
          16    the number of people who has access to it has
          17    to be reduced to the minimum, well then, you
          18    know, yes, it's become more important,
          19    professionally those changes are making it
          20    more costly for us and we have to do more to
          21    be sure that this thing, you know gets out at
          22    exactly you know at the time that the market
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            165
           1    is expecting it and that everyone gets it at
           2    the same time.
           3              These are all things that we have
           4    never worried about.  I mean -- you guys are
           5    more general than we are but we went over to
           6    NAS, and over to the Department of
           7    Agriculture, it was, you know their system of
           8    lock-up and -- I don't know if any of you
           9    have been, it's really a treat, but it's -- I
          10    would say it's quite deluxe compared to the
          11    systems that we either have or are
          12    contemplating.
          13              Now, they have separate switch,
          14    separate telecommunication switches and
          15    almost -- it's almost like going into a
          16    submarine where you close the water tight
          17    doors and it just -- you know we are putting
          18    out surveys every week, that you say now
          19    these derivatives are startling and all but
          20    almost that it's worked the other way, you
          21    know, have it, the fact that is, that these
          22    products have come into the market, have
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            166
           1    added sometimes more things that we have to
           2    do and have required us maybe to give up a
           3    little bit of efficiency to improve security.
           4              So at least in the short term it's
           5    been adding -- this is so important it merits
           6    additional resources.  So, it's interesting
           7    but your right, those things are potential
           8    opportunities but I think it would -- again
           9    I'm not the king, the king's over here but I
          10    have a feeling that the weekly petroleum
          11    inventory survey will not be the first one on
          12    our list.
          13              CHAIRMAN BREIDT:  Right.
          14              MR. GRUENSPECHT:  Although it will
          15    probably be the one that would generate the
          16    most -- if you're trying to generate maximum
          17    pain.
          18              CHAIRMAN BREIDT:  Right, I just
          19    wonder if there are some slightly subtle ways
          20    to do the same thing, you know, it's not
          21    quite that in your face but it's similar --
          22              DR. NEERCHAL:  Could I ask a
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            167
           1    related question, do you -- I think there was
           2    a session -- related session, couple of
           3    meetings ago, do you have a fairly good
           4    understanding of who they uses for each
           5    survey is and I think you know, you might
           6    give a power ranking on each of them -- each
           7    of your user, so you know where to push the
           8    button if you ever need to -- I think, I also
           9    agree, I think that you need to, people will
          10    know how an important survey is when it is
          11    gone, like we have been complaining about the
          12    BLS survey, and you see it is gone, you know
          13    how important it was.  So once in a while you
          14    just let people now, what it feels like if
          15    you're not there.
          16              So, it maybe an important thing for
          17    you to know, who the customers are for each
          18    survey and you know, who are the important
          19    players in each group.
          20              MS. KIRKENDALL:  We have spotty
          21    data on that, for some offices they have been
          22    very good in keeping tabs on who is using
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            168
           1    their data profiles and other offices haven't
           2    paid too much attention.  Dwight French's
           3    office survey, energy consumption survey, has
           4    always done a good job knowing who the users
           5    are.  The outside users, the outside federal
           6    users especially.
           7              MS. KHANNA:  Well, I think we are
           8    all saying this ma'am, I'm just going to
           9    restate.  One of the things that we want to
          10    keep in mind is what are the goals of the
          11    exercise?  Is it really -- this is our budget
          12    for this year and we have to live with it and
          13    so then we trim or is this our budget
          14    projections for next year and there is chance
          15    to actually getting an increase and then we
          16    cut where it hurts, you know because the -- I
          17    mean your strategy is going to depend on the
          18    goal and so I think we need to first -- each
          19    time actually define the goal before we
          20    decide what to do.
          21              DR. BURTON:  Howard, there is
          22    another question too and I think we've talked
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            169
           1    about this in the past, I can't recall for
           2    sure.  I mean certainly you all develop
           3    products that people would pay for and is
           4    there an opportunity to develop revenues
           5    through those products.  I like being able to
           6    get all the stuff for free, but I would damn
           7    sure pay for it.
           8              DR. FEDER:  I was going to say
           9    exactly what you said --
          10              DR. HENGARTNER:  The stock market
          11    there's the option product, given a
          12    subscription, those who want it at 9:30 on
          13    the Wednesdays and everybody else can have it
          14    hour later for free.
          15              DR. BURTON:  Having it three
          16    minutes later would be --
          17              DR. HENGARTNER:  Those who want it
          18    at 9:30 on Wednesday, you know there is a
          19    subscription and that's it.  It will get the
          20    data to the researchers.
          21              MR. GRUENSPECHT:  Essentially, I
          22    mean some of the stuff has a lot of --
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            170
           1    tremendous market value, others delivers on
           2    mission of the agencies that probably do not
           3    have a lot of market value but once you go
           4    down the path of user fees, which is I guess
           5    what's being described.  I mean it is an
           6    interesting path but it's a path where there
           7    maybe whole parts of the program that would
           8    not --
           9              DR. HENGARTNER:  No, no but you
          10    have to be careful, user fees from the
          11    private sector pay and the government pays
          12    the rest, because the government also
          13    requires the data.
          14              MR. GRUENSPECHT:  Right.
          15              DR. HENGARTNER:  And that's the
          16    current model, that's where we do it.
          17              MR. GRUENSPECHT:  But we don't have
          18    any user fees.
          19              DR. HENGARTNER:  You have no user
          20    fees?
          21              MR. GRUENSPECHT:  Right.
          22              DR. HENGARTNER:  Well, what we're
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            171
           1    suggesting is make the private sector who
           2    receives an economic value in portions of
           3    your data might be willing to pay.
           4              MS. KHANNA:  How do you know, I
           5    mean I could download the data and e-mail it
           6    to my friend.
           7              DR. FEDER:  You sign in there, I
           8    mean -- you say that you're not --
           9              MS. KHANNA:  But everybody does it
          10    anyway.
          11              DR. BURTON:  Oh, only in academics
          12    we do that, firms will do that, the risk of
          13    --
          14              DR. FEDER:  For instance I
          15    downloaded the paper -- yeah downloaded it
          16    but don't send it outside, because -- you're
          17    right companies really would pull with risk
          18    and be there.
          19              DR. SITTER:  And they'd get it at
          20    9:31.
          21              DR. FEDER:  True.
          22              MR. GRUENSPECHT:  There is really a
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            172
           1    tremendous secondary industry of taking EIA
           2    data and repackaging it.
           3              DR. BURTON:  I know.
           4              MR. GRUENSPECHT:  About the data
           5    but --
           6              MR. BLAIR:  I wanted to comment
           7    that I think relates to a number of these
           8    things, whether you're cutting back
           9    particular products or whole surveys, or
          10    you're doing tinkering inside to try to make
          11    adjustments and that is visibility, don't do
          12    these things without the outside world kind
          13    of being aware of what's going on and what's
          14    in the pipeline, particularly it's, you know,
          15    you're making small changes inside, nobody
          16    sees that, nobody cares you know as long as
          17    they are getting so.
          18              But if you've got things in the
          19    works or you're considering something like a
          20    feed based program or whatever I don't know
          21    the politics and administration issues but
          22    wherever possible, I think that, you know,
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            173
           1    you want to be visible, you don't want this
           2    to be an internal thing that, you know, you
           3    suffered, to let your users and the other
           4    people related, suffer along with you as
           5    you're making these decisions and perhaps
           6    advocate for you and decisions are under
           7    considerations that people have feelings
           8    about.  Well, one of the --
           9              DR. FEDER:  Sorry, again summing
          10    from my Canadian experience, when provinces
          11    need -- province specific estimates or they
          12    want more position to augment a simple size,
          13    setting a province or some transfer of
          14    funding from the province, I don't know if --
          15              MR. GRUENSPECHT:  Yes, we've had
          16    some of that, I think we had working with
          17    certain States --
          18              MR. FRENCH:  We've mostly worked
          19    with other federal agencies to do things to
          20    add questions on surveys and do things
          21    generally.  Now with our consumption survey
          22    at the time we're doing the re-designs of
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            174
           1    surveys to try to have the capability of
           2    going down the smaller geographic areas, so
           3    if there's somebody who does want to pay for
           4    something to supplemental survey for a
           5    particular area, we're more capable of doing
           6    that, but we thought a few years ago that
           7    that might be a distinct possibility as there
           8    was some States that maybe interested in
           9    doing that.  But I think States feel squeezed
          10    and we're not hearing that the same way as we
          11    were a few years ago.  Well, we can still do
          12    our re-designs to try to provide as much
          13    capability for that as possible, if somebody
          14    comes out of his work.
          15              MR. WOOD:  By the way, maybe 15
          16    years ago at an ASA meeting, we had as a
          17    topic, kind of the value of publicly
          18    available data and it was one of the MIT
          19    professors and it was just a brilliant
          20    presentation, but the fundamental conclusion
          21    was the same as we already heard this
          22    morning, that if you've got some important
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            175
           1    information, it's worth a million dollar to
           2    an individual to know it, 24 hours prior to
           3    the rest of the world.
           4              That is, it loses it's monetary
           5    value, even though it keeps it's societal
           6    value if you release it to everyone at the
           7    same time and it looks like we could have our
           8    cake and icing and eat it and everything is
           9    good, I like that idea, there's a subscriber
          10    list, if you want to know it on Thursday and
          11    it's a million dollars a year.  And the
          12    others gets to it on Friday, and we do do
          13    sort of something like that with other
          14    federal agencies who fund individual work
          15    that they want done, give us money to do it
          16    but I think, there might be a few
          17    constitutional amendments.
          18              There might be some ethical
          19    question that might be to put on board, but
          20    you know and who really -- what organization,
          21    what group really wants to know that
          22    information and can they bring pressure on
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            176
           1    the people that set our budget.  You know we
           2    don't go all the way and start really
           3    delicate.  An agency that has the mechanism
           4    in fact to sell it, I think this might be
           5    pushing the envelop you know if you had a
           6    subscription survey but that's the way huge
           7    numbers of consulting firms are actually
           8    doing.
           9              Many of them are in that secondary
          10    market and every single slide in a million
          11    dollar customer oriented cost survey and EIA
          12    is that data collector and often the analyses
          13    that we send out, yet people pay a great deal
          14    of money to revive.
          15              MR. GRUENSPECHT:  If BLS could
          16    follow that model they'd be a real cost
          17    center for the government.
          18              DR. SITTER:  I think that this
          19    issue of visibility in your customers and who
          20    is willing to pay and so forth really tides
          21    in well with the discussion on your self
          22    evaluation and external review process that
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            177
           1    was discussed yesterday.  One of the main
           2    ideas about what you do well and don't do
           3    well when presenting is, you know, who is
           4    using your data and the other thing is how
           5    much do they value your data and I think that
           6    if your going to go through the exercise
           7    anyway, it might be a good idea to you know
           8    have some of your big customers, take a look
           9    at you and talk this things over with them as
          10    part of your OMG thing and maybe they'll tell
          11    you that they really value your data and when
          12    you start talking about well we've really got
          13    problems, maybe they will be some help for
          14    you there both in your OMG evaluation and at
          15    the same you'll get some input, maybe you're
          16    wrong, maybe they don't value your data that
          17    much.  Good to know.
          18              MR. BLAIR:  I like that idea a lot
          19    and I think if there was some way to make it
          20    part of the process in the sense that almost
          21    -- like you know, part of an advisory board
          22    or whatever that, you know would comment and
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            178
           1    those kinds of things, build that into the
           2    process that they're talking about in terms
           3    of improving the agency's effectiveness, that
           4    there could be, you know, depending on how
           5    you get on the board but you know,
           6    potentially powerful source, I think of
           7    support for you.
           8              MR. GRUENSPECHT:  I don't think you
           9    made Guy's life any easier, but by that
          10    criterion I think we do reach out, but I
          11    think there are a lot of good ideas here,
          12    some of which would have significant
          13    implementation problems, but others I think
          14    that we could pursue, and it's really
          15    appreciated.  Guy, do you have any?
          16              MR. CARUSO:  No, I appreciate it.
          17              MR. GRUENSPECHT:  Well, thanks a
          18    lot, those are the little, you know beyond
          19    your stripes, narrow functional area
          20    expertise but I think it's something we need
          21    to be looking at.  I appreciate your --
          22              MS. KHANNA:  It doesn't matter we
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            179
           1    like to talk anyway.
           2              MR. GRUENSPECHT:  What's that?
           3              MS. KHANNA:  It doesn't matter we
           4    like to talk anyway, we're researchers.
           5              MR. GRUENSPECHT:  And being really
           6    patient, I appreciate it, thank you very
           7    much.
           8              CHAIRMAN BREIDT:  Okay, we're right
           9    on time here, maybe just a few minutes early.
          10    There's an agenda item on committee
          11    suggestions for the spring '05 meeting, but
          12    we'll defer that till lunch.  So, what we're
          13    doing now is I'll invite public comment.
          14              Yeah, see the public as we make
          15    this history.  So, before someone has --
          16    yeah, can you come to the mike, please?  And
          17    if you can just say your name and
          18    affiliation?
          19              MS. CARROLL:  My name is Carolyn
          20    Carroll and I'm affiliated with a small
          21    statistical consulting company, Stattac.
          22    What I would like to say and I could probably
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            180
           1    do that -- what I would like to say is that I
           2    was struck by how willing and cooperative and
           3    collegial this group was and how interested
           4    the group was in trying to do better and also
           5    how hard this group works.  I come out of a
           6    couple of different communities, I've spend a
           7    lot of time in the Department of Defense and
           8    a lot of time in industry, IBM being one of
           9    the places and so I wanted to say that this
          10    sort of willingness to work through things I
          11    think it's very positive and that there are a
          12    number of communities that have some of those
          13    same interest and that by talking you can
          14    find ways to go forward.  So that's what I
          15    think.
          16              CHAIRMAN BREIDT:  Thank you.  And
          17    Neha you had something?
          18              MS. KHANNA:  Yes, before the
          19    meeting is officially adjourned, I want to
          20    take the liberty of speaking on behalf of
          21    every member of the committee to say thank
          22    you to you Jay.
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            181
           1              CHAIRMAN BREIDT:  Well, thank you
           2    very much.  Before we did adjourn I wanted to
           3    say thank you as well, so.  First I wanted to
           4    thank the members of the committee for
           5    agreeing to participate in the committee and
           6    being very robust and flexible and dealing
           7    with all of the different kinds of problems
           8    we've dealt with over the years.  It's really
           9    been a privilege to work with such a talented
          10    and diverse group, I've learned a lot, I
          11    don't normally interact with econometricians
          12    and survey methodologist, my normal herd is
          13    math-stat type people but it's nice to get
          14    out and meet new people, so that's been a
          15    pleasure.
          16              It's also really been a pleasure to
          17    work with EIA.  I didn't really know anything
          18    about the agency before serving on this
          19    committee but I learned a lot about it and I
          20    now feel this allegiance that Nancy and Randy
          21    were talking about yesterday, so.  In the six
          22    years I've been on this committee, we've
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            182
           1    talked about many many different kinds of
           2    problems but almost every time it was not a
           3    question of how do we do this but how do we
           4    do this better and it's just an obvious
           5    dedication to quality throughout this agency
           6    that's really impressive and a pleasure to
           7    work with.
           8              So, I would like to thank the
           9    people who -- the EIA staff who regularly
          10    interact with committee but they all slipped
          11    off right when -- but you know there's many
          12    many people that regularly interact and it's
          13    -- we see them again and again and get to
          14    know them, which is nice and then of course I
          15    want to thank Nancy for all her work and Bill
          16    in particular for all the logistical support,
          17    handling everything making this Chairmanship
          18    job very easy, so.  So, then it's been real
          19    pleasure, thanks very much and we're
          20    adjourned.
          21                   (Whereupon, at 11:53 a.m., the
          22                   PROCEEDINGS were adjourned.)
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382

                                                            183
           1                       *  *  *  *  *
           2
           3
           4
           5
           6
           7
           8
           9
          10
          11
          12
          13
          14
          15
          16
          17
          18
          19
          20
          21
          22
                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382